QA reviews
In the app: AI → Advanced → QA reviews
Needs one of these permissions: 'View QA Reports' or 'Edit QA Settings'.
QA reviews let Squid grade a sample of tickets that the AI handled, on a schedule, against a written rubric. The page holds the schedule, the list of runs, the graded tickets with their findings, the versions of the rubric, and two views that sort the failures by who has to fix them.
What QA reviews are and who can use them
A QA run takes a random sample of closed tickets on which the AI ran, lets a reviewing model read each ticket and compare it with the rubric, and stores a verdict and a list of findings per ticket. It reviews the AI's work only; it does not grade your team. It covers every store of the organization.
Three things are needed:
- Squid Agent for the organization. The row "QA reviews" in the AI menu is only shown in organizations that have Squid Agent, and scheduled runs only start while it is on. Squid switches Squid Agent on per organization; there is no switch for it in the app.
- A plan that includes QA reviews, to switch them on: Pro, Advanced or Enterprise. A trial does not count, whatever plan it is a trial of.
- A permission. 'View QA Reports' shows the tabs "Runs", "Results", "Rubric versions", "Causes" and "Issues". 'Edit QA Settings' shows the tab "Settings". The "Not a real fail" and "Count it again" buttons are on the "Causes" tab, so they need both permissions. With neither, the page says which permission is needed.
A teammate whose access is limited to certain stores only sees the tickets and findings of those stores.
Settings
Needs the 'Edit QA Settings' permission. Available on the Pro, Advanced and Enterprise plans.
"QA reviews" (switch): off by default. On, Squid starts a run on the schedule below. On a plan that does not include QA reviews the switch is locked and the page says "QA reviews are included on the Pro and Advanced plans" with a link to Settings → Billing. If QA reviews were already on when the plan changed, no runs start any more; you can still switch them off, and they resume when the plan includes them again.
"Run every (days)": 1 to 5. The default is 3. It cannot be longer than 5 days, because the data a run needs about each ticket is kept for about 6 days.
"Tickets per run": 1 to 200. The default is 40.
Nothing is saved until you click "Save". After saving, the page says when the next run starts. Runs start once a day, at 06:00 UTC. Switching QA reviews on makes the first run start at the next 06:00 UTC; after that a run starts every so many days as you set.
"Rubric" shows which rubric version is in use. The rubric cannot be edited here. A change goes through a Squid Agent proposal that a person approves, so every version is kept. "Change it with Squid Agent" opens Squid Agent with the request started; the button is shown to people who can use Squid Agent.
"Usage this month" shows how many tickets were reviewed this month, out of the number your schedule plans for the month, and when the next run is. Squid sets a monthly limit per organization. When it is reached, runs are skipped until the next month.
Runs
Needs the 'View QA Reports' permission.
The list shows the 50 most recent runs. Each row shows:
- the run's name and when it started. Click the name to open its results; a run that reviewed no tickets cannot be opened;
- its stage: "Queued", "Judging", "Completed", "Skipped" or "Failed", and when it finished. A skipped run says why: "Skipped: this month's QA limit is reached";
- how many tickets it sampled. "38 of 40 tickets" means fewer tickets were available than asked for, with a note saying why;
- the rubric version it used.
A run samples tickets that are closed now, were closed in the last 6 days, had at least one AI run and were not reviewed by an earlier run. The sample is random.
When a run finishes, is skipped or fails, people with the 'View QA Reports' permission get a notification if they have left QA run notifications on. For a completed run with reviewed tickets it opens the run's results; otherwise it opens this list.
Results
Needs the 'View QA Reports' permission.
"Results" shows one run: the one you opened from "Runs" or from a notification, otherwise the latest run that reviewed tickets among the 50 most recent runs. If a linked run cannot be loaded, the page also falls back to that selection.
At the top are the counts per verdict: "Pass", "Fail", "Severe fail", "No spec" and "Insufficient evidence". The last two mean the ticket was not graded. The buttons under the counts filter the list by verdict.
Each ticket in the list shows its number, its store, three verdicts, the severity of its worst finding and its main finding:
- Headline: the main verdict. It fails when at least one finding that matters breaks a process rule of the rubric.
- Damage: looks only at findings that matter about money, links, made-up facts or exposed data.
- Strict: the worst finding of any kind, including small ones that would not fail the ticket.
Severity runs from 1 (severe: money or exposure) to 5 (minor or improvement).
The list loads 100 tickets at a time; "Show more" loads the next 100.
Click a ticket number to open it. On the left are the messages as the reviewer read them, labelled "Customer", "AI reply", "Agent", "Note" or "Saved FAQ". Customer email addresses and phone numbers are blanked out. On the right are the three verdicts and every finding with its severity, or "No findings." "All results" goes back to the list.
Rubric versions
Needs the 'View QA Reports' permission.
The rubric is the list of checks a ticket is graded against. An organization that has not changed it uses Squid's default rubric, version 1.
Each version shows its number, an "In use" label on the one that runs now, when it was made, what changed in plain words (or "The first version."), and, when recorded, who approved it and the Squid Agent proposal, with a link to that proposal.
Earlier versions are kept. Going back to one is also done through a proposal. "Change it with Squid Agent" is shown to people who can use Squid Agent.
Causes
Needs the 'View QA Reports' permission.
"Causes" shows the failed findings of one run, the latest completed run unless a link named another, and who has to fix each one.
Squid checks its own judgement of "who fixes this" and of the cause against manual audits. Each of the two is shown only once it agrees with those audits often enough. Until then the tab says so, and shows the failures without them.
When they are shown:
- "Who fixes this: You": the fix is in your prompt, your store's rules or a connection. "Who fixes this: Squid": the fix is in Squid's code, tools or the AI itself. A finding owned by Squid can carry a "Squid issue" label with its status.
- The four causes: "Prompt gap" (your store has a rule, but the prompt does not follow it), "Missing policy" (your store has no rule for the situation), "Squid harness" (Squid's code, tools or data gave the AI wrong or missing information) and "Hallucination" (the data and the prompt were right and the AI still got it wrong).
- "Why Squid thinks so" explains the choice. Where a broken connection is the reason, "Fix the connection" opens the page to repair it.
Once owners or causes are shown, a finding without an owner says "Not attributed yet", "Monthly analysis limit reached" or "Can't be determined".
Each row also shows the ticket, its store, the kind of failure, the check it failed and the part of the reply it is about.
At the bottom, "Auto-dropped (…) and dismissed (…)" opens the findings that do not count. Auto-dropped findings are false alarms that Squid removed by itself, for example because the quoted text is not in the ticket or the order data contradicts the finding. Dismissed findings were marked "Not a real fail" by a person, with their reason.
Not a real fail
Needs the 'Edit QA Settings' permission.
"Not a real fail" on a failure, on the "Causes" tab, removes it from the QA score. You must write why, in up to 500 characters, and choose what was wrong:
- "The check doesn't fit this case (Squid suggests a rubric change)" (the default): Squid tries to draft a change to that check of the rubric, with your reason as an exception. Approving it also needs Squid Agent access and access to every store, as well as 'Edit QA Settings'. No proposal is made if the check cannot be found, already has a waiting proposal, or the proposal is refused. The finding still leaves the score.
- "The grader misread the ticket (reported to Squid)": the mistake is reported to Squid.
"Dismiss it" confirms. The ticket's verdict is worked out again without the finding, and the dismissal is recorded in the audit log with your name.
"Count it again", in the list of auto-dropped and dismissed findings, puts a finding back so it counts in the score again. It needs 'Edit QA Settings' too and is also recorded in the audit log.
Issues
Needs the 'View QA Reports' permission.
"Issues" groups the open failures of the last 30 days, across runs. Like "Causes", it shows who fixes them and their causes only once Squid's judgement agrees with manual audits often enough.
At the top, "Who fixes this: You" and "Who fixes this: Squid" each appear when that group has attributed failures, with its number of tickets. Under them is one row per issue: failures with the same cause in the same place. A row shows a title, the number of tickets, who fixes it, the cause, where it happens and the next step:
- "Open the fix": a Squid Agent proposal with a fix exists; this opens it.
- "Fix it with Squid Agent" or "Answer the question": opens Squid Agent with the request started. Shown to people who can use Squid Agent.
- "Fix the connection": opens the page to repair the connection.
- a "Squid issue" label: Squid has it and shows its status.
A line at the bottom says how many more issues there are about stores or data you cannot see.