Quality Assurance
In the app: QA (in the main menu)
Needs one of these permissions: 'Submit QA Reviews', 'View QA Reports' or 'View QA Reviews'.
The QA page is where reviewers find the tickets they have to check and where leads see how the checks turn out. It lists the work; the checking itself is done on the ticket, in the Inbox.
The two kinds of QA on this page
The page has two modes, with a switch at the top right when you may use both:
- "Mixed QA": closed conversations, handled by your team, by the AI or by both, are scored against your own rubric with a score from 1 to 5.
- "AI QA": tickets the AI answered are graded "Good" or "Bad", in daily batches and per AI release.
The two are separate. They sample tickets by different rules and keep different scores.
Who sees what:
- "Mixed QA" is shown with the 'Submit QA Reviews' or the 'View QA Reports' permission.
- "AI QA" is shown with the 'View QA Reviews' permission.
- The "QA" item in the main menu is shown to people with 'Submit QA Reviews' or 'View QA Reports'. Someone who only has 'View QA Reviews' can use AI QA on this page but does not get the menu item.
"QA settings" at the top right opens Settings → Quality Assurance, where sampling and the rubric are set up.
A teammate whose access is limited to certain stores only sees tickets of those stores.
Mixed QA
Needs one of these permissions: 'Submit QA Reviews' or 'View QA Reports'.
Mixed QA has two lists, switched with "Pending" and "Reviewed".
"Pending" (needs 'Submit QA Reviews') lists the conversations that wait for a review. A conversation gets here in one of two ways, shown as a label on the row:
- "Sampled": Squid picked it when it was closed, following the sampling rules under Settings → Quality Assurance.
- "Flagged": a teammate flagged it for QA by hand on the ticket. Only a closed ticket can be flagged.
The list is newest first by when a conversation was queued. A row shows the subject, whether the conversation was handled by "Human", "AI" or "Mixed", the store, the customer and when it was queued. Click it to open the ticket in the Inbox with its QA tab open, and score it there. A conversation leaves the list once its review is submitted. A conversation that is reopened also leaves the list and comes back when it is closed again.
"Reviewed" (needs 'View QA Reports') lists the reviews that are done: the ticket number, whether it was handled by "Human", "AI" or "Mixed", who handled the conversation, who reviewed it, when, and the overall "Score".
The filter next to the switch narrows either list to "Human-handled", "AI-handled" or "Mixed" conversations. Lists load 50 rows at a time, with "Load more".
People with only 'Submit QA Reviews' see "Pending"; people with only 'View QA Reports' see "Reviewed" and the metrics.
Metrics
Needs the 'View QA Reports' permission.
Above the lists, people with 'View QA Reports' see the metrics of the reviews submitted in the last 30 days, counted back from now. A teammate limited to certain stores sees the figures of those stores only.
- one tile per part of your rubric that has been scored in the period, with the average score out of 5 and the number of reviews that scored it;
- "Score distribution": how many reviews got an overall score of 1, 2, 3, 4 and 5;
- "Per-handler average": the average overall score per teammate who handled the conversation, with the number of reviews, lowest average first;
- "Per-config-set (AI tickets)": the average overall score per AI config set, lowest average first.
Without any review in the last 30 days the block says so. When the numbers cannot be loaded it shows a message with "Try again" instead of empty figures.
AI QA
Needs the 'View QA Reviews' permission.
AI QA collects tickets the AI answered into buckets, for a person to grade. A bucket has a target number of tickets and a pass mark: the share of reviews that must be "Good".
How a ticket gets into a bucket: each time the AI sends a customer a message, Squid checks the ticket against the AI QA sampling settings under Settings → Quality Assurance. A ticket is sampled at most once and into one bucket. It is not sampled when the ticket was created longer ago than the look-back period, when it has fewer AI messages than the minimum, when someone already graded it, or when it was removed from the queue before. Then the first of these that applies takes it:
- the bucket of the AI release that answered it, when update compliance is on and that release has an open bucket with room;
- today's bucket of the first enabled flow split, in the order of the list, whose conditions match, when flow splits are on and that bucket is not full; a full first match falls through to the daily bucket, rather than trying another split;
- today's daily bucket, when the daily number of tickets is above zero and the bucket is not full.
A ticket that fits none of these is not sampled.
There are five views: "Queue", "History", "Errors", "Assignees" (with 'Edit QA Settings') and "My work" (for assignees).
Grading is done on the ticket's "AI QA" tab in the Inbox. Every ticket in these views links there.
Queue
Needs the 'View QA Reviews' permission.
"Queue" shows the buckets that are open now. The cards at the top choose what you see:
- "Main queue": today's buckets together. The card shows how many of today's tickets are reviewed, "Split by flows" when flow splits are in use, and today's "QA score": the share of today's reviews rated "Good", green at or above the pass mark and red below. "… below pass mark" counts the buckets that score under their own pass mark.
- "Update Compliance QA": one card per release with an open bucket. See the next section.
"QA buckets reset in …" counts down to midnight in your organization's timezone. A new day's buckets open with the first ticket sampled that day, and the buckets of earlier days close then. "Refresh" reloads the queue.
Under "Daily sample" each of today's buckets has a card:
- a flow split's bucket carries the split's name, the label "Flow" and its conditions;
- "Main queue" is the daily bucket. Its label is "Remaining" when today also has flow-split buckets, which take their tickets first, and "All" when there are none.
A bucket shows "… / … reviewed", its target and how many tickets are sampled so far, or "Bucket complete for today". Its tickets are in two lists you open by clicking: "Pending review" and "Reviewed". A pending ticket shows the store, the customer, the channel, when it was sampled and who it is assigned to, with "Open ticket to review". A reviewed ticket shows its rating and who reviewed it.
People with 'Manage QA Reviews' can remove a pending ticket with the bin. The ticket stays in the Inbox, is never sampled again, and its place in the bucket becomes free for another ticket. A ticket that is already reviewed cannot be removed. A pending ticket that is moved to the trash or merged also gives its place back.
A teammate limited to certain stores sees only the buckets that hold a ticket of those stores, with counts over those tickets alone.
Nothing happens automatically when a daily bucket ends below its pass mark; it is only shown as failed.
Update Compliance QA
Needs the 'View QA Reviews' permission.
When update compliance is on and its number of tickets is above zero, every new release of an AI config set gets its own bucket, filled with tickets that release answered. The release that is made by switching version control on gets none. Its card shows the release's name, when it was released and how many tickets are reviewed. Opening it shows:
- "Reviewed": reviews done out of the number the release needs.
- "QA score (share rated Good)" with the pass mark and the number of "Good" and "Bad" reviews.
- "Outcome": "Waiting for reviews", "On track to pass", "Below pass mark", "Passed" or "Failed"; "In review" for a bucket without a pass mark.
- "View release": opens the release under AI → Config Sets.
- "Tickets handled by this release": the pending and reviewed tickets, as in the main queue.
The verdict is given automatically, at the moment a review is saved. A release fails as soon as it can no longer reach the pass mark, even if every remaining review were "Good"; that can be before all reviews are in, earlier than the note under "Below pass mark" says. It passes once every review is in and the pass mark is met. No verdict is given while update compliance is switched off.
When a release fails while it is the live one, Squid takes it out of use and points the AI back at the latest stable release before it: the newest older release that passed, or that has no verdict. Failed and superseded releases are excluded. If there is no such release, the failed release stays live and is marked failed.
When the AI stops using a release before its verdict, the bucket closes without a result; "History" shows it as "Superseded" if it had a pass mark, or "Not judged" if it had none.
With update compliance off, the page says "Update compliance is off": releases are not sampled, and their buckets are hidden but kept.
History
Needs the 'View QA Reviews' permission.
"History" lists every bucket there has been, newest first, as "QA runs". The filter shows "All", "Update Compliance" or "Daily QA", and the line next to it counts the runs and how many passed and failed.
- A release shows its config set, its name, when it was released, reviews done, pass mark, "QA score", its status and "View release".
- A day shows the date, reviews done over all its buckets, the number of flow splits, how many buckets were below their pass mark, the day's "QA score" and its status. Click a day to see each bucket with its own reviews and score. Today is open by default.
A day passes when none of its buckets is still being evaluated and the share of "Good" reviews over all its buckets meets the day's pass mark. The day uses its daily bucket's pass mark, or its earliest bucket's when there is no daily bucket. A run that is still open shows "In progress"; a finished day on which nothing was reviewed shows "No reviews"; a release the AI stopped using before its verdict shows "Superseded".
Errors
Needs the 'View QA Reviews' permission.
When a reviewer grades a ticket "Bad", they pick what the AI did wrong from a shared list of errors. "Errors" shows how often each error was reported and holds the list.
Period. "Last 3 days" (the default), "Last 7 days" or "Custom", which goes back at most 30 days.
"All errors" has one row per error reported in the period: its name and category, the number of tickets, the change against the period of equal length before it ("new" when it did not occur then), a small trend chart and its status.
"Error catalogue" is the full list, with a filter for "All", "Open" and "Resolved". A row shows the error's short name, status, category and description, the number of tickets it is linked to and when it was last reported. Click a row to see its tickets.
The statuses are "Occurring", "Fix in progress", "Resolved" and "Reoccurring".
- "+ New error" (needs 'Submit QA Reviews'): add an error with a description, a short name that must be unique, and a category.
- "Import CSV" (needs 'Manage QA Reviews'): add many errors from a file with the columns category, name and description, at most 500 rows. Names that already exist are skipped.
- On an open row, with 'Manage QA Reviews': "Mark fix in progress"; "Mark as resolved", which clears its active ticket links while keeping the error visible on those tickets and in historical error counts, and if it is reported again it comes back as "Reoccurring"; "Reopen" on a resolved error; and "Delete error", which removes it from the list, from the reviewer's picker and from every ticket it was reported on.
Assignees
Needs the 'Edit QA Settings' permission.
"Assignees" is shown to people with 'Edit QA Settings'. It holds the teammates the pending AI QA tickets are divided between.
Searching the team also needs the 'View Team' permission. Search by name or email and click "Add" to add someone; the bin removes them. A newly sampled ticket goes to the assignee with the fewest pending tickets; on a tie, to whoever was added first. Adding or removing someone spreads the tickets that still wait for review evenly again, moving as few as possible. With nobody on the list, pending tickets are unassigned. Tickets that are already reviewed keep their assignee.
Being assigned only organizes the work. Anyone who may grade can still grade any ticket.
My work
Needs the 'View QA Reviews' permission.
"My work" is shown to people who are on the assignee list. It shows how many tickets are assigned to you in the open buckets, split into "Pending review" and "Reviewed", and under "Your tickets to review" the tickets that still wait for you. Each opens the ticket's "AI QA" tab in the Inbox. "Refresh" reloads the list.