SquidSupport
Product reference / Settings

Quality Assurance settings

In the app: Settings → Quality Assurance

Needs the 'Edit QA Settings' permission.

Decides which conversations are put in front of reviewers and what they score. The page holds two separate samplers: rubric sampling, which picks closed conversations for a scored review, and AI QA sampling, which picks tickets the AI answered for a Good or Bad rating on the AI QA tab in the inbox. Both are set once for the organization and cover every store.

The page is listed in Settings for people with the 'Edit QA Settings' permission, which every change needs. Loading the settings also needs 'View QA Reviews'; without it the page shows that it could not load them.

Rubric sampling

Rubric sampling fills the QA review queue with closed conversations. A reviewer gives each one an overall score from 1 to 5 and a score on each rubric dimension.

"Sample closed conversations into the QA queue" switches the sampler on or off. It is off by default. When it is off, closing a conversation never adds it to the queue, and the two sampling boxes below are disabled.

Sampling happens at the moment a conversation is closed by a teammate, by the AI or by the "Close" action of an automation rule. A rule that closes a conversation with its "Set Status" action does not offer it for sampling. An open conversation cannot be sampled, and a conversation is sampled at most once. A conversation with no reply from a teammate or the AI is never sampled.

There are two boxes, each with its own rule:

  • "Human-handled tickets": conversations where only teammates replied.
  • "AI-handled tickets (and mixed)": conversations where the AI replied, including those where both the AI and a teammate replied. An answer that came from a saved FAQ in the chat widget does not count as an AI reply.

Each box has a mode:

  • "Percent of closes": every closed conversation has that chance of being sampled, from 0 to 100%. The default is 10%.
  • "Fixed number per period": conversations are sampled as they close until the number is reached, then no more until the next period. The number can be 0 to 10,000, and the period is "per week" or "per month". The number is counted per store and per box, not for the organization as a whole: with 5, each store gets up to 5. A week starts on Monday and a month on the 1st, at midnight UTC.
  • "Manual flag only": nothing is sampled automatically. Only conversations a reviewer flags by hand reach the queue.

"Overall scores at or below … count as "negative" and require a comment." A reviewer who gives an overall score at or below this number must write a comment, or the review is refused. The number can be 3, 4 or 5, and the default is 3. Negative reviews can be read by the AI Builder when a Builder session is tied to a config set and you ask it about quality. They never change a live prompt by themselves.

Every change in this section is saved the moment you make it, including each change to a number while you type; there is no save button. It applies to conversations closed after that.

Rubric dimensions

Dimensions are what a reviewer scores besides the overall score, each from 1 to 5. A new organization starts with four: "Tone", "Accuracy", "Empathy" and "Resolution".

Each row shows the dimension's name and, in brackets, its key.

  • "Rename" changes the name reviewers see. The key stays the same.
  • "Archive" takes the dimension off the review form. It is shown struck through here and its earlier scores are kept. "Restore" puts it back.
  • A dimension cannot be deleted.

To add one, fill in "Dimension key" and "Dimension display label" and click "Add". The key is a short internal name: it must start with a letter and may hold only letters, digits and underscores, up to 64 characters. Capital letters are stored as lower case. It must be unique and cannot be changed later. The label can be up to 128 characters. Renaming, archiving, restoring and adding are saved at once.

Without any dimension, sampling still runs, but reviewers have no dimensions to score.

AI QA sampling

AI QA sampling picks tickets the AI answered for reviewers to rate Good or Bad on the AI QA tab in the inbox. It is separate from rubric sampling and does not wait for a ticket to be closed: each time the AI sends a customer a message, including an AI draft that a teammate approves, the ticket may be sampled.

A ticket is sampled at most once, into the first bucket that has room, in this order: the update-compliance bucket of the release the AI is running, then the first matching flow split, then the daily sample. A ticket that was already rated on the AI QA tab, or was removed from the queue, is not sampled again. Days are counted in your organization's business-hours timezone, the one set under Settings → Organization, which the (i) next to the heading names.

"Update compliance" is off by default. When it is on, every config-set release created from then on gets its own bucket of tickets answered under that release, so you can check that an update behaves. It needs version control on the config set; the first release, made by turning version control on, gets no bucket. Turning it off stops this sampling and hides those buckets on the AI QA tab; nothing is deleted. The switch saves at once.

"Update Compliance threshold" is how many tickets a release's bucket asks for. The slider goes from 10 to 200; the default is 20. A bucket keeps the number it was opened with: a change applies to releases created after it.

"Acceptable QA %" is the share of Good ratings a bucket needs to pass, 0 to 100. The default is 50. A new release starts from this value. A release's bucket fails as soon as the pass mark can no longer be reached, and passes once every ticket is rated and the mark is met. When a release fails, the AI goes back automatically to the latest stable release that is older than it. Nothing changes when the failed release is no longer the one in use, or when there is no older stable release: the failed release then stays in use, marked as failed. A daily bucket is only shown against its pass mark; nothing happens when it falls short.

"Minimum AI messages" is how many replies the AI must have sent to the customer on a ticket before it can be sampled. Approved drafts count; notes, saved FAQ answers, back-office templates and campaign messages do not. A reply split into several chat bubbles counts once. From 1 to 100; the default is 1.

"Overall daily sample" is how many AI tickets reviewers should rate per day, from all traffic. From 0 to 1,000. The default is 0, which turns the daily sample off. Today's bucket keeps the number and the pass mark it opened with. A changed number applies when the next bucket opens, but setting it to 0 stops daily sampling immediately. Turning it back on resumes today's bucket if one already exists.

"Look back" leaves out tickets that were created longer ago than this. The slider goes from 1 to 30 days; the default is 7.

The numbers wait for "Save", which stores only the ones you changed. "Discard changes" puts them back. The switches save straight away. Leaving the page with unsaved numbers asks you to confirm.

Flow splits

"Enable flow-splits (advanced)" is off by default. When it is on, you can reserve a number of reviews per day for one kind of ticket, such as one store's returns. The switch saves at once and shows how many flows exist. Turning it on also fills enabled splits from eligible tickets the AI already answered today, including unrated tickets in today's daily sample.

Each flow split has:

  • "Name", up to 255 characters.
  • "Tickets to review per day", 0 to 1,000. A new split starts at 5. Today's bucket keeps the number it opened with; a change applies from the next day.
  • "Enabled". When off, tickets skip this split and go on to the next one.
  • "Conditions". A ticket must match every condition, and a split needs at least one. Splits are matched at the moment the AI sends a message, so only things known then can be used: store, channel, language, mailbox, priority, department, customer, assignee, tags, whether the AI escalated or ignored the ticket, hallucination detection, creation date and ticket fields. "Assigned to me" cannot be used.

Each row of the list shows the split's name, its number per day and its conditions, with a switch that turns it on or off at once. Flows are tried from top to bottom and the first enabled match takes the ticket. Use the up and down arrows to change the order; the new order is saved at once. When a split is full for the day, further matching tickets go to the daily sample instead; they are not offered to a later split.

"Add flow" opens "Add flow split"; "Edit" opens "Edit flow split". A new or changed split that is enabled does not wait for the AI's next reply: its bucket opens at once and is filled, up to its number, from tickets the AI already answered today. That includes matching tickets that were waiting unrated in today's daily sample, which move to the split. "Delete" asks for confirmation. A deleted split stops taking tickets; tickets it already sampled stay in their buckets.

AI QA audit sheet

"Save AI QA audits to a Google Sheet" is off by default. When it is on, each review a reviewer saves on the AI QA tab is written to the chosen spreadsheet:

  • one row per reviewed ticket, in a daily tab named "MM/DD Audit". The tab is named after the day before, because a day's reviews audit the previous day: reviews saved on 10/20 go to "10/19 Audit". Days follow your organization's business-hours timezone. The row holds the ticket number, the errors the reviewer picked, "Pass" for Good or "Fail" for Bad, the comment, the reviewer's name and the customer's email address. Saving a review again on the same day rewrites the ticket's row in that tab; saving it again on a later day adds a row to that day's tab;
  • the AI QA errors, in a tab named "Error lookup": an error created from then on gets a row, and a deleted error's row is removed.

Only reviews saved by a person are written, never automatic grading. Nothing is backfilled: it starts with the next review and the next new error. Writing happens in the background, so a slow or failing Google never blocks a review.

"Google Sheet" picks the spreadsheet. The list holds the spreadsheets from Settings → Google Sheets that Squid can open and edit; "Manage Google Sheets" opens that page. Someone who cannot see the Google Sheets settings sees only the spreadsheet that is already chosen. The switch can only be turned on once a spreadsheet is chosen, and is refused when Google Sheets is not connected or the chosen spreadsheet is no longer available with edit access. Both the picker and the switch save at once.