Simulation of a proposal
In the app: AI → Squid Agent → Proposals → a proposal → "See the conversations side by side"
Needs the 'Use Squid Agent' permission.
A simulation shows what a proposed change to your AI would have answered to real customers, next to what the AI answered live. This page lists the conversations of one simulation side by side. No simulated reply was sent to a customer.
What a simulation is
"Simulate it first" on a proposal card starts a simulation. You can also ask Squid Agent in a thread to simulate a pending change. It is available for a pending proposal that edits your AI configuration, deploys a config set or changes a knowledge item. Changes to automation rules, widget FAQ entries and product knowledge cannot be simulated. A direct edit to a specialist's instructions or tool access cannot be simulated. Builder drafts use the Builder's draft configuration instead.
Squid can switch simulations off. The button is then not shown, and asking in a thread answers that simulations are switched off and that nothing was queued.
Which conversations. Squid starts from at most 500 conversations whose saved AI-run records were created in the last 30 days, newest record first, limited to the stores the proposal itself touches. It combines the channels selected across those stores into one filter. When stores use different channel assignments, this can include a conversation from a channel the change does not affect for that particular store. Only conversations for which Squid kept a record of what the AI looked up are considered. A conversation qualifies when a customer wrote before its first outgoing message, that outgoing message came from the AI, and the saved record includes its first AI run. Internal notes are skipped; only the first 40 messages are examined. From those Squid takes up to 40, in a fixed random order, not simply the 40 newest. If fewer than 10 qualify, the simulation fails and says there were not enough conversations. For a direct configuration or knowledge-item edit, a store must use the configuration for at least one channel family, or be set to use it when AI is turned back on. Otherwise the simulation fails for lack of stores. A Builder draft uses its proposed target stores.
What runs. For each conversation the AI answers the customer's opening turn again, in a test sandbox, using the model resolved for that store and configuration. It combines customer messages before the first AI reply, limited to 4,000 characters. Direct edits use the selected configuration with the proposed change applied; Builder changes use the draft. No conversation or message is created, and nothing is sent. What the AI looks up is answered in this order:
- with what the same look-up returned when the conversation happened;
- otherwise, for a supported read-only look-up, with today's data;
- otherwise with a made-up result.
Anything that is not a read of your data, such as a refund, a tag, closing the conversation or a hand-over, is never carried out. The sandbox blocks customer-support changes. Reading from a connected service can still refresh its connection credentials.
Grading. The live first reply and the simulated reply are both graded with your organization's active QA rubric, by the automatic QA reviewer. Customer identifiers are removed from what the reviewer reads.
Starting one needs to be the person who asked for the proposal, or another person who can see it and meets its permission, store-access and readiness requirements. The latter can simulate even when approval is reserved for the person who asked. It is refused while Squid Agent is paused. A simulation is paid from your organization's monthly analysis budget, which Squid sets; you never see an amount. Asking for a simulation of a proposal that already has one for the same change, whether queued, running or finished with a result, returns that one; it is not run twice. A simulation that failed can be asked for again.
Opening this page needs the 'Use Squid Agent' permission and access to the proposal. A simulation you may not see shows "Not found". The link at the top goes back to the proposal.
Summary
The top of the page shows the proposal's number and title and two comparisons:
- "QA pass rate": the share passing QA among completed conversations where both replies received a pass or fail grade, live against simulated. Ungraded replies are excluded.
- "Handed to a person": the share of completed conversations the AI handed to your team on the first turn, live against simulated. These can count even when QA could not grade the replies.
On this page each comparison shows the difference in percentage points with its range. The live percentage, the simulated percentage and the number of replies compared for each one are on the proposal card, not here. "Not enough replies to compare" appears when no pair of replies could be compared for that figure; with only a few pairs a difference and a range are still shown. The range can be narrow when those sampled pairs all have the same outcome. While the simulation runs, the comparisons say "Shown when the simulation finishes" and a line counts the conversations that are done.
Under them is how many conversations from the last 30 days were simulated out of those chosen, the version of the QA rubric used, and "partial result" when the simulation stopped early or some conversations failed or timed out. A simulation stops early when Squid Agent is paused or switched off, when a later batch cannot reload the original QA rubric version unchanged, when the month's analysis budget runs out or when it runs out of time. When a pause, switch-off or budget limit is detected, no new replay starts. Replays in progress may finish; a replay whose next grading call exceeds the remaining budget can stop partway through. A partial result needs at least 10 finished conversations; with fewer there is no result and the page says why.
Conversations side by side
Each conversation is one row with the ticket number (a link to the conversation in the Inbox when you may read its text), the store and the channel, the customer messages before the first AI reply, combined and limited to 4,000 characters, and two columns:
- "Live first reply": what the AI sent, with its QA verdict.
- "Simulated reply": what the AI would have sent with the change, with its QA verdict and the tools it used.
The verdict is "QA pass", "QA fail", "QA severe fail" or "Not graded", or "No verdict" when there is none, plus "Handed to a person" when the reply was a hand-over. Under a reply are the findings behind its verdict, each as a rubric rule and its category.
Each tool in the simulated reply says where its answer came from: "as it answered live" (the result recorded when the conversation happened), "replayed against today's data" (looked up again now), "Simulated, not executed" (an action that was not carried out, or a made-up result) or "refused".
A conversation that did not get a result says why: "Running", "Waiting for the next batch", "Timed out" with how long it took, "Failed", "Interrupted (not compared)", "Stopped (analysis budget ran out)", or "Not run" with the reason.
The customer's message and both replies are shown only to people who have the 'View Conversations' permission and access to that store. For others the row says "Customer text hidden" and still shows the verdicts. A conversation from a store you have no access to shows neither the store's name nor its verdicts.
What a simulation does not tell you
- It replays the opening customer turn before the first AI reply. What the AI does in later turns is not simulated.
- It runs on past conversations. Look-ups that were replayed against today's data can differ from what was true then.
- It is graded by the automatic QA reviewer, not by a person.
- It is an estimate on at most 40 conversations; the range next to each difference shows how uncertain it is.
A simulation changes nothing and approves nothing. The proposal still waits for a person to approve or reject it.