SquidSupport
Product reference / AI

Experiment result

In the app: AI → Advanced → Live experiments → Experiment results → a test

Needs the 'View Configuration Tests' permission.

This page shows one live experiment: how far its orders have matured, the verdict once it is fixed, the rule the verdict is judged by, and the buttons to roll out the winner or to stop the test.

Test header

The header shows the test's name and status, the trigger in the words of the person who made it with the categories it was read as, the day it started, the number of stores, any store labels and the channels. A test that came from a Squid Agent proposal has the line "Proposed by Squid Agent (proposal #…), approved by …", which opens that proposal. A test with a written hypothesis shows it in a grey box.

"All configuration tests" at the top goes back to the list. The statuses are explained on the Experiment results page.

A teammate whose access is limited to certain stores sees the page only for a test that runs, or ran, in one of their stores. They see the numbers of their own stores. Numbers that are counted over all stores, such as the overall verdict and the spend monitor, are left out for them, and the page says so.

Maturity

A test is judged per order. An order joins the test when it can be identified on a ticket where the AI ran with the customer’s assigned test configuration. The run counts even if it escalated, was blocked or failed before replying. Assigning a customer to a group alone adds no order. A refund or dispute from before the AI’s first run in the test is not counted for either group. An order has matured when the maturity period has passed, counted from the order date or from the AI’s last run about it, whichever is later. For tests made on the Live experiments page that period is 30 days. Refunds and disputes arrive late, which is why the test waits.

The bar and its legend split the orders into "Matured", "Within 7 days", "8–30 days", "31–90 days", "Beyond 90 days" and "No date yet". "No date yet" holds orders without a maturity date, because the order date or a qualifying AI run has not been found yet. Under the bar the page says when the last order that is still maturing is due.

Before any order is linked, this block shows how many customers are in the test.

Spend safety monitor

This is the one money figure shown before the verdict. It keeps the highest recorded excess cost of the test group over the same number of orders in the control group, using the control group’s average cost per order. The cost counted is refunds, payment disputes and vouchers, a voucher at its full face value. The figure never goes below zero, and while the control group has no linked orders yet, the whole cost of the test group counts. Next to it is the limit when one is set, or "(no limit set)". Tests made on the Live experiments page have no limit.

It is a safety figure, not a result: it can only go up, so it does not say which group is cheaper. When the spend reaches a limit, the page shows "breaker tripped". The test then stops adding new customers by itself; customers already in the test stay in their group.

The verdict is locked

Until the verdict is fixed, this block says why it is locked and lists what is held back: which group won, how big the difference is, the results per group, the take-rate per group and the spend per group. These are held back on purpose, so that nobody acts on half a result.

The verdict is fixed once for each version of the pre-registered rule, by an automatic check that runs every hour, after the test has concluded and every linked order has matured. A test with no linked orders stays locked. Refreshing the page does not unlock it.

Reading the verdict

When the status is "Ready to read", the page shows one panel for all stores together, then one per store and one per store label. A label panel pools stores with that label. Each panel is judged on its own customers, so a store's panel can say something else than the panel for all stores. Each panel shows, from top to bottom: the warnings, the outcome, the measured effect, the two groups and the six metrics. "Variant" is the test group.

The outcome line

The outcome is one of four lines. Under it is the range of the difference, test set minus control, in your reporting currency.

  • "Variant wins": the whole range lies on the side that is better for the test set. For a cost metric that means the whole range is below zero: the test set cost less per order.
  • "Control wins": the whole range lies on the side that is better for the control.
  • "Not established": no group is named. This is shown when the range contains zero, so it is not known which group is better. It is also shown when the range is clear of zero but the measured difference is smaller than the threshold in the pre-registered rule: the rule said beforehand that a difference that small would not be acted on.
  • "Not enough data": there is nothing to compare. A group has fewer customers than the rule requires (at least 2 in each group, or the higher number in the rule), or one group has no observations for the deciding metric, such as no rated responses when CSAT decides. No range is shown.

Only "Variant wins" and "Control wins" name a winner that "Promote" can roll out without an override.

A grey note under the range says why a result is "Not established". For tests made on the Live experiments page the range is sampling error only: it shows how much the result could differ by chance with these customers, and nothing else.

"Measured effect" is the single best estimate of the difference. It is context for the range, not the result.

Warnings above the number

The warnings are printed above the outcome on purpose, so they are read first. Each one says in which direction the result leans ("overstates" or "understates"), which group it affects and how much. Which warnings appear:

  • Vouchers and return costs are not priced. Shown on every test that was created without a cost model, which is every test made on the Live experiments page. The comparison counts refunds and chargebacks only. What a voucher costs and what it costs to take goods back are not in the number, so a group that hands out vouchers or routes more returns to the warehouse looks cheaper than it is.
  • Refunds scored as zero. Shown when at least one refund in either group had an amount Squid could not determine. Such a refund counts as zero and is not left out. The warning gives the number per group; if one group has many more, the result leans towards that group.

Tests that were created with a cost model, which the Live experiments page does not make, show other warnings in place of the first one:

  • that a voucher is counted at its face value and then reduced by an assumed redemption rate, which overstates the cost of the group that offers vouchers when fewer are redeemed;
  • that return shipping and restocking are an assumption and not a measurement, which understates the cost of the group that routes returns when they cost more than assumed;
  • that the range shows sampling error only, when the cost model has one fixed value for everything or when the deciding metric is not a cost.

For those tests the range can be wider: it covers every cost assumption that was fixed beforehand, and "How the band was derived" lists the result under each of them.

The two groups and the six metrics

  • "Control" and "Variant": for each group the number of customers (this is the sample size), "Orders", "Runs served" (how many AI runs the group got), "Order join rate" (the share of linked orders matched to stored order information) and "Refunds nothing could price". A large gap between the groups in the last two means the comparison is less reliable.
  • The six metrics: "Refund + dispute cost per order", "Refund cost per order", "Dispute cost per order", "Offer take-rate", "CSAT" and "Messages per ticket", each for control and for the test set, with the difference and its 95% range. One of them is the verdict metric, named in the panel's header; the other five are context and never change the outcome.

Pre-registered rule

The rule a test is judged by is fixed before the first result exists. This block shows it: the metric that decides, how the groups are compared, the number of days after which refunds and disputes are counted, the threshold for a decision and the number of resampling rounds used for the range.

For tests made on the Live experiments page the rule is set automatically at creation: "Refund + dispute cost per order" decides, and refunds and disputes are counted over 30 days.

A test without a rule says so and cannot get a verdict while it has none. This page does not add or change rules. A first rule can be registered elsewhere only before any customer has enrolled. An existing rule can be superseded with a new version and a written reason; earlier results stay tied to their original rule.

Promote

Needs the 'Edit Configuration Tests' permission.

Promoting rolls a group's config set out to every store the test ran in, on the channels the test ran on. For a test on one channel only that channel's set of the store is changed; chat and WhatsApp share one. For a test on all channels the store's default set is changed, and also the separate set a store has for chat or for email, so no channel keeps the losing set.

Promote is locked until the verdict is fixed. No group can be rolled out before that, not even with an override.

  • "Arm to roll out": "Variant" (the test set) or "Control". It starts on the group the verdict named. The button rolls out the selected group. Choosing a group the verdict did not name requires an override and a written reason.
  • "Promote to every scoped store" is available when the verdict named a winner. Promoting control assigns its measured configuration to the stores too. This can change stores that no longer use that configuration.
  • "Promote an arm the frozen verdict did not name": tick this to roll out a group after a result that named no winner, or to choose the losing group. You must write a reason. The selected group is then rolled out. The override and the reason are recorded under "What was done to this test". An override cannot skip the requirement for a frozen verdict.

What is rolled out is the config set as it was measured. When the group's set is still the same as when the test started, the stores are pointed at that set. If the set was edited or deleted after the test started, Squid makes a new config set from the version that ran in the test and points the stores at that copy, not at the edited set. The exception is a test that was started with "Freeze the config set used for the live experiment" off: there the set as it is now is rolled out.

Promoting is refused, with the reason and nothing changed, when the test has not ended, when no store is left to roll out to, or when the configuration to roll out has a failed AI Builder policy check that nobody reviewed.

Stores are handled one by one. "Per store" lists each store with the channels that were changed and "applied", "failed" or "pending", with the error for a failed store. When some stores failed, the button reads "Resume for … store(s)" and retries only those.

Promoting needs the 'Edit Configuration Tests' permission. A teammate whose access is limited to certain stores rolls the set out to their own stores only; the other stores of the test are left as they are.

Launch a draft

Needs the 'Edit Configuration Tests' permission.

A test with the status "Draft" shows the block "Launch" in place of "Promote". Tests made on the Live experiments page never are drafts; this applies to tests created another way.

The block checks whether the test can start and lists every reason it cannot, such as no store in scope or a store you cannot access. "Launch this test" starts the split in every store of the test; the button stays off while a reason is listed.

Stop test

Needs the 'Edit Configuration Tests' permission.

"Stop test" is shown for a draft or a running test to people with 'Edit Configuration Tests'. You must write why, in up to 500 characters.

  • A running test ("Stop the test"): it stops taking new customers. Customers who are already in the test keep their group until their conversations finish, and the orders keep maturing towards a verdict.
  • A draft ("Abandon the draft"): the test is dropped and never runs.

This is gentler than "Stop test" on the Live experiments page, which puts every customer back on the control at once.

What was done to this test

A log of what happened to the test, such as its start, a stop, the verdict being fixed, a tripped spend limit and each store's promotion. Each line shows when it happened, what was done and who or what did it. The block only appears once there is something to show.