SquidSupport
Product reference / AI

Experiment results

In the app: AI → Advanced → Live experiments → Experiment results

Needs the 'View Configuration Tests' permission.

Experiment results lists every live experiment of the organization with how far it is: whether it still runs, how many of its orders are old enough to be judged, and its verdict once there is one. Click a test to open it. The heading on the page itself reads "Live experiments".

Test list

The list shows up to 100 tests: running tests, stopped tests, finished tests and drafts. "New experiment" at the top right goes to AI → Advanced → Live experiments, where tests are started. Without any test the page says "No live experiment exists yet." with the link "Start one".

Each row shows:

  • the test's name;
  • its trigger, in the words of the person who made it, with the categories it was read as on a second line;
  • the day it started, the kind of trigger ("Category" for tests made on the Live experiments page), the number of stores it runs in and, where its stores have them, their store labels;
  • its status;
  • the three blocks described below: "Verdict metric", "Take-rate (pooled)" and "Maturity".

A teammate whose access is limited to certain stores only sees tests that run, or ran, in at least one of those stores, and the store count only counts their stores.

This page only shows what has already been worked out. Opening or refreshing it never calculates a verdict.

Status of a test

  • "Draft": the test was created but never started. Tests made on the Live experiments page start at once, so a draft comes from another route, such as a Squid Agent proposal.
  • "Running": new matching customers are still being added.
  • "Collecting": no new customers are added, and the orders in the test are still getting old enough to be judged.
  • "Stopped": the test has ended and its verdict is not fixed yet. Either its orders are still maturing, or every order has matured and the automatic check that fixes the verdict has not run yet. That check runs once an hour. The line under "Verdict metric" says which of the two it is.
  • "Ready to read": the verdict is fixed. Only this status can carry a result. If your store access covers only part of the test, the overall verdict is withheld even when the status is "Ready to read".
  • "Abandoned": the test was dropped before it finished. It never gets a verdict.

Verdict metric and take-rate

"Verdict metric" is the one number that decides the winner. It comes from the test’s current pre-registered rule; changing that rule requires a new revision and a reason. For tests made on the Live experiments page it starts as "Refund + dispute cost per order". "No rule pre-registered yet" means the test has no such rule and cannot get a verdict until one is registered. A first rule cannot be registered after customers have already been enrolled.

Under it is the verdict in one line when the status is "Ready to read". Before that the row says why the verdict is locked, for example that the test is still enrolling or how many orders are still maturing. No number about either group is shown before the verdict is fixed, so that nobody picks a winner early.

"Take-rate (pooled)" is the share of offers that customers in the test accepted, out of all offers the AI made to them, for both groups together. "No offer presented yet" means the AI has made no offer in the test so far. It is never shown per group before the verdict, because that would give the winner away.

Maturity on a row

The bar shows how many of the test's orders have matured, as "… of … orders matured", with the share that is still maturing and, when there are any, the number of orders that are "not priced yet, so undated". The "not priced yet" label means Squid has no maturity date for the order yet. This can mean the order date or a treatment date has not been found. An order joins a test when a ticket that actually ran under the test is linked to that order. Before any order is linked, the row shows how many customers are in the test and that results unlock at the test’s maturity window. For tests started on the Live experiments page this is 30 days after the later of the order date and the latest AI run under the test.

The pooled take-rate and maturity counts include only stores you may access. Each order matures on its own date, so a test does not finish on one day. The test's own page shows the spread.