Live experiments
In the app: AI → Advanced → Live experiments
Needs the 'View Configuration Tests' permission.
A live experiment is an A/B test of two AI config sets on real customer conversations. On this page you start a test, see the tests that are running, change their priority and stop them. The outcome of a test is on the "Experiment results" page, linked at the top.
What a live experiment is
A test compares the config set your customers get today, the control, with another config set, the test set. Conversations that match the test's trigger are split in two halves: one half is answered with the control, the other half with the test set. A test made here uses a 50/50 random split and no percentage limit on eligible traffic. A matching conversation can still stay out, for example if another test wins priority or the conversation has no identified customer.
The split is per customer, not per conversation. A customer keeps the same group on later matching conversations and on conversations already answered in that group. Other conversations use the store’s ordinary setup. A conversation already handled outside the test stays outside it, even if a later message matches. A customer who shops in several of your stores is one customer and gets one group. The group is decided by chance when the customer's first matching conversation comes in after the test started. An older conversation can enter the test when a new message matches; the conversation’s creation date does not exclude it.
A test runs in the stores where the control config set was in use at the moment the test started. In those stores it takes a conversation only when the control is the set that answers that conversation's channel. A store that starts using the control after the test started is not added. The test has no scheduled end date or size you choose on this page. It runs until stopped; each group has a technical limit of two billion customers.
The AI decides whether a conversation matches the trigger right after it has categorized the customer's message, before it writes a reply.
Seeing this page needs the 'View Configuration Tests' permission. Starting a test, changing a priority and stopping a test need 'Edit Configuration Tests'; without it the page shows the tests and says "Starting or stopping a test requires permission to edit live experiments."
New A/B test
Needs the 'Edit Configuration Tests' permission.
The form opens when you load the page, even when tests already exist. After you start a test it closes; "New A/B test" opens it again.
"Control — what customers get today": the config set to compare against. Only config sets that are in use in at least one store can be chosen; the others are listed under "Not live anywhere" and cannot be picked. Under the list the page says where the set runs today. When exactly one config set is in use, it is filled in for you. Changing the control clears the trigger conversation, because a trigger is measured where the control runs.
"Test — what the test tries instead": any config set, including one that is not in use anywhere. It must differ from the control. The test set’s name is lowercased, extra spaces are collapsed and long names are shortened to at most 50 characters to form the ticket tag; tickets in the control group get the same name followed by "(control)". A test cannot start while another running test already uses the same tag.
The two config set choices and the trigger chat are remembered in this browser tab when browser storage is available. Priority and the freeze switch reset on refresh.
A teammate whose access is limited to certain stores cannot start a test with a control that also runs in a store they cannot see.
Describe the trigger
Needs the 'Edit Configuration Tests' permission.
The trigger says which conversations are in the test. You describe it in a chat, in your own words, and a builder model turns your words into filters. Before the first message the chat lists what a trigger can filter on; click a row to see every value:
- "Ticket category": what the customer writes about, as the AI labels the ticket before it answers. You can include categories, exclude categories, or both. The list holds up to 120 categories your organization uses, most used first.
- "Channel": where the conversation came in: email, chat or WhatsApp.
- "Payment method": how the order the conversation is about was paid, read from Shopify when the conversation comes in. An order paid partly with a method counts as paid with it. A method can also be excluded.
A conversation is in the test only when every filter you name matches.
After each message the builder shows "How I read your trigger": what is in the test, what is not, how many eligible conversations created in the last 30 days would have matched, how many that is per group, where the test would run and up to four example tickets marked "in test" or "not in test". Reply "yes" or "ja" to confirm, or say what to change; every change gives a new version of the reading.
The estimate uses the latest recorded category and only the stores and channels where the control runs. Deleted and merged conversations and conversations without a recorded category are left out. The 30 days are measured back from now in UTC.
Things to know:
- With a payment filter the count says "up to", because past conversations do not record how the order was paid. Once the test runs, a conversation whose order or payment method cannot be read stays out of the test.
- When nothing matched in the last 30 days, "yes" does not confirm. Describe the trigger differently, or click "I still wish to proceed" to keep it. The test then starts empty and takes new matching conversations as they come in.
- A trigger that can never match anything is refused with "This trigger can never start a test".
Confirming the trigger does not start the test. It opens the summary "Ready to go live".
Ready to go live
Needs the 'Edit Configuration Tests' permission.
The summary shows what will start: the 50/50 split, the trigger with its estimate, the payment filter, the two tags, the stores and channels, and that a returning customer stays in the same group. Two things can still be set:
"Priority": "Highest", "High", "Medium" (the default), "Low" or "Lowest". It only matters when a ticket matches more than one running test: the test with the higher priority gets it, and on equal priority the tests share such tickets at random in equal parts. A customer who is already in a test stays in it. The priority can be changed later on the test's card.
"Freeze the config set used for the live experiment": on by default.
- On: the test runs both config sets exactly as they are at this moment. Edits made to either set afterwards do not reach the test.
- Off: every conversation gets the config sets as they are at that moment, so later edits reach the test at once. The results then mix conversations from before and after each edit.
This switch cannot be changed once the test is live.
"Change the trigger" goes back to the chat. "Confirm and go live" starts the test at once: from that moment new matching conversations are split. Nothing changes for customers before you click it.
Running tests
The list shows every running test and the tests that were stopped in the last 14 days. The heading reads "Running tests" when all of them run and "Tests" when some are stopped. Each card shows:
- the test set's name, the day it started and the control it runs against;
- "Running" or "Stopped";
- who started it and when, when it was stopped, and the stores it runs in;
- the trigger in the words of the person who made it, with the categories, channels and payment methods it was read as;
- the two tags. A test that was started another way, for example through a Squid Agent proposal, may say "This test writes no ticket tags.";
- "Priority": a list you can change while the test runs. The change applies to tickets that come in from then on;
- the split bar with the number of customers in each group so far;
- a link to the test's results page.
Results are not shown here. A verdict is computed on the results page after the test has concluded and every tracked order has matured. For tests started here, maturity is 30 days after the later of the order date and its last recorded test treatment.
Stop a test
Needs the 'Edit Configuration Tests' permission.
"Stop test" on a running test's card asks for confirmation and then stops the test at once:
- the test stops serving both groups, including existing test customers. Conversations use the store’s ordinary setup, which may differ from the original control if store assignments changed;
- tickets that already got a tag keep it;
- the results stay available, and the orders already in the test keep maturing towards a verdict;
- the card stays on this page for 14 days, marked "Stopped".
A stopped test cannot be started again; make a new one.
This differs from "Stop test" on the test's results page, which lets customers who are already in the test finish in their group.
A teammate whose access is limited to certain stores cannot stop a test, or change its priority, when it also runs in a store they cannot see.