UAE · ENBook a demo
RetailProductDecision-maker

Experimentation Agent

Designs A/B tests and reads the results.

Experimentation AgentDiscovery and Delivery
Industry
Retail
Department
Product
Team
Discovery and Delivery
Kind
Decision-maker
Approver
E-commerce Director
Code
PRD-11

It designs A/B tests for the online store and app (search ranking, product page layout, basket upsell, checkout steps, delivery option order) with hypothesis, primary and guardrail metrics, sample size and duration, and reads the results, deciding ship, iterate, stop or keep running. It forecasts the effect of rolling a winner out on orders and margin and, once approved, sets the next rollout step on the feature flag service, never testing prices on shoppers.

How it works

7 steps, and a person signs off before step 7.

  1. 01Looks it up

    Finds the record the request is about in the experimentation platform: users, conversions, revenue and guardrail metrics by variant for the experiment named..

  2. 02Reads the request

    Works out what is being asked and what a complete answer needs.

  3. 03Checks the facts

    Checks figures, dates and rules against the source records, not from memory.

  4. 04Forecasts, when it decides

    Projects what each option will cost or return before it recommends one.

  5. 05Works it through

    Prepares the draft, match, recommendation or reply, with its sources.

  6. 06Asks a person

    Hands the work to E-commerce Director with its reasoning before anything goes out.

  7. 07Acts

    Once approved, it acts in the feature flag service: it sets the next rollout step for the winning variant. and records what was done.

What it works with

Your knowledge, your systems, your people.

Knows

The experimentation standard: statistics settings, primary metrics (conversion, revenue per visitor, average basket value), guardrails (returns rate, page load, payment errors, complaint contacts), peak freeze rules and the no-price-testing rule.

Reads from

the experimentation platform: users, conversions, revenue and guardrail metrics by variant for the experiment named.

Acts in

the feature flag service: it sets the next rollout step for the winning variant.

Hands back

An object with call (ship, iterate, stop or keep_running), variants (a list, each with variant, users, primary_metric_value, lift_percent and p_value), guardrails (a list, each with metric and change_percent), forecast (orders_change and margin_change per week) and next_rollout_percent.

Asks a person when

A winner changes checkout or payment, a guardrail worsened beyond 1 percent, or the test is stopped before its planned sample.

ApproverE-commerce Director

Settings

Your limits, set by your managers.

SettingDefaultWhat it controls
Significance level0.05The false positive rate accepted.
Statistical power0.8The chance of detecting a real effect of the minimum size.
Smallest effect to detect (%)2The relative lift the sample is sized for.
Longest test (days)28Tests are stopped or decided by this point.
Guardrail metricsreturns rate, page load time, payment error rate, customer service contactsMetrics a winner must not worsen.
Guardrail tolerance (%)1A guardrail worsening more than this blocks shipping.
Rollout step (%)25The share of traffic each rollout step adds.

Limits are settings, not code: a manager changes them without a developer.

Rules it always follows

  • Size every test for 2 percent at significance 0.05 and power 0.8, covering whole weeks and no more than 28 days.
  • Never test prices, fees or discounts by showing shoppers different prices for the same item at the same time.
  • A winner must not worsen returns rate, page load time, payment error rate, customer service contacts by more than 1 percent; a winner that lifts conversion but raises returns is not a win.
  • No test starts or ships inside the peak freeze.
  • Roll a winner out in steps of 25 percent; a checkout or payment change goes to the E-commerce Director before each step.

See the Experimentation Agent agent on your own work.

We’ll run it on a sample of your work and show you what your approver would see.

Experimentation Agent · Retail | Basinwright Industries