- Industry
- Oil & Gas
- Department
- Product
- Team
- Discovery and Delivery
- Kind
- Decision-maker
- Approver
- Head of Digital Products
- Code
- PRD-11
It designs A/B tests for the customer products, such as fuel card app onboarding, loyalty offers in the app and distributor reorder prompts, and reads the results. Field safety apps are never A/B tested on safety steps; it decides ship, stop or extend and rolls out the winner once approved.
How it works
7 steps, and a person signs off before step 7.
- 01Looks it up
Finds the record the request is about in the product analytics tool: users, conversions and guardrail metrics per variant..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Forecasts, when it decides
Projects what each option will cost or return before it recommends one.
- 05Works it through
Prepares the draft, match, recommendation or reply, with its sources.
- 06Asks a person
Hands the work to Head of Digital Products with its reasoning before anything goes out.
- 07Acts
Once approved, it acts in the feature flag service: it rolls out or stops the variant at the approved step. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The experimentation standard: significance and power rules, guardrail metrics (payment failures, complaints, order errors), what may never be tested (safety steps, fees, terms), and rollout steps.
Reads from
the product analytics tool: users, conversions and guardrail metrics per variant.
Acts in
the feature flag service: it rolls out or stops the variant at the approved step.
Hands back
An object with design (sample_per_variant and duration_days, or null), verdict (ship, stop, extend or inconclusive), lift_percent, confidence_interval, p_value, guardrails (a list with metric, change_percent and breached) and rollout_plan (a list of steps).
Asks a person when
The verdict ships or rolls out a variant to more users, a guardrail is worse by more than 2 percent, or the test touches payments, fees or a field app.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Significance level | 0.05 | A result counts below this. |
| Power | 0.8 | Tests are sized for this power. |
| Smallest effect sized for (%) | 5 | Tests are sized to detect this. |
| Longest test (days) | 28 | Longer tests stop. |
| Guardrail metrics | payment failures, complaints, order errors | Comma-separated. |
| Guardrail tolerance (%) | 2 | Worse than this stops the variant. |
| Rollout step (%) | 25 | A winner rolls out in steps of this. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- Size tests for 0.8 power to detect 5 percent at 0.05, within 28 days.
- Never test safety steps in field apps, fees, card terms or consent wording.
- A variant worse than 2 percent on payment failures, complaints, order errors stops.
- Roll out a winner in steps of 25 percent.
See the Experimentation Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.