- Industry
- Procure AI
- Department
- Product
- Team
- Discovery and Delivery
- Kind
- Decision-maker
- Approver
- Head of Product
- Code
- PRD-11
It designs A/B tests on the platform, such as a new guided-buying search that steers requisitioners to catalogues, a reworded supplier onboarding form or an approval reminder cadence, and reads the results: ship, iterate, stop or keep running. It forecasts the effect of rolling a winner out on maverick spend, PO cycle time or onboarding completion and, once approved, sets the next rollout step on the feature flag.
How it works
7 steps, and a person signs off before step 7.
- 01Looks it up
Finds the record the request is about in the spend analytics warehouse: experiment exposure and metric results by variant and client..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Forecasts, when it decides
Projects what each option will cost or return before it recommends one.
- 05Works it through
Prepares the draft, match, recommendation or reply, with its sources.
- 06Asks a person
Hands the work to Head of Product with its reasoning before anything goes out.
- 07Acts
Once approved, it acts in the source-to-pay suite: it sets the next rollout percentage on the feature flag. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The experimentation standards, metric definitions (catalogue share, maverick-spend share, PO cycle time, onboarding completion), guardrails, client consent to experiments and past test results.
Reads from
the spend analytics warehouse: experiment exposure and metric results by variant and client.
Acts in
the source-to-pay suite: it sets the next rollout percentage on the feature flag.
Hands back
An object with hypothesis, design (an object with metric, sample_size and duration_days, or null), verdict (ship, iterate, stop or keep_running, or null), effect_percent, guardrails (a list with metric and change_percent) and next_rollout_percent.
Asks a person when
The verdict ships or rolls out a variant, a guardrail is worse by more than 1 percent, or the test touches approvals, payments or supplier data.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Significance level | 0.05 | The threshold for a significant result. |
| Smallest effect to detect (%) | 3 | Sample size is set to detect this. |
| Longest test (days) | 35 | Long enough to cover a month-end cycle. |
| Guardrail metrics | approval cycle time, invoice exception rate, supplier onboarding drop-off | Metrics that must not get worse. |
| Guardrail tolerance (%) | 1 | A guardrail worse by more than this stops a ship. |
| Rollout step (%) | 25 | The share of users added at each rollout step. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- Tests run only on clients whose contract allows experiments; approval steps, limits and controls are never varied.
- Size the test to detect 3 percent at 0.05 and run at most 35 days, spanning a month-end.
- A variant ships only when significant and no guardrail in approval cycle time, invoice exception rate, supplier onboarding drop-off is worse by more than 1 percent.
- Roll out in steps of 25 percent of users.
- State the forecast effect on the client KPI, with its range.
See the Experimentation Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.