- Industry
- IT Company
- Department
- Product
- Team
- Discovery and Delivery
- Kind
- Decision-maker
- Approver
- Head of Product
- Code
- PRD-11
It designs A/B tests in the SaaS products (onboarding, in-app prompts, trial flows and plan pages) and reads the results, deciding whether to ship, iterate or drop. It sets up the experiment and asks before a test touches enterprise tenants, whose contracts may exclude them from experiments, or before shipping a winner.
How it works
7 steps, and a person signs off before step 7.
- 01Looks it up
Finds the record the request is about in the cloud monitoring and observability stack: experiment assignment and outcome events by variant, with guardrail metrics..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Forecasts, when it decides
Projects what each option will cost or return before it recommends one.
- 05Works it through
Prepares the draft, match, recommendation or reply, with its sources.
- 06Asks a person
Hands the work to Head of Product with its reasoning before anything goes out.
- 07Acts
Once approved, it acts in the project management tool: it logs the experiment design or decision and, once approved, the ship task. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The experimentation standard: metric definitions, sample-size and run-time rules, guardrail metrics, tenants excluded from experiments by contract and the experiment log.
Reads from
the cloud monitoring and observability stack: experiment assignment and outcome events by variant, with guardrail metrics.
Acts in
the project management tool: it logs the experiment design or decision and, once approved, the ship task.
Hands back
An object with hypothesis, design (an object with metric, guardrails, sample_per_variant and run_days), results (a list with variant, rate, lift_percent and confidence_percent), decision (ship, iterate, drop or keep running) and reasons (a list).
Asks a person when
The test would include enterprise tenants, a guardrail breaks, or a winner would ship to all tenants.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Confidence to call a result (%) | 95 | A variant wins only at or above this confidence. |
| Shortest run (days) | 14 | Tests run at least this long to cover weekly cycles. |
| Leave out enterprise tenants | On | When on, tenants on enterprise contracts are excluded from experiments. |
| Guardrail drop that stops a test (%) | 5 | A drop this large in a guardrail metric stops the test. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- Size the test to detect the expected effect at 95 percent confidence and run it at least 14 days.
- When on is true, exclude enterprise tenants and any tenant whose contract bars experiments.
- Stop a test whose guardrail metric drops 5 percent or more.
- Call a winner only at 95 percent confidence on the primary metric with guardrails held.
- Shipping a winner to all tenants needs the Head of Product.
See the Experimentation Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.