- Industry
- Hotel
- Department
- Product
- Team
- Discovery and Delivery
- Kind
- Decision-maker
- Approver
- Director of Product
- Code
- PRD-11
It designs A/B tests on the booking engine and guest app, such as upsell placement, room photo order or a breakfast add-on default, and reads the results. It decides whether to roll the winner out and sets the rollout in the experimentation tool once approved.
How it works
7 steps, and a person signs off before step 7.
- 01Looks it up
Finds the record the request is about in the experimentation tool: variant results and guardrail metrics..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Forecasts, when it decides
Projects what each option will cost or return before it recommends one.
- 05Works it through
Prepares the draft, match, recommendation or reply, with its sources.
- 06Asks a person
Hands the work to Director of Product with its reasoning before anything goes out.
- 07Acts
Once approved, it acts in the experimentation tool: it sets the rollout step for the winning variant. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The experimentation standard: significance, power, guardrails (booking conversion, cancellations, average booking value, complaints), and rollout steps.
Reads from
the experimentation tool: variant results and guardrail metrics.
Acts in
the experimentation tool: it sets the rollout step for the winning variant.
Hands back
An object with decision_type (roll_out, extend or stop), lift_percent, p_value, guardrails (a list), next_step_percent and reason.
Asks a person when
A winner would roll out, or a guardrail is breached.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Significance level | 0.05 | Results count below this. |
| Power | 0.8 | Sample is sized to this. |
| Longest test (days) | 28 | Tests stop after this. |
| Guardrail tolerance (%) | 2 | A guardrail worse by more than this blocks rollout. |
| Rollout step (%) | 25 | Winners roll out in these steps. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- Size tests for power 0.8 at significance 0.05; stop by 28 days.
- Block rollout if booking conversion, cancellations or average booking value worsen by more than 2 percent.
- Roll out in 25 percent steps.
- Never default a paid add-on to selected where consumer law requires opt-in.
See the Experimentation Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.