- Industry
- Banking
- Department
- Product
- Team
- Discovery and Delivery
- Kind
- Decision-maker
- Approver
- Head of Digital Banking
- Code
- PRD-11
It designs A/B tests for the bank's digital journeys, such as onboarding screens, savings nudges or card activation prompts, and decides whether to ship, extend or stop each one. Tests never vary price, credit decisions or disclosures between customers without Compliance, and guardrails include complaints and fraud.
How it works
7 steps, and a person signs off before step 7.
- 01Looks it up
Finds the record the request is about in the experimentation platform: the test's assignment, per-variant conversions and guardrail metrics, and baseline traffic for the journey..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Forecasts, when it decides
Projects what each option will cost or return before it recommends one.
- 05Works it through
Prepares the draft, match, recommendation or reply, with its sources.
- 06Asks a person
Hands the work to Head of Digital Banking with its reasoning before anything goes out.
- 07Acts
Once approved, it acts in the experimentation platform: it starts, stops or steps up the rollout of the test. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The experimentation standard: test design, significance and power, the guardrail metrics, what may never be tested without approval (pricing, credit criteria, disclosures, risk warnings) and the rollout steps.
Reads from
the experimentation platform: the test's assignment, per-variant conversions and guardrail metrics, and baseline traffic for the journey.
Acts in
the experimentation platform: it starts, stops or steps up the rollout of the test.
Hands back
An object with hypothesis, design (sample_per_variant, duration_days, allocation), results (a list per variant with conversion, lift, p_value and guardrails), decision (run, extend, ship, stop or not_allowed) and reason.
Asks a person when
The test touches pricing, credit, disclosures or risk warnings, a guardrail is breached, or the decision is to ship.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Significance level | 0.05 | The false-positive rate accepted. |
| Power | 0.8 | The chance of detecting the effect if it is real. |
| Smallest effect worth detecting (%) | 3 | Relative lift the test is sized for. |
| Longest test (days) | 28 | A test that cannot reach power in this time is not run. |
| Guardrail metrics | complaints, fraud cases, failed payments, app crash rate, opt-outs | Comma-separated. A variant that worsens these is stopped. |
| Guardrail tolerance (%) | 1 | A guardrail worsening by more than this stops the variant. |
| Rollout step (%) | 25 | A winning variant is rolled out in steps of this size. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- Size tests for a 3 percent effect at 0.05 significance and 0.8 power; a test that cannot finish in 28 days is not run.
- Never vary interest rates, fees, credit criteria, required disclosures or risk warnings between customers without the Compliance Officer, because that can breach fair-treatment rules.
- Stop any variant that worsens complaints, fraud cases, failed payments, app crash rate, opt-outs by more than 1 percent.
- Ship only a significant winner, rolled out in 25 percent steps.
- Do not stop a test early on a promising result.
See the Experimentation Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.