UAE · ENBook a demo
BankingProductDecision-maker

Experimentation Agent

Designs A/B tests and reads the results.

Experimentation AgentDiscovery and Delivery
Industry
Banking
Department
Product
Team
Discovery and Delivery
Kind
Decision-maker
Approver
Head of Digital Banking
Code
PRD-11

It designs A/B tests for the bank's digital journeys, such as onboarding screens, savings nudges or card activation prompts, and decides whether to ship, extend or stop each one. Tests never vary price, credit decisions or disclosures between customers without Compliance, and guardrails include complaints and fraud.

How it works

7 steps, and a person signs off before step 7.

  1. 01Looks it up

    Finds the record the request is about in the experimentation platform: the test's assignment, per-variant conversions and guardrail metrics, and baseline traffic for the journey..

  2. 02Reads the request

    Works out what is being asked and what a complete answer needs.

  3. 03Checks the facts

    Checks figures, dates and rules against the source records, not from memory.

  4. 04Forecasts, when it decides

    Projects what each option will cost or return before it recommends one.

  5. 05Works it through

    Prepares the draft, match, recommendation or reply, with its sources.

  6. 06Asks a person

    Hands the work to Head of Digital Banking with its reasoning before anything goes out.

  7. 07Acts

    Once approved, it acts in the experimentation platform: it starts, stops or steps up the rollout of the test. and records what was done.

What it works with

Your knowledge, your systems, your people.

Knows

The experimentation standard: test design, significance and power, the guardrail metrics, what may never be tested without approval (pricing, credit criteria, disclosures, risk warnings) and the rollout steps.

Reads from

the experimentation platform: the test's assignment, per-variant conversions and guardrail metrics, and baseline traffic for the journey.

Acts in

the experimentation platform: it starts, stops or steps up the rollout of the test.

Hands back

An object with hypothesis, design (sample_per_variant, duration_days, allocation), results (a list per variant with conversion, lift, p_value and guardrails), decision (run, extend, ship, stop or not_allowed) and reason.

Asks a person when

The test touches pricing, credit, disclosures or risk warnings, a guardrail is breached, or the decision is to ship.

ApproverHead of Digital Banking

Settings

Your limits, set by your managers.

SettingDefaultWhat it controls
Significance level0.05The false-positive rate accepted.
Power0.8The chance of detecting the effect if it is real.
Smallest effect worth detecting (%)3Relative lift the test is sized for.
Longest test (days)28A test that cannot reach power in this time is not run.
Guardrail metricscomplaints, fraud cases, failed payments, app crash rate, opt-outsComma-separated. A variant that worsens these is stopped.
Guardrail tolerance (%)1A guardrail worsening by more than this stops the variant.
Rollout step (%)25A winning variant is rolled out in steps of this size.

Limits are settings, not code: a manager changes them without a developer.

Rules it always follows

  • Size tests for a 3 percent effect at 0.05 significance and 0.8 power; a test that cannot finish in 28 days is not run.
  • Never vary interest rates, fees, credit criteria, required disclosures or risk warnings between customers without the Compliance Officer, because that can breach fair-treatment rules.
  • Stop any variant that worsens complaints, fraud cases, failed payments, app crash rate, opt-outs by more than 1 percent.
  • Ship only a significant winner, rolled out in 25 percent steps.
  • Do not stop a test early on a promising result.

See the Experimentation Agent agent on your own work.

We’ll run it on a sample of your work and show you what your approver would see.

Experimentation Agent · Banking | Basinwright Industries