UAE · ENBook a demo
IT CompanyProductDecision-maker

Experimentation Agent

Designs A/B tests and reads the results.

Experimentation AgentDiscovery and Delivery
Industry
IT Company
Department
Product
Team
Discovery and Delivery
Kind
Decision-maker
Approver
Head of Product
Code
PRD-11

It designs A/B tests in the SaaS products (onboarding, in-app prompts, trial flows and plan pages) and reads the results, deciding whether to ship, iterate or drop. It sets up the experiment and asks before a test touches enterprise tenants, whose contracts may exclude them from experiments, or before shipping a winner.

How it works

7 steps, and a person signs off before step 7.

  1. 01Looks it up

    Finds the record the request is about in the cloud monitoring and observability stack: experiment assignment and outcome events by variant, with guardrail metrics..

  2. 02Reads the request

    Works out what is being asked and what a complete answer needs.

  3. 03Checks the facts

    Checks figures, dates and rules against the source records, not from memory.

  4. 04Forecasts, when it decides

    Projects what each option will cost or return before it recommends one.

  5. 05Works it through

    Prepares the draft, match, recommendation or reply, with its sources.

  6. 06Asks a person

    Hands the work to Head of Product with its reasoning before anything goes out.

  7. 07Acts

    Once approved, it acts in the project management tool: it logs the experiment design or decision and, once approved, the ship task. and records what was done.

What it works with

Your knowledge, your systems, your people.

Knows

The experimentation standard: metric definitions, sample-size and run-time rules, guardrail metrics, tenants excluded from experiments by contract and the experiment log.

Reads from

the cloud monitoring and observability stack: experiment assignment and outcome events by variant, with guardrail metrics.

Acts in

the project management tool: it logs the experiment design or decision and, once approved, the ship task.

Hands back

An object with hypothesis, design (an object with metric, guardrails, sample_per_variant and run_days), results (a list with variant, rate, lift_percent and confidence_percent), decision (ship, iterate, drop or keep running) and reasons (a list).

Asks a person when

The test would include enterprise tenants, a guardrail breaks, or a winner would ship to all tenants.

ApproverHead of Product

Settings

Your limits, set by your managers.

SettingDefaultWhat it controls
Confidence to call a result (%)95A variant wins only at or above this confidence.
Shortest run (days)14Tests run at least this long to cover weekly cycles.
Leave out enterprise tenantsOnWhen on, tenants on enterprise contracts are excluded from experiments.
Guardrail drop that stops a test (%)5A drop this large in a guardrail metric stops the test.

Limits are settings, not code: a manager changes them without a developer.

Rules it always follows

  • Size the test to detect the expected effect at 95 percent confidence and run it at least 14 days.
  • When on is true, exclude enterprise tenants and any tenant whose contract bars experiments.
  • Stop a test whose guardrail metric drops 5 percent or more.
  • Call a winner only at 95 percent confidence on the primary metric with guardrails held.
  • Shipping a winner to all tenants needs the Head of Product.

See the Experimentation Agent agent on your own work.

We’ll run it on a sample of your work and show you what your approver would see.

Experimentation Agent · IT Company | Basinwright Industries