UAE · ENBook a demo
Oil & GasProductDecision-maker

Experimentation Agent

Designs A/B tests and reads the results.

Experimentation AgentDiscovery and Delivery
Industry
Oil & Gas
Department
Product
Team
Discovery and Delivery
Kind
Decision-maker
Approver
Head of Digital Products
Code
PRD-11

It designs A/B tests for the customer products, such as fuel card app onboarding, loyalty offers in the app and distributor reorder prompts, and reads the results. Field safety apps are never A/B tested on safety steps; it decides ship, stop or extend and rolls out the winner once approved.

How it works

7 steps, and a person signs off before step 7.

  1. 01Looks it up

    Finds the record the request is about in the product analytics tool: users, conversions and guardrail metrics per variant..

  2. 02Reads the request

    Works out what is being asked and what a complete answer needs.

  3. 03Checks the facts

    Checks figures, dates and rules against the source records, not from memory.

  4. 04Forecasts, when it decides

    Projects what each option will cost or return before it recommends one.

  5. 05Works it through

    Prepares the draft, match, recommendation or reply, with its sources.

  6. 06Asks a person

    Hands the work to Head of Digital Products with its reasoning before anything goes out.

  7. 07Acts

    Once approved, it acts in the feature flag service: it rolls out or stops the variant at the approved step. and records what was done.

What it works with

Your knowledge, your systems, your people.

Knows

The experimentation standard: significance and power rules, guardrail metrics (payment failures, complaints, order errors), what may never be tested (safety steps, fees, terms), and rollout steps.

Reads from

the product analytics tool: users, conversions and guardrail metrics per variant.

Acts in

the feature flag service: it rolls out or stops the variant at the approved step.

Hands back

An object with design (sample_per_variant and duration_days, or null), verdict (ship, stop, extend or inconclusive), lift_percent, confidence_interval, p_value, guardrails (a list with metric, change_percent and breached) and rollout_plan (a list of steps).

Asks a person when

The verdict ships or rolls out a variant to more users, a guardrail is worse by more than 2 percent, or the test touches payments, fees or a field app.

ApproverHead of Digital Products

Settings

Your limits, set by your managers.

SettingDefaultWhat it controls
Significance level0.05A result counts below this.
Power0.8Tests are sized for this power.
Smallest effect sized for (%)5Tests are sized to detect this.
Longest test (days)28Longer tests stop.
Guardrail metricspayment failures, complaints, order errorsComma-separated.
Guardrail tolerance (%)2Worse than this stops the variant.
Rollout step (%)25A winner rolls out in steps of this.

Limits are settings, not code: a manager changes them without a developer.

Rules it always follows

  • Size tests for 0.8 power to detect 5 percent at 0.05, within 28 days.
  • Never test safety steps in field apps, fees, card terms or consent wording.
  • A variant worse than 2 percent on payment failures, complaints, order errors stops.
  • Roll out a winner in steps of 25 percent.

See the Experimentation Agent agent on your own work.

We’ll run it on a sample of your work and show you what your approver would see.

Experimentation Agent · Oil & Gas | Basinwright Industries