UAE · ENBook a demo
Insure AIProductDecision-maker

Experimentation Agent

Designs A/B tests and reads the results.

Experimentation AgentDiscovery and Delivery
Industry
Insure AI
Department
Product
Team
Discovery and Delivery
Kind
Decision-maker
Approver
Head of Product
Code
PRD-11

It designs A/B tests on the digital quote-and-bind, renewal and claims journeys (hypothesis, primary and guardrail metrics, sample size and duration) and reads the results, deciding ship, iterate, stop or keep running. It never tests price differences on identical risks, and once approved it sets the rollout of the winning variant in the journey configuration.

How it works

7 steps, and a person signs off before step 7.

  1. 01Looks it up

    Finds the record the request is about in the policy administration system: journey traffic, quotes, binds, add-on attaches, FNOL completions, free-look cancellations and complaints by test variant..

  2. 02Reads the request

    Works out what is being asked and what a complete answer needs.

  3. 03Checks the facts

    Checks figures, dates and rules against the source records, not from memory.

  4. 04Forecasts, when it decides

    Projects what each option will cost or return before it recommends one.

  5. 05Works it through

    Prepares the draft, match, recommendation or reply, with its sources.

  6. 06Asks a person

    Hands the work to Head of Product with its reasoning before anything goes out.

  7. 07Acts

    Once approved, it acts in the policy administration system: it sets the journey configuration's variant traffic split for the rollout. and records what was done.

What it works with

Your knowledge, your systems, your people.

Knows

The experimentation guide: allowed test types, metric definitions (quote-to-bind, add-on attach, FNOL completion, complaint rate), the rule against price tests on identical risks, statistical standards and past test results.

Reads from

the policy administration system: journey traffic, quotes, binds, add-on attaches, FNOL completions, free-look cancellations and complaints by test variant.

Acts in

the policy administration system: it sets the journey configuration's variant traffic split for the rollout.

Hands back

An object with stage, design (hypothesis, primary_metric, guardrails, sample_per_variant, duration_days, or null), results (a list of metric, variant, lift_percent, p_value, or null), guardrail_breaches (a list), decision (ship, iterate, stop or keep_running) and rollout_plan (a list of step_percent and date, or null).

Asks a person when

A variant changes a disclosure or how customers cancel, claim or complain, a guardrail worsens by more than 2 percent, or the decision is to ship.

ApproverHead of Product

Settings

Your limits, set by your managers.

SettingDefaultWhat it controls
Significance level0.05The p-value threshold for a winner.
Statistical power0.8Power used to size the test.
Smallest effect worth detecting (%)5Relative lift the test is sized to detect.
Longest test (days)28Tests needing longer than this are redesigned.
Guardrail metricscomplaint rate, free-look cancellations, FNOL abandonmentComma-separated metrics a winner must not harm.
Guardrail tolerance (%)2A winner that worsens a guardrail by more than this is not shipped.
Rollout step (%)25The winner is rolled out in steps of this share of traffic.

Limits are settings, not code: a manager changes them without a developer.

Rules it always follows

  • Never test different premiums, excesses or cover for identical risks; test only wording, layout, order and journey steps.
  • Size the test for 5 percent at significance 0.05 and power 0.8; redesign if it needs more than 28 days.
  • A variant wins only when the primary metric clears significance 0.05 and no guardrail in complaint rate, free-look cancellations, FNOL abandonment worsens by more than 2 percent.
  • A variant that changes a regulatory disclosure, key facts or how cancellation and complaints are offered needs the Head of Product before it runs.
  • Roll out a winner in steps of 25 percent, watching guardrails at each step.
  • When results are inconclusive at 28 days, stop and record it rather than run on.

See the Experimentation Agent agent on your own work.

We’ll run it on a sample of your work and show you what your approver would see.

Experimentation Agent · Insure AI | Basinwright Industries