UAE · ENBook a demo
RetailEngineeringInvestigator

Incident Response Agent

Diagnoses outages and drafts incident reports.

Incident Response AgentPlatform and Reliability
Industry
Retail
Department
Engineering
Team
Platform and Reliability
Kind
Investigator
Approver
Head of Platform
Code
ENG-15

It diagnoses outages across the online store, app, checkout, payment, order management, store tills and the warehouse and carrier feeds from the alerts, logs and recent changes it is given, ranks the likely causes, proposes the safest mitigation and drafts the incident report. It sizes each incident by lost orders, stores affected and shoppers charged wrongly. **The problem it solves:** Why the online store, checkout, store tills or order flows failed, and which cause the evidence supports.

How it works

6 steps, and a person signs off before step 6.

  1. 01Looks it up

    Finds the record the request is about in the monitoring service: the alerts, logs, deploys, promotion launches and business signals for the services and stores named..

  2. 02Reads the request

    Works out what is being asked and what a complete answer needs.

  3. 03Checks the facts

    Checks figures, dates and rules against the source records, not from memory.

  4. 04Works it through

    Weighs competing explanations against the evidence and shows which one holds up.

  5. 05Asks a person

    Hands the work to Head of Platform with its reasoning before anything goes out.

  6. 06Acts

    Once approved, it acts in the incident tracker: it opens or updates the incident with severity, causes, mitigation and timeline. and records what was done.

What it works with

Your knowledge, your systems, your people.

Knows

The engineering runbook: services, owners, known failure modes (payment provider timeouts, promotion engine errors, stock feed stalls, till offline mode), mitigations and the incident severity scale.

Reads from

the monitoring service: the alerts, logs, deploys, promotion launches and business signals for the services and stores named.

Acts in

the incident tracker: it opens or updates the incident with severity, causes, mitigation and timeline.

Hands back

An object with severity, likely_causes (a list of cause, evidence and confidence), mitigation (steps), impact (lost_orders_estimate, stores_affected and shoppers_charged_wrongly), timeline (a list of time and event) and next_checks.

Asks a person when

The incident is SEV1, the mitigation is a rollback, failover or promotion switch-off, or shoppers may have been charged wrongly.

ApproverHead of Platform

Settings

Your limits, set by your managers.

SettingDefaultWhat it controls
Error rate that makes it SEV1 (%)5At or above this error rate on checkout, payment or order capture the incident is SEV1.
Stores unable to trade that make it SEV110This many stores unable to take payment makes the incident SEV1.
May propose a rollback without approvalOffWhen off, any rollback, failover or promotion switch-off it proposes needs the incident commander.
Update the status page after (minutes)15A shopper-facing incident gets a status update after this long.

Limits are settings, not code: a manager changes them without a developer.

Rules it always follows

  • An error rate at or above 5 percent on checkout, payment or order capture, or 10 or more stores unable to take payment, is SEV1.
  • Rank causes by evidence, starting with changes and promotion launches in the hour before the symptoms began.
  • Prefer the mitigation that restores trading fastest with the least risk, such as switching off a faulty promotion or failing over a payment route; when off is false any rollback needs the incident commander.
  • When shoppers may have been charged wrongly or twice, list the orders affected for refund before closing.
  • Update the status page after 15 minutes of a shopper-facing incident.
  • Never guess a cause the evidence does not support; list what to check next.

See the Incident Response Agent agent on your own work.

We’ll run it on a sample of your work and show you what your approver would see.

Incident Response Agent · Retail | Basinwright Industries