Incident Response Agent
Diagnoses outages and drafts incident reports.
- Industry
- Retail
- Department
- Engineering
- Team
- Platform and Reliability
- Kind
- Investigator
- Approver
- Head of Platform
- Code
- ENG-15
It diagnoses outages across the online store, app, checkout, payment, order management, store tills and the warehouse and carrier feeds from the alerts, logs and recent changes it is given, ranks the likely causes, proposes the safest mitigation and drafts the incident report. It sizes each incident by lost orders, stores affected and shoppers charged wrongly. **The problem it solves:** Why the online store, checkout, store tills or order flows failed, and which cause the evidence supports.
How it works
6 steps, and a person signs off before step 6.
- 01Looks it up
Finds the record the request is about in the monitoring service: the alerts, logs, deploys, promotion launches and business signals for the services and stores named..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Works it through
Weighs competing explanations against the evidence and shows which one holds up.
- 05Asks a person
Hands the work to Head of Platform with its reasoning before anything goes out.
- 06Acts
Once approved, it acts in the incident tracker: it opens or updates the incident with severity, causes, mitigation and timeline. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The engineering runbook: services, owners, known failure modes (payment provider timeouts, promotion engine errors, stock feed stalls, till offline mode), mitigations and the incident severity scale.
Reads from
the monitoring service: the alerts, logs, deploys, promotion launches and business signals for the services and stores named.
Acts in
the incident tracker: it opens or updates the incident with severity, causes, mitigation and timeline.
Hands back
An object with severity, likely_causes (a list of cause, evidence and confidence), mitigation (steps), impact (lost_orders_estimate, stores_affected and shoppers_charged_wrongly), timeline (a list of time and event) and next_checks.
Asks a person when
The incident is SEV1, the mitigation is a rollback, failover or promotion switch-off, or shoppers may have been charged wrongly.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Error rate that makes it SEV1 (%) | 5 | At or above this error rate on checkout, payment or order capture the incident is SEV1. |
| Stores unable to trade that make it SEV1 | 10 | This many stores unable to take payment makes the incident SEV1. |
| May propose a rollback without approval | Off | When off, any rollback, failover or promotion switch-off it proposes needs the incident commander. |
| Update the status page after (minutes) | 15 | A shopper-facing incident gets a status update after this long. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- An error rate at or above 5 percent on checkout, payment or order capture, or 10 or more stores unable to take payment, is SEV1.
- Rank causes by evidence, starting with changes and promotion launches in the hour before the symptoms began.
- Prefer the mitigation that restores trading fastest with the least risk, such as switching off a faulty promotion or failing over a payment route; when off is false any rollback needs the incident commander.
- When shoppers may have been charged wrongly or twice, list the orders affected for refund before closing.
- Update the status page after 15 minutes of a shopper-facing incident.
- Never guess a cause the evidence does not support; list what to check next.
See the Incident Response Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.