Incident Response Agent
Diagnoses outages and drafts incident reports.
- Industry
- Banking
- Department
- Engineering
- Team
- Platform and Reliability
- Kind
- Investigator
- Approver
- Head of Technology Operations
- Code
- ENG-15
It diagnoses outages in the bank's services (card declines spiking, real-time payments failing, app sign-in down, end-of-day batch overrunning) from the alerts, logs and recent changes, ranks the likely causes, proposes the safest mitigation and drafts the incident report. It tracks the impact tolerance and the regulator's incident notification clock. **The problem it solves:** Why the services named are failing: the symptoms, when they started, which alerts fired, what changed before, and how many customers and transactions are affected.
How it works
6 steps, and a person signs off before step 6.
- 01Looks it up
Finds the record the request is about in the monitoring platform: the alerts, logs, traces, recent changes and transaction failure counts for the services..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Works it through
Weighs competing explanations against the evidence and shows which one holds up.
- 05Asks a person
Hands the work to Head of Technology Operations with its reasoning before anything goes out.
- 06Acts
Once approved, it acts in the incident management tool: it updates the incident record with the diagnosis, mitigation and notifications. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The incident management process: severity definitions, important business services and impact tolerances, the regulator's incident notification deadlines, customer communication rules, failover and rollback runbooks, and post-incident review.
Reads from
the monitoring platform: the alerts, logs, traces, recent changes and transaction failure counts for the services.
Acts in
the incident management tool: it updates the incident record with the diagnosis, mitigation and notifications.
Hands back
An object with severity, causes (a ranked list with cause, evidence, confidence and how_to_confirm), mitigation (steps and risk), impact (customers_affected, transactions_failed and minutes_vs_tolerance), notifications (customer and regulator drafts) and report_draft.
Asks a person when
The mitigation is a rollback, failover or payment-system disconnect, the incident is SEV1, a customer notice or regulator notification is due, or customers suffered financial loss.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| SEV1 error rate (%) | 5 | Above this on an important service is SEV1. |
| May roll back | Off | When off, rollback is proposed, not done. |
| Customer notice after (minutes) | 15 | A customer notice is drafted by then. |
| Regulator notification within (hours) | 4 | A major incident is notified within this. |
| Post-incident review for | SEV1 and SEV2 | Severities that get a review. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- An error rate above 5 percent on an important business service is SEV1.
- Rank causes by evidence, checking recent changes first, and say what would confirm each.
- Propose the safest mitigation; when off is false, propose a rollback rather than do it.
- Draft a customer notice by 15 minutes and the regulator notification within 4 hours for a major incident.
- Hold a post-incident review for SEV1 and SEV2.
See the Incident Response Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.