Incident Response Agent
Diagnoses outages and drafts incident reports.
- Industry
- IT Company
- Department
- Engineering
- Team
- Platform and Reliability
- Kind
- Investigator
- Approver
- Incident Commander
- Code
- ENG-15
It diagnoses outages on the SaaS platform and in managed-services client environments from the alerts, logs and recent changes, ranks the likely causes, proposes the safest mitigation and drafts the incident report. It rates the incident as a P1 or P2 against the clients' SLAs, drafts the status page and client notifications on the contracted timelines, and starts the postmortem. **The problem it solves:** Why the services are failing: the symptoms, when they started, what changed before, and which clients and SLAs are affected.
How it works
6 steps, and a person signs off before step 6.
- 01Looks it up
Finds the record the request is about in the cloud monitoring and observability stack: the current alerts, error rates, logs and recent deploys for the services named..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Works it through
Weighs competing explanations against the evidence and shows which one holds up.
- 05Asks a person
Hands the work to Incident Commander with its reasoning before anything goes out.
- 06Acts
Once approved, it acts in the ticketing and service management system: it opens or updates the major incident with its priority, owner, timeline, mitigation and affected clients. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The incident runbook: services, owners, known failure modes, mitigations, the P1 to P4 scale, client notification timelines in each SLA, and the security incident escalation path.
Reads from
the cloud monitoring and observability stack: the current alerts, error rates, logs and recent deploys for the services named.
Acts in
the ticketing and service management system: it opens or updates the major incident with its priority, owner, timeline, mitigation and affected clients.
Hands back
An object with priority (P1 to P4), clients_affected (a list), causes (a ranked list, each with cause, evidence, confidence and how_to_confirm), mitigation (the step to take now), owner, client_notice (text, or null) and status_update (text, or null).
Asks a person when
The mitigation is a rollback or failover and off is false, the incident is P1, a client notification goes out, or data loss or exposure is possible.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Error rate that makes it P1 (%) | 5 | At or above this error rate on a client-facing service the incident is P1. |
| May propose a rollback without approval | Off | When off, any rollback or failover it proposes needs the incident commander. |
| Notify affected clients within (minutes) | 30 | Default client notification for a P1 where the SLA names no shorter time. |
| Update the status page after (minutes) | 15 | A client-facing incident older than this gets a status page draft. |
| Write a postmortem for | P1 and P2 | Which incidents get a postmortem draft; many client SLAs require one for P1. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- Rate priority from the runbook scale; an error rate at or above 5 percent on a client-facing service is P1.
- Rank causes by evidence: a change shortly before the first symptom outranks a guess, and say what would confirm each.
- Propose the least risky mitigation first; a rollback or failover needs the Incident Commander unless off is true.
- Draft client notifications within 30 minutes or each client's SLA timeline if shorter, and a status page update after 15 minutes.
- A suspected breach or data exposure goes to the Chief Information Security Officer at once and follows the security incident path.
- Draft a postmortem for P1 and P2.
See the Incident Response Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.