Incident Response Agent
Diagnoses outages and drafts incident reports.
- Industry
- Procure AI
- Department
- Engineering
- Team
- Platform and Reliability
- Kind
- Investigator
- Approver
- Incident Commander
- Code
- ENG-15
It diagnoses platform outages, such as approvals not routing, invoices stuck between the e-invoicing network and the match engine, payment files not reaching banks or a client ERP integration failing, from the alerts, logs and recent changes, ranks the likely causes, proposes the safest mitigation and drafts the incident report. It rates impact by the clients, documents and payment runs affected, not only by error rates. **The problem it solves:** Why the platform services are failing: the symptoms, when they started, what changed before, and which clients, documents and payment runs are affected.
How it works
6 steps, and a person signs off before step 6.
- 01Looks it up
Finds the record the request is about in the monitoring system: the current alerts, error rates, queue depths, recent deploys and credential rotations for the services named, and the payment runs due..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Works it through
Weighs competing explanations against the evidence and shows which one holds up.
- 05Asks a person
Hands the work to Incident Commander with its reasoning before anything goes out.
- 06Acts
Once approved, it acts in the incident tool: it opens or updates the incident with its severity, owner, timeline and mitigation. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The engineering runbook: services, owners, known failure modes (expired client ERP credentials, network endpoint outages, OCR backlog, match engine locks), mitigations, the incident severity scale and the client notification commitments.
Reads from
the monitoring system: the current alerts, error rates, queue depths, recent deploys and credential rotations for the services named, and the payment runs due.
Acts in
the incident tool: it opens or updates the incident with its severity, owner, timeline and mitigation.
Hands back
An object with severity (SEV1 to SEV4), impact (an object with clients, documents_stuck and payment_runs_at_risk), causes (a ranked list, each with cause, evidence, confidence and how_to_confirm), mitigation (the step to take now), owner (the service owner from the runbook) and status_update (text, or null).
Asks a person when
The mitigation is a rollback or failover and off is false, the severity is SEV1, data loss or a duplicate payment is possible, or invoices or payment files would be replayed.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Error rate that makes it SEV1 (%) | 5 | At or above this error rate on a client-facing service the incident is SEV1. |
| Payment run at risk within (hours) makes it SEV1 | 24 | An incident that blocks invoices or payment files for a client payment run due within this many hours is SEV1. |
| May propose a rollback without approval | Off | When off, any rollback or failover needs the incident commander. |
| Update the status page after (minutes) | 15 | A client-facing incident older than this gets a status page draft. |
| Write a postmortem for | SEV1 and SEV2 | Which incidents get a postmortem draft. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- Rate severity from the runbook scale; an error rate at or above 5 percent on a client-facing service, or a payment run at risk within 24 hours, is SEV1.
- Rank causes by evidence: a change or credential rotation shortly before the first symptom outranks a guess, and say what would confirm each.
- Propose the least risky mitigation first; a rollback or failover needs the Incident Commander unless off is true.
- Never replay invoices or regenerate payment files without proving the originals did not go through, so no supplier is paid twice.
- Draft a status page update when clients are affected for more than 15 minutes, naming no client.
- Draft a postmortem for SEV1 and SEV2.
See the Incident Response Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.