Data Pipeline Agent
Maintains data ingestion and transformation jobs.
- Industry
- IT Company
- Department
- Engineering
- Team
- Platform and Reliability
- Kind
- Investigator
- Approver
- Engineering Manager
- Code
- ENG-18
It diagnoses failed, late and wrong data jobs in client data-platform engagements and the SaaS analytics pipeline, tracing the fault through the job graph to a source schema change, bad records, a credential, capacity or a code change. It proposes the fix and any backfill, keeping client data inside its contracted region and the backfill cost within limit. **The problem it solves:** Why the data job failed, ran late or produced wrong data: where in the graph it went wrong, what changed, and what the impact on clients is.
How it works
6 steps, and a person signs off before step 6.
- 01Looks it up
Finds the record the request is about in the cloud monitoring and observability stack: the job runs, logs, row counts and data quality results for the job named..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Works it through
Weighs competing explanations against the evidence and shows which one holds up.
- 05Asks a person
Hands the work to Engineering Manager with its reasoning before anything goes out.
- 06Acts
Once approved, it acts in the ticketing and service management system: it records the diagnosis and schedules the fix and backfill. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The data platform runbook: job graph, freshness SLAs per client, data quality checks, backfill rules, data-residency terms and cost limits.
Reads from
the cloud monitoring and observability stack: the job runs, logs, row counts and data quality results for the job named.
Acts in
the ticketing and service management system: it records the diagnosis and schedules the fix and backfill.
Hands back
An object with causes (a ranked list with cause, evidence and confidence), fix, backfill (an object with days and cost_usd, or null), quarantined_records and sla_breached (true or false).
Asks a person when
The backfill exceeds 30 days or 500 USD, or client-facing data was wrong.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Freshness SLA (hours) | 6 | Data later than this breaches the SLA unless the client's contract says otherwise. |
| Row count drop that is wrong (%) | 20 | A drop larger than this is a fault. |
| Longest backfill (days) | 30 | Longer backfills need the approver. |
| Quarantine bad records | On | When on, bad records are held aside, not dropped. |
| Backfill cost limit (USD) | 500 | Backfills estimated above this need the approver. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- Trace the fault through the job graph to its first bad step and rank causes by evidence.
- Treat a row drop over 20 percent as a fault even if the job succeeded.
- When on is true, hold bad records aside and report them.
- A backfill over 30 days or 500 USD needs the Engineering Manager.
- Keep client data in its contracted region during any fix or backfill.
- Report a freshness breach of 6 hours to the client's account team.
See the Data Pipeline Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.