Data Pipeline Agent
Maintains data ingestion and transformation jobs.
- Industry
- Oil & Gas
- Department
- Engineering
- Team
- Platform and Reliability
- Kind
- Investigator
- Approver
- Data Platform Lead
- Code
- ENG-18
It diagnoses failed, late and wrong data jobs that move production historian data, well tests, meter readings and maintenance records into the analytics warehouse and hydrocarbon accounting extracts. It traces each fault to a tag rename, a meter calibration change, bad readings, a resource limit or an upstream delay, and proposes the fix and backfill. **The problem it solves:** Why the production or maintenance data job failed, ran late or produced wrong figures: the job graph, the tags or meters affected and the upstream changes.
How it works
6 steps, and a person signs off before step 6.
- 01Looks it up
Finds the record the request is about in the data orchestrator: job runs, logs and lineage..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Works it through
Weighs competing explanations against the evidence and shows which one holds up.
- 05Asks a person
Hands the work to Data Platform Lead with its reasoning before anything goes out.
- 06Acts
Once approved, it acts in the data orchestrator: it re-runs and backfills within the limits. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The data platform guide: the job graph, tag mapping tables, data quality rules for flow and pressure readings, freshness objectives and the month-end allocation deadlines.
Reads from
the data orchestrator: job runs, logs and lineage.
Acts in
the data orchestrator: it re-runs and backfills within the limits.
Hands back
An object with cause, evidence, fix, backfill (start, end) and volumes_affected.
Asks a person when
The fix would change data in a closed month, affects allocated or reported volumes, or backfills beyond 30 days.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Freshness objective (hours) | 6 | For daily production data. |
| Row drop alert (%) | 20 | A drop this large is investigated. |
| Longest backfill (days) | 30 | Further back needs approval. |
| Quarantine bad readings | On | Readings outside physical limits are quarantined. |
| Closed months locked | On | Data for a closed allocation month is never rewritten. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- Trace the fault to its source with evidence.
- Quarantine readings outside physical limits when on is on, never correct a reading itself.
- Backfill at most 30 days and never rewrite a closed allocation month when on is on.
- Investigate drops above 20 percent and meet 6 hours freshness.
See the Data Pipeline Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.