UAE · ENBook a demo
Oil & GasEngineeringInvestigator

Data Pipeline Agent

Maintains data ingestion and transformation jobs.

Data Pipeline AgentPlatform and Reliability
Industry
Oil & Gas
Department
Engineering
Team
Platform and Reliability
Kind
Investigator
Approver
Data Platform Lead
Code
ENG-18

It diagnoses failed, late and wrong data jobs that move production historian data, well tests, meter readings and maintenance records into the analytics warehouse and hydrocarbon accounting extracts. It traces each fault to a tag rename, a meter calibration change, bad readings, a resource limit or an upstream delay, and proposes the fix and backfill. **The problem it solves:** Why the production or maintenance data job failed, ran late or produced wrong figures: the job graph, the tags or meters affected and the upstream changes.

How it works

6 steps, and a person signs off before step 6.

  1. 01Looks it up

    Finds the record the request is about in the data orchestrator: job runs, logs and lineage..

  2. 02Reads the request

    Works out what is being asked and what a complete answer needs.

  3. 03Checks the facts

    Checks figures, dates and rules against the source records, not from memory.

  4. 04Works it through

    Weighs competing explanations against the evidence and shows which one holds up.

  5. 05Asks a person

    Hands the work to Data Platform Lead with its reasoning before anything goes out.

  6. 06Acts

    Once approved, it acts in the data orchestrator: it re-runs and backfills within the limits. and records what was done.

What it works with

Your knowledge, your systems, your people.

Knows

The data platform guide: the job graph, tag mapping tables, data quality rules for flow and pressure readings, freshness objectives and the month-end allocation deadlines.

Reads from

the data orchestrator: job runs, logs and lineage.

Acts in

the data orchestrator: it re-runs and backfills within the limits.

Hands back

An object with cause, evidence, fix, backfill (start, end) and volumes_affected.

Asks a person when

The fix would change data in a closed month, affects allocated or reported volumes, or backfills beyond 30 days.

ApproverData Platform Lead

Settings

Your limits, set by your managers.

SettingDefaultWhat it controls
Freshness objective (hours)6For daily production data.
Row drop alert (%)20A drop this large is investigated.
Longest backfill (days)30Further back needs approval.
Quarantine bad readingsOnReadings outside physical limits are quarantined.
Closed months lockedOnData for a closed allocation month is never rewritten.

Limits are settings, not code: a manager changes them without a developer.

Rules it always follows

  • Trace the fault to its source with evidence.
  • Quarantine readings outside physical limits when on is on, never correct a reading itself.
  • Backfill at most 30 days and never rewrite a closed allocation month when on is on.
  • Investigate drops above 20 percent and meet 6 hours freshness.

See the Data Pipeline Agent agent on your own work.

We’ll run it on a sample of your work and show you what your approver would see.

Data Pipeline Agent · Oil & Gas | Basinwright Industries