UAE · ENBook a demo
IT CompanyEngineeringInvestigator

Data Pipeline Agent

Maintains data ingestion and transformation jobs.

Data Pipeline AgentPlatform and Reliability
Industry
IT Company
Department
Engineering
Team
Platform and Reliability
Kind
Investigator
Approver
Engineering Manager
Code
ENG-18

It diagnoses failed, late and wrong data jobs in client data-platform engagements and the SaaS analytics pipeline, tracing the fault through the job graph to a source schema change, bad records, a credential, capacity or a code change. It proposes the fix and any backfill, keeping client data inside its contracted region and the backfill cost within limit. **The problem it solves:** Why the data job failed, ran late or produced wrong data: where in the graph it went wrong, what changed, and what the impact on clients is.

How it works

6 steps, and a person signs off before step 6.

  1. 01Looks it up

    Finds the record the request is about in the cloud monitoring and observability stack: the job runs, logs, row counts and data quality results for the job named..

  2. 02Reads the request

    Works out what is being asked and what a complete answer needs.

  3. 03Checks the facts

    Checks figures, dates and rules against the source records, not from memory.

  4. 04Works it through

    Weighs competing explanations against the evidence and shows which one holds up.

  5. 05Asks a person

    Hands the work to Engineering Manager with its reasoning before anything goes out.

  6. 06Acts

    Once approved, it acts in the ticketing and service management system: it records the diagnosis and schedules the fix and backfill. and records what was done.

What it works with

Your knowledge, your systems, your people.

Knows

The data platform runbook: job graph, freshness SLAs per client, data quality checks, backfill rules, data-residency terms and cost limits.

Reads from

the cloud monitoring and observability stack: the job runs, logs, row counts and data quality results for the job named.

Acts in

the ticketing and service management system: it records the diagnosis and schedules the fix and backfill.

Hands back

An object with causes (a ranked list with cause, evidence and confidence), fix, backfill (an object with days and cost_usd, or null), quarantined_records and sla_breached (true or false).

Asks a person when

The backfill exceeds 30 days or 500 USD, or client-facing data was wrong.

ApproverEngineering Manager

Settings

Your limits, set by your managers.

SettingDefaultWhat it controls
Freshness SLA (hours)6Data later than this breaches the SLA unless the client's contract says otherwise.
Row count drop that is wrong (%)20A drop larger than this is a fault.
Longest backfill (days)30Longer backfills need the approver.
Quarantine bad recordsOnWhen on, bad records are held aside, not dropped.
Backfill cost limit (USD)500Backfills estimated above this need the approver.

Limits are settings, not code: a manager changes them without a developer.

Rules it always follows

  • Trace the fault through the job graph to its first bad step and rank causes by evidence.
  • Treat a row drop over 20 percent as a fault even if the job succeeded.
  • When on is true, hold bad records aside and report them.
  • A backfill over 30 days or 500 USD needs the Engineering Manager.
  • Keep client data in its contracted region during any fix or backfill.
  • Report a freshness breach of 6 hours to the client's account team.

See the Data Pipeline Agent agent on your own work.

We’ll run it on a sample of your work and show you what your approver would see.

Data Pipeline Agent · IT Company | Basinwright Industries