UAE · ENBook a demo
RetailEngineeringInvestigator

Data Pipeline Agent

Maintains data ingestion and transformation jobs.

Data Pipeline AgentPlatform and Reliability
Industry
Retail
Department
Engineering
Team
Platform and Reliability
Kind
Investigator
Approver
Head of Platform
Code
ENG-18

It maintains the data jobs that load till transactions, online orders, stock positions, loyalty activity and product data into the analytics warehouse, and explains why a job failed, ran late or produced numbers that do not reconcile. It proposes the fix and the backfill. **The problem it solves:** Why the data job failed, ran late or produced figures that do not reconcile to the source.

How it works

6 steps, and a person signs off before step 6.

  1. 01Looks it up

    Finds the record the request is about in the analytics warehouse: the job runs, row counts, source file arrivals by store and the reconciliation results for the dates affected..

  2. 02Reads the request

    Works out what is being asked and what a complete answer needs.

  3. 03Checks the facts

    Checks figures, dates and rules against the source records, not from memory.

  4. 04Works it through

    Weighs competing explanations against the evidence and shows which one holds up.

  5. 05Asks a person

    Hands the work to Head of Platform with its reasoning before anything goes out.

  6. 06Acts

    Once approved, it acts in the data pipeline scheduler: it queues the fix and the backfill for the dates affected. and records what was done.

What it works with

Your knowledge, your systems, your people.

Knows

The data platform guide: the pipelines and schedules, the source systems, the reconciliation checks (till sales to finance, stock to the warehouse count), known late-arriving feeds from stores, and the retention and masking rules.

Reads from

the analytics warehouse: the job runs, row counts, source file arrivals by store and the reconciliation results for the dates affected.

Acts in

the data pipeline scheduler: it queues the fix and the backfill for the dates affected.

Hands back

An object with likely_cause, alternative_causes (a list), evidence (a list), missing_stores (a list), fix (steps), backfill (dates and estimated_rows) and finance_impact (text or null).

Asks a person when

The fix restates figures already reported to finance or the board, or the backfill touches closed periods.

ApproverHead of Platform

Settings

Your limits, set by your managers.

SettingDefaultWhat it controls
Reconciliation tolerance (%)0.1Warehouse totals may differ from the source by no more than this.
Wait for late store files (hours)6Stores whose till files are later than this are reported as missing.
Backfill without approvalOffWhen off, a backfill over finance-reported dates needs the approver.

Limits are settings, not code: a manager changes them without a developer.

Rules it always follows

  • Weigh source outages, late store files, schema changes and job defects before naming a cause.
  • Totals that differ from the source by more than 0.1 percent do not reconcile and are flagged to finance.
  • Report stores whose till files are more than 6 hours late as missing, by store.
  • When off is false, a backfill over dates already reported to finance needs the Head of Platform.
  • Keep shopper data masked in the warehouse as the masking rules say.

See the Data Pipeline Agent agent on your own work.

We’ll run it on a sample of your work and show you what your approver would see.

Data Pipeline Agent · Retail | Basinwright Industries