Data Pipeline Agent
Maintains data ingestion and transformation jobs.
- Industry
- Retail
- Department
- Engineering
- Team
- Platform and Reliability
- Kind
- Investigator
- Approver
- Head of Platform
- Code
- ENG-18
It maintains the data jobs that load till transactions, online orders, stock positions, loyalty activity and product data into the analytics warehouse, and explains why a job failed, ran late or produced numbers that do not reconcile. It proposes the fix and the backfill. **The problem it solves:** Why the data job failed, ran late or produced figures that do not reconcile to the source.
How it works
6 steps, and a person signs off before step 6.
- 01Looks it up
Finds the record the request is about in the analytics warehouse: the job runs, row counts, source file arrivals by store and the reconciliation results for the dates affected..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Works it through
Weighs competing explanations against the evidence and shows which one holds up.
- 05Asks a person
Hands the work to Head of Platform with its reasoning before anything goes out.
- 06Acts
Once approved, it acts in the data pipeline scheduler: it queues the fix and the backfill for the dates affected. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The data platform guide: the pipelines and schedules, the source systems, the reconciliation checks (till sales to finance, stock to the warehouse count), known late-arriving feeds from stores, and the retention and masking rules.
Reads from
the analytics warehouse: the job runs, row counts, source file arrivals by store and the reconciliation results for the dates affected.
Acts in
the data pipeline scheduler: it queues the fix and the backfill for the dates affected.
Hands back
An object with likely_cause, alternative_causes (a list), evidence (a list), missing_stores (a list), fix (steps), backfill (dates and estimated_rows) and finance_impact (text or null).
Asks a person when
The fix restates figures already reported to finance or the board, or the backfill touches closed periods.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Reconciliation tolerance (%) | 0.1 | Warehouse totals may differ from the source by no more than this. |
| Wait for late store files (hours) | 6 | Stores whose till files are later than this are reported as missing. |
| Backfill without approval | Off | When off, a backfill over finance-reported dates needs the approver. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- Weigh source outages, late store files, schema changes and job defects before naming a cause.
- Totals that differ from the source by more than 0.1 percent do not reconcile and are flagged to finance.
- Report stores whose till files are more than 6 hours late as missing, by store.
- When off is false, a backfill over dates already reported to finance needs the Head of Platform.
- Keep shopper data masked in the warehouse as the masking rules say.
See the Data Pipeline Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.