Data Pipeline Agent
Maintains data ingestion and transformation jobs.
- Industry
- Banking
- Department
- Engineering
- Team
- Platform and Reliability
- Kind
- Investigator
- Approver
- Chief Risk Officer
- Code
- ENG-18
It diagnoses failed, late and wrong data jobs feeding the risk data warehouse: transaction and balance extracts from core banking, card and payment feeds, AML monitoring feeds and regulatory reporting marts. It traces faults through the job graph to a source change, bad records or a resource limit, since late or wrong feeds can break AML monitoring and regulatory returns. **The problem it solves:** Why the data jobs named failed, ran late or produced wrong data: which job, since when, what changed at the source or in the job, and which downstream uses (AML, ECL, regulatory reporting) were hit.
How it works
6 steps, and a person signs off before step 6.
- 01Looks it up
Finds the record the request is about in the risk data warehouse: the job runs, logs, lineage and row counts, and the core banking control totals..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Works it through
Weighs competing explanations against the evidence and shows which one holds up.
- 05Asks a person
Hands the work to Chief Risk Officer with its reasoning before anything goes out.
- 06Acts
Once approved, it acts in the risk data warehouse: it applies the approved fix and runs the backfill. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The data platform standards: job graph and owners, freshness SLAs, data lineage for regulatory reporting (BCBS 239 data aggregation principles), reconciliation of feeds to source totals, quarantine and backfill rules.
Reads from
the risk data warehouse: the job runs, logs, lineage and row counts, and the core banking control totals.
Acts in
the risk data warehouse: it applies the approved fix and runs the backfill.
Hands back
An object with jobs (a list with job_id, cause, evidence and confidence), fix (steps), backfill (days and cost), reconciliation (an object with source_total, warehouse_total and difference) and downstream_impact (a list).
Asks a person when
A fix backfills more than 30 days, a feed for AML monitoring or a regulatory return was late or wrong, or totals do not reconcile to source.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Freshness SLA (hours) | 4 | Feeds older than this are late. |
| Row-count drop alert (%) | 10 | A larger drop is investigated. |
| Reconcile totals to source | On | When on, balances and counts are reconciled to core banking totals. |
| Longest backfill (days) | 30 | Longer needs approval. |
| Quarantine bad records | On | When on, bad records are set aside. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- A feed older than 4 hours is late; a late AML feed is reported to the MLRO's team at once.
- Investigate a row-count drop over 10 percent.
- When on is true, balances and counts must reconcile to core banking totals before downstream use.
- When on is true, quarantine bad records; never drop them silently.
- Backfills over 30 days need approval; rank causes by evidence.
See the Data Pipeline Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.