UAE · ENBook a demo
BankingEngineeringInvestigator

Data Pipeline Agent

Maintains data ingestion and transformation jobs.

Data Pipeline AgentPlatform and Reliability
Industry
Banking
Department
Engineering
Team
Platform and Reliability
Kind
Investigator
Approver
Chief Risk Officer
Code
ENG-18

It diagnoses failed, late and wrong data jobs feeding the risk data warehouse: transaction and balance extracts from core banking, card and payment feeds, AML monitoring feeds and regulatory reporting marts. It traces faults through the job graph to a source change, bad records or a resource limit, since late or wrong feeds can break AML monitoring and regulatory returns. **The problem it solves:** Why the data jobs named failed, ran late or produced wrong data: which job, since when, what changed at the source or in the job, and which downstream uses (AML, ECL, regulatory reporting) were hit.

How it works

6 steps, and a person signs off before step 6.

  1. 01Looks it up

    Finds the record the request is about in the risk data warehouse: the job runs, logs, lineage and row counts, and the core banking control totals..

  2. 02Reads the request

    Works out what is being asked and what a complete answer needs.

  3. 03Checks the facts

    Checks figures, dates and rules against the source records, not from memory.

  4. 04Works it through

    Weighs competing explanations against the evidence and shows which one holds up.

  5. 05Asks a person

    Hands the work to Chief Risk Officer with its reasoning before anything goes out.

  6. 06Acts

    Once approved, it acts in the risk data warehouse: it applies the approved fix and runs the backfill. and records what was done.

What it works with

Your knowledge, your systems, your people.

Knows

The data platform standards: job graph and owners, freshness SLAs, data lineage for regulatory reporting (BCBS 239 data aggregation principles), reconciliation of feeds to source totals, quarantine and backfill rules.

Reads from

the risk data warehouse: the job runs, logs, lineage and row counts, and the core banking control totals.

Acts in

the risk data warehouse: it applies the approved fix and runs the backfill.

Hands back

An object with jobs (a list with job_id, cause, evidence and confidence), fix (steps), backfill (days and cost), reconciliation (an object with source_total, warehouse_total and difference) and downstream_impact (a list).

Asks a person when

A fix backfills more than 30 days, a feed for AML monitoring or a regulatory return was late or wrong, or totals do not reconcile to source.

ApproverChief Risk Officer

Settings

Your limits, set by your managers.

SettingDefaultWhat it controls
Freshness SLA (hours)4Feeds older than this are late.
Row-count drop alert (%)10A larger drop is investigated.
Reconcile totals to sourceOnWhen on, balances and counts are reconciled to core banking totals.
Longest backfill (days)30Longer needs approval.
Quarantine bad recordsOnWhen on, bad records are set aside.

Limits are settings, not code: a manager changes them without a developer.

Rules it always follows

  • A feed older than 4 hours is late; a late AML feed is reported to the MLRO's team at once.
  • Investigate a row-count drop over 10 percent.
  • When on is true, balances and counts must reconcile to core banking totals before downstream use.
  • When on is true, quarantine bad records; never drop them silently.
  • Backfills over 30 days need approval; rank causes by evidence.

See the Data Pipeline Agent agent on your own work.

We’ll run it on a sample of your work and show you what your approver would see.

Data Pipeline Agent · Banking | Basinwright Industries