Data Pipeline Agent
Maintains data ingestion and transformation jobs.
- Industry
- Insure AI
- Department
- Engineering
- Team
- Platform and Reliability
- Kind
- Investigator
- Approver
- Data Platform Lead
- Code
- ENG-18
It maintains the data pipelines that feed the actuarial and reserving warehouse, IFRS 17 models, the fraud-screening service and management reporting from policy, claims and payment systems. It diagnoses late, short or wrong loads before they reach reserving or regulatory reports. **The problem it solves:** Why the data load is late, short or wrong: which pipelines, since when, and what changed at source or in the jobs.
How it works
6 steps, and a person signs off before step 6.
- 01Looks it up
Finds the record the request is about in the actuarial and reserving warehouse: load history, row counts and reconciliation results for the pipelines..
- 02Reads the request
Works out what is being asked and what a complete answer needs.
- 03Checks the facts
Checks figures, dates and rules against the source records, not from memory.
- 04Works it through
Weighs competing explanations against the evidence and shows which one holds up.
- 05Asks a person
Hands the work to Data Platform Lead with its reasoning before anything goes out.
- 06Acts
Once approved, it acts in the actuarial and reserving warehouse: it reruns or backfills the pipeline after approval and records the fix. and records what was done.
What it works with
Your knowledge, your systems, your people.
Knows
The pipeline catalogue, freshness SLAs, data contracts from policy and claims systems, the month-end and quarter-end reserving calendar and known failure modes.
Reads from
the actuarial and reserving warehouse: load history, row counts and reconciliation results for the pipelines.
Acts in
the actuarial and reserving warehouse: it reruns or backfills the pipeline after approval and records the fix.
Hands back
An object with causes (a ranked list), affected_reports, fix, backfill and reconciliation.
Asks a person when
The issue affects a reserving, IFRS 17 or regulatory report, or a backfill exceeds 90 days.
Settings
Your limits, set by your managers.
| Setting | Default | What it controls |
|---|---|---|
| Freshness SLA (hours) | 6 | Data older is late. |
| Row drop that alerts (%) | 10 | Day on day. |
| Longest backfill (days) | 90 | Longer needs approval. |
| Quarantine bad records | On | When on, bad records are held, not loaded. |
| Reconcile premium and claims totals to source | On | When on, totals must match the source within a cent. |
Limits are settings, not code: a manager changes them without a developer.
Rules it always follows
- Alert when data is older than 6 hours or rows drop by 10 percent.
- When on is true, written premium, paid claims and reserves must reconcile to source.
- When on is true, hold bad records.
- Backfills over 90 days need the Data Platform Lead.
- Rank causes by evidence: a source schema change or late source extract outranks a guess.
See the Data Pipeline Agent agent on your own work.
We’ll run it on a sample of your work and show you what your approver would see.