UAE · ENBook a demo
Procure AIEngineeringInvestigator

Spend Data Pipeline Agent

Maintains the jobs that load and classify client spend into the spend analytics warehouse.

Spend Data Pipeline AgentData Pipeline Agent · Platform and Reliability
Industry
Procure AI
Department
Engineering
Team
Platform and Reliability
Kind
Investigator
Approver
Data Platform Lead
Code
ENG-18

It diagnoses failed, late and wrong jobs that load client ERP extracts, POs, invoices and card data into the spend analytics warehouse and classify them into the category taxonomy, tracing the fault to a client extract schema change, a supplier-normalisation or currency-conversion error, a resource limit or an upstream delay. It proposes the fix and the backfill, re-runs within freshness and cost limits, and holds anything that rewrites published spend cubes or savings figures for approval. **The problem it solves:** Why the spend data job failed, ran late or produced wrong or misclassified data: the job and client, the error or the wrong figures, when it started, and what changed in the client's extract, the taxonomy, the code or the schedule.

How it works

6 steps, and a person signs off before step 6.

  1. 01Looks it up

    Finds the record the request is about in the data orchestrator: the run history, task logs, schedules, control totals and lineage for the jobs named..

  2. 02Reads the request

    Works out what is being asked and what a complete answer needs.

  3. 03Checks the facts

    Checks figures, dates and rules against the source records, not from memory.

  4. 04Works it through

    Weighs competing explanations against the evidence and shows which one holds up.

  5. 05Asks a person

    Hands the work to Data Platform Lead with its reasoning before anything goes out.

  6. 06Acts

    Once approved, it acts in the data orchestrator: it re-runs or backfills the jobs for the chosen range and adds the data-quality checks. and records what was done.

What it works with

Your knowledge, your systems, your people.

Knows

The data platform standards: the job catalogue with owners and lineage, freshness targets per client dataset, the category taxonomy and classification rules, supplier normalisation, exchange-rate sources, data-quality checks, and the backfill and retention rules.

Reads from

the data orchestrator: the run history, task logs, schedules, control totals and lineage for the jobs named.

Acts in

the data orchestrator: it re-runs or backfills the jobs for the chosen range and adds the data-quality checks.

Hands back

An object with root_cause (one sentence), causes (a ranked list, each with cause, evidence, confidence and how_to_confirm), affected_tables (a list), client_tenants (a list), downstream_impact (a list), fix, backfill (an object with from, to, tables and estimated_cost, or null) and quality_checks (a list of checks to add).

Asks a person when

The fix overwrites published spend cubes or savings figures, backfills more than 90 days, costs more than 200, or changes data already shared with a client.

ApproverData Platform Lead

Settings

Your limits, set by your managers.

SettingDefaultWhat it controls
Freshness target (hours)24Client spend data older than this, without its own target, counts as late.
Spend total drift that means wrong data (%)2A load whose spend total differs from the client's ERP control total by more than this is wrong data.
Most spend left unclassified (%)5A load leaving more than this share of spend outside the category taxonomy is flagged.
Longest backfill without approval (days)90Backfills over a longer range need the approver.
Quarantine bad recordsOnWhen on, records that fail validation are set aside so the rest of the load proceeds.
Backfill compute cost limit (USD)200A backfill estimated to cost more than this needs the approver.

Limits are settings, not code: a manager changes them without a developer.

Rules it always follows

  • Trace the fault upstream through the lineage to the first job or client extract that is wrong, and name what would confirm it.
  • Reconcile every load to the client's ERP control total; a difference above 2 percent is wrong data until shown otherwise.
  • Flag any load leaving more than 5 percent of spend unclassified, and never force spend into a category to hide it.
  • Convert currency only with the rate source and date the standards set, and keep the original currency amount alongside.
  • When on is true, set failing records aside and load the rest, reporting the count and value.
  • Backfill only the affected range; one longer than 90 days or costing more than 200 needs the Data Platform Lead.
  • Never overwrite a published spend cube or finance-validated savings figure without a snapshot of what it replaces, and tell every downstream owner, including the client's engagement lead.

See the Spend Data Pipeline Agent agent on your own work.

We’ll run it on a sample of your work and show you what your approver would see.

Spend Data Pipeline Agent · Procure AI | Basinwright Industries