AP Exception Desk: an AI agent investigates invoice exceptions, a person decides

Accounts payable teams at distribution companies spend much of their day on invoices that don't match their purchase order. AP Exception Desk is the system we built to take that investigation work off their desks. It reads supplier invoices and runs a three-way match against purchase orders and goods receipts. When something doesn't match, an AI agent works out why and drafts the next step. An AP clerk approves, rejects or escalates every proposal, and every step is recorded in a tamper-evident audit log.

The challenge: invoice exceptions eat AP time and invite fraud

A supplier invoice should agree with two documents: the purchase order (what was ordered, at what price) and the goods receipts (what actually arrived). When they disagree, the invoice becomes an exception. Someone has to find out why, contact the buyer or the supplier, and decide what to do. The work is manual and slow, mistakes cost money, and some exceptions are attempted fraud.

For mid-size distributors in the Baltics, exceptions are routine, not an edge case:

• Most suppliers are abroad, so every invoice raises a VAT question: Latvian VAT, EU reverse charge or import.
• Goods arrive in parts or damaged, so invoices often run ahead of goods receipts.
• Supplier price lists change during the year, while purchase orders keep the old price.
• Invoices arrive by email from many senders, which makes “new bank details” fraud and look-alike sender domains a real risk.
• A small AP team handles every exception, so time per case and correct escalation matter.

That gave the system five requirements:

1. 1Detect every mismatch, the same way every time.
2. 2Cut the time a clerk spends investigating each exception.
3. 3Catch payment fraud before money leaves the company.
4. 4Keep a person in control of every decision with consequences.
5. 5Record who decided what, and why.

The solution: AI investigates, people decide

We split the work by what each part does best. Code parses structured data and does every calculation. The AI agent does the reading and reasoning a clerk would otherwise do by hand. A person makes every decision that has consequences.

Invoice data capture: exact where possible, AI where needed

Structured e-invoices (Peppol BIS / EN 16931 UBL, UN/CEFACT CII and Factur-X / ZUGFeRD) are parsed exactly, with no model involved. Only PDFs without embedded e-invoice data, scans included, go to a vision model. A reading that fails validation gets one retry, along with the checks it failed.

Every reading is then validated. Lines must add up to the subtotal, VAT and total, the IBAN checksum must hold, and the VAT number and dates must be valid. Instead of a confidence percentage, each field gets one of three statuses: checked (an independent check confirms it), unchecked (read, but nothing confirms it) or failed. The clerk sees each field next to the original document and knows exactly what to verify.

Deterministic three-way matching

The matching engine is plain Python with exact decimal arithmetic. It pairs invoice lines with purchase order lines (by SKU, then description, then price), calculates price and quantity variances, tracks quantities already invoiced, detects duplicates and checks VAT against the supplier's country. By default, a price difference is flagged only when it exceeds both 2% and €10. The tolerance can be set per supplier.

An AI agent that investigates, but can't act

For each exception, the agent gathers evidence through read-only tools: the invoice, purchase order, goods receipts, contract terms, the supplier record, the supplier's earlier invoices and the email thread. It returns a plain-language diagnosis, up to ten pieces of evidence that each cite the tool call they came from, a confidence label, and one proposed action from the actions allowed for that exception type.

Code reviews the answer before any person sees it. Every figure the agent quotes is checked against its sources (more on that below). Policy rules can only make a case more cautious: an action outside the allowed set becomes an escalation, and possible fraud and high-risk cases always go to a person.

The same split, where the model reads and drafts, code enforces the rules and a person signs off, works for many back-office processes. It's how we approach AI automation in general.

Invoice fraud checks the AI can't override

The matching engine checks seven risk signals: a changed bank account, an invalid IBAN, a sender domain that doesn't match the supplier, a look-alike domain, shortened payment terms, urgent language and an unusual amount. Any one of them marks the case as high risk, and nothing that runs later, the agent included, can remove that flag.

Bank details read from an invoice never overwrite the supplier record. Changing a bank account is a separate, verified action that keeps the old account on file. A supplier's email reply can't settle a bank-detail case either: those are verified by phone with a known contact.

Follow-up emails, sent exactly once

Approved emails go from the AP mailbox to the buyer or the supplier. The recipient always comes from master data, never from the model. If nobody answers, fixed-text reminders go out after five days, and after two reminders the case escalates to the AP manager.

Replies are linked to their case, ignored if they come from a different domain than the one we wrote to, screened by a low-cost model and handed back to the agent. The agent can propose closing the case, but closing it still takes a person's approval.

A tamper-evident audit trail for every invoice

Every step is written to an append-only audit log. Each entry's hash covers its content and the previous entry, so editing or deleting a row breaks the chain, and a database trigger rejects changes. For each invoice, a workflow view shows what happened and who did it: the AI, a person or the system.

Technical architecture

The system has two parts: a Python backend with the API, the matching engine, the agent, the workflow and a background worker, and a Next.js console for AP clerks. PostgreSQL stores the business data, the audit log, the job queue and the workflow state.

Agent

Our own agent loop rather than a framework, with a budget per investigation (12 turns, 300k tokens, €1, 300 seconds) and parallel tool calls. If the budget runs out or a model fails, the case goes to a person instead of crashing.

Model choice

Every call goes through LiteLLM, so the model is a setting: Anthropic Claude, OpenAI, Google Gemini, OpenRouter or local models through Ollama. Reading, investigating and screening replies each have their own model, with fallbacks for rate limits and outages.

Workflow

LangGraph runs the case workflow with a PostgreSQL checkpointer, so a case can wait days for a reviewer or a supplier and resume where it stopped. The database stays the source of truth.

Background jobs

A transactional outbox: side effects are saved as jobs in the same transaction as the decision, claimed with a lease, retried with backoff and deduplicated with idempotency keys.

Observability

Every AI run is traced in self-hosted Langfuse (steps, tokens, cost and latency), with IBANs and email addresses masked.

ERP integration

Purchase orders and goods receipts live in the customer's ERP, such as SAP Business One, Microsoft Dynamics or Visma. Connecting it is the first step of every rollout and a typical integration project.

Engineering decisions

No AI in the money

Language models can get arithmetic wrong, and a payment check has to give the same answer every time and be explainable to an auditor. So every calculation is deterministic code with decimal arithmetic, covered by property-based tests. The AI reads and explains. It never decides whether the numbers match.

Verify the agent's figures instead of trusting them

A model can state a figure that looks right but doesn't exist, and asking it to be careful doesn't fix that. Instead, code checks every amount, percentage and document number the agent quotes against the tool output it cites. A failed check gets one retry. After that, the case is flagged for a person.

Treat supplier text as untrusted

Invoices and emails come from outside the company, so instructions hidden in them (prompt injection) are a real attack path. Supplier text reaches the model as data, never as instructions, and email recipients come from master data. The evaluation suite includes injected instructions in both an email and an invoice line.

Side effects that run exactly once

A double click, a retry or a crashed worker must never send two emails or apply a decision twice. Decisions carry an idempotency key and a version check. Emails are saved with a fixed Message-ID before sending, so a resend after a crash is recognised as the same message.

Results

We measure the system against test and evaluation suites with known answers. The current results:

• Investigation evaluation: 22 synthetic case types with known answers, covering clean matches, every exception type, fraud and prompt injection. Matching was correct in 22 of 22, agent actions in 15 of 15, with 0 safety violations.
• Extraction evaluation: 35 known invoices in five formats (UBL, CII, Factur-X, PDF and scans), compared field by field. The pass rule: no field marked as checked may be wrong.
• 272 automated backend tests, with type checking and linting passing for the backend and the console.

For an AP clerk, the work now starts from a diagnosis with sources and a draft email, decided on the invoice page, instead of from an empty screen and three documents to compare by hand.

What comes next

The system works end to end, from the uploaded invoice to the final email. Next on the roadmap: user accounts and four-eyes approval for high-risk decisions, the ERP connection, a dashboard with exception volumes, resolution times and AI cost, and supplier memory, so the agent can learn from similar past cases.

Questions AP and finance teams ask

Can the AI approve or pay an invoice?

No. The agent only reads and proposes. Every action that leaves the system needs a person's approval, and possible fraud is never resolved automatically.

Does it work with e-invoices?

Yes. UBL, CII and Factur-X / ZUGFeRD e-invoices are parsed exactly, without a model. Scans and other PDFs go to a vision model and are checked field by field.

Can we see who decided what?

Yes. Every step, whether taken by the AI, a person or the system, is written to an append-only audit log, and each invoice has a workflow view with its full history.

Can the AI run on our own servers?

Yes. The model is a setting, and local models through Ollama are supported for data that must stay in-house.

If your AP team spends its days chasing invoices that don't match, tell us about your invoice process. We'll tell you honestly what we'd automate and what should stay with a person.