Automating Incoming Invoice Processing: OCR, LLM Extraction, and the Pitfalls In Between

Every company that buys things receives invoices, and in a surprising number of them a person still opens each PDF, reads off the invoice number, the net amount, the VAT, and the due date, and types all of it into an ERP or accounting system. That process is slow, mind-numbing, and error-prone — and it scales linearly with headcount. Digitizing it is one of the highest-leverage automation projects a finance or operations team can run.

This article is about the extraction step in that pipeline: turning an unstructured invoice document into structured data your systems can act on. There are two dominant approaches today — classic OCR with template- or rule-based parsing, and LLM-based extraction — and they fail in very different ways. Having built pipelines with both, here is an honest look at the trade-offs, the pitfalls, and the stack I would choose today.

Why bother automating at all

Before the how, briefly the why:

• Cost per invoice. Industry benchmarks put fully manual invoice processing at several euros per document once you account for keying, matching, approval routing, and correcting mistakes. Automated pipelines push that toward cents.
• Error rates. Humans transpose digits. A wrong IBAN or a misread amount is not a cosmetic bug — it is money leaving the company in the wrong direction.
• Cycle time. Early-payment discounts and supplier relationships depend on invoices moving through approval quickly, not sitting in a shared mailbox.
• Compliance and auditability. A structured pipeline gives you a complete, queryable record of what was received, what was extracted, and who approved it.

One important caveat up front: in the EU, structured e-invoicing (XRechnung, ZUGFeRD/Factur-X, Peppol) is becoming mandatory for more and more B2B transactions. When an invoice arrives as structured XML, there is nothing to extract — you parse it. Extraction is the bridge technology for the long tail of PDFs, scans, and photographed receipts that will realistically be with us for years. Design your pipeline so that structured e-invoices bypass the extraction step entirely.

Approach one: classic OCR + rules

The traditional stack: an OCR engine (Tesseract, ABBYY, or a cloud service) converts the document to text with coordinates, and a layer of templates, regexes, and positional rules pulls out the fields.

Where it shines:

• Deterministic. The same document produces the same output, every time. That property is worth a lot in accounting.
• Cheap and local. Tesseract is free and runs on your own hardware. No per-page API costs, no data leaving your infrastructure — relevant under GDPR and for anyone with strict data-residency requirements.
• Mature and explainable. When a field is wrong, you can point at the rule that misfired and fix it.

Where it breaks down:

• Templates don't scale with supplier diversity. Rules written for one supplier's layout break the moment they redesign their invoice — or the moment supplier number 201 shows up. Maintaining hundreds of templates becomes its own full-time job.
• No semantics. OCR gives you characters, not meaning. It cannot tell a delivery date from an invoice date, or a net total from a gross total, unless a rule encodes exactly where and how that distinction appears.
• Layout brittleness. Skewed scans, stamps over text, multi-column layouts, and line-item tables that wrap across pages all degrade rule-based parsing badly.

Approach two: LLM-based extraction

The modern alternative: hand the document (as text, or as an image to a vision-capable model) to an LLM together with a target schema, and get structured JSON back.

Where it shines:

• Layout-agnostic. One well-written prompt plus a schema handles the invoice from supplier 1 and supplier 500. New supplier, new layout, unusual phrasing — usually no code change.
• Actual semantics. The model understands that "Zahlbar innerhalb von 14 Tagen" is a payment term, that a "Gutschrift" is a credit note that flips the sign, and that the number next to "Gesamtbetrag" is probably the gross total.
• Speed of development. A working prototype is an afternoon of work, not a quarter of template engineering. For a mixed corpus of suppliers this is transformative.

Where it bites:

• Hallucination. This is the failure mode that should keep you up at night. An LLM will not leave a field blank because it is unsure — unless you force it to. It will produce a plausible-looking invoice number that does not exist on the document. Plausible-but-wrong is far more dangerous in accounting than obviously-missing.
• Non-determinism. The same document can yield slightly different output across runs. Auditors dislike this; so do your reconciliation scripts.
• Cost and latency. Per-document API calls add up at volume, and a vision model reading a 12-page scan is not instant.
• Privacy and data residency. Invoices contain bank details, names, and commercial terms. Sending them to a third-party API needs a data-processing agreement, and possibly an EU-hosted or self-hosted model depending on your requirements.
• Confidence is hard. Classic OCR gives you per-character confidence scores. An LLM's self-reported confidence is poorly calibrated — you cannot build your review queue on "the model says it's 95% sure."

The pitfalls nobody warns you about

Whichever approach you pick, these are the stumbling blocks that show up in real projects:

1. Garbage in, garbage out. A blurry photo of a crumpled thermal receipt defeats every extraction method. Reject or flag low-quality inputs at the door instead of producing low-quality data downstream.
2. Line items are the hard part. Header fields (invoice number, date, totals) are the easy 80%. Line-item tables — merged cells, wrapped descriptions, tables spanning pages, discounts as negative lines — are where both OCR templates and LLMs earn their keep. If your use case only needs header data, say so explicitly and skip the pain.
3. Validate with arithmetic, not vibes. An invoice is a self-checking document: net + VAT must equal gross, line items must sum to the net total, VAT must be consistent with the stated rate, and IBANs have a checksum. These invariants catch a large share of both OCR misreads and LLM hallucinations for free. Build them in from day one.
4. You need a human-in-the-loop — permanently. No pipeline reaches 100%. The goal is not to remove the human but to change their job from typing everything to reviewing the flagged 5–10%. Budget for the review UI; it is not an afterthought, it is half the product.
5. You need an evaluation set before you need a model. Collect a few hundred real invoices with hand-verified ground truth first. Without it you cannot compare approaches, catch regressions after a prompt change, or honestly answer "how accurate is it?"
6. Duplicates and near-duplicates. The same invoice arrives by email, then again via the supplier portal, then a dunning copy. Extraction quality is irrelevant if you pay it twice. Deduplication (supplier + invoice number + amount) belongs in the pipeline.
7. Don't let the model do math or lookups. Asking an LLM to compute totals or resolve a supplier against your master data invites errors. The model extracts what is on the page; deterministic code computes, validates, and matches.

The stack I would build today

For a mixed corpus of PDFs and scans at small-to-medium volume, my current default is a Python pipeline with LLM extraction at the core and deterministic guardrails around it:

• Ingestion: watch a mailbox / upload endpoint; detect structured e-invoices (XRechnung, ZUGFeRD) and route them straight to a parser — no extraction needed.
• Text extraction: try the PDF text layer first (PyMuPDF / pdfplumber — free and lossless); fall back to OCR or a vision model only for scans without a text layer.
• Extraction: an LLM call with a strict schema and explicit permission to return null for anything not clearly present on the document.
• Validation: Pydantic for shape, plus the arithmetic and checksum invariants described above.
• Persistence and review: PostgreSQL for extracted data and audit trail, and a small review UI where flagged documents land.

The heart of it is pleasantly small. Define the schema:

from decimal import Decimal
from pydantic import BaseModel

class LineItem(BaseModel):
    description: str
    quantity: Decimal | None
    unit_price: Decimal | None
    total: Decimal

class Invoice(BaseModel):
    supplier_name: str
    invoice_number: str
    invoice_date: str          # ISO 8601
    currency: str
    net_total: Decimal
    vat_amount: Decimal
    gross_total: Decimal
    iban: str | None
    line_items: list[LineItem]

Extract against it, then verify with arithmetic — never trust the model's numbers unchecked:

def validate(inv: Invoice) -> list[str]:
    issues = []
    if inv.net_total + inv.vat_amount != inv.gross_total:
        issues.append("net + VAT does not equal gross")
    line_sum = sum(item.total for item in inv.line_items)
    if inv.line_items and line_sum != inv.net_total:
        issues.append("line items do not sum to net total")
    if inv.iban and not iban_checksum_ok(inv.iban):
        issues.append("IBAN checksum failed")
    return issues  # any issue -> human review queue

Documents that pass validation flow through automatically; anything with an issue — or a null in a required field — goes to the review queue. In practice this hybrid lands at a high straight-through rate while making the dangerous failure mode (confidently wrong data entering the ERP) rare.

Conclusion

OCR versus LLM is a false dichotomy. Classic OCR is deterministic, cheap, and private, but drowns in template maintenance. LLM extraction generalizes beautifully across suppliers and layouts, but hallucinates with a straight face and needs guardrails you must build yourself. The pipelines that actually work in production combine them: cheap deterministic text extraction where possible, an LLM for the semantic heavy lifting, arithmetic validation as the safety net, and a human review queue for everything the pipeline is not sure about.

And keep an eye on the horizon: as structured e-invoicing mandates roll out across the EU, the best extraction step will increasingly be the one you get to skip.