The importance of AI multi-modality

Why the future of intelligent document processing is not one model, but a garden of them. And why the platform that grows and tends that garden matters more than any single model inside it. 

01 · THE ONE-MODEL FALLACY 

Intelligence is not one thing 

There is a comfortable story going around that artificial intelligence has become a solved problem. You take one large model, you show it a document, you ask it a question, and the answer comes back. If that were true, every company drowning in invoices, purchase orders, credit notes, and contracts would already be finished, and intelligent document processing would be a closed chapter rather than a living field. The story is seductive because the demo always works: one clean invoice in, one tidy answer out. 

It is not true, and anyone who has fed real business documents to a single model learns why within an afternoon. Real documents are hostile. They arrive as phone photos shot at an angle, as scans with a coffee ring across page three, as forty-page fee schedules whose header repeats on every page, as forms with a signature scrawled through the total and a rubber stamp half over the date. One vendor prints a total as 1,234.56; another, one border away, prints the very same amount as 1.234,56. A Canadian invoice stacks GST and PST on a single line, while a Spanish one stacks IVA and RE. Ask one general-purpose model to read the layout, identify the logo, add the column, and follow a customer's instruction all in a single pass, and it will do most of it well and quietly get a slice of it wrong. There is a structural reason for that slice. A language model predicts the next token from patterns it has seen, so when you ask it to total a column it is not calculating, it is guessing the most plausible-looking number. In a chat reply that is harmless. In accounts payable, the plausible wrong number is the one that reconciles to nothing three weeks later. 

A single model asked to do everything at once will get most things right, and quietly get the expensive things wrong. 

Docupath is built on the opposite premise: that intelligence is not one thing, and a document is not one problem. Reading the geometry of a table is a different task from recognizing a company by its logo, which is different again from adding a column of currency, which is different again from interpreting a sentence a customer typed in plain English. Each of those wants the tool that is genuinely best at it, and each of those tools can be measured, swapped, and improved on its own. We call that collection our AI model garden. The rest of this article walks through the beds of that garden, the deterministic machinery wrapped around them, and why the whole is something quite different from a thin shell over a chat model. 

02 · HOW WE THINK ABOUT MODELS 

The engine, not the car 

Start with a picture. A high-performance car is not its engine. It is the chassis, the transmission, the brakes, the aerodynamics, the cooling, the telemetry on the dashboard. The engine matters enormously, yet drop a newer, stronger engine into the same car and it is still your car, with every other system and every hard-won setup intact. The engine is replaceable. The car is the asset. 

We treat AI models exactly that way. A model is an engine: astonishing, improving every few months, and certain to keep improving for years. But it is never the product, and we are careful never to build as though it were. In our platform no model name is hard-coded; each is a value in configuration. The model that classifies a document, the model that extracts the fields, the model that reads a customer's natural-language rule are all settings, and every one of them is reached through a single internal gateway rather than through calls scattered across the codebase. That gateway owns prompting, retries, token accounting, and output parsing in one place, so pointing a task at a different engine, or routing different tasks to different engines, is a configuration change rather than a rebuild. When a promising new model appears, we register it, run it against our own benchmark set of real customer documents, and promote it only where it measurably wins. 

This matters because of the direction of travel. Models keep getting cheaper, faster, and sharper on a curve with no plateau in sight. If your product is a wrapper around one model, every gain in that model is equally a gain for every competitor calling the same endpoint, and every price rise or quiet regression becomes your emergency. If your product is the car, each new engine is a gift you accept on your own terms: you built the brakes and the telemetry once, and they keep earning no matter what sits under the hood. So the day a materially better model ships, we adopt it deliberately, prove it on our own benchmarks, and hand customers the improvement without asking them to change a single line of their integration. 

03 · THE MODEL GARDEN 

The garden, bed by bed 

Garden is the right word because a garden is arranged, not piled. Different plants sit in different beds, in the soil and light that suit them, and the gardener tends the whole so it yields more than any single plant could alone. Ours has five beds, with a spine of deterministic code running through all of them. Here is what each one actually does, and where the hard engineering lives. 

BED 01 · THE SPINE 

Deterministic algorithms 

The first thing to say, and the most important, is that a large share of the work is done by no model at all. It is plain, exact code, because for whole classes of task a model is simply the wrong instrument and determinism is the entire point. 

Arithmetic is the clearest case. We never ask a model to total a column or work out a tax, for the reason above: it approximates by pattern, and money does not tolerate approximation. Instead the model extracts only the values actually printed on the page, and our own code derives everything else in exact decimal arithmetic, the kind that never leaks a floating-point cent. Line net is quantity times unit price. Tax is net times rate. The document total is the sum of the lines. Where a jurisdiction splits tax into parts, the code adds every component in full: GST plus PST on that Canadian invoice, IVA plus RE on the Spanish one, each treated as its own line rather than mistaken for a single lump. 

Then comes a discipline that finance and compliance teams care about more than any headline accuracy figure. When the total our code computes disagrees with the total printed on the page, we do not silently rewrite the page to make it agree. We keep the printed value exactly as it is and raise a flag that records three things: what was printed, what we computed, and why they differ. If a document prints 104.49 but net plus tax comes to 104.50, we keep 104.49, mark a one-cent discrepancy, and show a human the arithmetic behind it. Rounding conventions vary by vendor and by country, so a one-cent gap is often perfectly legitimate; the point is that the system surfaces it instead of burying it. It shows its work. 

Formatting travels the same deterministic path. Amounts are normalized to one canonical form and dates to one unambiguous standard. Where a value is genuinely undecidable, a bare 1,234 that could mean one thousand two hundred or one point two three four depending on locale, we do not gamble on a reading. We flag it for a human. Certainty is corrected automatically; ambiguity is surfaced honestly rather than smoothed over and hoped about. 

BED 02 · CAPTURE 

Machine learning for capture 

Before a document can be reasoned about at all, it has to become structure. Turning pixels into structure is itself a machine-learning problem, and we run a capture layer trained on an enormous corpus of real-world documents to solve it. What matters is the shape of what it hands back. It does not return a flat ribbon of characters. It returns the page as a hierarchy: tables as true grids of rows and columns, paragraphs as blocks, key-value pairs where a printed label sits beside its value (Invoice No. next to 90231, VAT next to 20%), barcodes with their decoded payloads, and a confidence score on every word so a smudged fax can be treated with the caution it deserves. Each element also carries its bounding box, so we always know not only what was read but exactly where on the page it sat. 

That difference, structure instead of characters, is the line between machine learning and the flat character recognition of the past. The model has learned, from millions of examples, where information tends to live and how a table stays coherent even when the rule lines between cells are faint or missing entirely. Picture a fee schedule broken by a sub-header that repeats across pages: a naive reader runs the quantity from one row into the date of the next and fuses them into a single mangled cell. Our capture layer, run in a high-resolution mode, keeps them apart, because it has seen that exact shape of table thousands of times and is modelling the grid rather than the ink. 

BED 03 · MEMORY 

Retrieval-augmented extraction 

This is the idea we are proudest of, and it is worth slowing down for, because it is genuinely unlike the way most systems work. 

A document that lands today is almost never the first of its kind. A given supplier tends to send invoices that look like their own past invoices, month after month, down to the position of the logo and the order of the columns. So instead of treating each arrival as a stranger, Docupath asks one question first: have we seen documents that share this one's structure before, and if so, what did a clean, corrected extraction of them look like? That prior clean extraction becomes a consistency reference, an anchor, that holds the shape of the new extraction steady. This is retrieval-augmented extraction, or RAE: the same instinct behind retrieval-augmented generation in chat systems, aimed at a stricter goal, extracting the same fields the same way every single time. 

Tuesday's answer exactly matches Monday's answer. 

How we find the matching documents is deliberately boring, and boring is the point. We do not lean on fuzzy vector similarity, which can quietly drift from one run to the next. We group documents by a structural fingerprint built from the document type and the normalized names of the two trading parties, buyer and seller. Normalization strips the noise first (legal suffixes, casing, punctuation, extra whitespace) so that ACME Ltd., Acme Limited, and ACME LIMITED all collapse to one identity, and the fingerprint is then a stable hash of those parts. Every invoice of the same type, from the same supplier, to the same customer falls into the same bucket and shares one canonical anchor. It is deterministic, it is explainable, and it depends on no external lookup, so the grouping means precisely what it says. 

The anchor steers shape, and it never invents a value. Every figure in the output must still come from the document in front of us. Where today's document genuinely differs from its predecessors, today's document wins. Where a customer's own instruction disagrees with the anchor, the instruction wins outright. The anchor's one job is to stop the same field being labelled three different ways across three near-identical documents, so that Total Due, Amount Payable, and Balance Owing all land in the single field your ledger is expecting. 

The payoff is consistency you can rely on and cost you do not pay twice. When a burst of near-identical documents arrives together, the platform recognizes that they resolve to the same unit of work and collapses them: ten identical delivery receipts become one model call and ten byte-for-byte identical results, not ten separate calls that each phrase the answer a little differently. Submit the very same document again next week and the earlier result is served straight from cache, with no model call at all. In a domain where the same page quietly giving a different answer on Tuesday than it gave on Monday is a real and expensive failure, that determinism is something customers feel on the first day. 

BED 04 · SIGHT 

Vision 

Some things on a page are not text at all, and some are text that a purely textual reader will never place correctly. This is where vision models, which look at the rendered image of the page rather than a stream of characters, earn their bed in the garden. 

Vision decides who the parties are. Which company issued this document, and which company is it billed to? The answer usually lives in spatial layout, not in the words themselves: the logo and letterhead at the top, the remit-to block, the address sitting above “Ship To” against the one above “Bill To.” A vision model reading the page as an image can tell buyer from seller by where each name sits and how the page is composed, even on a vendor template it has never encountered, with nobody writing a rule for that vendor. It is reasoning about geometry, which is exactly what pixels and coordinates give it and a flat token stream throws away. 

Vision reads the human marks. The “PAID” stamp with a date across the total, the company seal, the handwritten “approved, J.K.” in the margin, the signature at the bottom. A text-only reader ignores all of it. A vision pass captures it, so the fact that a purchase order was approved by hand, and when, becomes part of the structured record rather than being lost. 

Vision rescues damaged tables, and this is one of the sharpest examples of why multi-modality matters. A long legal invoice once arrived with a fee schedule that ran to hundreds of rows across many pages. A first, text-driven pass reached the bottom of an early page and stopped, returning only fifty-eight rows. Because our capture layer independently reports how many rows the table structure actually contains, the system could see that fifty-eight was far too few. A vision pass then re-read the pages as images and recovered the full table, in this case four hundred and eighty-three rows. Crucially, the recovered table only replaced the first one because it was both closer to the true row count and at least as arithmetically sound, so a rescue can repair a truncated table but can never make one worse. That guard, take the better answer only when it is provably better, runs throughout the platform. 

Vision settles ambiguity at the level of a single cell. When two codes are stacked in one line, a supplier's article number and the buyer's own code, a naive extraction can wobble between them from run to run. A vision pass reads that row as a picture, identifies which column holds the value the customer actually wants, and, importantly, we then record that decision as a deterministic rule for that template. The model is consulted once. Every document after that is handled by exact code, with no model call and no wobble. Artificial intelligence resolves the hard case once, and determinism carries it forever. 

Vision also acts as a second opinion. When a field comes back with low confidence because the value could not be found in the captured text, an independent vision read of the page can confirm it and raise the confidence, or disagree with it and raise a flag for a human. The two modes check each other. 

Even barcodes get a second, sharper pass. After the capture layer's first read, a rotation-tolerant decoder re-scans the page images at high resolution to catch the codes the first pass misses: a QR code printed at an angle, a small data-matrix code on a medical sticker, a dense sheet of shipping labels. On a run of pharmaceutical stickers, the difference between the low-resolution first pass and the high-resolution second pass was the difference between finding no codes at all and reading every one. 

BED 05 · REASONING 

Reasoning, and the right engine for the job 

Not every task deserves the same size of engine, and the garden lets us match the tool to the job. Take a customer's written instructions. When someone types a rule in plain English, say, if the supplier is Acme, post it to cost centre 4021, we do not simply hope a model reads it the same way on every future document. We compile that sentence, once, into a precise, deterministic program, an explicit rule with typed conditions and actions, which our own engine then executes identically forever after. 

Because that compilation happens exactly once and the compiled program is cached, we can afford to spend a heavier, reasoning-grade model on it. Getting the meaning of the rule exactly right matters far more than the one-time cost of thinking hard about it, and what we keep is a program we can read, diff, and version rather than a prompt we have to trust on every run. The expensive reasoning happens a single time; the cheap, exact execution happens on every document after. That is the whole philosophy in miniature: spend intelligence where it changes the outcome, and spend nothing where plain determinism does the job better. 

04 · LEAVING LEGACY BEHIND 

Why this leaves the old world behind 

For decades, document automation rested on two moves: optical character recognition to turn a scan into text, then pattern matching or fixed templates to pull fields out of that text at known coordinates. If the invoice matched the template, it worked beautifully. But if a supplier nudged their invoice number two centimetres to the left, or a brand-new supplier appeared, or a vendor quietly redesigned their layout, the template missed and someone rebuilt it by hand. Every new document shape was a new maintenance ticket. 

What separates Docupath from that world, and cannot be reduced back to it, is contextual understanding. The platform does not hunt for the invoice number at fixed coordinates. It understands what an invoice number is and finds it wherever it sits, under whatever label, in whatever language, on a layout it has never seen before. It reads the picture to tell buyer from seller. It remembers what this supplier's documents looked like last time and holds the new extraction consistent with that memory through retrieval-augmented extraction. It copes with the messy reality, stamps, signatures, rotated barcodes, split tables, clashing number formats, that shatters a rigid template on contact. Where the old world needed a fresh template for every new shape of document, the garden simply generalizes, because meaning generalizes and pixel positions do not. 

Meaning generalizes across new layouts. Matching pixels to a fixed template does not. 

05 · NOT A WRAPPER 

Why this is not a ChatGPT wrapper 

It is fair to ask how any of this differs from just sending a document to a chat model and reading back its reply. The answer is everything above, and it is the same difference as the one between the car and the engine. 

A wrapper is a single engine doing everything with nothing built around it. No separate capture layer returning real structure. No deterministic arithmetic refusing to let a model guess at money. No retrieval-augmented extraction keeping the tenth invoice consistent with the first. No vision pass to rescue a table that stopped at row 58. No compiler turning a plain-English instruction into an exact, versioned program. No cache guaranteeing that the same document yields the same answer. No ledger accounting for every call. Ask a raw model the same document twice and you may get two different answers. Ask it to add a column and it may miscalculate without a flicker of doubt. Show it a new layout and it has no memory of the last one it saw. 

The garden is the inverse. The model is one contributor among many, wrapped in deterministic checks, structural capture, visual understanding, retrieval, and verification, all of which we built and all of which keep their value no matter which model we plug in. That surrounding machinery is the moat, and it is also what makes the swappable-engine promise real: because the engine is only one bed in the garden, we can lift it out and drop in the next, better one without disturbing anything else. 

06 · COST, SPEED, ACCURACY 

Tuned for all three at once 

A common belief holds that cost, speed, and accuracy form a triangle: pick two, sacrifice the third. The garden dissolves that triangle, because for each task we get to choose between deterministic gains that are essentially free and model-driven gains that cost a call, and we lean on the free ones wherever they will do. 

Deterministic work is fast, exact, and effectively free once written. Every total computed in code instead of guessed by a model is accuracy bought at zero marginal cost. Every instruction compiled once into an exact program, and every stubborn cell that vision resolves once into a permanent rule, moves work out of the slow, priced, variable model path and into cheap, repeatable code that runs in microseconds. 

Where we do spend a model call, we spend it deliberately. Near-identical documents are detected and collapsed, so a burst of them becomes one call and any repeat is served from cache, which cuts cost and latency at the same time and makes the output more consistent, not less. Work that does not need the top, most expensive tier is routed automatically to a cheaper, faster model. And heavy reasoning is spent only where it amortizes across every future document, compiling a rule for instance, rather than paid again on every single page. 

Under all of it sits measurement. Every external call the platform makes, every model invocation and every page processed, is written to a usage ledger with the exact tokens or pages consumed and the prices in force at that moment. The full cost of any one document can be reconstructed to the cent, after the fact, line by line. You cannot tune what you cannot see, and because Docupath sees every fraction of a cent, we can keep driving cost down while pushing accuracy up instead of trading one for the other. 

07 · GOVERNANCE 

Built to be trusted 

For the people who decide whether a platform like this may touch their documents, capability is only half the question. The other half is governance, and multi-modality, done our way, turns out to be a governance advantage rather than a risk. 

Every tenant's data is isolated. Access is authenticated with signed, federated identity, and that identity is bound into every request and re-checked on every read and write, so one customer's documents can never surface in another customer's results. Older, weaker credential schemes have been retired completely rather than left switched off. 

Everything is explainable. Because so much of the pipeline is deterministic, and because every change is recorded, we can answer the question auditors and regulators actually ask, which is not “what did the AI say” but “why did it say that.” Every value an instruction changes is logged with the value before, the value after, and the rule that changed it. Nothing is edited silently. When a printed total and a computed total disagree, both are shown side by side. When a field is uncertain, it is flagged rather than dressed up as confident. A finance or compliance team gets a system that shows its work, not a black box that asks to be trusted on faith. 

Everything is accounted for. The same usage ledger that lets us tune cost gives customers a complete, per-document record of exactly what was processed and what it cost. And we deliberately keep the contents of requests out of ordinary logs, so sensitive document data is never scattered into places it does not belong. 

Finally, deletion stays in your hands. By default Docupath retains documents and their extraction history on purpose, because the consistency and explainability described throughout this article depend on remembering what a supplier's documents looked like last time. Deletion is therefore explicit and customer-directed rather than automatic: a single call removes a document and everything derived from it in one clean transaction. Teams with strict data-minimization requirements should plan around that deliberate choice, and the control to act on it sits with them, in the API, on demand. 

08 · THE GARDEN KEEPS GROWING 

The garden keeps growing 

The models will keep getting better. That is the one safe prediction in this field, and it is the assumption we built the entire platform around. A company whose product is a single model is a hostage to that curve. A company whose product is the garden, the deterministic spine, the structural capture, the retrieval-augmented memory, the vision, the reasoning spent where it counts, the caching, the ledger, and the governance, gets to treat every advance in every model as another engine it can drop in on its own terms. 

That is what the importance of AI multi-modality really comes down to. It is not a slogan about using more than one kind of model. It is a way of building. Choose the genuinely best tool for each part of the problem: algorithm, machine learning, retrieval, vision, reasoning, and deterministic verification. Let each one do what it is best at. Surround them all with the tooling that makes the whole trustworthy, consistent, and cheap to run. And keep the engines swappable, so the platform gets better every time the technology does, without ever asking the customer to start over. 

THE QUESTION WORTH ASKING

If you are deciding how to handle documents at scale, do not ask which model do you use, because models change. Ask what did you build around the model, because that is what you are really buying.

Put the garden to work on your toughest document. 

Contact us