When you benchmark models on your own documents, the findings that feel most insightful are the ones that confirm a good intuition. Those are also the ones to distrust. This is the story of one we published, why it was wrong, and how the model we least expected caught it.
The finding we published
Most teams reach for a vision-language model for document extraction, because it reads scans and photos without a separate OCR step. So we benchmarked one against its same-size, same-price, text-only sibling on clean digital invoices, real text, no scanning. The vision model came out well behind. It fit a clean intuition: a vision model turns a page into pixels and inherits the errors that come with that, even on text it could have parsed as text. So we wrote it up, and we routed our own pipeline around it. Here is the table we published:
| Model (7B class) | Type | F1 on clean digital |
|---|---|---|
| Qwen2.5-7B | text-only | 99.0% |
| Phi-3.5-mini | text-only, cheapest | 92.7% |
| Qwen2.5-VL-7B | vision-language | 87.5% |
It was a tidy result, and the intuition behind it felt solid. It was also wrong.
How a reasoning model broke it
A few weeks later we added the largest open model we had ever benchmarked, a 1.6-trillion-parameter reasoning model. On one field, a provider identifier, it scored strangely low, and not by guessing wrong. It was declining to answer: it had decided the value in front of it was not a valid identifier and refused to emit it. It was right. Our answer key for that field held an internal record number, a long hyphenated UUID, where a real form carries a standardized ten-digit code. Most models had transcribed whatever string sat in that slot and scored fine. The reasoning model would not, and that is what surfaced the bug.
That one field had been deciding the vision finding. A text model reads the value straight off the document's text layer and copies it, UUID or not. A vision model has to read it off the rendered pixels, and a thirty-six-character UUID smeared across an image is the kind of string a small vision model fumbles. It is one of eight scored fields, so getting it wrong cost about twelve points, almost all of the gap we had reported. And the original comparison ran on just twelve documents, a smoke test, far too few to trust a gap that size. A bad answer key and a tiny sample had manufactured a finding.
The corrected numbers
We regenerated the answer key with valid identifiers, re-ran the eval at proper scale (about 130 documents per tier), and re-scored all twenty-one models in the Index. On clean documents the vision "penalty" disappears:
| Model (7B class) | Type | F1, corrected |
|---|---|---|
| Qwen2.5-7B | text-only | 96.9% |
| Phi-3.5-mini | text-only, cheapest | 90.9% |
| Qwen2.5-VL-7B | vision-language | 97.7% |
At the same size and the same cost, the vision model reading the page as an image now matches the text model reading the text layer, a fraction of a point apart, well inside the noise. The 11.5-point gap we had reported was one broken field, amplified by a twelve-document sample.
What held up: scale pays only where vision is hard
The first post had a second finding, and this one survived the re-run intact. The reflexive thought is, "a bigger model will fix it." We ran the same extraction eval across three difficulty tiers (clean digital, clean scan, and degraded scan, the skewed, blurred, compressed way a real fax or phone photo arrives) on a 7B vision model and its 72B sibling, about ten times the cost per token, now on the corrected answer key:
| Tier | 7B F1 | 72B F1 | Δ |
|---|---|---|---|
| Medical, clean digital | 97.5% | 99.6% | +2.1 |
| Medical, degraded scan | 97.8% | 99.3% | +1.5 |
| Commercial, degraded scan | 70.6% | 74.5% | +3.9 |
On the clean and clean-scan tiers the two are a point or two apart, and paying ten times more buys almost nothing. The 72B's edge shows up only on the hardest commercial scans, on the fields the small model fails: long identifiers, account numbers, the alphanumeric strings that fall apart on a noisy image. And even there the 72B reaches only 74.5%. Scale helps at the margin. It does not solve the hard tier.
So the two decisions the Index informs are unchanged. Route by input type (digital PDFs to the text path, scans and photos to the vision path), but the reason is efficiency: read a text layer instead of paying a model to OCR it, and the corrected numbers say you give up nothing in quality by doing so. And route by tier, not by model: the cheap model where it ties the expensive one, the expensive model reserved for the degraded inputs where those few points count. Preprocessing (de-skew, upscale) is the other lever, and often the cheaper one.
Why we run our own documents
None of this shows up on a generic leaderboard, because a leaderboard does not run your documents at your difficulty. But the sharper lesson is the one the bug taught. A finding that flatters your intuition, measured on a handful of documents against an answer key you never audited, is the finding that slips through. The fixes were ordinary: a bigger sample, and an answer key we regenerated and checked. The catch was not: it took a model willing to disagree with the test data, and it cost us re-scoring all twenty-one models in the Index to be sure. We would rather run a benchmark that corrects itself in public than one that never has to.