Correction, July 2026: the first version of this post reported DeepSeek-V4-Pro leading our retrieval measure at 40%. That number came from a defective retrieval metric: the judge never saw the retrieved passages, and the question set contained duplicated and ambiguous items. We repaired both, validated the new judge against a human-scored anchor set, and re-ran the entire lineup. On the repaired metric every model lands in one 80-86% band and no model separates from the pack, so the ranking this post was built around no longer exists. We have corrected the text below; the cost findings are unchanged. Details in the benchmark writeup.
Each quarter we benchmark open-weight models on the document workloads our customers run: field extraction from invoices and forms, and retrieval-augmented answers over regulated records. This quarter we added DeepSeek-V4-Pro, the newest and largest open flagship from DeepSeek. It matched the strongest models we have measured on retrieval faithfulness and read fields off clean documents about as well as anything we have tested. It is also the clearest illustration this Index has produced of a point we keep making: the model that wins the benchmark is not automatically the model you should run.
What it is
DeepSeek-V4-Pro is a mixture-of-experts model with 1.6 trillion parameters in total and 49 billion active for any one token. It is text-only and a reasoning model — it works through a problem before it answers. The weights ship in a mixed 4-bit and 8-bit format that lands at about 865 gigabytes. That is too large for a box of last-generation H100s, but it fits a single eight-GPU H200 box with room for the cache, so we served it on one machine under the stable vLLM release, version 0.23.0. One detail cost us a serve: the checkpoint uses a block-scaled 8-bit format that the default math path could not dispatch, and the engine failed while sizing its cache. Turning on the DeepGEMM path — the opposite of the setting the previous DeepSeek needed — cleared it, and the model came up.
What it scored
On retrieval faithfulness, the hardest measure we run and the one that decides whether an answer stays grounded in the records it retrieved instead of inventing detail, DeepSeek-V4-Pro scored 80.1% on our repaired retrieval metric, inside the band where the entire lineup lands, frontier models included. (The first version of this post reported 40% and a first-place rank; that figure came from a defective metric we have since repaired and validated against a human-scored anchor set. On the repaired metric no model in the lineup separates from the others on retrieval.) On clean digital extraction, the field-level accuracy that decides whether an invoice or a form comes through correct, and the tier most documents sit in, it reached 98.4%, inside the band where the whole current lineup now sits. As a text-only model we measure it there, on real text rather than scanned images.
It also caught a bug in our own test data, and the way it did is worth a sentence. One extraction field, a provider identifier, had been stored in our answer key as an internal record number rather than the standardized ten-digit value a real form carries. Most models copied whatever string sat in that spot and scored fine. DeepSeek-V4-Pro, reasoning about the field, decided the value was not a valid identifier and declined to emit it — and in doing so scored badly on a field that was itself wrong. Chasing that down surfaced the bug. We fixed the answer key, re-ran all twenty-one models in the Index, and the corrected numbers are what the benchmark now shows. A model good enough to disagree with your test data is a model worth measuring carefully.
What it costs
This is where the trade-off lands. At peak utilization on a spot H200 box, DeepSeek-V4-Pro served about 516 output tokens per second, which works out to $11.71 per million output tokens — roughly three times the next most expensive model in the lineup. Three things compound to get there. It is the largest model we run, so every token moves more weight. It is a reasoning model, so it spends tokens thinking before it answers. And its 4-bit experts run on our Hopper GPUs through a software path that saves memory but not speed, so the arithmetic lands at 8-bit rates rather than the format's native 4-bit. On the newest Blackwell hardware, built for that format, the same model would be materially faster and cheaper. At more typical H200 spot prices the figure eases to around $9.70, but it stays the most expensive serve in the set.
What this means for self-hosting
DeepSeek-V4-Pro is a quality decision, not a cost decision. If you want the largest open model available running entirely inside your own account, nothing sent to a third party, it delivers that, at a price. But on retrieval quality the lineup is statistically tied, so a model at the same measured quality and a third of the serving cost will almost always be the better plan — and this quarter's Index has several. That is what the benchmark is for: not a single trophy, but the numbers that tell you which trade-off you are actually making. The best open model we tested is real, and it is open. Whether it is the one to run is a question about your workload, not the leaderboard.