Correction, July 2026: two numbers in this post were revised after publication. Extraction rose from 96.3% to 98.3% when we fixed a defect in our own gold data (an internal ID standing in for a real provider NPI). The retrieval score changed from 36.4% to 83.2% when we repaired the retrieval metric itself: the original judge never saw the retrieved passages and rejected correct answers, a defect we confirmed against a human-scored anchor set and describe in the benchmark writeup. The serving result this post describes is unchanged.
Each quarter we benchmark open-weight models on the document workloads our customers run: field extraction from invoices and forms, and retrieval-augmented answers over regulated records. This quarter we added NVIDIA's Nemotron-3-Ultra-550B. It was the hardest model to serve we have tried, for three separate reasons, and it came up on a single eight-GPU box on the first real attempt. It also landed in the retrieval band where our strongest models sit. The result was good. Getting it to serve at all was the surprising part.
What should have blocked the serve, and what cleared each barrier.
Three reasons it should not have run
Three facts about this model each point at a different wall, and large open models seldom clear all three at once.
A new architecture
Nemotron-3-Ultra is a hybrid of Mamba-2, Transformer attention, and a mixture of experts: 560 billion parameters in total, 55 billion active for any one token. The Mamba layers carry a fixed-size running state through the sequence in place of attention's cache that grows with every token, which is part of how a model this size stays affordable to serve. Most large open models are pure Transformers, and a new architecture like this one often needs a bleeding-edge build of the inference engine to run. This one ran on the stable vLLM release, version 0.23.0, with no patches. The open-source inference stack had already taken it in.
A 4-bit format for hardware we do not own
The weights ship in NVFP4, NVIDIA's 4-bit floating-point format. NVFP4 is accelerated only on Blackwell, NVIDIA's newest GPU generation. Our cluster runs the generation before it, Hopper. vLLM still loaded the 4-bit weights on Hopper, through a software path called Marlin: it holds each weight at 4 bits in memory, then converts it to a format Hopper can multiply as the math runs. We got the memory saving without the chips the format was supposed to require.
Too large for one box at full precision
At full 16-bit precision the model is about 1.1 terabytes. That does not fit one machine, so it would force a serve split across several nodes. H200 capacity has been scarce for weeks, so the multi-node path was closed to us in any case. The 4-bit build is about 310 gigabytes. It fits one eight-GPU H100 box, and H100 capacity was open.
What it scored
On clean digital extraction, the field-level accuracy that decides whether an invoice or a form comes through correct, it reached 98.3%. That number is the clean digital tier, the one most invoices and forms sit in. Nemotron is text-only, so we measure it there, on inputs that are real text rather than a scanned image. On retrieval faithfulness, the hardest measure we run and the one that decides whether an answer stays grounded in the records it retrieved instead of inventing detail, it scored 83.2%, inside the band where the whole lineup lands, frontier models included. Served this way it produced about 2,000 output tokens per second, which on a spot H100 box works out to $1.96 per million output tokens. For a 550-billion-parameter reasoning model running inside our own account, that is a number you can put in a budget.
What the number leaves out
The Marlin path buys memory. It does not buy speed. Hopper does the arithmetic in 8-bit, not the format's native 4-bit, so throughput comes out around what an 8-bit serve gives, below the two to four times a Blackwell box would reach with real 4-bit hardware. So $1.96 is a Hopper figure. The same model on Blackwell would be faster and cost less. We publish the number we measured, on the hardware we can get this quarter, and we name the hardware.
What this means for self-hosting
The trend is the reason to care. Self-hosting a frontier-scale open model used to mean the newest GPUs, a cluster of them, and a patched inference engine. Here none of that held. A stable release supported a brand-new architecture. A software path ran a next-generation format on current hardware. Quantization to 4 bits shrank a terabyte-class model onto one machine. The bar for running large open models inside your own account keeps dropping, and it drops faster than the hardware refresh cycle. That gap is the difference between renting a capability through a vendor's API and owning it. For a lender or an insurer whose records cannot leave its environment, that is the point: the model runs where the data already sits, and nothing goes to a third party.