A 4-bit GPT-OSS derivative beat its own bfloat16 checkpoint on seven of nine benchmarks, yet the bit width is the least interesting part of the result. The quantized model received a second round of supervision from the original 120B model. Its bfloat16 comparison did not. For developers sizing an inference deployment, Multiverse Computing's experiment suggests that quantization can double as another training stage, provided a team still has logit access to a stronger teacher and the compute to run it.
The company calls the recipe Quantization-Aware Healing, or QAH. It began with OpenAI's GPT-OSS 120B, compressed the architecture to 60 billion parameters, recovered that smaller model in bfloat16, and then quantized it to MXFP4 while distilling from the original model. The resulting 60B checkpoint scored above the recovered bfloat16 version on long-context reasoning, mathematics, agentic coding, tool use, science, instruction following, and general coding. It trailed on MMLU-Pro and SciCode, according to the paper released on August 21.
That is a narrower claim than saying 4-bit models are more accurate than 16-bit models. The authors spell out the distinction: QAH performs an additional distillation pass during quantization, so the deployable artifact receives training that the bfloat16 checkpoint never saw. The result tests a production recipe, rather than the isolated effect of changing numerical precision.
Healing after two kinds of compression
A model can be made cheaper in two separate ways. Structural compression removes parts of the network, such as layers, heads, or neurons. Quantization stores the remaining weights at lower precision. The QAH paper combines both: GPT-OSS 120B becomes a 60B architecture, then that smaller network moves from bfloat16 to the 4-bit MXFP4 format. The authors report about four times less weight memory than the 60B bfloat16 student and roughly half the compute per token of the 120B teacher.
GPT-OSS already makes this comparison slightly unusual. OpenAI ships the mixture-of-experts weights of GPT-OSS 120B largely in MXFP4, with some tensors in bfloat16. Its model card lists 117 billion total parameters and 5.1 billion active parameters, and says the released model fits on a single 80GB GPU. In the QAH setup, "120B teacher" describes the original uncompressed architecture, while "full precision" refers to the recovered 60B student stored in bfloat16. The teacher itself is already mostly low precision.
The usual quantization-aware training approach, QAT, uses fake quantization during the forward pass and continues training against hard labels. A related method, quantization-aware distillation, learns from the output distribution of a full-precision checkpoint with the same architecture. Multiverse Computing argues that this same-architecture teacher is a weak reference after structural compression because the recovered 60B checkpoint is already an approximation of the original 120B model.
QAH points the 4-bit student back at the original model instead. The frozen teacher's top 100 logits are precomputed for each example, and the student minimizes KL divergence against that distribution. The paper's training setup used a mixture of NVIDIA Nemotron and SmolTalk data, a 32,000-token sequence length, a global batch size of 64, a learning rate of 5 x 10^-6, and 400 steps for the 60B quantization stage. Embeddings, layer norms, and selected attention components were frozen because the authors found those parts sensitive to destructive drift.
Long sequences make this kind of distillation expensive because a dense loss can materialize scores across every token and vocabulary item. The researchers cached the teacher logits and calculated KL divergence in chunks along the sequence. Their paper says this reduced peak intermediate memory enough to train at a 32,000-token context within the same hardware budget used for shorter-context QAT.
Seven wins, with two losses and plenty of noise
The biggest gain over the 60B bfloat16 checkpoint came on AA-LCR, a long-context reasoning test: 42.7 for QAH against 35.3, a 7.4-point difference. AIME 2025 rose from 70.7 to 76.3. The gains were smaller on Aider, tau2-bench, GPQA Diamond, IFBench, and LiveCodeBench. MMLU-Pro slipped from 74.0 to 73.8, while SciCode fell from 35.6 to 34.2. The full nine-benchmark table is published with the QAH report.
Against the original 120B teacher, the 60B QAH model scored 66.5 versus 66.0 on LiveCodeBench and 67.4 versus 69.0 on GPQA Diamond. The authors treat the half-point LiveCodeBench difference as a match because it sits within likely run-to-run variation. The largest remaining gap was AA-LCR, where the teacher scored 50.0. Structural compression still appears to cost capacity on the longest reasoning task in the set.
The statistical limits deserve as much attention as the seven wins. Every reported number comes from a single run, with no seed variance or confidence interval. AIME 2025 contains only 30 problems. The authors describe one- or two-point differences as indicative, and they explicitly avoid calling the LiveCodeBench result a victory over the teacher. Those cautions make the larger AA-LCR and AIME gaps interesting, while leaving the small deltas unsettled.
There is another comparison missing. The paper's main explanation says learning from the original model should beat distilling from the recovered bfloat16 checkpoint. It never runs that direct, matched experiment. QAH is compared with QAT, which uses hard-label cross-entropy, rather than with quantization-aware distillation from the recovered 60B teacher. The authors call this the most useful experiment still to add and describe their proposed "recovered-teacher ceiling" as motivated but unmeasured.
Faster convergence may matter more than the final score
A smaller 20B-to-9B experiment compared QAH and QAT through 1,200 training steps. Both reached nearly the same peak average across MMLU-Pro, LiveCodeBench, and GPQA Diamond: 54.9 for QAH and 54.6 for QAT. QAH reached its peak around step 100, while QAT took about 700 steps. After that peak, QAH stayed within roughly two points; QAT fell by nearly 19 points by step 1,200, according to Figure 3 in the paper.
For a training team, this may be more useful than a modest benchmark gain. A method that holds near its best checkpoint reduces the cost of choosing an early-stopping point and the risk of serving a later, degraded checkpoint. The authors attribute the difference to the objective: a frozen teacher distribution gives the QAH student a stable target, while cross-entropy against hard labels can keep moving the model after its inherited abilities begin to erode. That mechanism is their interpretation of two GPT-OSS compression runs, rather than a general result across model families.
The training system also changed model quality in ways the researchers did not expect. Across an 11-configuration QAT sweep, the best DeepSpeed ZeRO-3 run reached 65.15 on GPQA Diamond, while the best FSDP2 run reached 73.74. The team suspects an interaction between mixed-precision communication and MXFP4 weights, but did not isolate the cause. Their operational conclusion is modest: for this workload, the distributed backend had to be treated as a tuned training variable.
The open weights do not make the method fully reproducible
Multiverse Computing says QAH produced Hypernova-60B, and the current model repository is public under Apache 2.0 with Transformers loading instructions. The paper also says the released checkpoint received more training after the evaluated pipeline. Its public model-card scores therefore differ from the paper's QAH table. Anyone checking the claim needs to compare the paper's three experimental checkpoints, rather than treating the current repository as the exact artifact behind every reported number.
One central component remains proprietary: the tensor-network operator used to turn the 120B and 20B models into the 60B and 9B students. QAH itself needs only output-logit access to the original teacher, but the researchers have not tested it with layer pruning, SliceGPT, low-rank decomposition, or other structural compression techniques. They also tested one model family, one MXFP4 format, and one data mixture. There are no results here for Llama, Qwen, Mistral, INT4, NVFP4, or FP8.
The next evidence to watch is therefore concrete. A matched comparison against distillation from the recovered bfloat16 checkpoint would test the paper's central teacher-choice argument. Repeated runs would show whether the smaller benchmark gaps survive ordinary variance. Tests on a public compression pipeline and another model family would reveal whether QAH travels beyond Multiverse Computing's GPT-OSS setup. Until then, the useful finding is bounded: extra teacher supervision during 4-bit conversion recovered more capability than this team's bfloat16 stopping point, and it did so with a steadier training curve.