The two independently-published Phase 1 (RQ) and Phase 2 (NF4) techniques, applied to the same model at the same time — the RQ-compressed embedding and NF4-quantized attention/MLP weights both active simultaneously — to measure what stacking compressions actually costs and buys, rather than assuming the benefits simply add up.
The embedding matrix (256,000 × 2,304, ~590M params) is substituted in-place with the exact same Phase 1 RQ reconstruction (M=384, K=1024, L=2, 3.59 bits/param) already validated and deployed on its own site. Every attention and MLP weight is separately loaded as NF4 4-bit, the same Phase 2 technique already validated and deployed on its own site. Nothing here was retrained — both compressions are read-only reuses of already-computed artifacts.
Every variant answered the identical 200 held-out questions, judged the same blinded way. Storage/memory/speed were measured in one consistent benchmark (same GPU, same prompts, same warmup/sync discipline) so these numbers are directly comparable to each other, not just to baseline.
| variant | grounded acc. | open acc. | grounding rate |
|---|---|---|---|
| Uncompressed (baseline) | 92.9% | 80.0% | 93.5% |
| RQ (embedding) | 92.9% | 86.7% | 93.5% |
| NF4 (weights) | 91.6% | 86.7% | 94.2% |
| LUT (serving strategy) | 92.3% | 84.4% | 93.5% |
| Combined RQ+NF4 (this page) | 91.0% | 80.0% | 94.2% |
| MLP RQ v2 | 92.3% | 84.4% | 93.5% |
| variant | storage | peak GPU memory (generation) | speed |
|---|---|---|---|
| Uncompressed | 5.28 GB | 5,278.8 MB (1.0x) | 46.65 ms/tok (1.0x) |
| RQ | 412 MB embedding (vs. 1,180 MB bf16) | 5,278.8 MB (1.0x — no runtime savings) | 46.57 ms/tok (1.0x) |
| NF4 | ~2.2 GB whole model | 3,533.2 MB (0.67x) | 80.32 ms/tok (1.72x) |
| LUT | same 412 MB artifact | 7,830.8 MB (1.48x — worse than baseline) | 85.22 ms/tok (1.83x — slowest) |
| Combined RQ+NF4 (this page) | ~2.2 GB | 2,353.6 MB (0.45x — best of all six) | 80.12 ms/tok (1.72x) |
| MLP RQ v2 | ~1.66B params @ 9.44 bpp (~41% smaller on 63.6% of the model) | 5,279 MB (1.0x — no runtime savings) | 65.18 ms/tok (1.40x) |
The honest takeaway: combining doesn't naively add the individual benefits. Combined has the worst grounded accuracy of any variant tested (91.0%) and its open-domain accuracy (80.0%) doesn't beat NF4 alone (86.7%) — the two compressions' approximation errors compound rather than average out. Its real selling point is memory: 2,353.6 MB peak is the lowest of all six variants, genuinely useful if the priority is fitting on the smallest possible GPU and the accuracy trade-off is acceptable — not a strictly-better replacement for NF4 or RQ alone.
The embedding portion of this model still gets fully reconstructed to bf16 at load time (same limitation as RQ alone) — only the NF4-quantized attention/MLP weights genuinely stay compressed throughout serving. The measured memory number above reflects both effects together, not a clean per-technique breakdown.
Generation is capped at 80 tokens, same as every other Gemma-2-2B demo in this project.