Architecture-specific floors: why we publish negative results

Sipsa Labs ships our near-lossless 5-bit transformer compression substrate across 22 architectures spanning dense (0.6B–405B), Mixture-of-Experts (47B–141B), and state-space (Mamba-2.8B). Of those, 14 are fully PPL-verified end-to-end, 6 are in active eval, and 2 are in compression. Of the 14 verified, 13 land at sub-1.01× perplexity ratio against the bf16 reference. The remaining one — Llama-3.1-8B — sits at 1.0125×.

That single non-record number is more useful to our customers than any of the records.

The result: 14 fully PPL-verified, one architecture-specific floor

Here is the verified subset of the matrix that ships in our public benchmark page (full machine-readable JSON at docs/benchmarks.json):

^* Phi-3-mini-4k-instruct evaluated at seq_len=128 (not apples-to-apples with the rest of the matrix at seq_len=1024); reported here for parity with our internal benchmark JSON.

An additional 8 architectures sit in our pipeline as of this post: Qwen3-235B-A22B, Phi-3.5-MoE-instruct, Llama-3.1-70B, Qwen3-32B, Mixtral-8x22B-v0.1, and TinyLlama-1.1B-Chat are mid-eval; Mamba-2.8B and Qwen3-1.7B-instruct are in compression with PPL provenance pending re-run. We publish them once the source JSONs round-trip through scripts/verify_all_benchmarks.py — not before.

Why one number gets a different label

Most quantization libraries publish only the best per-architecture result. We do too — but for Llama-3.1-8B we ran three independent perturbation experiments from the production substrate baseline, in three completely different directions, to test whether the 1.0125× was a real architecture-specific floor or just a local minimum we hadn't escaped:

1. Capacity perturbation: increase substrate capacity by ~33%. Result: 1.0137×. Worse.
2. Schedule perturbation: extend the build schedule by ~67%. Result: 1.0135×. Worse.
3. Objective perturbation: add a held-out output-distribution agreement regularizer at lambda=0.1. Result: 1.0698×. Catastrophically worse, despite the regularizer's signal converging cleanly.

Three independent perturbations, three independent failures, three different mechanisms. That converges on a real claim: 1.0125× is the substrate floor for Llama-3.1-8B at 5 bits per weight, not a tunable parameter. Further reduction requires a substrate-level change (a different bit-budget, a different codec family) — not a knob inside this substrate.

Why this matters more than another <1.005× record

Three reasons.

1. Customer trust comes from honest negative results.

Every quantization paper publishes the architecture where their method shines. The customer who tries to apply that method to their architecture finds out the hard way which architectures don't work. We publish the verified architectures with their measured ratios, including the one where the substrate plateaus, because that is what enterprise customers under SOC 2 / SR-11-7 / FDA review actually need to know.

2. Empirically-bounded floors are stronger than asymptotic claims.

"Our method is asymptotically optimal" is unfalsifiable. "We perturbed the substrate three independent ways and the PPL ratio got worse every time, so 1.0125× is the empirical floor" is falsifiable in a single run by anyone with a 32 GB GPU. The reproducibility floor is the trust floor.

3. Architecture-specific floors are a real engineering signal.

Llama-3.1-8B is in the same bf16-loss regime as our other dense architectures, but its 5-bit substrate floor is ~2.5× higher than the 7B-class Mistral floor. That is a real architecture-specific property — likely tied to GQA head density, MLP intermediate width, or pretraining-data distribution. Knowing the floor lets us choose substrate parameters per-architecture instead of globally.

What's verifiable, today, in 5 minutes

Everything in the table above is reproducible from the customer side. Pick any verified row:

pip install ultracompress
huggingface-cli download SipsaLabs/mistral-7b-v0.3-uc-v3-bpw5 --local-dir ./mistral-7b
uc verify ./mistral-7b

The uc verify command pins SHA-256 over the reconstructed model state and confirms reproducible reconstruction against the published manifest. Different from AWQ / GPTQ in that the reconstruction is pinned to the SHA-256 manifest recorded at pack time.

What we do next

The 13 verified architectures already in production stay there. The Llama-3.1-8B floor at 1.0125× is what ships, marked as the verified architecture-specific minimum. The 8 architectures still in eval or compression publish only after their PPL ratios are reproducible from disk. Where headroom remains, it is at the substrate level, not in parameter-tuning the existing substrate against any single architecture.

If you are evaluating compressed-model substrates against your own architecture stack, our benchmark page shows the verified PPL ratios per architecture, and Phase 0 POCs are 1 week, $5K, on the architecture of your choice.