Uncheatable Eval: Scoring LLMs by How Few Bits They Need

A September 2026 paper takes the least glamorous number in language modeling — negative log-likelihood — and promotes it to the benchmark itself. Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models (Tan, Li and Shen, arXiv:2609.27510, cs.CL, submitted September 23, 2026) argues that if you want to compare base models without benchmark rot, you should stop scoring answers and start scoring how many bits each model needs to losslessly encode text that did not exist when its training data was frozen¹. The mechanism is the prediction–compression equivalence the paper draws on explicitly: a model that assigns higher probability to the observed next token needs fewer code bits for that token, so a rolling snapshot of newly published text becomes an eval whose answer key is simply the text itself¹.

The design answers a real disease. Benchmark leakage has complicated evaluation to the point where static question sets cannot be trusted for frontier training runs, and base models — pretrained, not post-trained — cannot reliably follow the instructions a task benchmark assumes in the first place¹. Uncheatable Eval sidesteps both: no questions, no instructions, no reference answers, just next-token probability on documents collected after any plausible training cutoff.

This guide derives the meter honestly — bits per byte from cross-entropy, the compression-rate↔bpb↔ratio conversion, a rewording demonstration of what the meter cannot see — reproduces the paper's headline numbers (80 models, 14 categories, the 3.200·P^-0.290 + 5.328 scaling fit, the long-context architecture split), and then applies the pressure the paper invites and then some: what compression doesn't measure, what can still leak through the curation pipeline, and why the leaderboard may be partially a scoreboard of pretraining data choices.

1. The construction: a benchmark whose answer key is the text

The evaluation corpus is a July 2026 snapshot of newly published text from public sources, 14 categories: English fiction, News, English encyclopedia, Non-English encyclopedia, five scientific-paper categories (Biology preprints, Computer science, Mathematics, Physics, Other scientific papers), and five code categories (C++, JavaScript, Markdown, Other, Python)¹. Each category holds 500 samples — 7,000 documents total. A curation pipeline drops too-short and anomalous samples, removes near-duplicates with MinHash locality-sensitive hashing, and NFC-normalizes Unicode text, so a model cannot farm the score by compressing the same press release five hundred times¹.

For scoring, each sample is truncated to at most 3,584 tokens under each of eight tokenizers, producing one shared text per sample that every model sees under a common budget — a deliberate reconciliation of the tokenizer mismatches that otherwise make cross-model bits incomparable¹. Longer-context evaluation keeps up to 32,768 UTF-8 bytes of the same documents.

2. The meter: −log₂ p, divided by bytes

The paper's compression rate is the ideal code length from arithmetic coding, not a gzipped file size. For a document s with token sequence x₁…x_T under model θ,

B_θ(s) = −log₂ p_θ(x₁:T) = −Σₜ log₂ p_θ(x_t | x_(1:t-1)),

i.e. the next-token cross-entropy in bits. Normalizing by the document's UTF-8 byte count n_byte(s) and pooling over a document collection gives the reported compression rate:

CR_θ(𝒮) = 100 · Σ_s B_θ(s) / (8 · Σ_s n_byte(s)).

Lower CR is better prediction; the paper reports it as a percentage¹. Note the two quiet scope decisions: this is the theoretical code length — finite-precision coding overhead and the storage cost of the model's own parameters are excluded — and each score is a property of the model–tokenizer pair, not the model alone¹.

Before trusting the paper's numbers, run the arithmetic on a toy. The following cell scores one fixed sentence with an order-0 (byte-frequency) and an order-1 (byte-bigram, add-one smoothed) model, standard library only, no seeding needed for the fixed text:

import math
from collections import Counter, defaultdict
 
# Cell 1 (FINAL) — bpb from cross-entropy, order-0 and order-1 toy models
TEXT = "compression scoring measures prediction, not intelligence"
b = TEXT.encode("utf-8")
n = len(b)
f = Counter(b)
bits0 = sum(-c*math.log2(c/n) for c in f.values())
bpb0 = bits0/n
 
ctx = defaultdict(Counter)
for a, c in zip(TEXT, TEXT[1:]):
    ctx[a][c] += 1
alpha = sorted(set(TEXT))
V = len(alpha)
bits1 = -math.log2(1/V)  # first char uniform
for a, c in zip(TEXT, TEXT[1:]):
    bits1 += -math.log2((ctx[a][c]+1)/(sum(ctx[a].values())+V))
bpb1 = bits1/n
 
print(f"text = {TEXT!r}")
print(f"n_byte = {n}")
print(f"order-0 : code length {bits0:.4f} bits  -> bpb = {bpb0:.6f}  -> CR = {100*bpb0/8:.4f} %")
print(f"order-1 : code length {bits1:.4f} bits  -> bpb = {bpb1:.6f}  -> CR = {100*bpb1/8:.4f} %")

Output from a real run (Python 3):

text = 'compression scoring measures prediction, not intelligence'
n_byte = 57
order-0 : code length 218.2199 bits  -> bpb = 3.828419  -> CR = 47.8552 %
order-1 : code length 185.1509 bits  -> bpb = 3.248262  -> CR = 40.6033 %

The two readings worth keeping: (1) CR is not an exotic statistic — it is bits-per-byte expressed as a percentage of the 8-bit original (CR = 100·bpb/8), and bpb is the average base-2 cross-entropy per byte. (2) The only thing that moved between order-0 and order-1 was context — same text, same alphabet, 33 fewer bits — which is exactly the lever the paper's long-context study pulls on real architectures¹.

3. Unit translation: what the paper's percentages mean in gzip terms

The paper's headline numbers live in the 3–10 percent range, which sounds magical until you convert. A CR of 5.328 % means the model encodes 1,000 bytes into 53.28 bytes' worth of bits (66.6 bits per 100 bytes). The cell below converts paper-typical CRs into bpb and into the gzip-style "ratio 1:N" figure (my toy CR above, ~40 %, is deliberately terrible — real LLMs are two orders of magnitude better):

import math
from collections import Counter
 
# Cell 2 — CR -> bpb -> ratio conversion on paper-typical CRs
for cr in [3.16, 5.328, 6.31, 7.315]:
    bpb = cr*8/100
    print(f"CR {cr:6.3f} % -> bpb {bpb:.4f} -> 1 byte in {bpb:.4f} bits -> ratio 1 : {8/bpb:.3f}")

Output from a real run (Python 3):

CR  3.160 % -> bpb 0.2528 -> 1 byte in 0.2528 bits -> ratio 1 : 31.646
CR  5.328 % -> bpb 0.4262 -> 1 byte in 0.4262 bits -> ratio 1 : 18.769
CR  6.310 % -> bpb 0.5048 -> 1 byte in 0.5048 bits -> ratio 1 : 15.848
CR  7.315 % -> bpb 0.5852 -> 1 byte in 0.5852 bits -> ratio 1 : 13.671

Gzip-class compressors sit around 2.2–2.9 bpb on English text, so a strong model's 0.25–0.5 bpb on its best categories is roughly a 5–10x improvement — the paper's aggregate fitted CR for large models is ~5.3 % (0.43 bpb), and the best 30B-class per-category scores in its Table 2 reach about 3.16 % (0.25 bpb) on code¹. These are believable, physical numbers, which is precisely why the meter is worth taking apart carefully.

4. Findings, re-verified against the full paper

Scaling. Across the 80-model cohort, byte-weighted CR across all 14 categories follows a power law with an additive constant: CR(P) = a·P^b + c fits with the aggregate estimate CR(P) = 3.200·P^−0.290 + 5.328, R² = 0.915, RMSE 0.293 percentage points, with P in billions of parameters (mixture-of-experts models count all experts)¹. The asymptote c ≈ 5.33 says the meter sees diminishing returns in the way every scaling law does — the fitted curve flattens as size grows¹. The Pareto frontier of models not dominated by any other fits even tighter (3.244·P^−0.304 + 5.016, R² = 0.992)¹. But fit quality varies by category: R² runs from 0.594 for non-English encyclopedia articles to 0.935 for biology preprints¹ — compression scaling is cleanest exactly where the text is most formulaic.

Architecture families and context. The 54-model long-context study on four scientific-paper categories is the paper's most original result, and the summary is a slope. The paper buckets byte positions into intervals (1–2, 2–4, 4–8, 8–16, 16–32 KiB), then fits CR as α + β·log₂(position): β is the CR drop in percentage points per doubling of available context. For similar-sized representatives — Qwen3-8B-Base (attention), Falcon-H1-7B (hybrid), RWKV7-G1J-7.2B (recurrent) — the pooled slopes are −0.426, −0.454 and −0.314, with total CR gains of 1.700, 1.807 and 1.266 points from first to last interval¹. The recurrent model is not stuck — its absolute CR keeps improving — but it improves more slowly, so RWKV's early-position advantage (0.067 points below Qwen3 in the 1–2 KiB interval) decays into a 0.367-point deficit at 16–32 KiB, while Falcon-H1 crosses the other way (0.025 worse to 0.082 better)¹. And the decay is partly a training artifact: older RWKV checkpoints trained with 4,096-token contexts decay more (0.41–0.65 points) than matched 16,384-token G1J checkpoints (0.33–0.55)¹. How compression improves with context is an architecture-plus-training signature — the paper's own framing, and the finding a static-loss benchmark cannot see at all.

The MMLU correlation. All 80 models were also run on zero-shot MMLU (14,042 questions, highest next-token logit among A–D), and lower technical-text CR — pooled over five Scientific Paper and four Code categories — is associated with higher MMLU accuracy at Spearman ρ = −0.884, bootstrap 95% CI [−0.942, −0.783]. Pooling all 14 categories gives ρ = −0.871, and each of the nine technical categories individually shows ρ between −0.883 and −0.853¹. The sign is expected; a model that predicts everything better knows more of everything. The paper's verb is "associated", and it should stay that verb.

5. What the meter cannot see: a rewording demonstration

Compression rate is a property of this byte sequence. Change the wording, keep the content, and the meter moves — or fails to. The cell below scores two one-sentence documents with the same toy order-0 model; they assert the same idea at the same difficulty, but B is shorter and worded differently:

import math
from collections import Counter
 
# Cell 3 — identical content, reworded: bpb moves
def order0_bpb(s):
    b = s.encode("utf-8"); n = len(b); f = Counter(b)
    return sum(-c*math.log2(c/n) for c in f.values())/n, n
A = "compression scoring measures prediction, not intelligence"
B = "compression scoring gauges foresight, not cognition"
for name, s in [("original ", A), ("reworded ", B)]:
    bpb, n = order0_bpb(s)
    print(f"{name}: {n} bytes, bpb {bpb:.4f}, CR {100*bpb/8:.2f} %")

Output from a real run (Python 3):

original : 57 bytes, bpb 3.8284, CR 47.86 %
reworded : 51 bytes, bpb 3.7760, CR 47.20 %

Same claim, same model, different score. That is not a bug — CR was never claimed to measure difficulty or truth, only predictability — but it is the honest seed of every criticism below: the meter scores rendering, and everything a model does that is not next-token prediction on the sampled corpus is invisible to it.

6. Anti-hype: four ways to cheat an uncheatable eval

It measures text prediction, full stop. The paper says it itself: "compression rate does not capture the instruction-following abilities acquired through post-training" — the protocol is limited to base models, and no tool use, no math, no agentic behavior, no long-form coherence enters the score¹. A model with identical next-token quality and wildly different downstream usefulness gets the same CR. The ρ = −0.884 MMLU link does not close that gap: it is a correlation across a 2026 cohort of 80 models, pooled over technical categories; MMLU itself is contamination-susceptible, and a rank agreement leaves most of the models' MMLU variance tied to things compression does not touch. Nothing here certifies CR as a general-capability proxy.

Contamination has not been eliminated — it has been moved upstream. The paper's own limitations section: "publication dates and model training cutoffs may be incomplete, and models may have seen copies of the same content"¹. Text that is newly published in July 2026 can be old in substance — syndicated wire copy, boilerplate documentation, translated encyclopedia fragments, re-posted papers, source files from long-lived repositories. MinHash near-duplicate removal operates within the snapshot, not against each model's training set, so pre-snapshot near-duplicates glide through as "new". A model that overdosed on C++ boilerplate before its cutoff compresses July 2026 boilerplate well without ever seeing the snapshot. The curation pipeline narrows the leak; it does not close it.

The correlation may be a snapshot of pretraining data mixes. ρ = −0.884 is measured on one cohort of prevailing base models. Those models' pretraining mixes are heavily correlated with each other (shared web crawls, shared code corpora, shared paper scrapes), so the CR↔MMLU agreement may partly reflect a shared 2025-era data recipe rather than a law of capability. The paper partially controls for this — regressing CR residuals across text types shows models do have type-specific fingerprints (pairwise Pearson r = 0.803–0.969 across the five text types, Scientific Paper and Code correlating most strongly at 0.969)¹ — but a model generation trained on a deliberately different mix could break the mapping without losing any capability. Correlations fitted on one data-recipe generation are exactly what the meter's own history warns about.

The eval pays out for recency-dense pretraining choices. If the score is compression of July-2026 text, the cheapest legal way to move it is not more capability but more distribution-matched pretraining data: upweight current news, fresh preprints, new repository code. A vendor who mirrors the benchmark's source categories monthly is buying CR points the honest way under the rules, and the rules cannot tell that from intelligence. The paper is aware the risk compounds over time — "this contamination risk grows as the dataset ages: new models may train on its public texts, so we must refresh" the corpus¹ — which makes refresh cadence, not the method, the load-bearing component of "uncheatable".

7. Verdict

The core idea is sound, old, and finally operationalized at scale: a model is a compressor, and freshly published text is the one test set a frozen model provably has not seen at publication time. The math is beyond dispute — CR is average cross-entropy per byte, rescaled — and the empirical work is better than the abstract suggests: the architecture-vs-context slopes and the training-context comparison are genuinely new signal, and the curation pipeline is a real attempt to kill the farming strategies previous compression-based evals tolerated. The claims are also scoped more honestly than the title implies, which is to the authors' credit: no instruction-following claim, no post-training claim, explicit residual contamination risk.

The overreach to resist is reading the leaderboard as a capability ranking. It is a prediction-quality ranking on one month's corpus, in the categories those 80 models happen to be fed on — with a measured correlation to a leaked-prone legacy benchmark and structural levers (data mix, recency density, syndicated near-duplicates) that move the score without moving anything you would call intelligence. Use it for what it is: the cleanest public meter of next-token prediction on unseen text, and a warning about what every static benchmark cannot give you.

Footnotes