Quantization Breaks Retrieval While Accuracy Holds: The Top-2 Gap Test

A September 2026 paper documents a failure mode that standard quantization evaluation cannot see: a model whose classification accuracy survives 4-bit weight quantization still changes a large fraction of its retrieval top-1 results. The Undetected Damage of Quantization on Retrieval and How to Fix It (Zhou, Zirilli, Solombrino, Dessì and Rodolà, arXiv:2609.24322, cs.LG, submitted September 21, 2026) reports that across three backbones and two modalities, top-1 retrieval results change for 14 to 46 percent of queries at 4 bits, while the same checkpoints, under the same quantizer, change their classification answer for only a few percent of inputs¹. The benchmark numbers move a fraction as much: on the paper's CLIP text-to-image setting, Recall@1 falls by 1.3 percent while 24.6 percent of top-1 results change and about one in ten correctly answered queries loses its image¹.

The mechanism is almost embarrassingly simple, and that is the point. Every score the model produces — a class logit or a query-document similarity — moves by at most some bounded displacement ε under bounded rounding error. The top-1 result is then guaranteed to survive only when its lead over the runner-up exceeds 2ε: any smaller gap admits a perturbation that swaps the two. Classification's cross-entropy loss pushes the correct class logit away from all others, so most classification inputs have the gap for free; contrastive retrieval training separates a positive only from its paired negatives and never asks the top item to stay far from second place¹. The gap is measurable without labels, which makes it both a pre-deployment predictor and a per-input runtime trust signal — and it drives two task-specific fixes: spend extra bit-width on the layers that move the gap most, or route only low-gap inputs to full precision.

This guide verifies the numbers at primary, walks the bound and the assumptions it quietly makes, reproduces the failure mode, the bound check and the selective-precision economics in runnable Python, and then prices the anti-hype: what a "top-1 change" is and is not, and where the clean bound stops being clean.

1. The effect: same model, same weights, different task, different damage

The paper's core experiment holds everything fixed except the task. Three fine-tuned backbones — ViT-B/16 and ViT-L/16 for images, Qwen3-Embedding-0.6B for text, 53 fine-tuned checkpoints in total — are quantized weight-only at 4 and 3 bits (W4/W3), symmetric per-channel or per-128-weight-group scales, every linear layer quantized, and then read two ways: as a classifier (top-1 predicted class) and as a retriever (first-ranked item over a corpus)¹. At W4 round-to-nearest with group-128 scales, the top-1 change rate versus the full-precision (FP) model splits by task¹:

The same checkpoint, the same quantizer, the same bit-width: a few percent of classification answers change, and up to 45.5 percent of retrieval top-1 results do. The abstract's headline range of 14 to 46 percent spans the paper's full measurement set — 14 percent is the best case (a well-separated text embedder such as GTE or BGE under a stronger quantizer like GPTQ at W4), and the upper end is image retrieval under plain round-to-nearest¹. The single-checkpoint row of the paper's Table 1 makes the asymmetry un-ignorable for the "different models" objection: one set of embeddings, read through a frozen linear probe versus cosine self-retrieval, still changes retrieval top-1 for 41.2 percent of queries versus 8.6 percent for classification, a 4.8× ratio on identical vectors¹.

The aggregate metrics that would clear this model for deployment see almost none of it. On the paper's Table 2 (round-to-nearest, group-128, four BEIR corpora): Qwen3-Emb-8B at W4 loses 3.1 percent of mean nDCG@10 while 15.6 percent of queries whose gold document ranked first lose it; Qwen3-Emb-0.6B loses 7.5 percent of nDCG@10 while 22.2 percent of gold-top-1 queries lose the gold from rank 1. The document that lost first place usually stays inside the top-10 window that nDCG@10 averages over, so the metric absorbs the damage. The paper's summary: at W4 the gold document leaves the top-1 for 16-22 percent of queries that had one, and leaves the top-10 just as often, while nDCG@10 falls only 2.7-7.6 percent¹.

2. The mechanism: gap, contender set, and the 2ε bound

The analysis is a worst-case argument over the score vector z over N candidates, where a "score" is whatever the system ranks by — a class logit in classification, a query-candidate similarity in retrieval. Quantization is modeled as replacing z with z + δ, where δ is a perturbation with ‖δ‖∞ ≤ ε, the largest amount by which quantization displaces any score. Writing g_2(z) for the gap between the top-1 and the runner-up, the paper's Lemma 1 states: the top-1 survives every ε-bounded perturbation if and only if g_2(z) ≥ 2ε¹.

Both directions matter. Necessity is the two-line argument: each score moves by at most ε, so a candidate can overtake the winner only if it started within 2ε of it; when nobody did, no perturbation can produce a new leader. Sufficiency is the constructed swap: if the gap is below 2ε, subtract ε from the winner and add ε to the runner-up and the order inverts. The proof sits on older ground than the paper — when every score is known only to an interval of the same width, the induced comparison relation is the semiorder studied by Luce, Scott and Suppes, and Fishburn, and the rankings a bounded perturbation can reach are its linear extensions². The paper's own credit is precise: "what is new here is the measurement, not the condition"¹.

Three refinements make the condition deployable rather than merely true.

The contender set, not the runner-up. The set C_ε(z) = {k ≥ 2 : g_k(z) < 2ε} — all positions a perturbation could push past the top-1 — can reach deep into the ranking. Being within 2ε is not transitive, so a chain of near-ties lets a candidate far below the top overtake it. This is where retrieval flips actually come from: when classification changes its top-1, the new top-1 is the old runner-up 82.2 percent of the time; in the paper's image retrieval it is only 39.7 percent, with the rest coming from deeper in the list¹. Retrieval top-1 flanks are crowded; classification margins are not.

Differences, not levels. Corollary 1 observes that whether the top-1 survives depends on δ only through the differences δ_j − δ_a involving the winner — a perturbation that shifts every score equally changes nothing, however big ε is. So the quantity to calibrate is S(x) = max_j(δ_a′ − δ_j) ≤ 2‖δ‖∞ on the realized perturbation, not the global 2ε; the paper's stability check therefore accepts strictly more inputs than the raw bound would (calibrating 2ε instead of S accepts 67.2 percent of classification inputs against 85.2 percent at α = 0.10, and no retrieval query at all)¹.

Measured, not assumed. Crucially, the paper does not assume ε: it runs both the FP model and the quantized model on the input and measures δ = z^PTQ − z^FP, taking ε = ‖δ‖∞. The bound's job is structural — it tells you which statistic predicts flips — while the perturbation size is an empirical quantity, which is what lets one threshold generalize across queries.

The mechanism explains the task asymmetry without any appeal to quantizer weakness. At W4 the contender set is empty for 74.3 percent of ViT-B/16 classification inputs and for none of the retrieval queries; over everything the paper measures, 85.8 percent of classification inputs sit above the stability threshold (median separation ratio g_2/2ε = 5.73) against 3.5 percent of retrieval queries (median 0.081)¹. Classification training pays the gap; retrieval training does not, because a contrastive loss separates a positive only from the negatives it was paired with, and the second-ranked document is a comparison training never made.

3. What the gap buys: prediction, per-input trust, and two fixes

The separation ratio g_2/2ε turns out to explain the damage across everything the paper measures: the median separation correlates with the top-1 change rate at Spearman −0.88 over 2,118 measured configurations, and −0.98 across twelve model/bit-width configurations (p < 10⁻⁷)¹. Neither ingredient alone is as predictive — the median gap reaches −0.58 and the median ε just +0.05, against −0.88 for the ratio. A practitioner's consequence the paper makes explicit: pick the retrieval model with higher separation, not the larger one — GTE-large at 0.34B parameters changes fewer top-1 results at both bit-widths than Qwen3-Embedding-4B at twelve times the size, because its gaps are wider¹.

Per-input stability check (no labels). The top-1 condition needs FP scores and the perturbation size, neither of which is available at deployment time when only the quantized model runs. The paper's rewrite uses the quantized model's own top-1/top-2 gap against a single threshold τ̂ calibrated once on n unlabeled inputs as the split-conformal quantile of S(x). When calibration and test inputs come from the same distribution, conformal calibration guarantees Pr(accepted and top-1 changed) ≤ α with no test-time labels³. Measured violations stay inside α in every split: on ViT-B/16 classification at α = 0.10, coverage is 85.2 ± 0.1 percent with a 1.47 percent conditional violation rate; on text retrieval coverage is only 11.3 percent — the check correctly refuses to vouch for almost every retrieval query — and on CLIP ViT-L/14 it accepts 15.7 percent¹. The paper prices this honestly: it inherits conformal prediction's exchangeability assumption and degrades under distribution shift, "the standard conformal assumption" being precisely what its Limitations section lists¹.

Fix 1, retrieval: allocate bits where the gap moves. Weight-only PTQ methods decide how to round, not where to spend bits — bit-width is set globally and existing allocators rank layers by reconstruction error. The paper's criterion instead spends the extra bit-width on the layers whose quantization moves the top-1/top-2 gap most: for each linear layer ℓ, quantize that layer alone with plain round-to-nearest at 3 bits, rerun 128 calibration queries, and take s_ℓ = median_x |g_2(x) − g_2^(ℓ)(x)| — how much the one-layer quantization displaces the gap between the same two documents. Layers are ranked by s_ℓ divided by parameter count, and at a fixed budget of 3.5 bits per weight, the top half receives 4 bits and the rest stay at 3. The whole measurement is L+1 forward passes over 128 unlabeled queries and trains nothing; the sensitivities are measured once with RTN and the resulting assignment is applied under whatever quantizer the deployment uses¹. The result: gap allocation recovers 60-73 percent of the full W3→W4 benefit at half the extra bit's cost, ranks first among all seven forward-only criteria under every quantizer tested, and its lead over the next-best criterion runs from 5.5 to 18.4 points across 24 model/corpus/quantizer configurations. The allocator it replaces — reconstruction error, the objective of existing allocation methods — captures only 40-48 percent and is nearly uncorrelated with gap sensitivity across layers on text embedders (mean Spearman −0.04)¹. In absolute terms at 3.5 bits under RTN: 33.3 percent of top-1 results change against 43.1 percent for reconstruction-driven allocation, and 13.8 percent of correct top-1 documents are lost against 19.9 percent¹.

Fix 2, classification: route low-gap inputs to FP. In classification the at-risk minority is small, so the fix is per-input. The quantized model's own top-1/top-2 gap comes free from the forward pass the deployment already runs; route x to the FP model iff g_q(x) < τ, with τ set at the 25th percentile of g_q on an unlabeled validation slice¹. Routing 25 percent of inputs recovers 85-93 percent of the accuracy lost to quantization on the paper's three backbones (tasks with at least a 0.5-point FP→PTQ gap), at a cost the paper computes as 47 percent of the FP cost at the 4.5× speedup GPTQ reports. A Proposition 3 identifiability result adds a theoretical floor: up to order-equivalence, the gap is the only score of the quantized logits whose threshold sets match the worst-case flippable sets at every budget¹.

4. Runnable cells

Three self-contained cells (standard library only, seed fixed, run with Python 3.9.6). The first reproduces the asymmetry: identical bounded perturbations, two different gap distributions, and the flip-rate gap that results.

import random
 
# CELL 1: the gap>2*eps criterion -- retrieval vs classification gap distributions
random.seed(11)
 
N = 200            # candidates
Q = 500            # queries/inputs per task
 
def retrieval_scores():
    # retrieval-like: scores cluster near the top (nothing separates 1st from 2nd)
    return sorted((random.uniform(0.85, 1.0) for _ in range(N)), reverse=True)
 
def classification_scores():
    # classification-like: winner pulled away from the pack by a wide margin
    rest = sorted((random.uniform(0.0, 0.35) for _ in range(N - 1)), reverse=True)
    return [0.9 + random.uniform(0.0, 0.1)] + rest
 
eps = 0.01          # largest rounding-induced score displacement
 
def flip_stats(gen):
    flips = contenders = 0
    for _ in range(Q):
        z = gen()
        g2 = z[0] - z[1]
        if g2 < 2 * eps:
            contenders += 1
        # simulate the actual perturbation: independent uniform in [-eps, eps]
        d = [random.uniform(-eps, eps) for _ in range(N)]
        zp = [a + b for a, b in zip(z, d)]
        if zp.index(max(zp)) != 0:
            flips += 1
    return flips / Q, contenders / Q
 
rf, rc = flip_stats(retrieval_scores)
cf, cc = flip_stats(classification_scores)
 
print(f"eps = {eps}  (stability threshold 2*eps = {2*eps})")
print()
print("task          | top-1 flip rate | at-risk inputs (g2 < 2*eps)")
print(f"retrieval     |     {rf:6.1%}      |     {rc:6.1%}")
print(f"classification|     {cf:6.1%}      |     {cc:6.1%}")
print()
print("same perturbation size; only the gap distribution differs")

Output from a real run (Python 3.9.6, seed 11):

eps = 0.01  (stability threshold 2*eps = 0.02)
 
task          | top-1 flip rate | at-risk inputs (g2 < 2*eps)
retrieval     |      78.4%      |     100.0%
classification|       0.0%      |       0.0%
 
same perturbation size; only the gap distribution differs

One caveat the cell also demonstrates: with this deliberately extreme gap contrast, the classification side never flips — the paper's real classification inputs flip a few percent of the time because real gaps are not infinite. The at-risk column is the paper's real separation: retrieval leaves 100 percent of queries below the threshold, classification none.

The second cell makes the bound concrete on an actual quantization grid: a weight matrix quantized with per-group scales, the largest realized score displacement, and which candidates sit inside the resulting contender set.

import random
from math import ceil
 
# CELL 2: block-quantization rounding-error bound -- where the 2x bound bites
random.seed(23)
 
# a weight matrix: out=64 channels, in=256 inputs, group size 128
OUT, IN, GROUP, BITS = 64, 256, 128, 4
 
W = [[random.gauss(0.0, 0.08) for _ in range(IN)] for _ in range(OUT)]
X = [random.gauss(0.0, 1.0) for _ in range(IN)]     # one typical query input
 
def quantize(w, bits, group):
    step = lambda mx: 2 * mx / (2**bits - 1)
    q = [[0.0] * len(w[0]) for _ in w]
    max_round = 0.0                                   # largest per-score disp basis
    for r, row in enumerate(w):
        for g0 in range(0, len(row), group):
            g = row[g0:g0 + group]
            mx = max(abs(v) for v in g)
            s = step(mx)
            for j in range(len(g)):
                q[r][g0 + j] = round(g[j] / s) * s
                max_round = max(max_round, abs(q[r][g0 + j] - g[j]))
    return q, max_round
 
Wq, werr = quantize(W, BITS, GROUP)
 
def scores(M):
    return [sum(M[i][j] * X[j] for j in range(IN)) for i in range(OUT)]
 
z, zq = scores(W), scores(Wq)
disp = [abs(a - b) for a, b in zip(z, zq)]
eps = max(disp)
 
order = sorted(range(OUT), key=lambda i: -z[i])
g2 = z[order[0]] - z[order[1]]
violators = [i for i in range(1, OUT) if z[order[0]] - z[order[i]] < 2 * eps]
 
print(f"max per-weight rounding error (group_{GROUP}, {BITS} bits): {werr:.5f}")
print(f"largest realized score displacement eps:        {eps:.5f}")
print(f"stability threshold 2*eps:                     {2 * eps:.5f}")
print()
print(f"top-1 score           {z[order[0]]:.4f}")
print(f"top-2 score           {z[order[1]]:.4f}")
print(f"top-1/top-2 gap g2:   {g2:.5f}")
print()
print(f"contender set |C_eps| (candidates within 2*eps of top-1): {len(violators)}")
print(f"separation ratio g2 / 2*eps: {g2 / (2 * eps):.3f}  "
      f"({'stable' if g2 >= 2 * eps else 'NOT stable'})")
print()
print("the bound assumes |displacement| <= eps on every score;")
print("with a per-channel / per-group scale this holds exactly, and the")
print("criterion reduces to comparing one measured gap against 2*eps")

Output from a real run (Python 3.9.6, seed 23):

max per-weight rounding error (group_128, 4 bits): 0.02315
largest realized score displacement eps:        0.42153
stability threshold 2*eps:                     0.84307
 
top-1 score           2.5917
top-2 score           2.3443
top-1/top-2 gap g2:   0.24734
 
contender set |C_eps| (candidates within 2*eps of top-1): 4
separation ratio g2 / 2*eps: 0.293  (NOT stable)
 
the bound assumes |displacement| <= eps on every score;
with a per-channel / per-group scale this holds exactly, and the
criterion reduces to comparing one measured gap against 2*eps

Two readings. The separation ratio of 0.293 is squarely in the paper's retrieval regime (median 0.081) — nowhere near the threshold and therefore flippable. And the gap between the per-weight rounding error (0.02) and the score-level displacement (0.42) is the honest scale problem: eps at the output is an accumulated, input-dependent quantity, which is exactly why the paper measures it rather than deriving it.

The third cell mirrors the allocation economics: at a fixed budget of 3.5 bits per weight, does spending the extra half-bit on the gap-moving layers buy a disproportionate share of the full extra bit's benefit?

import random
 
# CELL 3: selective-precision economics -- gap-moving layers vs a uniform extra bit
random.seed(31)
 
L = 12               # linear layers
Q = 400              # queries
N = 200              # candidates
 
# Each layer, quantized at 3 bits, displaces every query-document score by an
# independent error uniform in [-e_l, e_l]. A few layers are far more
# gap-sensitive than the rest (paper finding: sensitivity is concentrated).
E3 = [0.080 if l in (2, 5, 8) else 0.003 for l in range(L)]
E4 = [e / 8 for e in E3]          # one extra bit: 1/8 the rounding error
 
def flip_rate(bits_per_layer):
    flips = 0
    for _ in range(Q):
        # retrieval-like scores: top items nearly tied
        z = [random.uniform(0.85, 1.0) for _ in range(N)]
        d = [sum(random.uniform(-e, e) for e in
                 (E3[l] if b == 3 else E4[l] for l, b in
                  enumerate(bits_per_layer))) for _ in range(N)]
        zp = [a + b for a, b in zip(z, d)]
        if zp.index(max(zp)) != z.index(max(z)):
            flips += 1
    return flips / Q
 
# rank layers by sensitivity / parameter count (here: equal sizes, so by e_l),
# give the extra bit to the half that moves the gap most -> avg 3.5 bits
ranked = sorted(range(L), key=lambda l: -E3[l])
gap_guided = [4 if l in ranked[:L // 2] else 3 for l in range(L)]
 
r_w3 = flip_rate([3] * L)
r_gap = flip_rate(gap_guided)
r_w4 = flip_rate([4] * L)
 
benefit_full = r_w3 - r_w4
benefit_gap  = r_w3 - r_gap
capture = benefit_gap / benefit_full
 
print("allocation                   | avg bits | top-1 flip rate")
print(f"uniform W3                   |   3.00   |     {r_w3:5.1%}")
print(f"gap-guided (6 layers at 4b)  |   3.50   |     {r_gap:5.1%}")
print(f"uniform W4                   |   4.00   |     {r_w4:5.1%}")
print()
print(f"full extra bit's benefit (fewer flips):  {benefit_full:5.1%}")
print(f"gap-guided captures:                    {capture:5.1%} of it")
print("for 50% of the extra bit's memory cost")

Output from a real run (Python 3.9.6, seed 31):

allocation                   | avg bits | top-1 flip rate
uniform W3                   |   3.00   |     97.0%
gap-guided (6 layers at 4b)  |   3.50   |     89.5%
uniform W4                   |   4.00   |     86.0%
 
full extra bit's benefit (fewer flips):  11.0%
gap-guided captures:                    68.2% of it
for 50% of the extra bit's memory cost

The toy lands inside the paper's measured band: gap-guided allocation at half the extra bit's cost captures 68.2 percent of the full bit's benefit, against the paper's 60-73 percent. The parameter that drives it is the concentration of sensitivity: when three of twelve layers dominate the score displacement, the extra half-bit buys almost what a full bit buys everywhere; make the sensitivity spread uniform and the capture collapses toward what reconstruction-error allocation buys (the paper's 40-48 percent). The method's value is empirical — it measures where the concentration actually is, per model, with L+1 cheap label-free passes.

5. Anti-hype: what "14-46 percent" is, and what it is not

The numbers are against the FP model, not against ground truth. A "top-1 change" means the quantized model returns a different first result than the FP model would — not necessarily a worse one. The paper measures this distinction: judged against BEIR relevance labels, 66 percent of flips whose FP top-1 was a relevant document replace it with a non-relevant one at W4 (77 percent at W3), and on CLIP, where each query has one correct image, 26-30 percent of changed results lose it while 13-23 percent gain it¹. So the damage is majority-harmful but not uniformly so, and a downstream system matters even for the neutral flips: a different retrieved document yields a different generated answer, and two servers running the same model at different precisions answer the same query differently. The paper's own framing of "undetected" is a benchmarking-practice claim — Recall@k and nDCG@10 on a fixed corpus can hide a 24.6 percent top-1 flip rate behind a 1.3 percent metric dip — and the correct response is to report ranking-change rates alongside aggregate metrics, not to treat every flip as an accuracy loss.

Strong quantizers reduce but do not remove this. The paper repeats the measurement with GPTQ (with and without activation ordering), AWQ, HQQ and AdaRound: all five reduce the top-1 change rate — GPTQ with activation ordering takes ViT-L/16 W4 from 42.1 to 29.6 percent and the best text cases to 14.4-14.5 percent — and none approaches the stability threshold on retrieval¹. The strongest single datapoint: GPTQ on ViT-L/16 restores classification accuracy from 55.5 to 88.6 percent and cuts the classification change rate eleven-fold, while retrieval on the same model still changes most of its top-1 results. If your mental model is "a good quantizer fixes PTQ," retrieval is the counterexample.

The bound is only as clean as the rounding-error model. The 2ε argument presumes a bounded, per-score displacement. The paper measures ε empirically per input (which sidesteps deriving it), but the regime matters: with per-channel or per-128-group scales the per-weight rounding error is bounded by half a grid step, and the accumulation to a score-level ε is measurable — this is the paper's setting, and activation quantization is explicitly untested (its Limitations list weight-only, untested activations, exact cosine over a finite corpus with approximate-nearest-neighbor indexes out of scope)¹. Schemes that transform weights non-uniformly — per-channel outlier splits, learned scales under clipping — make the displacement less cleanly bounded, and the conformal threshold inherits the exchangeability assumption. The structure of the criterion survives; the ε in "g_2 ≥ 2ε" is an empirical quantity per model per bit-width, not a constant of nature.

The routing fix is classification-only, and the paper says why. At W4 the at-risk fraction is 94-100 percent of retrieval queries against 9-26 percent of classification inputs¹; routing a minority only works when the minority is small, and for retrieval the per-input check correctly recovers almost nothing to certify. The retrieval-side answer is global (allocate bits), the classification-side answer is per-input (route), and mixing them up — routing retrieval queries on gap — would route nearly everything.

The question to ask of any quantized retrieval deployment this paper should provoke: what fraction of top-1 results differ from the FP model, measured on a held-out slice, and — if that number is not reported — why not?

6. Verdict

The paper's contribution divides cleanly. The condition (top-1 survives iff g_2 ≥ 2ε) is old mathematics, correctly credited to the semiorder and interval-order literature; the measurement is new and sharp — the same checkpoint flipping a few percent of classification answers and 33-46 percent of retrieval top-1 results at W4, with aggregate metrics blind to it, is a benchmarking-practice finding production systems should act on. The gap statistic is the right object: label-free, cheap, predictive across 2,118 configurations at Spearman −0.88, and it powers two fixes whose economics the paper measures rather than asserts (60-73 percent of an extra bit's benefit at half its cost; 85-93 percent of lost accuracy recovered by routing a quarter of inputs). The paper's own accounting prices the open questions — whether training can be made to encourage the gap a deployment needs is stated as the open question, activations and ANN indexes as out of scope. For anyone shipping quantized retrievers, the actionable core is one number to start reporting tomorrow: the top-1 agreement rate with the FP model, per corpus, before and after quantization.

Footnotes