CXL-SSDs For LLM Prefix Caching: Byte-Addressability Buys Nothing Without Chunk Awareness

A September 2026 systems paper opens with a result that should temper a lot of CXL marketing decks: when you plug an unmodified, generic CXL-SSD into an LLM prefix-caching stack, it is about 3× slower than local DRAM and no faster than a plain NVMe SSD — and the standard sequential prefetcher you would reach for to fix that leaves average time-to-first-token (TTFT) nearly unchanged while performing irrelevant NAND reads¹. The paper, Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management (Chung, Noh, Kim and colleagues at Sogang University with Samsung Electronics and ETRI, arXiv:2609.26828, cs.AR, submitted September 20, 2026), then earns its constructive half: LM-CXD, a CXL-SSD specialized for LLM prefix caching that closes the semantic gap between the serving engine (which knows which KV chunks will be consumed) and the device (which controls their placement and movement), cutting average TTFT by up to 2.6× over the stock CXL-SSD with compute-asynchronous prefetching and 4.03× with layerwise prefetching — landing within 1.5× of local DRAM on average¹.

The negative result is the load-bearing part, because it cleanly separates two claims that CXL marketing perennially fuses. Claim one: the block I/O path is expensive for KV retrieval regardless of the medium beneath it — CPU cache contention, host-DRAM staging copies, syscall bookkeeping. Claim two: CXL's byte-addressable, memory-semantic interface removes those costs. The paper proves claim one with a beautiful experiment (a ramdisk that keeps DRAM behind the block interface remains 1.8× slower than direct DRAM, with no NAND anywhere in the system) and then refutes the naive reading of claim two: removing the block path does not make a CXL-SSD faster than an NVMe SSD for this workload, because the costs that remain — NAND latency and device-side thrash in a bounded DRAM region — are not addressed by addressability at all¹.

This guide walks the full stack in arithmetic: the interface-cost decomposition, the silicon reality of CXL versus PCIe bandwidth, why chunk granularities and NAND granularities collide, why any prefetcher without engine knowledge must speculate and lose, and the layerwise pipelining bound that decides whether NAND latency actually hides under compute. Every number cited from the paper was verified against the full HTML text, not the abstract².

1. The problem: prefix caching has outgrown DRAM, and the block path is why SSDs hurt

Modern LLM traffic — RAG contexts, multi-turn chat, agentic loops — re-sends prefixes of tens to hundreds of thousands of tokens. Prefix caching reuses the KV caches computed for those prefixes to skip redundant prefill, and systems like LMCache extend the cache beyond GPU memory into host DRAM and storage³. But host DRAM is costly and bounded by memory channels and DIMM capacity, so production stacks drop the cold tier onto NAND-backed NVMe SSDs. NAND exposes that capacity only through the block interface, and the paper's characterization of a prefix-reuse workload (Qwen3-4B on vLLM with LMCache, 128k prefix, 32-token chunks, four storage configurations) traces the degraded TTFT to precisely three costs¹:

1. CPU cache contention. Block I/O runs on host threads that share the last-level cache (LLC) with inference threads, using it for entirely different purposes. Isolating the I/O threads' L3 partition with Intel Cache Allocation Technology (CAT) recovers up to 44% of P99 TTFT — a large, tail-shaped cost for something as mundane as syscalls touching page cache⁴.
2. Host-DRAM staging. The block interface forces every chunk through host DRAM before the GPU can read it, because the GPU can DMA only from pinned pages mapped into the application's address space. Measured with VTune: on a ramdisk, writes rise from 6% to 33% of DRAM traffic, holding DRAM busy 3.6× as long per retrieval. A pinned-DRAM baseline at 1.34 s average TTFT degrades to 5.27 s on ext4-over-brd, and stripping ext4 recovers only 0.2 s of the 3.9 s gap — the staging copy is the filesystem-independent part¹.
3. NAND latency itself. Replacing NAND with DRAM (the ramdisk) closes 61% of the SSD's TTFT gap to direct DRAM in steady state — so a bit over half the SSD slowdown is the medium, and the rest is the interface.

The first two costs are properties of the interface, not the medium. That is the paper's first durable finding: a ramdisk — pure DRAM, served through the block path — is still 1.8× slower than direct DRAM. And the obvious escape hatches fail for structural reasons. Host-side prefetch overlaps NAND latency but reintroduces the DRAM-capacity tension: you use SSDs because host DRAM is limited, yet prefetched chunks must occupy host DRAM before consumption. GPUDirect Storage (GDS) bypasses CPU and host DRAM but cannot remove the block-orientation of the path: cuFileRead lands every chunk in a VRAM staging buffer, from which load_and_reshape kernels reformat and copy it into vLLM's paged KV layout — two GPU-side copies per chunk competing for VRAM with model weights and hot KV, while NAND latency stays on the request's critical path⁵.

2. Why byte-addressable does not mean faster: the silicon underneath

CXL is a cache-coherent interconnect built on PCIe physical infrastructure. A CXL Type-3 device exposes Host-managed Device Memory (HDM) through the CXL.mem protocol, mapped into the host physical address space and accessed with ordinary load/store instructions⁶. A CXL-SSD bolts that onto an SSD architecture: controller, on-device DRAM (the DRAM region, serving hits at memory latency), and NAND flash behind a flash translation layer¹.

Here is the arithmetic the marketing glosses over. A PCIe 5.0 x16 link carries roughly 64 GB/s raw in each direction; after 128b/130b encoding, packet overhead, and the fact that a CXL.mem read is a round trip (request plus data, plus the dHDM coherency machinery), effective sustained bandwidth for memory-semantic loads from a Type-3 device lands well under that — typically in the 30–50 GB/s class per link, usually shared with other traffic on the same root complex. An aggregate host DRAM system on a modern dual-socket server delivers several times that across its channels. So the ceiling of CXL-attached memory is not DRAM speed; it is PCIe speed — the same wall NVMe sits behind. And below the DRAM region, a CXL-SSD miss pays a CXL.io/FTL round trip plus NAND program-read latency (tR), on the order of tens of microseconds for conventional TLC and low single-digit microseconds for the Z-NAND the paper models⁷.

That is why "byte-addressable" is a programming-model property, not a performance property:

• The DMA toll is the same link either way. Whether the GPU's source buffer is pinned host DRAM or a cudaHostRegister-mapped CXL region, the host-to-GPU transfer traverses the same PCIe hierarchy. Addressability removes the staging copy, not the transfer.
• Cache-line traffic still crosses the link. A random cache-line pull from device DRAM costs a full CXL.mem transaction pair (request line, return line, with coherency state). Bursty KV fills at 2 MiB hugepage granularity amortize this fine; a miss-driven, pointer-chasing retrieval pattern does not.
• Contention doesn't vanish, it relocates. With the block path gone, I/O threads disappear — but under load, every CXL-SSD read still competes for the shared link, the device's on-device DRAM region bandwidth, and the FTL's NAND channels.

The next cell prices the whole retrieval path for a realistic single-request retrieval — 18 GiB of KV (a 128k-token prefix for Qwen3-4B: 36 layers, GQA with 8 KV heads of 128 dimensions, BF16, in 56-token chunks) — under each backend, both as wall time and as delivered GB/s. The bandwidth values are stylized round numbers, chosen to be defensible for a single-socket server with one x16 GPU link; the stage structure of each path is the point, and it is taken directly from the paper's characterization.

import random
 
# Interface-cost arithmetic for one 128k-token KV retrieval (Qwen3-4B shape:
# GQA 8 KV heads x 128 head dim, BF16, 36 layers, 56-token LMCache chunks).
# Stylized bandwidths; ratios, not absolute times, are the point.
random.seed(260926828)
 
TOK, LAYERS, CHUNK_TOK = 131072, 36, 56
KV_TOK_LAYER = 2 * 8 * 128 * 2                 # K+V bytes per token per layer
CHUNK_B  = CHUNK_TOK * KV_TOK_LAYER * LAYERS   # bytes per KV chunk
NCHUNK   = TOK // CHUNK_TOK
PAYLOAD  = NCHUNK * CHUNK_B
 
B_HOST, B_PCIE = 40e9, 20e9    # host DRAM rw, host<->GPU link (every path pays this)
B_NVME, B_GDS  = 7e9, 12e9     # NVMe steady read, GDS NVMe->VRAM DMA
B_VRAM         = 40e9          # VRAM-internal copy bandwidth (load_and_reshape)
SYS_US         = 15e-6         # per-chunk block-path kernel/bookkeeping charge
 
base = PAYLOAD / B_PCIE        # the DMA toll no path avoids
rows = [
 ("pinned host DRAM",      [(1, B_PCIE)]),
 ("ramdisk (brd+ext4)",   [(1, B_HOST), (1, B_PCIE)]),
 ("local NVMe O_DIRECT",   [(1, B_NVME), (1, B_PCIE)]),
 ("GDS NVMe -> VRAM",      [(1, B_GDS), (1.35, B_VRAM)]),
 ("CXL-SSD, DRAM hit",    [(1, B_PCIE)]),
 ("CXL-SSD, NAND miss",   [(1, B_NVME), (1, B_PCIE)]),
]
print(f"payload {PAYLOAD/2**30:.2f} GiB in {NCHUNK} chunks of {CHUNK_B/2**20:.2f} MiB "
      f"(multi-hugepage: {CHUNK_B/2**20 > 2})")
print(f"{'path':26s} {'stages':>6s} {'wall ms':>9s} {'deliv GB/s':>10s} {'vs pinned':>9s}")
for name, stages in rows:
    t = sum(PAYLOAD * f / b for f, b in stages) + SYS_US * (NCHUNK if "NVMe" in name or "ramdisk" in name else 0)
    print(f"{name:26s} {len(stages):5d}  {t*1000:9.1f} {PAYLOAD/t/1e9:9.2f} {t/base:9.2f}x")
 
# Why byte-addressable != faster, mechanically:
# 1) the DMA toll is the same PCIe link for pinned DRAM and CXL-mapped memory;
# 2) block paths add host-DRAM staging passes (writes 6% -> 33% of DRAM traffic
#    in the paper, DRAM busy 3.6x as long per retrieval);
# 3) I/O shares the LLC -- isolating the I/O partition with Intel CAT recovers
#    up to 44% of P99 TTFT in the paper. Overlapped degradation, stylized:
print("\nretrieval overlapped with decode, LLC partitioning off (stylized):")
for drop in (0.30, 0.44):
    t = PAYLOAD / (B_PCIE * (1 - drop))
    print(f"  effective DMA -{int(drop*100)}% -> {PAYLOAD/t/1e9:5.2f} GB/s ({t/base:4.2f}x slow-down)")

Output from a real run (Python 3, seed 260926828):

payload 18.00 GiB in 2340 chunks of 7.88 MiB (multi-hugepage: True)
path                       stages   wall ms deliv GB/s vs pinned
pinned host DRAM               1      966.1     20.00      1.00x
ramdisk (brd+ext4)             2     1484.3     13.02      1.54x
local NVMe O_DIRECT            2     3761.6      5.14      3.89x
GDS NVMe -> VRAM               2     2297.5      8.41      2.38x
CXL-SSD, DRAM hit              1      966.1     20.00      1.00x
CXL-SSD, NAND miss             2     3726.5      5.19      3.86x
 
retrieval overlapped with decode, LLC partitioning off (stylized):
  effective DMA -30% -> 14.00 GB/s (1.43x slow-down)
  effective DMA -44% -> 11.20 GB/s (1.79x slow-down)

Three readings, all matching the paper's measured structure. The ramdisk line reproduces the 1.8× block-interface surcharge the paper measures (1.54× here — same mechanism: one extra full-payload pass through host DRAM plus per-chunk kernel overhead, minus the LLC contention term the wall-clock column of this toy serializes away). The NVMe line reproduces the ~3× total SSD penalty, with the NAND-bandwidth term dominant. And the two CXL lines are the article's thesis in one table: a DRAM-region hit is as fast as pinned host DRAM — the staging copy really is gone — but a miss is exactly as slow as NVMe, because it is the same NAND behind the same FTL, and nothing about CXL.mem changes tR. Addressability moves the cost curve's hit branch to DRAM speed while leaving the miss branch untouched. Whether you get the good branch depends entirely on what the device keeps in its DRAM region — and that is where stock devices fail.

3. The CXL-SSD paradox: a better interface, no better TTFT

The paper's § 4.1 experiment is the one to commit to memory. They run the multi-turn Q&A workload against a stock CXL-SSD implementation — same 2 MiB hugepages and CLOCK eviction the paper's own device will use, LMCache asynchronous loading enabled, no device-level prefetch — and against the same device with Cylon's standard next-n sequential prefetch policy, n = 4 pages per DRAM miss⁸. Results: applying the CXL-SSD without specialization does not improve performance over the SSD baseline; P99 TTFT regresses above the SSD's; and next-n leaves average TTFT nearly unchanged, with some turns worse. The counter comparison is precise: next-n issues 19,198 async loads, but evictions rise from 369,104 to 387,840 and NAND reads from 1,494,131 to 1,504,029 — prefetch added NAND traffic and evictions instead of saving them¹.

Why does a prefetcher make things worse? Because all CXL-SSD I/O must go through the bounded DRAM region, and a stock device's placement policy has no idea what is coming. Sixteen concurrent sessions sharing a 16 GiB region while each needs GiB-scale prefixes means thrash: chunks repeatedly evicted and re-fetched between DRAM and NAND. Next-n cannot help because it is miss-driven and speculative: it must first incur a DRAM miss to trigger, guesses the next sequential pages by address, and when its guess is late or wrong it pays an extra miss and burns NAND bandwidth on pages nobody wanted. The vicious cycle the paper names is structural: miss-driven prefetch must incur the miss it is trying to prevent¹.

And here is the granularity collision that makes speculation hopeless. LMCache's unit is the chunk — 32 to 128 tokens depending on the model, roughly 8–40 MiB of KV bytes at these shapes — while the device's unit is the 2 MiB hugepage, itself an alignment imposed on 4 KiB OS pages, mapping down onto NAND pages (tens of KiB) grouped into erase blocks (typically hundreds of KiB to a few MiB). A layerwise-inference read wants one layer across many chunks, which is neither page-shaped nor chunk-shaped. A device that sees only page addresses cannot serve either access shape, and it certainly cannot predict it: KV reuse follows the tree structure of prefixes — system prompt, retrieved documents, conversation history — not the address layout of a flat array. The serving engine knows the shape of that tree at prefix-lookup time, before any byte moves. The device never sees it. That asymmetry — not the absence of prefetching — is the gap.

The next cell reproduces the DRAM-region dynamics under three policies at the paper's oversubscription: 16 concurrent sessions each re-touching a growing prefix per turn, against a region that holds a fraction of the working set. Stock is miss-driven; next-n speculates on chunk+1 addresses; the hint policy is a stand-in for LM-CXD's windowed approach — the engine tells the device which chunks the next turns will need, and prefetched windows are pinned against eviction until consumed.

import random
 
# Felt-NAND-latency accounting for one CxS turn: does the DRAM region hold a
# session's chunks when the engine reaches for them? Region 16 GiB = 4096
# chunk slots; 16 concurrent sessions, each turn re-touching the full
# accumulated prefix and appending 390 new chunks. CLOCK eviction, no GC.
random.seed(260926828)
 
SLOTS = 4096
SYS = 200
NEW_PER_TURN = 390
N_SESS, N_TURN = 16, 6
 
def prefix_len(turn):                    # chunks a turn t session must read
    return SYS + NEW_PER_TURN * (turn + 1)
 
def run(policy, windows=2, wsize=350):
    resident = {}                        # cid -> True, dict preserves CLOCK order
    pin = set()
    st = dict(on_demand=0, resident_hit=0, wasted_fetch=0, nand=0, evict=0)
    def clock_evict():
        while len(resident) >= SLOTS:
            v = next(iter(resident))
            if v in pin:
                resident[v] = resident.pop(v)
                continue
            del resident[v]; st["evict"] += 1
            return
    def fetch(cid, pin_it=False):
        if cid in resident:
            resident[cid] = resident.pop(cid)
            return
        st["nand"] += 1
        clock_evict()
        resident[cid] = True
        if pin_it: pin.add(cid)
    for turn in range(N_TURN):
        for s in range(N_SESS):
            base = 1000000 + s * 100000
            need = [ *range(SYS), *range(base, base + prefix_len(turn) - SYS) ]
            if policy == "stock":
                for cid in need:
                    if cid not in resident:
                        st["on_demand"] += 1
                        fetch(cid)
            elif policy == "next-n":     # speculative: grab the next chunk address
                for cid in need:
                    if cid not in resident:
                        st["on_demand"] += 1
                        fetch(cid)
                    nxt = cid + 1
                    if nxt not in need and nxt not in resident:
                        fetch(nxt); st["wasted_fetch"] += 1
            else:                        # windowed engine hints, pinned windows
                for w in range(windows):
                    for cid in need[w * wsize:(w + 1) * wsize]:
                        fetch(cid, pin_it=True)
                for cid in need:
                    if cid in resident:
                        st["resident_hit"] += 1
                    else:
                        st["on_demand"] += 1
                        fetch(cid)
                pin.clear()
    return st
 
print(f"region {SLOTS} chunk slots; per-turn demand per session grows "
      f"{prefix_len(0)} -> {prefix_len(N_TURN-1)} chunks")
print(f"{'policy':22s} {'on-demand':>9s} {'pre-hit':>8s} {'NAND':>8s} {'wasted':>7s} {'evict':>7s}")
for name, pol in (("stock (miss-driven)", "stock"), ("next-n (speculative)", "next-n"),
                  ("windowed hints+pin", "hints")):
    st = run(pol)
    print(f"{name:22s} {st['on_demand']:9d} {st['resident_hit']:8d} {st['nand']:8d} {st['wasted_fetch']:7d} {st['evict']:7d}")

Output from a real run (Python 3, seed 260926828):

region 4096 chunk slots; per-turn demand per session grows 590 -> 2540 chunks
policy                 on-demand  pre-hit     NAND  wasted   evict
stock (miss-driven)       137040        0   137040       0  132944
next-n (speculative)      137040        0   137166     126  133070
windowed hints+pin         84800    65440   131240       0  127144

The signature matches the paper's Table 2 exactly in shape. Next-n adds NAND reads (137,166 versus 137,040) and evictions (133,070 versus 132,944) over doing nothing — spent entirely on wrong guesses (126 wasted fetch-chunk-events in this toy; in the paper, nearly twenty thousand async loads that bought zero hit-rate improvement and negative thrash). The windowed hint policy converts 65,440 of the stock device's on-demand misses into pre-staged, pinned hits — chunks resident in the DRAM region before the engine reaches for them — with fewer total NAND reads and fewer evictions, because it never fetches what nobody will consume and never evicts what is pinned in-flight. That conversion, miss-before-need into hit-before-need, is the entire performance story of LM-CXD. The toy also shows its honest limit: the working set still does not fit, so even hint-driven prefetch cannot push NAND reads to zero — the remaining 84,800 on-demand reads in the toy correspond to the paper's finding that "hundreds of milliseconds still remain at the tail, so neither method's overlap windows hide NAND entirely."

4. LM-CXD: pricing the co-design, one property at a time

The paper's move is to stop treating the CXL-SSD as a generic memory device and make the KV chunk the unit the device indexes, moves, reports progress on, and transfers to the GPU. Three properties, each closing a specific measured failure¹:

Property 1: chunk identity becomes I/O. LMCache's prefix lookup already computes chunk hashes for every request at arrival — that is how it finds the longest cached prefix. LM-CXD's shared structures (prefix lookup table tracking chunk-hash to location, a pin map protecting in-flight windows from CLOCK eviction, a free queue for chunk deletion, a prefetch queue of chunk hashes, and a prefetch plan pool for layerwise) live in a reserved region of device DRAM, readable and writable by both LMCache and the device controller with no syscalls, interrupts, or copies. LM-CXD resolves chunk hashes against its own index, skips chunks already in the DRAM region, and issues NAND reads itself — the cost of chunk-matching on the host becomes 6–7 microseconds per chunk under compute-asynchronous prefetching (CAP), against the 160 microseconds per chunk the stock device spends copying¹.

Property 2: the engine acts on device progress. LM-CXD records per-session dispatch and completion state in device DRAM. LMCache polls it, admits a session once its first two windows' worth of chunks are dispatched (not completed), and DMAs finished chunks to the GPU while the session waits in vLLM's non-blocking WAITING_FOR_REMOTE_KVS state, under other sessions' compute. This is the bidirectional half: hints flow engine-to-device, progress flows device-to-engine, and admission stops waiting for completion of loads it can overlap. Promotion is capped per scheduling step with one unconditional promotion, so the dispatcher drains between admissions.

Property 3: device DRAM is the GPU's source buffer. The DRAM region is cudaHostRegister-mapped pinned memory; DMA pulls chunks straight from device DRAM to VRAM without host staging. The windowing discipline (a per-rank pin budget sized to the region, round count, and workload; primed two windows ahead; the next window starts only when another unpins) is what makes PREFETCH possible in a region that cannot hold an entire prefix — the paper's third design principle, hide NAND latency under bounded device DRAM, made concrete.

Then two prefetch methods, each matched to a workload regime. CAP paces windowed prefetch behind lookup hints while other sessions compute — the benchmark answer for concurrent multi-turn serving. Layerwise prefetching goes further: it tracks a per-request plan, converts chunks from chunk-major to layer-major layout in a dedicated staging area of device DRAM, and uses a strided cudaMemcpy3DAsync descriptor so the DMA itself expresses the layout change (two copies per layer instead of two per chunk). A constant n-deep lookahead buffer lets NAND reads run ahead of layer consumption; the paper runs prefetch degree 7 on Qwen3-VL-32B.

5. The layerwise pipelining bound

Whether layerwise prefetching can pay is pure overlap arithmetic: can the NAND-to-DRAM movement of the next layers complete inside the GPU's per-layer compute time on this model at this prefix length? The bound is per-layer transfer time against per-layer compute time; the lookahead depth needed is their ratio; if the ratio exceeds one, no amount of pipelining hides NAND and the tail stays. The next cell runs the arithmetic for three of the paper's evaluated models at their evaluated prefixes. The compute model is the standard prefill attention envelope (4-bound) at 35% MFU on an L40S's 362 TFLOP/s BF16 — stylized, but the structural conclusion only needs to be directionally right because the transfer and compute terms scale so differently with prefix length.

import random
 
# Layerwise pipelining arithmetic: can NAND->DRAM movement for layer L+1..L+k
# hide under the GPU's compute for layer L? Uses the paper's evaluated models
# (Table 4): Qwen3-4B (36 layers), Qwen3-VL-32B (64 layers, layerwise
# prefetch degree 7), Llama-3.1-70B (80 layers). GQA 8 KV heads x 128 head
# dim, BF16. NAND: 16-channel Z-NAND, tR = 3 us -- parallel-array bound.
random.seed(260926828)
 
KVH, DIM, BYTES = 8, 128, 2
NAND_BW = 6.0e9            # effective sustained NAND->DRAM region read
CXL_DMA  = 18e9            # device-DRAM -> VRAM strided cudaMemcpy3DAsync
GPU_TFLOP = 362e12          # L40S BF16 dense peak
MFU = 0.35                 # realized fraction on attention prefill work
 
MODELS = [
    ("Qwen3-4B @128k",   36, 131072, 56),
    ("Qwen3-VL-32B @64k", 64,  65536, 32),
    ("Llama-70B @16k",   80,  16384, 128),
]
 
def layer_bytes(prefix_tok, layers):
    return 2 * KVH * DIM * BYTES * prefix_tok          # one layer, K+V
 
for name, layers, tok, csize in MODELS:
    lb = layer_bytes(tok, layers)
    t_xfer = lb / NAND_BW                       # NAND -> device DRAM, one layer
    t_dma  = lb / CXL_DMA                       # device DRAM -> VRAM, one layer
    flops = 4 * tok * tok * KVH * DIM
    t_compute = flops / (GPU_TFLOP * MFU)
    degree_needed = t_xfer / t_compute          # lookahead k so k >= t_xfer/t_c
    total_chunk = layer_bytes(tok, layers) * layers
    t_serial = layers * (t_xfer + t_dma + t_compute)
    t_pipe = layers * max(t_compute, t_xfer + t_dma / degree_needed)
    t_ideal = layers * t_compute
    print(f"{name}: layer {lb/2**20:6.1f} MiB  t_xfer {t_xfer*1e3:7.2f} ms  "
          f"t_dma {t_dma*1e3:6.2f} ms  t_compute {t_compute*1e3:8.2f} ms")
    print(f"    full-prefix layer KV {(lb*layers)/2**30:6.2f} GiB  "
          f"lookahead needed {degree_needed:5.2f} layers  "
          f"pipelined/serial speedup {t_serial/t_pipe:4.2f}x  vs compute-only {t_pipe/t_ideal:4.2f}x")
 
print("\nQwen3-VL-32B, full 64k prefix retrieval vs pipelined layerwise:")
name, layers, tok, _ = MODELS[1]
lb = layer_bytes(tok, layers)
t_xfer, t_dma = lb / NAND_BW, lb / CXL_DMA
t_c = 4 * tok * tok * KVH * DIM / (GPU_TFLOP * MFU)
t_serial_all = layers * (t_xfer + t_dma + t_c)
print(f"  serial (retrieve, then compute): {t_serial_all*1e3:8.0f} ms")
for deg in (1, 4, 7):
    t_pipe = layers * max(t_c, (t_xfer + t_dma) / deg)
    print(f"  layerwise degree {deg}: total {t_pipe*1e3:7.0f} ms  "
          f"({t_pipe/t_c/layers:4.2f}x pure compute, {t_serial_all/t_pipe:4.2f}x vs serial)")

Output from a real run (Python 3, seed 260926828):

Qwen3-4B @128k: layer  512.0 MiB  t_xfer   89.48 ms  t_dma  29.83 ms  t_compute   555.40 ms
    full-prefix layer KV  18.00 GiB  lookahead needed  0.16 layers  pipelined/serial speedup 1.21x  vs compute-only 1.00x
Qwen3-VL-32B @64k: layer  256.0 MiB  t_xfer   44.74 ms  t_dma  14.91 ms  t_compute   138.85 ms
    full-prefix layer KV  16.00 GiB  lookahead needed  0.32 layers  pipelined/serial speedup 1.43x  vs compute-only 1.00x
Llama-70B @16k: layer   64.0 MiB  t_xfer   11.18 ms  t_dma   3.73 ms  t_compute     8.68 ms
    full-prefix layer KV   5.00 GiB  lookahead needed  1.29 layers  pipelined/serial speedup 1.68x  vs compute-only 1.62x
 
Qwen3-VL-32B, full 64k prefix retrieval vs pipelined layerwise:
  serial (retrieve, then compute):    12704 ms
  layerwise degree 1: total    8886 ms  (1.00x pure compute, 1.43x vs serial)
  layerwise degree 4: total    8886 ms  (1.00x pure compute, 1.43x vs serial)
  layerwise degree 7: total    8886 ms  (1.00x pure compute, 1.43x vs serial)

Read the table the way the paper's measurements do. Qwen3-4B at 128k has 555 ms of compute per layer to hide 119 ms of movement behind — trivially overlappable, lookahead depth 0.16, pipelining reaches compute-bound (1.00× of pure compute). Qwen3-VL-32B at 64k likewise: depth 0.32 against 60 ms of movement under 139 ms of compute. Llama-3.1-70B at only 16k tokens is the cautionary row: 15 ms of movement against 8.7 ms of compute, lookahead needed 1.29 layers — the threshold case. This is precisely the paper's measured anomaly: Llama-70B's layerwise gains are the smallest (1.74× average TTFT improvement versus Qwen3-VL-32B's 4.03×), its P99 far tail stays within 1.3× of the stock device in the NAND-wait CDFs where the other models gain 2.9–4.8×, and its P99 TTFT in turn 3 spikes back to stock levels — because at 16k the per-layer compute offers the least overlap time, and "the few that arrive late have the least per-layer compute to hide behind." Same physics, both directions: prefix 4× shorter and layer count 25% higher than Qwen3-VL-32B's configuration flips the sign of the budget.

The degree-1/4/7 tie in the second block is the bound teaching its own lesson: once the movement pipeline for the next layer fits inside the current layer's compute at degree 1 (depth needed 0.32 is less than 1), extra lookahead degrees buy nothing — the paper's degree-7 setting on this model is slack, not necessity, and the real constraint the paper identifies elsewhere is the DRAM cost of the lookahead buffers (about 9 GiB for 64k-prefix models, displacing more than half the 16 GiB region — the true price of layerwise, not the prefetch machinery).

6. What the co-design actually buys, measured

Across five models (Qwen3-4B, Ministral-8B, Qwen3-Coder-30B-A3B, Qwen3-VL-32B, Llama-3.1-70B — all GQA) on the CxS multi-turn benchmark (16 sessions, 6 turns) and an agentic-coding trace (AIPerf, 48 sessions, 128k–160k prefixes): TTFT falls 2.51× and 2.6× on average under CAP for Qwen3-4B and Ministral-8B; 4.03× for Qwen3-VL-32B under layerwise; 1.74× for Llama-70B; within 1.5× of local DRAM on average overall¹. Median felt-NAND time per request falls from 139–920 ms (stock, across the four models) by at least 3× at the median and 2.5–4.1× at P90 under LM-CXD — with honest hundreds-of-milliseconds tails remaining. On the agentic trace, mean TTFT is nearly identical to stock (most of that KV stays hot in the DRAM region by reuse frequency) and the win is a 17% lower P90 — chunk-aware prefetch pays exactly where reuse is sporadic, and nowhere else.

The DRAM-region sensitivity table is the cleanest pricing of the semantic gap: at 8/16/32 GiB regions, the stock CXL-SSD averages 6.26/6.53/1.98 s TTFT while LM-CXD holds 2.78/2.40/1.90 s. The stock device is highly region-size-sensitive (3× swing); LM-CXD moves less than 1.5× across the sweep and at 8 GiB beats the stock device at 16 GiB by 2.3×. The lifecycle split shows why: the stock device's queue and prefill times both rise about threefold at small regions, while LM-CXD admits on dispatch and copies pinned windows, so its growth lands in the queue alone. The shared-structure overhead is negligible in the other direction — 22 MiB, 0.13% of a 16 GiB region, for everything CAP needs¹.

7. Anti-hype: what this paper does not establish

Nothing here runs on commercial silicon. No CXL-SSD is commercially available; LM-CXD is implemented on a modified NVMeVirt platform (page-fault-intercepted DRAM with an injected Z-NAND timing model, NUMA-node memory standing in for CXL timings), validated against the Cylon emulator on bimodal latency and per-miss breakdown (matched ≈6.2 microsecond device time; the residual difference is Cylon's KVM/QEMU exit overhead, not device physics)⁸. Every absolute number inherits emulation error; the mechanism claims are the transferable part.

The 1.5× gap to DRAM is a floor with known causes. Even fully specialized, device DRAM is bounded, NAND reads remain on the tail, and LM-CXD's queue-time component does not vanish — at 32 GiB regions the paper reports the two devices converge and LM-CXD's prefill matches local DRAM's, leaving queue time as the residual. Chunk-aware design narrows the gap; it does not close it.

Co-design cuts both ways. LM-CXD's bidirectional interface costs host-side bookkeeping (1.2–2.4% of a retrieve — small but nonzero), the lookahead buffers displace more than half the chunk region at 64k prefixes, and layerwise is a net loss for small-prefix, fast-decode models (Qwen3-4B: +1.40 s prefill to remove 0.51 s of queue wait). The paper's own cross-model comparison shows no single prefetch method winning everywhere — CAP for concurrent serving, layerwise for large-single-session loads — which is the standard signature of a real system rather than a benchmark artifact.

KV cache regenerability is load-bearing. LM-CXD discards live chunks from erased NAND lines rather than relocating them — no garbage collection — because KV can always be recomputed. That is a prefix-cache-specific exemption; the design does not generalize to storage where data must survive erasures¹.

Byte-addressability remains the right primitive, for the right reason. The correct reading of the negative result is not "CXL-SSDs are pointless," it is "addressability is necessary but not sufficient." Every path that touched the block interface left 1.8× on the table; removing it was worth doing before any chunk-awareness — a DRAM hit on LM-CXD's mapped region is pinned-DRAM-fast. Then and only then does the semantic work pay its 2.6–4.03×.

For a practitioner the checklist distills to: (1) keep hot prefixes in plain DRAM first — the stock CXL-SSD loses to local DRAM by 3×, and even LM-CXD concedes 1.5×; (2) if a NAND tier is unavoidable and CXL-SSDs materialize, nothing generic helps — demand chunk-visible interfaces, windowed pinned prefetch, dispatch-progressive admission, and layerwise conversion; (3) budget the device DRAM region explicitly and check the per-layer transfer-versus-compute ratio at your prefix lengths before promising anyone 4×.

Footnotes