https://flozi.net/en Deep dives into servers, GPUs, AI infrastructure and modern IT systems. en Fri, 25 Sep 2026 10:10:37 GMT https://flozi.net/en/blog/glm-b300-mtp-nvls https://flozi.net/en/blog/glm-b300-mtp-nvls Fri, 25 Sep 2026 00:00:00 GMT Production notes from serving a GLM-class MoE on 8x B300: the acceptance-rate math behind speculative decoding, why the measured log_stats chain says k=2-k=3 captures most of k=5, and why NVLS failures are silent by default. llm-inference speculative-decoding mtp b300 nccl production https://flozi.net/en/guides/ai/agent-fleet-cost-cache-elasticity https://flozi.net/en/guides/ai/agent-fleet-cost-cache-elasticity Fri, 25 Sep 2026 00:00:00 GMT The self-reported day of 44 agents, 4 billion tokens and $1,300 versus $80,000 — recomputed line by line against live price sheets, decomposed into cache discount, model spread and looping fraction, and rebuilt as a capacity-planning checklist for anyone running an internal agent fleet. ai machine-learning llm inference prompt-caching agents economics gpu-memory https://flozi.net/en/guides/ai/boost-host-memory-concurrent-tiering https://flozi.net/en/guides/ai/boost-host-memory-concurrent-tiering Fri, 25 Sep 2026 00:00:00 GMT BOOST (arXiv:2609.13592) decomposed to the bandwidth ledger: why prefetch tiering structurally burns HBM writes during decode, why concurrent proportional access adds the host tier instead of stealing from it, the alpha-math behind +31% throughput and 4.3% TPOT at iso-batch, and the NVLink-C2C scope guard that keeps PCIe x86 hosts out of the claim. ai machine-learning gpu gpu-memory inference hardware kv-cache https://flozi.net/en/guides/ai/cache-read-price-war-sept-2026 https://flozi.net/en/guides/ai/cache-read-price-war-sept-2026 Fri, 25 Sep 2026 00:00:00 GMT The press told a story of three frontier labs cutting prices in unison on September 21-22, 2026. The price sheets say otherwise: one vendor cut cache reads to 0.05x, one shipped new SKUs with an untouched cache multiplier, and one did nothing at all. Decomposed to arithmetic, with per-vendor Barrie-shaped day costings in Python. ai machine-learning llm inference prompt-caching agents economics pricing https://flozi.net/en/guides/ai/cxl-ssd-chunk-aware-kv-cache https://flozi.net/en/guides/ai/cxl-ssd-chunk-aware-kv-cache Fri, 25 Sep 2026 00:00:00 GMT A September 2026 Sogang University paper (arXiv:2609.26828) asks whether CXL-SSDs can replace NVMe for LLM prefix caching and starts with a resounding no: a stock CXL-SSD is about 3× slower than local DRAM and no faster than an NVMe SSD, and a generic next-n prefetcher leaves TTFT essentially unchanged while burning NAND bandwidth. The interesting half is the constructive one — LM-CXD, a co-designed CXL-SSD that makes KV chunks device-visible I/O units, exposes NAND-to-DRAM progress to the serving engine, uses device DRAM as a GPU-accessible buffer, and pipelines layerwise KV movement under GPU compute, reaching up to 2.6× (compute-asynchronous) and 4.03× (layerwise) lower TTFT than the stock device, within 1.5× of local DRAM. This guide prices every step of the interface stack in bandwidth-and-latency arithmetic, works through why CXL's memory semantics do not make loads free, why prefetching without engine knowledge must speculate and lose, and what remains unverified because no CXL-SSD is commercially available yet. ai machine-learning llm inference caching hardware https://flozi.net/en/guides/ai/cxmt-10pct-dram-euv-ceiling https://flozi.net/en/guides/ai/cxmt-10pct-dram-euv-ceiling Fri, 25 Sep 2026 00:00:00 GMT Counterpoint puts CXMT at 10% of global DRAM revenue in Q2 2026, two years ahead of UBS projections. We decompose how a DUV-only memory maker using SAQP at 11.95nm half-pitch got here on the leaders' HBM pivot, run the share, cost-wedge, and yield economics in Python, and locate the structural ceiling: PAM3 timing at multi-patterning overlay noise, a 30% cost-per-bit wedge, and two-generation lag in HBM. ai hardware memory dram supply-chain https://flozi.net/en/guides/ai/deepseek-v41-flash-kv-compression-techniques https://flozi.net/en/guides/ai/deepseek-v41-flash-kv-compression-techniques Fri, 25 Sep 2026 00:00:00 GMT DeepSeek-V4.1-Flash (arXiv:2609.19969, DeepSeek-AI, September 17, 2026) claims 890 bytes of global KV cache per token — roughly 1/4 of its own V4-Flash predecessor in HBM and 1/8 in persistent storage — plus 1M-token context, 552B backbone parameters with only 8B active in prefill and 16B in decode. This guide takes the claim apart mechanically: how CED projects decoder global KV from the encoder's final hidden state (K and V become per-layer linear projections of one shared representation), how CSA2 stacks three multiplicative compression dimensions with statically assigned Full/Reindex/Reuse modes in a 3x6 encoder and 5x4 decoder cadence, how MXFP4 survives on the strength of a norm argument (largest RMSNorm weight ~1 → the 512-channel latent is bounded by sqrt(512) ≈ 22.6 against a representable range of 2688 — 118.8x headroom, so the omitted global scale costs nothing), and how SWA Bounded Replay trades exact L x n_win = 5,120-token reconstruction for 128 tokens. We hunt for a byte ledger that reproduces the 890 from the published config (our closest candidate: 864, 3.0% off — the paper publishes no ledger), price the replay overhead per prompt length, and name the falsification signals: single-layer selection errors poisoning all Reuse layers downstream, position-dependent cache-hit states that are not mathematically identical, and E2M1's 1-bit mantissa concentrated exactly where long-context needle retrieval lives. Every derived number is computed in runnable cells and labeled as ours. ai machine-learning llm inference efficiency kv-cache deepseek https://flozi.net/en/guides/ai/disaggregated-quantization-phase-split https://flozi.net/en/guides/ai/disaggregated-quantization-phase-split Fri, 25 Sep 2026 00:00:00 GMT A September 2026 paper (arXiv:2609.26333) stops treating a model as one quantization problem. Prefill is compute-bound, so low-precision NVFP4 arithmetic accelerates it; decode is bandwidth-bound, so 1-3-bit weights accelerate that. Disaggregated quantization specializes formats, weights, and storage placement per phase: dropping activation quantization on decode alone is free accuracy, a separately trained NVFP4 prefiller lifts a frozen 1-bit GGUF decoder by +32.5 MMLU-Pro points, and SSD-streamed prefill buys a 1.78x TTFT speedup at 8K context — at the price of a second checkpoint to store and qualify, an SSD amortization that only pays on long prompts, and a 1-bit tax that survives. This guide runs the roofline, the crossover math, and a bits-vs-accuracy toy on real cells. ai machine-learning llm inference quantization https://flozi.net/en/guides/ai/flashloop-looped-transformer-kv-redundancy https://flozi.net/en/guides/ai/flashloop-looped-transformer-kv-redundancy Fri, 25 Sep 2026 00:00:00 GMT Looped transformers (Ouro, Huginn) re-run a weight-shared block R times per token to buy depth without parameters — and the KV cache bills them for it: on an A100, a 32K prompt takes Ouro-2.6B 27 seconds versus 3 seconds for LLaMA-3.1-8B, and its cross-loop KV cache holds 48 GiB versus 4 GiB. FlashLoop (arXiv:2609.29812, Yang and Liu, Tübingen) claims that most of what looping computes and stores is redundant: converged tokens stop changing, a stable slice of attention columns dominates the output, and adjacent-loop KV residuals shrink enough to quantize to 4 bits. The result: up to 1.64x end-to-end speedup and up to 6x KV-cache reduction, 'lossless accuracy'. This guide checks the math and the silicon: why decode is bandwidth-bound at arithmetic intensity ~0.4 FLOPs/byte against an A100 ridge of 201, what token-sparse updates and top-K column reconstruction actually compute (with a runnable toy), what a naive storage model of their INT4 base-plus-residual scheme predicts versus the paper's 6.06x (answer: 1.75x — the gap is the interesting part), and how far the 1.64x survives an Amdahl read of a real serving stack. The \"lossless\" claim is priced honestly: deltas run from +0.37 to -0.88 percentage points, and the thinking variants lose the most. ai machine-learning llm inference efficiency kv-cache looped-transformers https://flozi.net/en/guides/ai/gumbel-watermark-art50-compliance-asymmetry https://flozi.net/en/guides/ai/gumbel-watermark-art50-compliance-asymmetry Fri, 25 Sep 2026 00:00:00 GMT In August 2026, two implementations of the same 2022 idea crossed the finish line within one legal window: Anthropic shipped SynthID-Text-style watermarking in Claude under the EU Code of Practice (Nature 2024: H=4 sliding-window seed, M=2^m candidate knockout tournament over m=30 Bernoulli g-value layers), and vLLM shipped keyed Gumbel-max sampling (--watermark-config gumbel, Philox PRF, context_width=4) with a weights-free detector. This guide does what the announcements do not: it derives the Gumbel-max trick (argmax of log p plus iid Gumbel noise is an exact categorical sampler), shows in Python how a keyed PRF turns the noise into a detectable signal (z=+28 at N=200 fully marked, z=+1.15 at 5% marking), prices Anthropic's own limitations section as math (sparse choice = sparse signal; the null per-token gap is a clean -ln 32 in our runs), and anchors the whole thing in Art. 50(2) EU AI Act (machine-readable marking, applied since 2 August 2026, with the Omnibus 2026/1744 four-month transition ending 2 December 2026 for pre-existing systems). The thesis: Art. 50(2) requires a property (provable marking) that closed endpoints can deliver and gatekeep at their own discretion, while open serving can only deliver it forwards — a deployer can prove provenance over their own stream, but nobody can prove absence, and downstream fine-tunes erase the mark entirely. ai machine-learning llm watermarking security compliance regulation https://flozi.net/en/guides/ai/hbf-flash-kv-tier-hot-cold https://flozi.net/en/guides/ai/hbf-flash-kv-tier-hot-cold Fri, 25 Sep 2026 00:00:00 GMT High-Bandwidth Flash as a KV tier decomposed: why arXiv:2609.25782 gets 24x concurrent sessions and -7.6 kW/node from the same medium that arXiv:2608.11668 measures at 2-5.5x worse end-to-end latency. The deciding variable is placement policy: write-on-evict cold pools vs Mooncake-style SSD offload streams. Endurance, latency-budget, and power arithmetic recomputed in Python. ai machine-learning gpu gpu-memory inference hardware kv-cache storage https://flozi.net/en/guides/ai/kernelopt-agentic-kernel-search-fade https://flozi.net/en/guides/ai/kernelopt-agentic-kernel-search-fade Fri, 25 Sep 2026 00:00:00 GMT A September 2026 Red Hat paper (arXiv:2609.30059) sends five profiling-guided LLM agents at compiler-generated Triton kernels — preserving every cuBLAS and cuDNN call untouched — and gates the result behind a four-gate verification cascade that falls back to the torch.compile baseline whenever anything fails. The engineering is genuinely good: dispatch-aware targeting, NCU-guided beam search, model-level float64 verification, and a performance gate that converts correct-but-useless per-kernel wins into honest fallbacks. The result curve is the interesting part: 1.40× geometric mean over torch.compile on Level 1 (51 of 100 optimized), 1.15× on Level 2 (31 of 100), and 1.07× on Level 3 (12 of 50) — across all problems, unsolved ones counted at the preserved 1.0× baseline. The agentic speedup collapses as problems get harder, and 61 of 85 fallbacks are simply library dominance: most of a compiled model's time is spent inside vendor calls the method refuses to touch, an Amdahl ceiling this guide puts in numbers, along with the effective-geomean arithmetic of pass-rate-weighted speedups and why float64 model-level verification catches numerics that per-kernel tests cannot. ai machine-learning llm gpu kernels compilers https://flozi.net/en/guides/ai/krex-shared-gpu-benchmark-corruption https://flozi.net/en/guides/ai/krex-shared-gpu-benchmark-corruption Fri, 25 Sep 2026 00:00:00 GMT A September 2026 HKUST/Alibaba systems paper (arXiv:2609.30057) attacks a hidden coupling that the LLM kernel-agent wave created: agents need trustworthy GPU timing, and existing systems buy it by reserving an entire GPU per benchmarking command — wasteful, because the timed loop is only ~8.5% of a median command's duration. KREX narrows exclusivity to the timing phase: agents mark critical regions, the runtime blocks new GPU submissions, drains outstanding work, freezes sibling process trees via freezer cgroups, and pins the measuring threads to reserved CPU cores — only inside those regions. Result: up to 3.4× benchmarking throughput on NVIDIA H20 (2.6× on AMD MI308X) at p95 timing inflation of 0.30% / 1.58% / 3.90% for kernels above 10 ms / 1 ms / 0.1 ms. This guide puts the two arithmetic cores of the paper in numbers: how a biased comparator misdirects a hill-climbing agent (a wrong measurement is worse than a slow one), and why region-granular exclusivity is nearly free — including why short kernels structurally suffer more and where the 1/d amortization story only half holds. ai machine-learning llm gpu kernels benchmarking systems https://flozi.net/en/guides/ai/kv-cache-metering-billing-attribution https://flozi.net/en/guides/ai/kv-cache-metering-billing-attribution Fri, 25 Sep 2026 00:00:00 GMT arXiv:2609.24991 takes one H100 running vLLM with four tenants and shows the meter — token counts or GPU time-share — moves a retrieval-heavy tenant from 16.5% to 4.8% of the bill. We verify the 11.7–13.6-point gap at the primary, decompose the KV-residency cost of capital with Python, and check the synthetic-scope caveats the paper itself stamps on its seam numbers. ai machine-learning llm inference prompt-caching economics gpu-memory kubernetes https://flozi.net/en/guides/ai/kv-eviction-risk-contract https://flozi.net/en/guides/ai/kv-eviction-risk-contract Fri, 25 Sep 2026 00:00:00 GMT A September 2026 Korea University paper (arXiv:2609.27981) flips KV-cache eviction from budget-first to risk-first: you specify how often a request may materially degrade, and a Learn-then-Test calibration picks the retention level — or falls back to full KV when nothing certifies. This guide decomposes the finite-sample machinery, reproduces the certification cutoffs in Python, and prices the honest parts: calibration-population scope, the RULER-32K full-KV fallback, and the 5-10 percentage points of extra retention that the guarantee costs. ai machine-learning llm inference kv-cache eviction statistics reliability long-context https://flozi.net/en/guides/ai/kv-invariant-two-tower-scaling https://flozi.net/en/guides/ai/kv-invariant-two-tower-scaling Fri, 25 Sep 2026 00:00:00 GMT StepFun's KITE paper (arXiv:2609.27294) scales a 33.8B source into a 67B two-tower MoE whose prefill KV cost stays pinned to the small tower. We verify the 67B/2.15B-active numbers, the 1.5900 vs 1.6006/1.5921 loss ladder, and the 6.7%/31.6% inference proxy at the primary — then decompose the three hype-breakers the abstract does not carry: the tower-separation co-training risk, the loss-is-not-capability gap, and the decode FLOPs the KV invariant does not touch. ai machine-learning llm inference kv-cache architecture economics gpu-memory https://flozi.net/en/guides/ai/kv-tensor-decomposition-full-rank-modes https://flozi.net/en/guides/ai/kv-tensor-decomposition-full-rank-modes Fri, 25 Sep 2026 00:00:00 GMT Singular-value spectra of all four KV-cache tensor modes on Mistral-7B-v0.3 and Llama-2-13B: token and feature axes are low-rank, but head and layer axes are effectively full-rank — so every '2x KV compression for free' claim that mixes across heads or layers is fighting linear algebra, not implementation noise. Tucker beats CP/TT/t-SVD at matched storage 2x-5x because it can pin the full-rank modes, keys prefer 2D unfoldings while values prefer 4-way Tucker, and a mode-pinning theorem certifies all of it from measured spectra alone. With a runnable mixed-rank Python demo whose pinned-vs-forced reconstruction sweep reproduces the paper's error floor. ai machine-learning llm inference kv-cache tensor-decomposition linear-algebra gpu-memory https://flozi.net/en/guides/ai/kvset-working-set-prefix-cache-sizing https://flozi.net/en/guides/ai/kvset-working-set-prefix-cache-sizing Fri, 25 Sep 2026 00:00:00 GMT KVSET (arXiv:2609.27746) is not a new method — it is Mattson's 1970 LRU stack-distance algorithm pointed at KV pages, and it converts capacity-by-capacity cache simulation into one pass over the trace. This guide runs the algorithm end to end on a toy agentic trace, verifies the one-pass curve against naive multi-simulation, extracts the working set, and marks where the LRU-scope and trace-shape caveats bite. ai machine-learning llm inference kv-cache prefix-caching gpu-memory capacity-planning https://flozi.net/en/guides/ai/leaderboard-hidden-selection-noise https://flozi.net/en/guides/ai/leaderboard-hidden-selection-noise Fri, 25 Sep 2026 00:00:00 GMT Not "leaderboards are useless." arXiv:2609.28177 decomposed: the Gaussian margin model behind hidden-selection sensitivity curves, why the required margin grows from z=1.645 toward 2.7 as k grows at rho_w=0.56, why the correlation must match the ranked score (0.90 pooled vs 0.46 item-resampled vs 0.92 subject-resampled), and the 394-claim audit that finds 391 unsupported before selection even enters. Simulation included. ai machine-learning llm evaluation leaderboards https://flozi.net/en/guides/ai/precision-variant-greedy-decode-flips https://flozi.net/en/guides/ai/precision-variant-greedy-decode-flips Fri, 25 Sep 2026 00:00:00 GMT Two September 2026 papers dismantle the assumption that greedy decoding is deterministic. The first (arXiv:2609.26621, TMLR 2026) shows the same model, prompt, and greedy algorithm produce different outputs in BF16 versus FP16 on identical hardware — 49-100% of prompts diverge across six 1.1B-7B models — and traces the flip to a single low top-two logit margin at the lm_head, not to 22 layers of accumulated body error. A margin-gated FP32 lm_head recomputation buys back 22-36pp of exact agreement at under 4% latency, and, counterintuitively, broader FP32 compute makes agreement worse. The second (arXiv:2609.25624) attacks the cross-device axis: frameworks pick different GEMM kernels per architecture, different reduction orders flip tokens, and the fix is fixed-configuration fused-upcast kernels whose reduction order is a pure function of problem shape — bitwise-identical linear outputs across Ampere, Ada, and Hopper, 1.17-3.1x faster than the FP32-compute state of the art. This guide runs the underlying float non-associativity, the flipping argmax, and the margin criterion in runnable Python with real outputs, and maps both papers' honest limits. ai machine-learning llm inference numerics