Skip to content
Friendly disclaimer: flozi00 TechHub is a solo side-project next to a full-time job — personal learning notes, no official statements. Verify critical steps yourself.
flozi00 Logoflozi00 TechHub
ENDE

How we serve GLM on B300 in production: MTP k=5 tuning

Production notes from serving a GLM-class MoE on 8x B300: the acceptance-rate math behind speculative decoding, why the measured log_stats chain says k=2-k=3 captures most of k=5, and why NVLS failures are silent by default.

6 min readflozi00
llm-inferencespeculative-decodingmtpb300ncclproduction

Multi-token prediction in production is an acceptance-economics problem. This post is our tuning notes from serving a GLM-class MoE on 8x B300 (Blackwell Ultra, sm_103)1: what speculative depth buys once the compounding acceptance math is applied - measured on our own node - and why NVLink-SHARP (NVLS) failures stay silent by default, plus how to make them visible.

The setup

GLM MTP head ships inside the checkpoint - one extra prediction layer, not a separate draft-model download. Z-AI's model card for GLM-5.3 (744B/40B-active sparse-attention MoE GlmMoeDsaForCausalLM, 1M context, one MTP layer) documents the native MTP head2; on our node it is physically there as separate mtp_bf16-*.safetensors files next to the trunk weights. vLLM (speculative_config={"method": "mtp", "num_speculative_tokens": k})3 and SGLang (--speculative-algorithm NEXTN)4 consume it directly: the vLLM recipe for GLM-5.3 recommends exactly the flagship setup this node runs - TP=8, MTP k=5 with --kv-cache-dtype fp8_e4m3, plus the glmb300 variant's --served-model-name glmb300 mapping; Blackwell nodes use the NVFP4 expert-quantized checkpoint (only the routed experts are NVFP4; attention, shared experts and dense layers stay BF16)5.

The acceptance math that predicts your speedup

Per-position acceptance rates are the only numbers that matter. Expected tokens per decoding step is not k plus one - it is driven by the chain probability of joint acceptance (the framework is speculative decoding as introduced by Leviathan, Kalman and Matias: draft tokens are verified against the target model, and accepted tokens are resampled under the target distribution)6:

E[tokens per step]=1+∑i=1k∏j=1iaj\mathbb{E}[\text{tokens per step}] = 1 + \sum_{i=1}^{k} \prod_{j=1}^{i} a_j

vLLM's periodic log_stats lines give you exactly these numbers under production load: Mean acceptance length, accepted-vs-drafted token counts, and Per-position acceptance rate for every draft position. On our B300 node (GLM-5.3-class MoE, TP=8 across 8x B300, MTP k=5, fp8 KV cache, vLLM v0.30.0) the measured chain looks like this:

kmeasured chain (B300 node)E[accepted]tokens/stepshare of k=5
10.720.721.7248%
20.72, 0.581.302.3064%
30.72, 0.58, 0.481.782.7877%
40.72, 0.58, 0.48, 0.442.223.2289%
50.72, 0.58, 0.48, 0.44, 0.402.623.62100%

Chains compound brutally: k=2 captures ~64% of the k=5 ceiling on our measured chain, k=3 ~77%. But the share you capture is a function of the per-position rates your workload actually produces: on the same checkpoint served on 8x RTX PRO 6000 Blackwell Server Edition as one of our production pools, the measured chain falls off more steeply (0.77, 0.56, 0.40, 0.30, 0.23 over a 22-hour, 663-window scrape) - the k=2 share rises to ~71% while k=5 only reaches a mean acceptance length of 3.3. Same model, same k - a steeper chain makes deep speculation pay even less, a flatter one pays k=4 and k=5 back. Public measurements show the same spread across hardware and workloads for GLM MTP models, and the vLLM recipe for the smaller GLM-4.5 still recommends num_speculative_tokens 1 for optimal throughput on that model7; the GLM-5.3 recipe ships k=5 as the flagship default5.

Deeper speculation buys marginal tokens while every draft position adds cost:

  • Wasted verify work: a rejected draft still consumed a verify slot.
  • Rejection latency: the corrected token must propagate before the next draft starts - deeper chains stall longer per rejection.
  • KV-cache pressure: num_speculative_tokens reserves KV entries for draft positions per request; vLLM docs recommend starting small3.

Depth is a per-workload decision driven by measured acceptance - public benchmarks find acceptance varies wildly across datasets, requests, and positions7. Measure first, then set k - the log_stats lines named above are the measurement.

The quiet failure mode: NVLS on sm_103

The second topic is a documented failure-mode property of the fabric path: when NVLS initialization fails, NCCL's default is to keep running without it - no error, no crash, just a slower collective.

The mechanism, from the documentation:

  • NVLS is NVLink SHARP: in-network reduction offloaded into the NVSwitch domain; NCCL supports it since 2.17 on third-gen NVSwitch with Hopper or later8.
  • Its default is silent: NCCL_NVLS_ENABLE defaults to 2 (auto-detect). If NVLink SHARP resources cannot be allocated, NCCL falls back to a slower algorithm without an error; NCCL 2.27.3 added graceful fallback, with NCCL_NVLS_ENABLE=1 preserving the old loud-failure behavior9.
  • The silence has bitten at scale - on NVIDIA's own tracker: the graceful fallback caused a hang in a large-scale training job because not all ranks saw the same NVLS state - some thought NVLS was active while others had silently fallen back. The NCCL team wrote that they spent substantial debugging time on exactly that inconsistency and reversed course: newer NCCL raises a hard error when NVLS is enabled but broken, and NCCL_NVLS_ENABLE=0 is the explicit opt-out. A documented trigger is NVSwitch multicast-slot exhaustion: NVSwitch hardware caps multicast at 128 slots, allocated per sub-communicator and freed only when the communicator is destroyed10.
  • The stack below NCCL is version-locked: Fabric Manager checks the loaded kernel driver stack at initialization and aborts on mismatch; HGX B200/B300 nodes (fourth-gen NVSwitch) additionally require the NVLSM service, from the driver-branch-matched nvidia-open and nvlink5 packages11. The fabric stack moves as a set by NVIDIA's own packaging.

The working rules that follow from the documentation:

  1. Pin driver + fabric-manager + NCCL as a set - FM is version-locked to the driver by design11.
  2. Probe with a tiny all-reduce benchmark before blaming the serving stack.
  3. NCCL_DEBUG=INFO names the algorithm that actually ran - trust the measured probe over assumed topology.

Transferable rule: on new silicon, verify the fast path is active - silence is a documented failure mode of the fabric.

Notes on output fidelity

Speculative decoding with verifier acceptance changes throughput, not the output distribution: accepted tokens are exactly the tokens the trunk model would have produced. The original speculative-decoding result shows the output distribution is unchanged6 - for MTP heads the same as for separate EAGLE draft models.

  • GLM-5.3 model card: 744B/40B-active GlmMoeDsaForCausalLM, sparse attention (DSA), 1M context, one MTP layer; NVFP4 checkpoint for Blackwell: https://z.ai/blog/glm-5.3

Footnotes

  1. NVIDIA CUDA GPU compute capability table - B300 (Blackwell Ultra) is compute capability 10.3 (sm_103); SM120 is the separate consumer/workstation Blackwell family (RTX 50-series, 12.0): https://developer.nvidia.com/cuda/gpus ↩

  2. Z-AI, "GLM-5.3: Frontier Coding with Emergent Cyber Capabilities" ↩

  3. vLLM MTP documentation (speculative_config, num_speculative_tokens): https://docs.vllm.ai/en/latest/features/speculative_decoding/mtp/ ↩ ↩2

  4. SGLang speculative decoding parameter reference - --speculative-algorithm NEXTN (an alias of EAGLE): https://docs.sglang.io/docs/advanced_features/speculative_decoding ↩

  5. vLLM Recipes, zai-org/GLM-5.3 - flagship FP8-B200 recipe: TP=8, MTP k=5, fp8_e4m3 KV; NVFP4 Blackwell variant (experts-only NVFP4): https://recipes.vllm.ai/zai-org/GLM-5.3 ↩ ↩2

  6. Yaniv Leviathan, Matan Kalman, Yossi Matias, "Fast Inference from Transformers via Speculative Decoding" (arXiv:2211.17192, ICML 2023) - speculative-decoding framework; verification leaves the output distribution unchanged: https://arxiv.org/abs/2211.17192 ↩ ↩2

  7. Xiaoxuan Liu, Jiaxiang Yu, Jongseok Park, Ion Stoica, Alvin Cheung, "Speculative Decoding: Performance or Illusion?" (arXiv:2601.11580) - MTP on GLM-4.5-Air-106B: k=3 speedup 1.3-1.8x on H100; acceptance varies wildly across positions, requests, and datasets; vLLM Recipe for zai-org/GLM-4.5 - num_speculative_tokens 1 recommended for optimal throughput on GLM-4.5: https://specdecode-bench.github.io/ https://recipes.vllm.ai/zai-org/GLM-4.5 ↩ ↩2

  8. NVIDIA NCCL user guide - NCCL_NVLS_ENABLE (default 2, auto-detect): https://docs.nvidia.com/deeplearning/nccl/archives/nccl_2283/user-guide/docs/env.html ↩

  9. NCCL 2.27.3 release notes - graceful NVLS fallback: https://docs.nvidia.com/deeplearning/nccl/release-notes/rel_2-27-3.html ↩

  10. NCCL issue #2077 (NVIDIA/nccl tracker) - 128 NVSwitch multicast slots; NCCL team comment: graceful fallback caused a large-scale hang via rank-inconsistent NVLS state, hence the hard error when NVLS is enabled but broken; NCCL_NVLS_ENABLE=0 as opt-out: https://github.com/NVIDIA/nccl/issues/2077 ↩

  11. NVIDIA Fabric Manager user guide - FM checks the loaded kernel driver stack at initialization and aborts on mismatch; HGX B200/B300 (fourth-gen NVSwitch) requires the NVLSM service from nvidia-open/nvlink5 packages: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html ↩ ↩2