Two interactive tools for planning LLM serving hardware. The planner estimates VRAM fit, decode throughput, and time-to-first-token for a concrete model + GPU + precision + batch configuration. The break-even analyzer answers the deeper capacity question: at which batch size does decode stop being bandwidth-limited, at which point is it compute-limited, and what is the maximum sequence length you can serve at that batch.
All numbers are theoretical ceilings derived from vendor specs and roofline math — real deployments typically land at 60–80% of these values. The inference math guide explains the underlying formulas.
1. The planner: capacity, throughput, latency
Interactive Estimator
LLM inference planner beta
Total parameters drive VRAM capacity; active parameters drive decode speed (MoE only fetches routed experts per token). MLA and hybrid-attention models use compressed KV-cache math.
Capacity check (Single GPU)
- Model weights (total)
- 65.52 GB
- Active weights per token
- 65.52 GB
- KV cache per token
- 0.26 MB
- Total KV cache (5,120 tok × 1)
- 1.34 GB
- Activations
- 0.8 GB
- Runtime overhead
- 1.2 GB
- Total VRAM needed
- 68.86 GB
- Available VRAM
- 96 GB
- Headroom
- 27.14 GB
- Max batch @ current context
- ~21
- Max context @ batch 1
- ~108,639 tok
Roofline alignment
- GPU ops:byte (BF16 (2 B))279.02 ops/byte
- Decode intensity (batch 1)1 ops/byte
- Effective intensity (batch 1)1 ops/byte
- Gap278.02 ops/byte
Memory-bound: raising batch size lifts effective intensity because more tokens share each active-weight fetch.
Latency snapshot
- Decode throughput
- 20.51 tok/s
- Time per token
- 48.75 ms
- Throughput limit
- Memory
- Time to first token (4,096 tok)
- 809.52 ms
- Total request time
- 50.73 s
- Memory-limited ceiling
- 20.51 tok/s
- Compute-limited ceiling
- 5,723.09 tok/s
Decode ≈ min(aggregate bandwidth ÷ per-step bytes, FLOPS(BF16) ÷ 2·active-params, per-layer sync latency). Sync model: 0 sync(s)/layer × PCIe Gen5 x16 latency, scaled by kernel efficiency — TP+EP on PCIe fabrics is sync-bound, not bandwidth-bound.TTFT = max(total weight stream, linear + quadratic attention FLOPs).MoE capacity still needs all 32.762B params in VRAM.
2. Break-even: when does batching stop helping?
At batch size 1, decode is bandwidth-bound: generating one token means streaming every active parameter from HBM once, so tokens/s ≈ memory bandwidth ÷ active weight bytes. Whether extra requests ride along for free depends on the architecture: a dense model already streams its full weight set once per step, so each new request rides on the same reads — aggregate throughput climbs almost linearly while per-request speed stays roughly constant. An MoE model reads more weights per step as the batch grows (each request touches new experts), so its aggregate ceiling stays flat at first and per-request speed drops — until expert coverage saturates (point 2 below).
This cannot continue forever. Two things end the free lunch:
- Compute crossover — the total FLOPs the batch needs per step exceed what the GPU's tensor cores can deliver in one memory round. Beyond that batch size, the GPU is compute-bound and per-request speed degrades. This is what defines B*.
- MoE expert coverage — below the saturation batch (≈ total ÷ active parameters), every additional request touches fresh experts, so the aggregate ceiling stays flat and per-request speed drops from the start. Past it, all routed experts are already read once per step and the aggregate climbs almost linearly again — that is where MoE batching starts paying off.
The break-even batch B* is the batch where the bandwidth ceiling meets the flat compute ceiling — the transition from memory-bound to compute-bound. Below it, batching is free (for MoE: once past the saturation batch); at and above it, every extra request dilutes everyone's speed. Run below B* for maximum interactive responsiveness; run above it only for raw aggregate throughput. For dense models B* ≈ GPU FLOPS × bytes-per-parameter ÷ (2 × bandwidth) — at BF16 that equals the GPU's ops:byte ratio, a few hundred on modern cards, frequently more than VRAM can actually hold; MoE models multiply that value by total ÷ active parameters, landing in the thousands.
Break-even Analyzer
Decode break-even calculator beta
At low batch, decode is bandwidth-bound: every request re-reads the active weights, so tokens/s scales with the batch. Past the break-even batch B* amortize weight reads no further (MoE expert coverage is complete, or the compute roofline is hit) — per-request speed saturates and then only degrades. This panel finds B* and the largest context that fits there.
Break-even analysis
B* ≈ 280 needs more KV memory than this setup holds (VRAM fits ~53 sequences at 2,048 tokens). The setup never reaches the compute-bound regime — batch as high as VRAM allows and per-request speed keeps rising toward the B* value..
Decode throughput vs batch size
Aggregate tok/s = min(bandwidth ceiling, compute ceiling). The bandwidth ceiling stays flat while MoE expert coverage is incomplete (each request adds its own expert reads), then grows linearly once every routed expert is read once per step. B* is the first batch where the bandwidth ceiling reaches the flat compute ceiling — beyond it, decode is compute-bound and per-request speed degrades. The VRAM cap line shows where the batch stops fitting at a 2,048-token context.
How the break-even is computed
Both calculators share the same decode model:
- Bandwidth ceiling (aggregate tok/s) = batch × effective bandwidth × GPUs ÷ per-step bytes, where per-step bytes are
min(activeBytes × batch, totalWeightBytes)— pre-saturation the batch cancels, so the aggregate ceiling stays flat, as described above. - Compute ceiling (aggregate tok/s) = effective FLOPS × GPUs ÷ (2 × active parameters) — batch-independent, because the weights are read once per step regardless of batch.
- B* = the batch where the two ceilings intersect, i.e. where the workload transitions from memory-bound to compute-bound. Dense: ≈ FLOPS × bytes-per-param ÷ (2 × bandwidth) (= ops:byte at BF16), GPU-determined (a few hundred). MoE: that value multiplied by total ÷ active parameters (thousands).
- Max sequence length at B* = (total VRAM − weights − runtime overhead) ÷ (KV bytes per token × B*), capped at the model's context window.
The analyzer also reports the VRAM-feasible maximum batch at a ~2k-token context. When B* is larger than that, the setup never actually reaches the compute-bound regime — batch as high as VRAM allows and per-request speed keeps improving toward the B* ceiling.
Related
- LLM Inference Economics: Costs & Cloud Break-Even — what the break-even curve costs in dollars.
- LLM VRAM Requirements: A Mathematical Deep Dive — the memory model behind the fit column.
Limitations
- No continuous-batching arrival dynamics: B* is a steady-state ceiling, not a scheduling policy.
- Attention decode FLOPs (growing with context length) are not in the compute ceiling; at very long contexts the real crossover arrives earlier.
- Multi-GPU assumes tensor parallelism with perfect bandwidth scaling; PCIe fabrics add sync latency the planner models but the curve does not.