LLM Inference Math: From Theory to Hardware

Large Language Models (LLMs) have become foundational in modern AI, but running them efficiently requires a deep understanding of the interplay between model architecture and hardware capabilities. Simply choosing the most powerful GPU isn't always the most cost-effective solution. The key is to know whether your workload is compute-bound or memory-bound.

This guide walks through the essential math to profile an LLM for inference, helping you select the right hardware and optimize its performance. We will apply these principles to a real-world example: running the Qwen/Qwen3-VL-32B-Instruct model on the powerful NVIDIA RTX PRO 6000 Blackwell Edition workstation GPU.

This article is inspired by the mathematical approach detailed in the Baseten blog post, "A guide to LLM inference and performance."

Step 1: Understanding Your Hardware's Capabilities

The first step is to analyze the key specifications of our GPU. These numbers define the theoretical limits of our hardware. For the NVIDIA RTX PRO 6000 Blackwell Edition, the critical specs are:

• GPU Memory (VRAM): 96 GB GDDR7 ECC ¹
• GPU Memory Bandwidth: 1792 GB/s ¹
• FP16/BF16 Tensor Core Performance (dense): 503.8 TFLOPS ²

These three metrics—capacity, speed, and raw power—are the pillars of our analysis.

Step 2: Calculating the GPU's Operational Intensity (Ops:Byte Ratio)

A GPU's operational intensity, or ops:byte ratio, tells us how many computations it can perform for every byte of data it moves from VRAM. This is a crucial, hardware-specific ratio that reveals the balance between computation and memory access.

The formula is straightforward:

ops:byte Ratio = Compute Bandwidth (FLOPS) / Memory Bandwidth (Bytes/s)

Let's calculate it for our RTX PRO 6000:

• Compute: 503.8 TFLOPS = 503,800,000,000,000 FLOPS
• Memory: 1792 GB/s = 1,792,000,000,000 Bytes/s

ops_to_byte_ratio = 503,800,000,000,000 / 1,792,000,000,000
                  = 281.1 ops/byte

This means for our hardware to be fully utilized, our application must perform approximately 281.1 floating-point operations for every single byte it fetches from VRAM.

• If our model performs fewer operations per byte, we are memory-bound.
• If our model requires more operations per byte, we are compute-bound.

Step 3: Calculating the Model's Arithmetic Intensity

Next, we need to calculate the arithmetic intensity of our model. For Transformers, attention is where prefill compute concentrates: processing a prompt means one pass in which every query is matched against every earlier key.

We'll use the parameters for the Qwen/Qwen3-VL-32B-Instruct model ³:

• Sequence Length (N): 4096
• Model Dimension (d_model): 5120
• Number of Attention Heads (n_heads): 64
• Number of KV Heads (n_kv_heads): 8 (GQA)
• Dimension per Head (d_head): 128

Let's calculate the arithmetic intensity using the simplified roofline model approach:

Arithmetic Intensity = (4 * N^2 * d_head) / (8 * N^2)
                     = d_head / 2
                     = 128 / 2
                     = 64.0 ops/byte

Our model's prefill arithmetic intensity is approximately 64.0 operations per byte. This is a prefill-phase quantity — at batch-1 decode the intensity collapses to ≈1 FLOP/byte (2 FLOPs per 2-byte weight read), yet decode is more memory-bound, not less: every active weight byte must be streamed once per generated token. The full roofline form (Baseten's +3N²-op numerator and +8N·d-byte denominator) gives ≈62.4 ops/byte at N=4096 — the simplified d_head/2 overstates it by ~3%.

Step 4: Identifying the Bottleneck

Now we compare the two ratios:

• GPU Ops:Byte Ratio: 281.1 ops/byte
• Model Arithmetic Intensity: 64.0 ops/byte

Since 64.0 < 281.1, our workload is overwhelmingly memory-bound. This is common in LLM inference and means memory bandwidth is the primary limiting factor for inference speed.

Step 5: VRAM and Performance Estimation

VRAM for Model Weights

A ~32.8-billion-parameter model (the Qwen3-32B text stack; the VL variant adds a vision tower for ~33.4B total) at half-precision (FP16) requires:

VRAM for Weights = 32.76 Billion Parameters * 2 Bytes/Parameter = 65.5 GB

VRAM for the KV Cache

The KV cache size per token is:

KV Cache per Token = 2 * num_hidden_layers * n_kv_heads * d_head * 2 = 262,144 Bytes/token (Qwen3-VL-32B uses Grouped-Query Attention with 8 KV heads instead of 64, so the KV cache is 8× smaller than a full multi-head layout would need.)

With our 96 GB GPU, after loading the ~65.5 GB model, we have:

Spare VRAM = 96 GB - 65.5 GB = 30.5 GB

This allows for a theoretical batch size of:

Batch Size = Spare VRAM / (KV Cache per Token * Sequence Length) ≈ 28 sequences

Estimating Performance

Time Per Output Token (Decoding Latency):

Time/Token = Model Size (Bytes) / Memory Bandwidth (Bytes/s) ≈ 36.6 ms/token

This translates to a theoretical throughput of ~27 tokens/second.

Time to First Token (Prefill Latency): For a prompt of 512 tokens:

Prefill Time = (512 tokens * 2 * 32.76e9 params) / 503.8 TFLOPS = 66.6 ms

Conclusion

Our analysis shows:

1. Inference is Memory-Bound: The primary bottleneck is the 1792 GB/s memory bandwidth, not the 503.8 TFLOPS of tensor compute.
2. VRAM for Batching: The 96 GB of VRAM is a significant advantage, allowing a theoretical batch size of ~28 concurrent 4k-token sequences (≈30.5 GB spare ÷ ~1.07 GB KV per sequence) to better utilize the GPU's compute.
3. Performance Expectations: We can expect a theoretical throughput of around 27 tokens/second and a prefill time of approximately 66.6 ms for a 512-token prompt.

These calculations provide a solid foundation for understanding LLM inference performance and making informed hardware decisions.

Related

• LLM VRAM Requirements: A Mathematical Deep Dive — the memory axis of the roofline.
• Tensor Parallelism: Slicing the Silicon — splitting one model across GPUs when a single card is not enough.

References