LLM VRAM Requirements: A Mathematical Deep Dive

Deploying Large Language Models (LLMs) requires careful consideration of GPU memory requirements. This guide breaks down the mathematical formulas used to calculate VRAM consumption for both inference and training, using the Qwen3-VL-32B-Instruct model as a practical example.

Why VRAM Calculation Matters

Before deploying an LLM, you need to answer critical questions:

• How much GPU memory will my model consume?
• Can I run this model on my current hardware?
• Which quantization method provides the best memory-performance tradeoff?
• How many concurrent users can I support?

This guide provides the mathematical foundation to answer these questions accurately.

Example Model: Qwen3-VL-32B-Instruct

We'll use Qwen3-VL-32B-Instruct as our reference model throughout this guide. This multimodal model combines vision and language capabilities with the following architecture:

Configuration Source: The model configuration is extracted from the text_config section of the model's config.json file on Hugging Face Hub.

Core Memory Components

VRAM consumption for LLMs consists of four primary components:

1. Model Weights Memory

The base memory required to store the model's parameters.

Model Weights (bytes) = Number of Parameters × Bytes per Parameter

Bytes per Parameter depends on the data type (quantization level):

Example Calculation for Qwen3-VL-32B:

Number of Parameters: 33,357,390,064 (~33.4B)

float32:  33,357,390,064 × 4.0   = 133,429,560,256 bytes = 133.43 GB
float16:  33,357,390,064 × 2.0   = 66,714,780,128 bytes  = 66.71 GB
int8:     33,357,390,064 × 1.0   = 33,357,390,064 bytes  = 33.36 GB
int4:     33,357,390,064 × 0.5   = 16,678,695,032 bytes  = 16.68 GB

2. KV Cache Memory

The Key-Value cache stores intermediate attention states for efficient autoregressive generation. This is the most significant dynamic memory component during inference.

KV Cache (bytes) = 2 × Batch Size × Sequence Length × Num Layers × Num KV Heads × Head Dimension × KV Data Type Size

Breaking Down the Formula:

• 2×: Separate storage for Keys and Values
• Batch Size: Number of concurrent requests
• Sequence Length: Maximum context length (input + output)
• Num Layers: Number of transformer blocks
• Num KV Heads: Number of key-value heads (8 for GQA in Qwen3-VL)
• Head Dimension: Size of each attention head (128)
• KV Data Type Size: Bytes per value (typically 2 for float16)

Example Calculation for Qwen3-VL-32B:

Scenario: 1 user, 8,192 token context, float16 KV cache

Batch Size: 1
Sequence Length: 8,192 tokens
Num Layers: 64
Num KV Heads: 8 (Grouped Query Attention)
Head Dimension: 128
KV Data Type: float16 (2 bytes)

KV Cache = 2 × 1 × 8,192 × 64 × 8 × 128 × 2
         = 2 × 1 × 8,192 × 64 × 8 × 256
         = 2 × 1,073,741,824 bytes
         = 2,147,483,648 bytes
         = 2.15 GB (decimal; = 2.00 GiB binary)

Scaling with Batch Size:

Scaling with Sequence Length:

3. Activation Memory

Memory required for intermediate computations during forward passes.

PyTorch Activation Memory = Batch Size × Sequence Length × (18 × Hidden Size + 4 × Intermediate Size)

Example Calculation for Qwen3-VL-32B:

Scenario: 1 user, 8,192 token context

Batch Size: 1
Sequence Length: 8,192
Hidden Size: 5,120
Intermediate Size: 25,600

Activation Memory = 1 × 8,192 × (18 × 5,120 + 4 × 25,600)
                  = 8,192 × (92,160 + 102,400)
                  = 8,192 × 194,560
                  = 1,593,835,520 bytes
                  = 1.59 GB (base value)

Data Type Multipliers:

Different quantization levels have different activation memory footprints:

4. Non-PyTorch Memory Overhead

System-level memory overhead for CUDA context, cuBLAS, and other framework components.

Non-PyTorch Memory ≈ 1.00 GB (constant)

This is a fixed overhead independent of model size or batch configuration. The interactive calculator models it slightly differently: 1.2 GB scaled by the quantization format's overhead factor (see below), i.e. ~1.26 GB for int8.

Complete Inference Memory Formula

Combining all components, the total VRAM required for inference:

Total Inference VRAM = (Model Weights + KV Cache + Non-PyTorch Memory + Activations) / GPU Utilization

GPU Utilization Factor: Typically set to 0.9 (90%) to provide safety margin for memory fragmentation and unexpected spikes.

Note (matching the interactive calculator): The worked examples on this page use this hand-planning convention. The calculator at the top of the tool page instead (a) multiplies weight bytes by a per-format overhead factor for scale/zero-point storage — fp8 ×1.03, int8 ×1.05, NVFP4 ×1.1, int4/AWQ ×1.12 — (b) models the non-PyTorch overhead as 1.2 GB × that factor, and (c) does not divide by a utilization factor (its "Hardware Utilization" slider only derates bandwidth/throughput, never capacity). Expect the tool's int8/8k/batch-1 figure (~39 GB) to differ modestly from the hand calculation (~42 GB); both are conservative estimates, not guarantees.

Example: Qwen3-VL-32B Inference (int8 quantization)

Configuration:

• Quantization: int8 (1 byte per parameter)
• Batch Size: 1 user
• Sequence Length: 8,192 tokens
• KV Cache Data Type: float16
• GPU Utilization: 0.9

Step-by-Step Calculation:

1. Model Weights:
   33,357,390,064 × 1 byte = 33,357,390,064 bytes = 33.36 GB

2. KV Cache (float16):
   2 × 1 × 8,192 × 64 × 8 × 128 × 2 = 2,147,483,648 bytes = 2.15 GB

3. Activations (int8 uses 1.0× multiplier):
   1 × 8,192 × (18 × 5,120 + 4 × 25,600) × 1.0 = 1,593,835,520 bytes = 1.59 GB

4. Non-PyTorch Memory:
   ≈ 1.00 GB

5. Total (before GPU utilization adjustment):
   33.36 + 2.15 + 1.59 + 1.00 = 38.10 GB

6. Adjusted for GPU Utilization (90%):
   38.10 / 0.9 = 42.33 GB

Result: You need approximately 42 GB of VRAM to run Qwen3-VL-32B in int8 quantization with 8k context for a single user.

Example: Qwen3-VL-32B Inference (int4 quantization)

Configuration:

• Quantization: int4 (0.5 bytes per parameter)
• Batch Size: 4 users
• Sequence Length: 8,192 tokens
• KV Cache Data Type: float16
• GPU Utilization: 0.9

Step-by-Step Calculation:

1. Model Weights:
   33,357,390,064 × 0.5 bytes = 16,678,695,032 bytes = 16.68 GB

2. KV Cache (float16, batch=4):
   2 × 4 × 8,192 × 64 × 8 × 128 × 2 = 8,589,934,592 bytes = 8.59 GB

3. Activations (int4 uses 1.0× multiplier, batch=4):
   4 × 8,192 × (18 × 5,120 + 4 × 25,600) × 1.0 = 6,375,342,080 bytes = 6.38 GB

4. Non-PyTorch Memory:
   ≈ 1.00 GB

5. Total (before GPU utilization adjustment):
   16.68 + 8.59 + 6.38 + 1.00 = 32.65 GB

6. Adjusted for GPU Utilization (90%):
   32.65 / 0.9 = 36.28 GB

Result: You need approximately 36 GB of VRAM to run Qwen3-VL-32B in int4 quantization with 8k context for 4 concurrent users.

Training Memory Requirements

Training requires significantly more memory than inference due to:

• Gradients: Same size as model weights
• Optimizer States: 2× model weights (for Adam optimizer)
• Larger Activation Memory: ~1.5× inference activations
• Training Overhead: 30% safety buffer

Note (assumptions): The training memory calculations in this section assume a standard in‑GPU training setup and do NOT apply memory-saving or bandwidth-optimizing techniques such as CPU/NVMe offloading, sliding-window or chunked attention, specialized low‑memory optimizers or ZeRO-style partitioning, aggressive gradient checkpointing beyond the basic multiplier used above, or custom attention kernels that change KV storage. If you plan to use any of these methods, expected VRAM requirements can be materially lower and should be recalculated for your specific setup.

Breakdown:

• 3× Model Weights: Weights + Gradients + Optimizer States (Adam)
• 1.3×: Training overhead factor (30% buffer)

Example: Qwen3-VL-32B Training (float16)

Configuration:

• Quantization: float16 (2 bytes per parameter)
• Batch Size: 1
• Sequence Length: 8,192 tokens
• GPU Utilization: 0.9

Step-by-Step Calculation:

1. Model Weights (×3 for training):
   33,357,390,064 × 2 bytes × 3 = 200,144,340,384 bytes = 200.14 GB

2. KV Cache (float16):
   2 × 1 × 8,192 × 64 × 8 × 128 × 2 = 2,147,483,648 bytes = 2.15 GB

3. Activations (float16 uses 1.0× multiplier, ×1.5 for training):
   1 × 8,192 × (18 × 5,120 + 4 × 25,600) × 1.0 × 1.5 = 2,390,753,280 bytes = 2.39 GB

4. Non-PyTorch Memory:
   ≈ 1.00 GB

5. Subtotal:
   200.14 + 2.15 + 2.39 + 1.00 = 205.68 GB

6. Apply Training Overhead (×1.3):
   205.68 × 1.3 = 267.38 GB

7. Adjusted for GPU Utilization (90%):
   267.38 / 0.9 = 297.09 GB

Result: You need approximately 297 GB of VRAM to train Qwen3-VL-32B in float16 with 8k context and batch size 1.

GPU Memory Comparison Table

Here's a comprehensive comparison for Qwen3-VL-32B across different quantization methods:

Configuration: Batch size 1, Sequence length 8,192 tokens, GPU utilization 0.9

Practical GPU Recommendations

Based on the calculations above, here are suitable GPU configurations for Qwen3-VL-32B:

Inference Deployment

Training Deployment

Key Takeaways

1. Model Weights Scale Linearly: Doubling parameters doubles weight memory
2. KV Cache Scales with Context: KV cache memory grows linearly with context length
3. Batch Size Multiplies KV Cache: Each concurrent user adds KV cache overhead
4. Quantization Dramatically Reduces Memory: int4 uses ~1/8th the memory of float32
5. Training Needs ~3-4× Inference Memory: Due to gradients and optimizer states
6. GPU Utilization Buffer is Critical: Always reserve 10-20% safety margin

Advanced Considerations

Grouped Query Attention (GQA) Impact

Qwen3-VL-32B uses Grouped Query Attention with 8 KV heads instead of 64 query heads. This reduces KV cache memory by 8× compared to standard Multi-Head Attention:

Standard MHA: 64 KV heads → KV Cache = 17.18 GB (8k context, float16)
GQA (Qwen3): 8 KV heads → KV Cache = 2.15 GB (8k context, float16)

Memory Savings: 15.03 GB (87.5% reduction)

Long Context Scaling

Qwen3-VL-32B supports up to 262k tokens. Here's how KV cache scales:

KV Cache Quantization

Using fp8 for KV cache can halve memory usage:

float16 KV Cache (8k, batch=1): 2.15 GB
fp8 KV Cache (8k, batch=1): 1.07 GB

Savings: 50% memory reduction with minimal quality loss

Conclusion

Accurate VRAM calculation is essential for efficient LLM deployment. By understanding the mathematical foundations:

1. Choose the right quantization for your quality-memory tradeoff
2. Plan for KV cache scaling with concurrent users and context length
3. Budget for training overhead if fine-tuning
4. Select appropriate GPUs based on actual requirements
5. Always include safety margins to prevent OOM errors

Use these formulas to estimate memory requirements for any Hugging Face model by extracting configuration parameters and applying the calculations shown in this guide.

Related

• LLM Inference Math: From Theory to Hardware — the compute side of the same roofline.
• LLM Inference Calculator & Break-Even Analyzer — run the numbers for your model and card.

References and Tools

• Model Configuration: Qwen3-VL-32B-Instruct config.json
• Hugging Face Hub: Model metadata and parameter counts available via API
• GPU Specifications: Check manufacturer specifications for exact VRAM capacities
• Quantization Methods: GPTQ, AWQ, GGUF for different precision levels