16 GB VRAM residency budgeting
Parent: Blackwell sm_120 local-LLM inference stack on Linux: llama.cpp, Ollama, PyTorch and vLLM over a Thunderbolt eGPU · Published reference · snapshot 2026-09-30
Also known as: 16 GB VRAM residency budgeting (weights + KV cache)
↓ Facts as markdownall context files
Source-anchored research for 16 GB VRAM residency budgeting. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- For Qwen2-14B with 48 layers, 8 KV heads, and 128 head dimension at FP16, KV cache consumes exactly 196.6 KB per token, or 3.14 GB for a 16,384-token context window. (high confidence) [source]
Measurements and reference values
- On an RTX 5080 with 16 GB GDDR7 VRAM, a 14B Q4 model reserves ~7.62 GB for weights, leaving ~8.38 GB for KV cache, which comfortably accommodates 16,384 tokens with standard FP16 KV cache or 32,768 tokens with FP8 KV cache. (high confidence) [source]
- For 7B/8B models at 4-bit precision (~4.5 GB weights), remaining VRAM exceeds 11.5 GB, allowing context windows to expand up to 65,536 tokens without host offloading. (high confidence) [source]
- FP8 KV cache quantization reduces per-token KV memory by 50% down to 98.3 KB per token, permitting up to 32,768 context tokens in 3.14 GB on 16 GB accelerators. (medium confidence) [source]
- A 14-billion parameter model: FP16 - 28.0 GB, FP8/INT8 - 14.0 GB, 4-bit (Q4_K_M) - 7.62 GB (high confidence) [source]
- On 16 GB RTX 5080: Q4_K_M leaves 8.38 GB for KV cache and scratch (high confidence) [source]
Facts and statements
- In GQA architectures such as Qwen2-14B and Llama-3.1-8B, KV cache memory footprint scales as 2 * num_layers * num_kv_heads * head_dim * precision_bytes per context token. (high confidence) [source]
Children
- Chunked Prefill Memory Bounds (frontier)
- Context Sizing Budgets on 16GB VRAM
- FP8 and INT4 KV Cache Quantization (frontier)
- GGUF k-quants (frontier)
- GQA KV Cache Memory Mechanics
- NVFP4 on Blackwell sm_120 (frontier)
- PagedAttention and Memory Fragmentation (frontier)
- Sliding Window Attention (frontier)
- Weight Quantization Footprints
Frontier under this node: Chunked Prefill Memory Bounds, FP8 and INT4 KV Cache Quantization, GGUF k-quants, NVFP4 on Blackwell sm_120, PagedAttention and Memory Fragmentation, Sliding Window Attention