Local Model Performance Evaluation: Ollama, MLX, and Hardware Architectures
Published reference · snapshot 2026-09-30
↓ Facts as markdownall context files
Source-anchored research for Local Model Performance Evaluation: Ollama, MLX, and Hardware Architectures. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- Attempting to run models larger than 16GB across a Thunderbolt eGPU requires layer-by-layer weight streaming that collapses token generation throughput to sub-1 token per second. (high confidence) [source]
How it works
- When KV cache size exceeds available headroom during extended context ingestion, Ollama dynamically reallocates context buffers, incurring allocation stalls unless context size is pinned via num_ctx in Modelfile. (high confidence) [source]
Measurements and reference values
- Gemma4-12B quantized at 4-bit occupies ~7.8 GB VRAM, leaving 8.2 GB headroom in a 16GB VRAM pool for up to 32k token KV cache and activation workspaces. (high confidence) [source]
- When running completely resident in RTX 5080 GDDR7 VRAM (~1000 GB/s), Gemma4-12B generates ~85-92 tokens per second, roughly double the ~42-45 tokens per second achieved on Apple Silicon M5 Max (~400 GB/s). (high confidence) [source]
- Qwen3.6-35B in 4-bit quantized representation requires ~21.9 GB base weight footprint, fitting comfortably in 64GB Unified Memory while exceeding the 16GB capacity of consumer discrete GPUs. (high confidence) [source]
- On Apple M5 Max (400 GB/s bandwidth), memory-bound token generation for 35B 4-bit weights achieves 15.5 to 17.2 tokens per second, closely tracking theoretical memory bus saturation (400 GB/s / 22 GB = ~18.1 tps max). (high confidence) [source]
- Expanding context length to 64k tokens adds ~9.0 GB to the KV cache in 16-bit float, requiring explicit dynamic context cache management or 8-bit quantized KV cache to maintain resident headroom. (high confidence) [source]
- Multi-Token Prediction (MTP) heads in Qwen3.6 allow speculative verification of subsequent tokens, improving generation throughput by 20-30% on high-bandwidth memory architectures. (high confidence) [source]
Problems, failure modes and limitations
- Discrete eGPU GDDR7 achieves ~1000 GB/s memory bandwidth, delivering over 2x the memory-bound token generation throughput of Apple Silicon unified memory for models that fit within its 16GB VRAM limit. (high confidence) [source]
Comparisons and alternatives
- Serving Gemma4-12B via Ollama Metal introduces a 10-15% throughput penalty compared to native MLX-LM due to runtime translation and generic Metal kernel dispatch in ggml. (high confidence) [source]
- Thunderbolt 5 eGPU links provide approximately 7.0 GB/s practical unidirectional bandwidth, creating a severe bottleneck if model weights are streamed from host RAM rather than fully resident in GDDR7 VRAM. (high confidence) [source]
Facts and statements
- Pre-allocation of static tensor workspaces in Tinygrad or TensorRT avoids CUDA and Metal memory fragmentation during extended generation passes. (high confidence) [source]
- Time to First Token (TTFT) is compute-bound, determined by prompt batch parallel FLOPs on GPU compute cores, whereas auto-regressive token generation (TPS) is memory bandwidth-bound. (high confidence) [source]
- Measuring peak VRAM consumption requires tracking resident set size (RSS), active Metal wired memory allocations, and discrete GPU allocated VRAM during maximum prompt length ingestion. (high confidence) [source]
- Reporting mean tokens per second without measuring inter-token latency variance masks stutter caused by periodic garbage collection, dynamic KV cache reallocation, and memory compaction. (high confidence) [source]
- Automated evaluation pipelines must record thermal throttling states and host bus bandwidth saturation alongside inference latency to ensure benchmark reproducibility. (high confidence) [source]
- Apple Silicon Unified Memory provides a shared address space allowing CPU and Metal GPU to operate on identical memory pointers without PCIe serialization or host-device DMA copies. (high confidence) [source]
- GGUF format structures model weights into contiguous tensor buffers aligned to 32-byte boundaries, enabling zero-copy mmap() loading directly into virtual memory pages. (high confidence) [source]
- On Apple Silicon macOS, Ollama orchestrates llama-server instances allocating shared Metal command buffers, mapping unified memory buffers without host-to-device PCIe serialization. (high confidence) [source]
- Quantization schemes such as Q4_K_M and Q8_0 balance tensor weight compression against perplexity degradation by utilizing mixed precision blocks for attention projection matrices. (high confidence) [source]
Children
- GDDR7 Channel Bandwidth (frontier)
- GDDR7 Peak Throughput (frontier)
- GGML Metal Kernel Dispatch Overhead (frontier)
- GGUF Alignment (frontier)
- Gemma4-12B Cross-Platform Residency and Benchmarking
- Inference Metrics Framework: TTFT TPS and Memory Profiling
- Inter-token Latency Jitter (frontier)
- MLX KV Cache Quantization (frontier)
- MLX Unified Memory vs eGPU GDDR7 Hardware Architecture
- Memory Bandwidth Scaling Limit (frontier)
- Metal Command Buffer Offload (frontier)
- Multi-Token Prediction Heads (frontier)
- Ollama Architecture and GGUF Memory-Mapping
- Quantization Block Matrix (frontier)
- Qwen3.6-35B-MLX Sizing and Throughput Profiling
- Static Activation Buffers (frontier)
- TTFT Compute Bound Analysis (frontier)
- Thunderbolt 5 Bus Latency (frontier)
- Wired Memory Instrumentation (frontier)
- Zero-Copy Unified Pointer Passing (frontier)
- Local Inference Acceleration: Memory Architectures, Metal Kernels, and Latency Optimization
Frontier under this node: GDDR7 Channel Bandwidth, GDDR7 Peak Throughput, GGML Metal Kernel Dispatch Overhead, GGUF Alignment, Inter-token Latency Jitter, MLX KV Cache Quantization, Memory Bandwidth Scaling Limit, Metal Command Buffer Offload, Multi-Token Prediction Heads, Quantization Block Matrix, Static Activation Buffers, TTFT Compute Bound Analysis, Thunderbolt 5 Bus Latency, Wired Memory Instrumentation, Zero-Copy Unified Pointer Passing