Mac local LLMs: MoE streaming and offload

Parent: Running LLM models locally on a Mac · Published reference · snapshot 2026-10-05

If the model fits RAM, run it resident: Qwen3.6-35B-A3B 4-bit 48.4 tok/s on GPU vs 14.1 streamed (M6 32 GB); mlx-lm offload on a model that fits costs ~5x decode, 8-12x prefill, so enable it only when the model cannot load.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Stream or stay resident

If the model fits RAM, run it resident: Qwen3.6-35B-A3B 4-bit 48.4 tok/s on GPU vs 14.1 streamed (M6 32 GB); mlx-lm offload on a model that fits costs ~5x decode, 8-12x prefill, so enable it only when the model cannot load. [source]
Streaming 397B at ~4-6 tok/s suits batch/overnight work; 122B at higher quant gives better quality per second than 397B streamed. [source]
ds4 sizing: 64 GB Mac = Flash Q2 `--ssd-streaming` start; 96 GB resident Q2; 128 GB Flash Q2 or GLM 5.3 Flash Q2; 256 GB Q4/MXFP4. [source]
Hygiene: model on internal SSD, out of Time Machine and Spotlight; near-full disk, external drive or thermals show as token latency. Cap context (8K): KV competes with expert pages. [source]

Runtimes and flags

ds4: `--ssd-streaming`, `--ssd-streaming-cache-experts 32GB` (byte target; plain number = dynamic slots), `--ssd-streaming-full-layers N`, `--ssd-streaming-cold` and `--ssd-streaming-preload-experts N` for measurement only. 128 GB M5 Max: GLM 5.3 Flash Q4 (177.77 GiB) 11.9-14.9 tok/s, DeepSeek Flash MXFP4 11.9-19.3. [source]
ds4 keeps non-routed weights resident; an oversized expert cache displaces them and slows decode. Generation is more miss-sensitive than prefill: test a short generation first. [source]
ds4 caches are seeded from compiled hotlists (priority 1+31*remaining/total, not pins); `DS4_METAL_DISABLE_STREAMING_EXPERT_HOTLIST` turns it off. [source]
SwiftLM `--stream-experts`, `SWIFTLM_TOP_K`: 122B 0.58 to 5.91 tok/s, ~10 GB resident. Do not add `--draft-model` (5x I/O fan-out, slower than solo). `--gpu-layers N` on MoE is OOM avoidance only (~3 s/token). [source]
Flash-MoE (danveloper, Qwen3.5-397B, M3 Max 48 GB): 4.36 tok/s at 4-bit; 5.74 at 2-bit, which breaks JSON tool calls. [source]
Anemll fork (branch m5-nax, M5 Max 128 GB): Q3 GGUF experts + `--cache-io-split 4` 12.9 tok/s; split 8 worse; re-sweep on other chips. Like-for-like 4-bit is 9.5 tok/s, not the "3.3x" headline (2-bit). [source]
mlx-lm: `load(lazy=True)` avoids the 18.2 GB load spike; PR 1588 adds opt-in `enable_offload(resident_slots, fetch_fn)`. [source]
llama.cpp: `--no-mmap`/`--mlock` merged into `-lm, --load-mode`; old flags give `error: invalid argument`. `--n-cpu-moe N` is for discrete-GPU VRAM. [source]
JangPress (mmap + `MADV_DONTNEED`, Kimi-K2.6 JANGTQ): `JANGPRESS_PRESTACK` (~150 GB overlay), `KIMI_LOW_RAM=1`, `pct=100` on 128 GB. [source]

Cache and I/O rules

Trust the OS page cache: app-level LRUs lost (Metal LRU 3.14 to 1.99 tok/s as it grew, SwiftLM 4.84 to 4.01). A 9.8 GB Metal cache wired memory and left ~25 GB not ~35 GB for page cache. [source]
Use parallel `pread`, not mmap: cold mmap 5x slower (240 faults per 3.9 MB expert). 4 parallel preads gave 9.2x. `F_NOCACHE` only helps 2-bit (+3%). [source]
Read hints fail: `MADV_RANDOM` harmful, `F_RDADVISE` -8%/-4%, `F_RDAHEAD` 0%. [source]
Policy: per-layer LRU hits 0.80-0.87 at 30% capacity; pinning hot experts lost 6-30 points; adjacent-layer prefetch has no signal. SpecMD Least-Stale beats LRU but was tested on an A100, not Apple SSD. [source]
Layout: mbolt reorder+interleave 1.55x end-to-end under explicit reads; Flash-MoE clustering 0%. Warm decode is serial: SSD DMA and GPU do not overlap. [source]
Prefill touches nearly all experts, so a 30% cache refetches the table; warm prefill is no faster than cold. [source]
Benchmark SSD with `F_NOCACHE`: BlackMagic and AmorphousDiskMark skip it and may be inflated by RAM cache on big-memory Macs (author claim; a reply disputes it, SATA and HDD looked device-true). DiskTruth (beta 2026-04-16) sets F_NOCACHE on every fd, measures random 4K QD1 (10,000 ops, avg and p99) and QD1/4/16/32 scaling; it claims Apple NVMe has "extraordinary QD1 latency" but gives no numbers. [source]

Failure modes

`broadcast_shapes` fatal on first request: SwiftLM b769/b773 quantized MoE; fixed in SharpAI/mlx-swift-lm PRs 69, 71. [source]
`insufficient memory: model requires ≈234 GB peak`: set `VMLX_MEMORY_BUDGET_OVERRIDE`. `Unsupported model type: kimi_k25`: needs shadow config. [source]
`Qwen n-grams must follow page-aligned weights`: repack with `gguf-tools/qwen4_native_ngrams.py`. [source]
Ollama default load of a 35B on 16 GB: 4.3 M swapouts, no token in 10 min; llama.cpp mmap on the 13 GB IQ3_XXS gave 17.3 tok/s. [source]
Long context kills streaming: DeepSeek-V4-Flash 126 GB 4.65 tok/s at 512 to 0.32 at 40K; TurboQuant KV holds 4.16. [source]

Top-k reduction

Flash-MoE runs K=4 of 10 (K=3 collapses). llama.cpp: `--override-kv llama.expert_used_count=int:N`. SwiftLM: top-k 8 4.95, 6 5.20, 4 5.91, 2 6.52 tok/s (GPU-bound). Apple PHDS: safe to ~25% cut; training-time, so no runtime flag helps. [source]
Stacking K=4 with 2-bit breaks tools; evaluate the combination. [source]

Bandwidth: numbers to trust

Apple peak 14.5 GB/s (M5 Max 8 TB, FIO), up to 2x M4. Third-party M4 Max reads 5-7 GB/s; speed follows NAND package count, not capacity. Zero-cache ceiling, 397B: 8.9 tok/s at 4-bit, 11.1 at Q3 (14.5 GB/s); 3.4 and 4.2 at cold 5.5 GB/s. [source]
Corrections: 17.5 GB/s (M3 Max) and Anemll "36 GB/s" (0.74 ms/27 MB) exceed Apple's ceiling and include page-cache hits. Flash-MoE 1.6 GB at 17.5 GB/s is 10.8 tok/s; 18.6 is the 2-bit figure. [source]
Correction: JangPress "167 GB Kimi on 128 GB" is load-validated only; no coherent token reported (~130 GB footprint before token 1). [source]

Clusters and Engram

Flash-MoE paper: session cache avoids ~98% of reprocessing in a 10-turn chat; speculative decoding with a 35B draft was a 4.5x net slowdown. [source]
ds4 TP (2 Macs, RDMA, `iogpu.wired_limit_mb=120000`) is resident: omit `--ssd-streaming`. Metal validator does not reject the pair (correction), but no command or tok/s is documented. [source]
optiq pipeline: 20.5 tok/s resident on two Macs vs 4.9 streamed on one; residency cliff ~73% of RAM. [source]
Engram/n-gram tables (189 GiB) stay on disk in every mode; prefill overlaps reads (2,048-token chunks, prompts of 1,024+ tokens), decode does not. They are read row by row, `F_NOCACHE`, 264 B rows, 48 reads/token, latency-bound; no hot-row cache exists. [source]

Open

No SSD wear data, no measured ds4 QD16 stall latency, no sequential-read numbers from DiskTruth, no Apple-SSD Engram latency, no head-to-head of own cache vs page cache. [source]

Corrections and disagreements

CONTRADICTS: the jangq README headline and the two existing dossiers say a 167 GB Kimi-K2.6 bundle serves from a 128 GB Mac. The runtime-scripts README in the same repo says: "Small and Med can load to /health, but neither has produced a single coherent token through vmlxctl serve. First prefill reaches ~130 GB process footprint and severe system memory pressure before token 1. Do not start MMLU until a one-token probe returns content." The JangPress guide's own validation section says only that the bundles "load-validate" and have a post-load footprint of about 0.7 GB. So the validated claim is loadable, not runnable; the "runnable" claim in the guide's introduction is not backed by any measured token. [source]
CONTRADICTS: moe-expert-offload-to-ssd-on-macos.md reads the Anemll 0.74 ms per layer for 27 MB as roughly 36 GB/s of "effective" SSD read; the Anemll docs state those timings come from warm-cache runs, so they overstate raw SSD speed on M5 Max. [source]
CONTRADICTS: ssd-expert-streaming-and-cluster-sharding-for-mo.md (reading that TP and streaming are exclusive): ds4's Metal validator does not reject `--ssd-streaming` with `--tensor-parallel`; only network CUDA TP rejects it. The docs simply show no combined command. [source]
CONTRADICTS: moe-expert-offload-to-ssd-on-macos.md (4-bit "about 1.6 GB per token ... ceiling of about 18.6 tok/s at 17.5 GB/s"): 1.6 GB at 17.5 GB/s is about 93 ms or 10.8 tok/s; the 18.6 tok/s ceiling is the paper's 2-bit figure for 943 MB. [source]
CONTRADICTS: flash-moe-paper-flash-moe-pdf-90-plus-experiment.md and the Flash-MoE README, which describe a 1 TB M3 Max with "17.5 GB/s sequential read (measured)". Apple's own peak for the newer M5 Max with 8 TB is 14.5 GB/s, and third-party M4 Max Blackmagic reads are 5 to 7 GB/s. A 17.5 GB/s figure on an M3 Max 1 TB is above every first-party and third-party raw rate found, so it likely includes page-cache hits. The same repo's cold rate on that machine (5.5 GB/s with `F_NOCACHE`) is consistent with the raw rates. [source]
CONTRADICTS: moe-expert-offload-to-ssd-on-macos.md, which reads the Anemll 0.74 ms per 27 MB layer as "about 36 GB/s effective" and treats it as an M5-generation SSD gain. The Anemll table also lists 0.44 ms for 21.8 MB at Q3, which is about 50 GB/s. Both exceed Apple's 14.5 GB/s raw ceiling by 2.5x to 3.4x and can only be cache-served reads. (The Anemll docs already say these runs are warm; this adds the Apple ceiling as a hard bound.) [source]

Concepts in this cluster

MoE expert offload to SSD on macOS [source]
Expert-cache policy and on-disk layout for MoE streaming [source]
Expert-count reduction at inference top-k override [source]
Flash-MoE SSD-streamed MoE expert runtime [source]
JangPress cold-expert eviction for MoE larger than RAM [source]
MoE expert SSD streaming and page-cache residency [source]
SSD expert streaming and cluster sharding for MoE on Mac [source]
ds4 antirez SSD streaming runtime [source]
Aggregate SSD bandwidth when each cluster node streams experts [source]
Engram and n-gram embedding tables read from disk per token [source]
Flash-MoE paper flash_moe.pdf 90+ experiment log [source]
MoE-PHDS post-hoc declared sparsity fine-tune [source]
rustane warm-page-cache pread fanout (ncdrone/rustane) [source]
Engram row prefetch overlapped with preceding transformer blocks in Mac runtimes [source]
SSD bandwidth growth per Apple chip generation and 397B projection [source]
Zipfian hot-row cache tier for n-gram tables (DRAM over NVMe) [source]
Apple SSD speed dependence on NAND capacity and package count [source]
ds4 decode-time Engram read latency on Apple SSD (48 random reads per token) [source]
Apple NVMe small random read latency under F_NOCACHE at queue depth 16 [source]
Sequential read throughput of expert-sized reads by Apple SSD configuration [source]

Children

← the whole tree · 3D view· how to read this page