Disk and SSD-tiered KV cache servers

Parent: Mac local LLMs: Prompt cache and persistent KV · Published reference · snapshot 2026-10-05

oMLX: block-based cache modeled on vLLM with prefix sharing and copy-on-write; hot tier is RAM (write-back), cold tier is `PagedSSDCacheManager` writing safetensors blocks; the stack is PagedCacheManager (GPU) -> hot cache -> SSD manager. A changed byte spoils only the block it lands in and later...

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Facts

oMLX: block-based cache modeled on vLLM with prefix sharing and copy-on-write; hot tier is RAM (write-back), cold tier is `PagedSSDCacheManager` writing safetensors blocks; the stack is PagedCacheManager (GPU) -> hot cache -> SSD manager. A changed byte spoils only the block it lands in and later blocks; earlier blocks are still restored from disk, which is why shifting agent prefixes still hit. [source]
oMLX default SSD limit `auto` = 50% of (free disk + existing cache files, including GDN sidecars); the budget is refreshed during use and does not shrink merely because the cache grows or the server restarts. Fixed cap: `--paged-ssd-cache-max-size 20GB`. Settings persist in `~/.omlx/settings.json`; CLI flags override. The "GDN sidecars" mean hybrid (Gated DeltaNet) recurrent state is stored alongside the blocks. [source]
oMLX other defaults: `--max-concurrent-requests` 8; memory guard tiers (`--memory-guard safe|balanced|aggressive`, or `--memory-guard-gb N`), default ceiling is total RAM minus 8 GB (on 24 GB the ceiling was 18.8 GB, effective working set 11-13 GB); LRU model eviction, pinning and per-model TTL. [source]
oMLX is built on mlx-lm's BatchGenerator; mlx-lm still does the token generation. What oMLX adds is serving, caching, pooling and operations, not faster decode. [source]
llama.cpp disk path: `--slot-save-path DIR` enables `POST /slots/{id}?action=save|restore|erase` with body `{"filename":"x.bin"}`; without the flag the endpoints return 501. The server provides mechanism, not policy: the client decides when to save and which file name (key) to use. [source]
llama.cpp save format is `llama_state_seq_save_file` of the token list plus the sequence's KV cells. It never contained context checkpoints, and restore calls `slot->prompt.clear()`, which discards them (server-context.cpp:2529-2531 and 2576-2577 per issue #25913). [source]
stillwarm (zero-dependency wrapper around llama-server) restores before each request and saves after the response; its cache key is model SHA, flash-attention setting, KV cache types, SWA mode and prefix hash; build tag is recorded as a warning only; LRU prune under a disk budget. [source]
Rapid-MLX: radix prefix cache in RAM with RNN/DeltaNet state snapshots for hybrids; at shutdown it saves to disk only if the predicted write time fits a short budget (3.5 s minus 0.4 s commit headroom), and reloads at start. [source]
Ollama (llama.cpp GGUF runner and MLX runner): in-memory only. MLX runner caches in a trie (`x/mlxrunner/cache_trie.go`) with an 8 GiB paged-out bound; "paged out" is RAM, not disk. [source]
LM Studio mlx-engine: records are safetensors blobs packed into one anonymous `/tmp` scratch file with an in-memory offset table and free list; metadata is model-lifetime only. Wall-clock effect: it removes RAM pressure and enables hybrid/SWA rewinding within a session, not cross-restart persistence. [source]
Per-agent Q4 persistent cache (arXiv 2603.04428): 256-token blocks organised by agent ID, each layer stored as uint32-packed Q4 data plus bfloat16 scales/biases in safetensors; loaded via `mx.load()`; deletion of an agent's file erases its state (privacy angle); a ConcurrentScheduler interleaves prefill (512-token chunks) with decode on one thread because MLX is not thread-safe. [source]
2025-05-02 llama.cpp slot save/restore payload last changed substantively (commit 1d36b3670); 2025-10-03 PR #16382 made checkpoints mandatory for hybrid/recurrent rewinding, which silently broke the disk path. [source]
2025-11-08 feature request #17107 (web UI KV persistence) closed "not planned" (stale); auto-persist is explicitly left to clients. [source]
2026-02 oMLX launched (Show-and-tell mlx#3203, 2026-03-04 post; claim: TTFT 30-90 s down to 1-3 s on long contexts). By 2026-08 release 0.6.3rc1 (118th tagged release); 0.6.0 added experimental multi-Mac pipeline-parallel inference; 0.6.3rc1 fixed a regression that had broken prefix-cache reuse in long Claude Code sessions; 0.7.0rc1 benchmarked 2026-09-30. [source]
2026-03-14 llama.cpp discussion #20572 tutorial (pre/post-message hooks around slot save/restore); 2026-03-17 issue #20697 `--cache-disk` for UMA machines; 2026-03-28 issue #21133 (mmproj loaded blocks slot save and `auto_save_slots()` for text-only chats; closed not planned, stale). [source]
2026-07-06 stillwarm benchmark (b9871, M3 Max 36 GB). 2026-07-20 issue #25913 (hybrid restore reports success but reuses nothing). 2026-07-22 PR #26004 persists checkpoints inside the slot file (single-file appendix); PR #26640 reworked SLOT_SAVE/SLOT_RESTORE to write packed `server_tokens::serialize()` with media chunks. [source]
Ollama MLX runner regression #16698 (date not read) appeared in v0.30.8 ("improved prompt caching, decoupled from context shift" plus "MLX runner snapshots during prompt processing"). [source]
llama.cpp hybrid restore is a silent no-op (issue #25913, Apple M5 Pro, Metal, build 10068, Qwen3.6-35B-A3B MXFP4_MOE, `--slot-save-path ./kv`): restore returns `n_restored` = full count and `/slots` shows the tokens, but the next identical-prefix request has `cache_n = 0`. 14,906-token prefix: restore 0.12 s versus 75.9 s recompute, then all 14,914 tokens recomputed anyway. Cause: restore leaves `slot.prompt.checkpoints` empty, so `do_reset` is always true (server-context.cpp around :3300-3317). Evidence only at `-lv 4`. [source]
Draft-model state is not saved either: `ctx_dft` is not persisted, so `--model-draft`/draft-MTP setups desync after restore (separate open issue named in #25913 and PR #26004). [source]
Synthesising a checkpoint at `pos_min = 0` on restore is NOT a valid shortcut: it makes the matcher decode over a recurrent state that already consumed that token, giving silently corrupt output. [source]
Restore can be "successful but useless" on sliding-window models: Gemma-3 without `--swa-full` returned correct answers while re-prefilling almost the whole document. Always check `prompt_n` (or `cache_n`) after a restore. [source]
Where you save matters: saving after generation bakes sampled answer tokens into the file, so a new question may not extend it cleanly and SWA models fall back to full re-prefill; saving right after document prefill gives a strict prefix of any future question. [source]
Quantized-KV restores are deterministic but not equal to a fresh recompute: q4_0 KV diverged from a same-config cold recompute in 5/5 reps (first different token near position 47), q8_0 in 1/5, f16 in 0/5; the restored q4 output matched the f16 baseline. [source]
Invalidators measured on M3 Max (46-cell matrix): flash-attention on versus off refuses cleanly in both directions; KV cache type is matched; saved state larger than the new context window refuses; changing `-ngl` still works; files from a build about five weeks older restore both ways (no format guarantee). [source]
`--mmproj` loaded: `has_mtmd` is a capability flag and blocks slot save/restore, cache reuse and context shift for text-only chats (issue #21133, closed not planned); PR #26640 later added media-aware slot save/restore. [source]
llama.cpp save files scale linearly: about 219 MB for ~4K tokens on a hybrid Qwen3.6-27B, ~2.7 GB extrapolated at 50K; a GLM-4.6 64K context file was 36 GB; Llama-3.1-8B f16 is ~128 KiB/token (q8_0 0.531x, q4_0 0.281x). No built-in pruning: stillwarm adds LRU under a disk budget. [source]
oMLX cold-benchmark trap: SSD cache persists across benchmark runs, so "cold" is falsely fast unless `rm -rf ~/.omlx/cache`; and oMLX lazy-loads the model, so first-request TTFT includes model load. [source]
oMLX memory guard refuses large prefills on small Macs (18 GB, ~16k tokens: "Prefill would require ~9.08 GB peak ... dynamic ceiling is 9.00 GB"); on a 24 GB M4 Pro a deleted model folder still listed in `settings.json` caused an infinite HTTP 500 retry loop with ~100-line tracebacks per attempt (issue #2063, open, v0.4.4). [source]
Rapid-MLX 0.15.2 on 18 GB: prefix cache sized itself to 0.86 GB and each ~0.96 GB session entry was "too large", so every agent turn was a full prefill and the shutdown save skipped the session (first turn after restart 69.1 s). 0.15.3 (commit 1394f16, #3794) gives a minimum agent-session budget: 13.0 s on 18 GB, 12.3 s on 48 GB. Save is still not guaranteed: one of four 18 GB shutdowns skipped it (predicted 3.1 s write vs 3.5 s budget). [source]
Ollama MLX runner v0.30.8 regression (#16698, M4 Max 64 GB, qwen3.6:35b-mlx): over a 7-request sequence of 1K to 131K prompts (~259K tokens total) RSS grew from ~24 GB to 75 GB with 31.6 GB swap and stayed ~68 GB idle; pp131072 decode fell from 58.7 to 13.35 tok/s and TTFT 177 s to 317 s; v0.30.7 was flat; only an Ollama restart recovers. A commenter localised it to `cache.go:32` `maxPagedOutBytes = 8<<30` bounding only paged-out snapshots, not the live trie path. [source]
LM Studio 0.3.35 (2025-12-19, issue #1319): MLX prompt cache stopped working on an M3 Ultra 512 GB (GLM-4.6 6.5-bit, also MiniMax): every request fully re-processed; reverting to 0.3.31 fixed it. In 0.4.2+2 the log `Tried to trim '3195' tokens from the prompt cache, but could not: Cache is not trimmable. Clearing the cache instead.` appeared every turn with Claude Code. A commenter on 2026-04-15 suggested MLX runtime 1.6.0 fixed it (unconfirmed). [source]
mlx-lm issue #980 (2026-03-10, M3 Ultra 512 GB, LM Studio 0.4.6): prefix caching works only for pure full-attention models: MiniMax M2.5 cold 29.33 s, warm 6.15 s then 2.79 s; GPT-OSS 120B (sliding + full attention) shows no caching. [source]
Does llama.cpp slot restore work on hybrid models? ai-muninn (hybrid Qwen3.6-27B on an RTX 2080 Ti 22 GB, proxy over `--slot-save-path`): save 211 ms / 219 MB, restore 87 ms with no crash; 5K chat 9.9 s to 1.4 s. Issue #25913 (Metal, Qwen3.6-35B-A3B): restore succeeds but yields zero reuse. PR #26004 author reports a fix across CUDA/Metal/Vulkan. Reconcile: ai-muninn only checked that restore did not crash and a timed turn; #25913 shows reuse depends on checkpoints being present when the new prompt diverges from the stored one (a strict extension may still reuse). Treat hybrid disk restore as unreliable until #26004 (open at fetch time) or an equivalent lands. [source]
Restore versus recompute cost, speed claims. stillwarm (Apple M3 Max 36 GB, Llama-3.1-8B Q4_K_M): 8K cold 18.995 s vs resume 0.266 s (71x), 32K 157.42 s vs 0.989 s (159x), 64K 441.58 s vs 2.197 s (201x); wrong first headline of 1191x then 225x were corrected. The issue #20697 author's estimate of 10-20 ms per 50-100 MiB checkpoint was challenged as spec-sheet arithmetic; observed checkpoints are 60-214 MiB. The two measured sets agree on sub-second to ~2 s restores for 8K-64K but are on different models, so do not extrapolate between them. [source]
Is oMLX's SSD tier a speed win or an operations win? oMLX site claims "5 seconds, not 90". macoclock (M4 Max 64 GB, Gemma 4 31B 4-bit, 8k prefix): warm in-memory ties mlx-lm, oMLX's benefit appears only after restart or eviction; oMLX true cold is slower (64 s vs 32 s). Rapid-MLX benchmark: cold first turn on a 22.8K-token Claude-Code-shaped request was 63.1 s on oMLX, i.e. no cold benefit; restart turn 0.27 s. Vendor (Rapid-MLX) bias disclosed on the benchmark page. [source]
SSD wear. The Medium piece "Is oMLX worth installing" (Sep 19, 2026, paywalled) frames the SSD write load as the cost nobody examined; the llama.cpp #20697 thread notes some users avoid heavy SSD writes though modern TB-class drives make the fear "often overblown". No measured write volume for oMLX was found. [source]
Is PR #26004 merged, and in which llama.cpp build? (Open when fetched; two earlier companion-file PRs #20819 and #24028 conflict with master.) [source]
What does oMLX's cold tier actually write per request (GB per 10K tokens, TBW per day of Claude Code use)? Nothing measured found. [source]
Do oMLX safetensors blocks use the quantized KV format or fp16, and what is the on-disk size versus llama.cpp's 128 KiB/token f16? [source]
Does Ollama plan any disk tier? No source found; stillwarm says Ollama and LM Studio do not expose slot endpoints, while one #26004 reporter lists an "Ollama integration" without detail. [source]
Cross-machine portability of llama.cpp slot files is only anecdotal: one CUDA GB10 to Metal M3 Ultra restore reported working (cache_n 0 to 310, SHA-256-identical output) with the #26004 patch. [source]
oMLX is a block-based, vLLM-inspired KV cache with prefix sharing and copy-on-write across a hot RAM tier and a cold SSD tier. [source]
oMLX cold-tier blocks are safetensors files and are restored after a server restart when a prefix matches. [source]
oMLX default SSD cache limit `auto` is 50% of free disk plus existing cache files, including GDN sidecars; it refreshes during use and does not shrink as the cache grows; fixed cap via `--paged-ssd-cache-max-size 20GB`. [source]
oMLX stores settings in `~/.omlx/settings.json`, with CLI flags taking precedence; it refuses to start on a non-loopback address without a main API key. [source]
oMLX `--max-concurrent-requests` defaults to 8 and memory guard tiers are safe/balanced/aggressive with `--memory-guard-gb` for a custom ceiling. [source]
oMLX's architecture stack is PagedCacheManager (GPU, block-based, CoW) over a write-back hot cache over PagedSSDCacheManager. [source]
oMLX writes the cache in chunks so a changed byte invalidates only its chunk and later chunks, not the whole prompt. [source]
The oMLX creator's stated motivation is that coding agents send requests with shifting prefixes and existing MLX servers invalidate the whole cache. [source]
oMLX was announced in the MLX discussions on 2026-03-04 with a claimed TTFT drop from 30-90 s to 1-3 s on long contexts. [source]
oMLX release 0.6.3rc1 (Aug 2026) fixed a regression that broke prefix-cache reuse in long Claude Code sessions; 0.6.1 restored reasoning-effort compatibility while preserving prefix-cache reuse. [source]
oMLX 0.6.0 added experimental distributed inference across Macs using MLX pipeline parallelism over Thunderbolt. [source]
oMLX `--memory-guard` defaults to total RAM minus 8 GB. [source]
oMLX requires macOS 15.0 or newer. [source]
oMLX 0.7.0rc1 with SSD cache on, M4 Pro 48 GB, Qwen3.5-9B 4-bit, 22,819-token Claude-Code-shaped session: turn 1 cold 63.1 s, turns 2-10 median 0.49 s, total 67.6 s, first turn after restart 0.27 s, turns 2-10 after restart 0.27 s. [source]
Same test on M3 Pro 18 GB: turn 1 cold 67.0 s, turns 2-10 median 0.66 s, total 73.0 s, after restart 0.56 s then 0.44 s median. [source]
In that benchmark mlx-lm 0.31.3 and Ollama 0.34.3 reused cache within a session as well as oMLX but lost it on restart; on 18 GB mlx-lm's server died with Metal OOM on turn 4 and again on turn 3 after restart. [source]
For Ollama in that benchmark, "restart" means `ollama stop` (unloading the model), which drops its KV cache. [source]
Rapid-MLX 0.15.3 first turn after restart restored 18,944 of 22,819 prompt tokens and prefilled 3,875, taking 12.3 s on 48 GB and 13.0 s on 18 GB. [source]
Rapid-MLX 0.15.2 on 18 GB sized its prefix cache at 0.86 GB; each ~0.96 GB session entry was "too large", so every turn paid full prefill (70.0 s median) and turn 1 after restart was 69.1 s. [source]
Rapid-MLX shutdown save is budgeted: one 18 GB shutdown skipped the save because predicted write time 3.1 s did not fit 3.5 s minus 0.4 s headroom; commit 1394f16 (#3794) set a minimum agent-session cache budget. [source]
Cold prefill at ~16K tokens was close for all engines, 41.7 s (oMLX, fastest) to 49.4 s, on M4 Pro 48 GB. [source]
Rapid-MLX's comparison table lists LM Studio's cache as in-memory and Ollama's as in-memory per loaded model. [source]
Rapid-MLX describes its cache as a radix prefix cache with state snapshots for hybrid models, saved to disk on shutdown and restored at startup. [source]
macoclock (M4 Max 64 GB, Gemma 4 31B 4-bit, 8k prefix): after a restart mlx-lm was ~5x slower than its warm path and recovered about half the way from cold; oMLX came back at 7.2 s, 97% of the way from its 64 s cold prefill to warm. [source]
A valid cold test of oMLX requires `omlx stop && rm -rf ~/.omlx/cache/` before starting, because SSD blocks persist across runs. [source]
oMLX on a 24 GB M4 Pro in aggressive tier showed ceiling=18.8 GB, soft 17.0, hard 19.0, Metal wired limit 20.0 GB, leaving ~11-13 GB for model plus KV. [source]
oMLX issue #2063 (open, v0.4.4): a model folder deleted from disk but still in `settings.json` causes every request to retry a VLM-then-LLM load and return HTTP 500 with a ~100-line traceback in `server.log`. [source]
Interrupted model downloads leave corrupt safetensors that oMLX reports only as "invalid data offsets ... Perhaps an incomplete download or corrupt file?". [source]
A reported Ollama issue measured a 65,303-token prompt on an M5 Max at 104.3 s first time, 0.3 s unchanged, and 17.5 s after one byte changed. [source]
oMLX had 21,820 GitHub stars about seven months after its February 2026 launch. [source]
llama-server with `--slot-save-path DIR` exposes `POST /slots/<id>?action=save` and `?action=restore` with `{"filename":"x.bin"}`; without the flag it returns 501. [source]
llama.cpp feature request #17107 for automatic disk KV persistence was closed as not planned, so the server leaves save/restore policy to the client. [source]
A 60-line stdlib reverse proxy (kvproxy.py) that restores before and saves after each request cut a 5K-token chat from 9.9 s to 1.4 s (7x) on an RTX 2080 Ti 22 GB with hybrid Qwen3.6-27B; the 28K 25-40x figure is an estimate. [source]
On that hybrid model a ~4K-token slot saved in 211 ms to a 219 MB file and restored in 87 ms with `/health` ok. [source]
The same author stayed on the RAM cache for daily use because restarts are rare, one slot (`--parallel 1`) causes contention between conversations, and each ~244 MB save file needs LRU pruning. [source]
With a purge-free warm page cache restore beats recompute beyond about 30 tokens; with `sudo purge` before each rep it wins beyond about 132-295 tokens. [source]
stillwarm M3 Max 36 GB, Llama-3.1-8B Q4_K_M, llama.cpp b9871, flash attention on: 8K cold 18.995 s vs 0.266 s restore+TTFT (71x), 32K 157.42 s vs 0.989 s (159x), 64K 441.58 s vs 2.197 s (201x). [source]
Purged-page-cache restores read at 5.27 GB/s versus 6.40 GB/s raw `dd` (84-95% of SSD speed). [source]
A cold 64K restart on that M3 Max took more than seven minutes before the first token. [source]
All 15 supervised restores were byte-identical to recompute with f16 KV; decode speed after restore was unchanged. [source]
Energy: a 32K cold prefill took 116.5 s at ~33.8 W (1.102 Wh); resume from disk ~0.021 Wh (upper bound), about 50x less. [source]
Thermal drift on a laptop changed prefill from ~473 tok/s cool to ~310 tok/s after about an hour of load; purged-but-cool 64K restores (1.50-1.69 s) beat page-cached-but-hot (1.78-1.86 s); `dd` read 21.2 GB/s warm vs 6.4 GB/s cold. [source]
Restoring a llama.cpp slot file saved with flash attention on and loaded with it off (or the reverse) is refused; larger-than-context state is refused; changing `-ngl` works; files from builds ~5 weeks apart restore in both directions. [source]
Gemma-3 without `--swa-full`: restore reports success and the answer is correct but almost the whole document is re-prefilled; the stillwarm reuse warning detects this. [source]
Saving after document prefill only (strict prefix) allows partial reuse; saving after generation can force full re-prefill on SWA models. [source]
Quantized KV restore vs cold recompute divergence: q4_0 5/5 reps, q8_0 1/5, f16 0/5; restores are deterministic against saved state, so stillwarm `--verify` compares to a probe hash captured at save time. [source]
stillwarm states Ollama and LM Studio do not expose the slot save/restore endpoints; downloading a pre-built 471 MB 8K Qwen2.5-7B cache beats recompute only above ~215 Mbps. [source]
llama.cpp slot-save files are ~128 KiB/token at f16 for Llama-3.1-8B; q8_0 is exactly 0.531x and q4_0 0.281x. [source]
A GLM-4.6 64K-context slot file was 36 GB; a restore across llama.cpp updates kept working as long as the KV logic did not change, and the same model, quant and flash-attention setting are required. [source]
llama-cli supports `--prompt-cache FILE` and `--prompt-cache-ro`, but llama-server has no equivalent; slot endpoints are the server route. [source]
`mlx_lm.cache_prompt --prompt-cache-file X.safetensors` precomputes a prompt cache that `mlx_lm.generate --prompt-cache-file` reuses (offline route, not the server). [source]
llama.cpp issue #25913 (open, Jul 20 2026): on hybrid/recurrent models `/slots` restore returns full `n_restored` but the next identical-prefix request has `cache_n = 0`; on a 14,906-token prefix restore took 0.12 s vs 75.9 s recompute, then everything was recomputed. [source]
Root cause per #25913: the save file holds only tokens plus KV cells, restore calls `slot->prompt.clear()` which clears checkpoints, so `do_reset` is true at server-context.cpp:3300 and `n_past` becomes 0 at :3317; the in-memory prompt cache (#16391) stores whole `server_prompt` including checkpoints, which is why RAM reuse works. [source]
Control experiment in #25913: with `--ctx-checkpoints 0` and no save/restore, two identical-prefix requests also both show `cache_n = 0`, confirming checkpoints are the load-bearing mechanism. [source]
`--no-kv-unified` is ignored while the slot count is auto; `kv_unified` stays true until `--parallel` is set explicitly. [source]
Synthesising a pos_min = 0 checkpoint on restore would silently corrupt output; the draft context `ctx_dft` is also not saved. [source]
PR #26004 appends a tagged `SCKP` payload (magic, version, count, per-checkpoint pos fields and state blobs) after the llama state payload; old files restore as before and new files load on older servers; count capped at 1024, per-blob size capped, truncation aborts cleanly. [source]
PR #26004 reported results: Vulkan Radeon 8050S Qwen3.5-35B-A3B 181.9 s to 4.7 s (58,202 to 516 tokens); CUDA GB10 to Metal M3 Ultra cross-machine restore cache_n 0 to 310 with identical output; RTX 5070 Ti 4,460 to 30 tokens; a 36K-token agent session resumed with 13 tokens processed. [source]
Companion-file approaches (#20819 `.checkpoints`, #24028 `.ckpt`) conflict with master and can drift out of sync with the state file; #26004 keeps one file. [source]
PR #26640 reworked slot save/restore to write a packed `server_tokens::serialize()` payload with media chunks and restore with a two-pass load plus `validate()`. [source]
llama.cpp issue #20697 asks for `--cache-disk <path>` and `--cache-disk-max <MiB>` because on UMA systems `--cache-ram` competes with weights for the same pool; it proposes ~10-20 ms restore of 50-100 MiB checkpoints from 5-7 GB/s NVMe, and was challenged in the thread; last activity 2026-07-17 per another source. [source]
Observed llama.cpp checkpoints reach 213.707 MiB, larger than the 50-100 MiB premise in #20697. [source]
Slot save/restore writes the full sequence state, whereas context checkpoints store only the partial (non-reconstructible) state. [source]
llama.cpp issue #21133 (closed not planned): with `--mmproj` loaded, `has_mtmd` is set for every slot, blocking slot save/restore, context shift and cache reuse for text-only chats and crashing `auto_save_slots()` on shutdown. [source]
Ollama 0.19 blog lists three cache changes: cross-conversation reuse (less memory, more hits with a shared Claude Code system prompt), intelligent checkpoints at locations in the prompt, and smarter eviction so shared prefixes survive longer; it does not mention disk. [source]
Ollama issue #16698 (v0.30.8, M4 Max 64 GB, qwen3.6:35b-mlx): over 7 requests (1K to 131K, ~259K total tokens) SIZE grew from ~24 GB to 56-75 GB, swap hit 31.6 GB, idle memory stayed ~68 GB, pp131072 tgTPS fell 58.7 to 13.35 and TTFT 177 s to 317 s; v0.30.7 stayed at ~33 GB. [source]
A commenter traced it to `x/mlxrunner/cache.go:32` (`maxPagedOutBytes = 8 << 30`, `enforceEvictionPolicy` at L505) bounding only paged-out snapshots, with live trie branches from unique-prefix requests never pruned until the next request. [source]
Ollama on M-series with Gemma 4 and a ~5,000-token system prompt: cold ~7 s (~690 tok/s), warm ~20 ms to first token. [source]
LM Studio's disk-backed cache is one `/tmp` scratch file of safetensors records with an in-memory offset table and free list; the file shrinks when free space reaches its end and is released on model unload or process exit. [source]
LM Studio's cache keeps only local-attention KV on disk and evicts it from memory, so footprint scales with active sequences; for a single ever-growing conversation earlier local KV is evicted to make room for global KV. [source]
LM Studio's own post says its disk cache is "temporary and will not leave persistent files". [source]
LM Studio 0.3.35 (Dec 2025) on M3 Ultra 512 GB with GLM-4.6 MLX 6.5-bit re-processed every prompt; 0.3.31 worked. [source]
LM Studio 0.4.2+2 logged `Tried to trim '3195' tokens from the prompt cache, but could not: Cache is not trimmable. Clearing the cache instead.` every Claude Code turn. [source]
mlx-lm issue #980 (closed): prefix reuse works only for pure full-attention models; MiniMax M2.5 on LM Studio 0.4.6 went 29.33 s, 6.15 s, 2.79 s over three requests; GPT-OSS 120B (sliding + full) got no speedup. [source]
arXiv 2603.04428 Table 3 (M4 Pro 24 GB, Q4 weights and Q4 KV): Gemma 3 12B cold 3,964 ms at 1K to 172,096 ms at 32K, warm 475 to 1,819 ms, hot 683 to 1,264 ms; DeepSeek-Coder-V2-Lite cold 1,043 to 47,315 ms, warm 234 to 633 ms; Llama 3.1 8B cold 2,500 to 47,629 ms at 16K, warm 260 to 431 ms. [source]
In that paper warm beats hot by 40-55% at 1K-8K because the warm path uses one `mx.load()` sequential read while the hot path does per-layer hash lookups; Gemma warm is 93x faster than cold at 16K, DeepSeek 45x, Llama 111x. [source]
Paper head-to-head vs vllm-mlx 0.2.6 (FP16 prefix cache, Llama 8B): vllm-mlx cold 4K is 4,394 ms vs 10,235 ms, but warm TTFT converges (260/274/290 ms at 1/2/4K vs vllm-mlx 273/204/290); vllm-mlx FP16 (~128 KB/token) exhausted a 9.6 GB cache budget and failed at 16K, while the Q4 pipeline completed to 32K. [source]
Paper capacity maths on a 24 GB M4 Pro with ~16 GB free: FP16 fits ~7 agents at 16K, Q4 about 30; at 8K Q4 fits 12 agents in 10.2 GB. [source]
Paper estimates a 5-agent round-robin hides the ~500 ms disk reload behind the 1-3 s decode of the active agent; an internal SSD reads ~7 GB/s into 273 GB/s unified memory (SSD-to-UMA ratio ~2.6%). [source]
Paper perplexity with Q4 KV: Gemma 3 -0.10 (-0.7%), Llama 3.1 8B +0.12 (+2.8%), DeepSeek-V2-Lite +0.19 (+3.0%), measured on 512-token windows (shorter than Gemma's 1024 window) with no confidence intervals. [source]
Paper 25-turn multi-phase test: persistent mode cut Phase 5 TTFT 1.9x on Gemma and 1.3x on DeepSeek and total wall time 23% and 17%; Wikipedia-expert routing: priming 20.5 s/expert on Gemma, queries 847 ms warm (24.2x). [source]
The paper notes MLX is not thread-safe (mlx issues #2067, #2133, #3078), so inference runs on one scheduler thread with an RLock serialising disk saves; and that LM Studio mlx-engine did not persist cache to disk and hit type-mismatch errors with Gemma 3 prompts beyond the window before release 0.15.1. [source]
Per-agent cache deletion is file deletion, giving a targeted erasure path relevant to GDPR/HIPAA data-handling arguments. [source]

Corrections and disagreements

Does LM Studio persist KV across restarts? rapidmlx.com comparison table lists LM Studio as "In-memory". The LM Studio blog describes a disk-backed cache that is intentionally temporary (`/tmp` scratch file, cleared on model unload). Both are consistent: disk is used as overflow inside a model lifetime, not as a restart tier. CONTRADICTS prompt-processing-versus-decode-on-apple-gpus-and-neural-accelerators.md line 23 ("appear to survive restarts and evictions" for LM Studio mlx-engine 1.8.5); the survivors are oMLX and, partially, Rapid-MLX. [source]
CONTRADICTS prompt-processing-versus-decode-on-apple-gpus-and-neural-accelerators.md line 23: mlx-engine 1.8.5's disk cache does not survive restarts. [source]

Children

No children recorded.

← the whole tree · 3D view· how to read this page