Reference self-flip floor from cache_prompt and batch-size nondeterminism

Facts

• A llama.cpp user running 8 server slots with temperature 0, a fixed seed and cache_prompt false got 5 to 8 distinct completions from the same prompt on an H100, reproduced on an M1 MacBook Pro and an A100, and always one completion with one slot but still small logit variations. [source]
• A llama.cpp maintainer's answer: bit-identical floating point output needs the exact same operations in the same order, and multiple slots are faster precisely because they do not run that way; multi-slot output stayed not fully deterministic on master, and a per-slot RNG fix (PR 6835) was needed because slots first shared one RNG state. [source]
• The issue was closed by the stale bot after 14 days with no root-cause fix. [source]
• vLLM documents "batch invariance" as an opt-in beta (VLLM_BATCH_INVARIANT=1) that makes output independent of batch size and request order, supported on NVIDIA GPUs of compute capability 8.0 or higher and Intel XPUs with Triton; Apple GPUs are not listed. [source]
• No llama.cpp or MLX switch with the vLLM guarantee was found, so on a Mac the floor has to be measured, not configured away. [source]
• KLD analogue: llama-perplexity derives n_seq from n_batch / n_ctx and runs sequences together, so the reference pass and each candidate pass use batch shapes set by -b and -c; the candidate pass must reuse the reference's -c, and equal -b is the only way to keep batch composition equal. [source]
• A direct floor measurement: run the reference file's own model through the --kl-divergence pass with a different -b and read mean KLD and Same-top p; any nonzero KLD or Same-top p under 100% is the batch-shape floor. [source]
• 2024-05: issue 7052 reports the multi-slot effect. 2025-2026: vLLM adds batch invariance (docs read 2026-10-04). [source]
• Single-slot variation in logits does not necessarily change argmax, but a near-tie position can flip, so the floor concentrates on low-margin positions, the same positions a quant flips. [source]
• The issue's hardware was CUDA plus one M1 report; Metal reduction order differs, so the size of the floor on Apple GPUs is unknown. [source]
• "Greedy plus cache_prompt false is deterministic enough" (implied by the harness recipe in the covering file) versus the 7052 evidence of 5 to 8 distinct outputs across slots. The issue used many slots; one slot gave one completion. [source]
• Self-flip rate of a Metal llama-server with one slot, cache_prompt on versus off, and across -ub values, on a 1,000-position token stream. [source]
• Whether MLX has a batch-size-dependent reduction order that makes mlx_lm.server self-flip. [source]
• Issue 7052 reports 5 to 8 unique completions from 8 slots at temperature 0 with cache_prompt false, and one completion with a single slot. [source]
• The same issue reports the effect on an M1 MacBook Pro as well as an H100 and an A100. [source]
• The llama.cpp maintainer attributes the multi-slot variation to differing floating-point operation order across slots. [source]
• vLLM batch invariance is beta, opt-in, and listed for NVIDIA compute capability 8.0 or higher and Intel XPU only. [source]
• llama-perplexity sets n_seq from n_batch divided by n_ctx, so batch settings shape how sequences are grouped in both reference and candidate passes. [source]
• No measurement of a Metal llama-server or mlx_lm.server self-flip floor was found. [source]