Scaling Ollama across replicas: load balancing and when to move to vLLM
Parent: Ollama templates and scale-out · Published reference · snapshot 2026-10-15 · skill ollama-expert/references/ollama-templates-and-scale-out.md
Also known as: Ollama scale-out and vLLM
↓ Facts as markdownall context files
Source-anchored research for Scaling Ollama across replicas: load balancing and when to move to vLLM. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Definitions
- vLLM Production Stack is a Kubernetes-native reference deployment that adds KV-cache-aware, load-aware and prefix-aware routing, KV cache sharing/offload, KEDA autoscaling and observability, which have no counterpart in a plain nginx-in-front-of-Ollama setup. (medium confidence) (verified as of 2026-10-14) [source]
How it works
- Switch trigger: Red Hat concludes Ollama suits local dev, prototyping and single-user apps while vLLM suits production and high-concurrency workloads. Because Ollama queues beyond its parallel slots (TTFT rises sharply) while vLLM batches, adding Ollama replicas buys slots linearly per GPU but not batching efficiency. (medium confidence) [source]
- Rule for sizing a replica: its concurrency ceiling is its OLLAMA_NUM_PARALLEL, because a tuned Ollama throughput plateaus there (McDermott) and an untuned one at a few requests (the Applied Sciences paper, bottleneck near 10 users at NUM_PARALLEL=4). To serve N concurrent clients, either raise NUM_PARALLEL on each replica (memory grows by NUM_PARALLEL x context length) or add replicas. Past the point where VRAM limits the parallel slots, the benchmarks favour vLLM's continuous batching. (low confidence) [source]
- LiteLLM routing strategies are `simple-shuffle` (default and recommended for production), `least-busy` (fewest active requests; suited to high concurrency), `usage-based-routing` (lowest TPM/RPM usage; documented as "bad for perf" because of Redis tracking), `latency-based-routing` and `cost-based-routing`. Set `rpm`/`tpm` on a deployment to cap it, and `order` to prioritise deployments. When running several LiteLLM proxy instances, set redis_host/redis_port so load-balancing state is shared. (medium confidence) (verified as of 2026-10-14) [source]
Measurements and reference values
- Do not extrapolate one benchmark: each is one model on one box with a particular Ollama build and NUM_PARALLEL. Red Hat (a vendor of vLLM-based products) used one A100 and Llama 3.1 8B; the Applied Sciences paper used one H100 and Qwen3-4B; McDermott used two A6000s and Qwen3-14B. Treat the direction as established and the multiples (3x to 29x) as workload specific. Reproduce on your own model, context length and concurrency before migrating. (high confidence) [source]
Problems, failure modes and limitations
- Ollama's documented concurrency has two levels inside one server: several models loaded at once if memory allows, and parallel request slots per loaded model. If memory cannot fit a new model, new requests queue and are processed in order while idle models unload. (low confidence) [source]
- OLLAMA_NUM_PARALLEL sets parallel requests per model and defaults to 1 (envconfig/config.go and the FAQ on main, Ollama v0.40.4 (2026-10-12)). OLLAMA_MAX_LOADED_MODELS defaults to 3 x GPU count (3 for CPU) per the FAQ, and OLLAMA_MAX_QUEUE defaults to 512; beyond the queue limit the server answers HTTP 503 (server overloaded). OLLAMA_SCHED_SPREAD=1 makes Ollama schedule a model across all GPUs. (high confidence) (verified as of 2026-10-14) [source]
- LiteLLM adds failure handling that Ollama lacks across replicas: `num_retries` and `timeout` in router_settings, fallbacks, and deployment cooldowns (`allowed_fails` failures in a minute cool a deployment for `cooldown_time` seconds). By default a failed request moves to the next entry in `fallbacks` (a different model group); with `enable_weighted_failover` (simple-shuffle only, async calls only, capped by max_fallbacks, default 5) the Router first retries a different deployment inside the same model group, so an Ollama replica that returns 503 can be routed around. Deployments can also carry a `weight` to be picked more often. (low confidence) (verified as of 2026-10-14) [source]
Comparisons and alternatives
- Red Hat's benchmark (single A100-40GB, Llama 3.1 8B, vLLM 0.9.1 vs Ollama 0.9.2, GuideLLM, concurrency 1-256) reported peak 793 TPS for vLLM vs 41 for Ollama and P99 80 ms vs 673 ms. Ollama throughput stayed flat at default settings; with OLLAMA_NUM_PARALLEL=32 (highest stable) it still plateaued below vLLM, with erratic inter-token latency attributed to head-of-line blocking. (low confidence) (verified as of 2025-08-08) [source]
- Independent peer-reviewed benchmark (Applied Sciences, 2026-05-28; one NVIDIA H100 80 GB, Qwen3-4B; Ollama 0.24.0 at OLLAMA_NUM_PARALLEL=4 vs vLLM 0.17.0 with FP8): vLLM delivered 20-29x higher throughput and 8-19x lower P95 latency. At 100 concurrent users on the baseline scenario Ollama managed 0.45 req/s with 100.48 s average latency and a 39.29% error rate, while vLLM managed 15.20 req/s at 6.37 s with no errors. Ollama hit a concurrency bottleneck near 10 users; vLLM stayed stable to 100 and degraded between 100 and 200. (medium confidence) (verified as of 2026-10-14) [source]
- A second independent test (R. McDermott, 2025-05-27; two NVIDIA A6000 with NVLink, Qwen3-14B, vLLM 0.8.5.post1 with tensor parallel 2 vs Ollama with OLLAMA_SCHED_SPREAD=1 and OLLAMA_NUM_PARALLEL=32) found a smaller gap on a tuned Ollama: vLLM reached up to 3.23x the requests per second at 128 concurrent requests. Ollama ramped to about 22 requests per second at 32 concurrent requests, matching its NUM_PARALLEL setting, and stayed flat beyond that; more clients only added latency. (medium confidence) (verified as of 2026-10-14) [source]
Facts and statements
- Raising OLLAMA_NUM_PARALLEL multiplies memory: required RAM/VRAM scales by NUM_PARALLEL x context length (a 2K context with 4 parallel requests becomes an 8K context allocation), so per-replica parallelism is capped by VRAM and a replica-per-GPU layout trades memory for slots. (low confidence) [source]
- The Ollama FAQ describes no multi-node clustering, shared queue or built-in balancer; scale-out is therefore done outside Ollama by running independent instances behind a load balancer and treating the 503 overload response as a signal to route elsewhere or shed load. (low confidence) [source]
- vLLM documents three load-balancing modes for data-parallel serving: internal (one endpoint, --data-parallel-size), hybrid (per-node API servers with an upstream router, --data-parallel-hybrid-lb), and external (each rank its own deployment behind a router using real-time telemetry; non-MoE models can simply run as independent instances). (low confidence) (verified as of 2026-10-14) [source]
- LiteLLM (proxy or Python Router) load-balances across several deployments that share the same `model_name` in `model_list`; the Router picks a deployment per request. The documented example uses three Azure deployments, but each entry is just a litellm_params block with its own api_base, so entries pointing at separate Ollama servers are the same shape (this application to Ollama is an inference from the docs, not a documented recipe). (low confidence) (verified as of 2026-10-14) [source]
- For Ollama upstreams LiteLLM documents the `ollama_chat/<model>` prefix (recommended over `ollama/` "for better responses") and passes the server address as api_base, for example http://localhost:11434. (low confidence) (verified as of 2026-10-14) [source]
Children
- No children recorded.