vLLM vs SGLang
Both engines batch continuously and page their KV cache. SGLang's radix-tree prefix sharing wins on agent and RAG traffic; vLLM's model coverage and ecosystem win elsewhere. How to measure which applies to you.
vLLM and SGLang are the two open-source serving engines most teams evaluate. Both implement continuous batching and block-based KV cache allocation, both expose an OpenAI-compatible API, and both run on the same hardware. Their differences are in scheduler design and in which workloads each was built around.
Origins
vLLM (UC Berkeley, 2023) introduced PagedAttention and set the baseline for memory-efficient serving. Its design goal was throughput for general chat and completion traffic across a wide range of models.
SGLang (LMSYS, 2024) began as a language for structured multi-call LLM programs and grew a runtime optimised for them. Its central contribution is RadixAttention: a radix tree over KV cache blocks that shares prefixes across requests automatically, together with a scheduler that orders requests to maximise cache hits.
Architecture
| vLLM | SGLang | |
|---|---|---|
| KV cache | paged blocks, block table per sequence | paged blocks indexed by a radix tree over token prefixes |
| Prefix sharing | hash-based prefix caching, exact prefix match | radix tree, shares any common prefix at token granularity |
| Scheduling | FCFS with priority option; chunked prefill | cache-aware: reorders waiting requests to group shared prefixes |
| Control loop | Python, with a multi-step scheduling option to amortise overhead | Python with a zero-overhead scheduler design that overlaps CPU scheduling and GPU execution |
| Constrained decoding | via outlines / xgrammar integration | native, with a compressed FSM for JSON and regex |
| Speculative decoding | draft model, n-gram, EAGLE, Medusa | EAGLE, and others |
| Parallelism | tensor, pipeline, expert, data | tensor, pipeline, expert, data |
Workload fit
Independent chat and completion requests, diverse prompts. Prefix sharing has little to offer, and throughput comes down to kernel efficiency and scheduler overhead. The two are close; recent benchmarks trade the lead by model and version. vLLM's larger model zoo and quantisation coverage often decide it.
Agent loops, retrieval-augmented generation, few-shot prompting. Requests share long prefixes: a system prompt, retrieved documents, or a conversation history that grows by one turn per call. SGLang's radix cache reuses those prefixes without recomputation and its scheduler groups requests to keep them resident. The SGLang authors report multi-fold throughput gains on such workloads in their paper; the size of the gain depends on how much prefix is shared, so measure on your own traffic.
Structured output. JSON schemas, regular expressions, grammars. SGLang's native constrained decoding is faster and more complete. vLLM supports the same through integrations with an overhead that depends on the backend.
Multi-modal and new architectures. vLLM typically gains support first and covers more models.
Operations
Maturity and community. vLLM has the larger user base, more deployment recipes, more Kubernetes integrations, and more third-party tooling. Problems are more likely to have been encountered and documented.
Release cadence. Both release frequently and both change behaviour between versions. Pin versions, keep a latency regression suite, and upgrade deliberately.
Observability. Both export Prometheus metrics for queue depth, cache utilisation, throughput, and latency. Metric names differ; dashboards do not transfer.
Migration. The OpenAI-compatible API means clients rarely change. Engine configuration, metrics, and tuning do not carry over.
Measurement
Do not decide from published benchmarks. Replay one day of production traffic through each engine on the same hardware and record time to first token and time per output token at p50 and p99, tokens per second at the target concurrency, and behaviour at twice that concurrency. The prefix-sharing advantage in particular only shows on traffic that actually shares prefixes, which a synthetic benchmark may not.
Decision
- Traffic with shared prefixes (agents, RAG, chat with system prompts), or strict structured output: SGLang.
- Broad model coverage, newest architectures, largest ecosystem, or no strong prefix sharing: vLLM.
- Either way, keep the router-to-engine interface narrow so the choice can be revisited. The engines are converging on each other's features, and the right answer in a year may differ.