sglang
SGLang is a serving engine for large language models and multimodal models: you point it at model weights, it launches an HTTP server with an OpenAI-compatible API, and it squeezes throughput out of your GPUs with RadixAttention prefix caching, continuous batching, a zero-overhead scheduler, speculative decoding, and tensor, pipeline, expert, and data parallelism. Hosted by the LMSYS organization, it serves production traffic at companies like xAI and Cursor and doubles as the rollout backend for several RL post-training frameworks. It competes directly with vLLM for self-hosted inference.
One of the two serious choices for self-hosted LLM serving, with real production pedigree and standout prefix caching for agentic workloads. Budget for GPU ops and a fast-moving dependency; if you lack either, use a hosted API instead.
Use it if
- You self-host open-weight models (Llama, Qwen, DeepSeek, Kimi, GLM) on your own GPUs and throughput per dollar is the metric that matters
- Your workload has heavy shared prefixes (agents, few-shot prompts, multi-turn chat): RadixAttention caches them automatically and prefill drops accordingly
- You need structured output at speed: grammar-constrained JSON decoding is built into the server rather than bolted on client-side
- You run RL post-training: frameworks like verl, AReaL, and slime already use SGLang as their rollout engine, so matching engines removes a variable
- You do not have serious GPU hardware: this is datacenter software aimed at NVIDIA and AMD accelerators; for a laptop or CPU box, llama-cpp-python or ollama is the practical path
- You just need completions for a product without ops appetite: a hosted API costs less than the engineering time to babysit GPU serving infrastructure
- You want a slow, boring dependency: releases ship constantly, the repo carries roughly 773 open issues (plus thousands of open PRs), and version-to-version behavior changes are normal for a project moving this fast
- Your team already runs vLLM in production: performance leadership trades places between the two per model and hardware generation, and vLLM has more third-party deployment guides; switching costs usually beat the delta
Setup reality
This is not a pip-install-and-go library. The dependency tree pins CUDA-13-era wheels (cuda-python, flashinfer, flash-attn), so your driver, CUDA toolkit, and torch build must line up, and installs on anything but a recent NVIDIA Linux box get creative; the project's docker images are the sane route. First launch downloads tens of gigabytes of weights from the Hub. Expect to tune --mem-fraction-static when you hit CUDA out-of-memory at startup, set --tp to your GPU count for big models, and hold a gated-model HF token. AMD, TPU, and CPU backends exist but each has its own install documentation for a reason.
Patterns
Launch an OpenAI-compatible serverlaunch-server
python -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 --port 30000Weights download from the Hub on first run; gated models need HF_TOKEN set. Readiness is when 'The server is fired up and ready to roll!' appears.
Call the server with the OpenAI SDKopenai-client-chat
import openai
client = openai.OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="none")
resp = client.chat.completions.create(
model="meta-llama/Llama-3.1-8B-Instruct",
messages=[{"role": "user", "content": "Three facts about GPUs"}],
temperature=0.2,
)
print(resp.choices[0].message.content)Any OpenAI-compatible client library or framework works unchanged; the api_key value is ignored unless you start the server with --api-key.
Stream tokens as they generatestream-response
stream = client.chat.completions.create(
model="meta-llama/Llama-3.1-8B-Instruct",
messages=[{"role": "user", "content": "Write a haiku about caching"}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)Standard OpenAI streaming semantics; the final chunk can carry a None delta, so guard before printing.
Batch inference without a serveroffline-engine
import sglang as sgl
llm = sgl.Engine(model_path="meta-llama/Llama-3.1-8B-Instruct")
prompts = ["The capital of France is", "The capital of Japan is"]
outputs = llm.generate(prompts, {"temperature": 0, "max_new_tokens": 32})
for o in outputs:
print(o["text"])Engine runs in-process for offline batch jobs and RL rollouts; it grabs the GPU on construction, so only one Engine per process.
Constrain output to a JSON schemastructured-json-output
resp = client.chat.completions.create(
model="meta-llama/Llama-3.1-8B-Instruct",
messages=[{"role": "user", "content": "Give me info about Paris"}],
response_format={
"type": "json_schema",
"json_schema": {"name": "city", "schema": {
"type": "object",
"properties": {"name": {"type": "string"}, "population": {"type": "integer"}},
"required": ["name", "population"]
}},
},
)Decoding is grammar-constrained server-side, so output always parses; still instruct the model toward JSON in the prompt for better content quality.
Shard a big model across GPUstensor-parallel
python -m sglang.launch_server \
--model-path Qwen/Qwen2.5-72B-Instruct \
--tp 4 --port 30000--tp must divide both your GPU count and the model's attention heads; NVLink-connected GPUs matter, tensor parallel over PCIe hurts.
Tune KV cache memory to stop startup OOMfix-oom-startup
python -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--mem-fraction-static 0.8 --port 30000The engine pre-reserves a fraction of VRAM for weights plus KV cache; lower it when other processes share the GPU, raise it for more concurrent requests.
Serve a model with quantizationquantized-serving
python -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--quantization fp8 --port 30000fp8 needs Hopper-class or newer GPUs; for pre-quantized AWQ or GPTQ checkpoints, just point --model-path at them and the config is detected.
Run the server in Dockerdocker-deploy
docker run --gpus all --shm-size 32g -p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e HF_TOKEN=$HF_TOKEN \
lmsysorg/sglang:latest \
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000The official image sidesteps the CUDA dependency matrix entirely; mount the HF cache or you re-download weights on every container start.
Use the native /generate endpointnative-generate-endpoint
import requests
r = requests.post("http://127.0.0.1:30000/generate", json={
"text": "Once upon a time,",
"sampling_params": {"temperature": 0.7, "max_new_tokens": 64},
})
print(r.json()["text"])The native API exposes engine-level knobs the OpenAI surface hides; note max_new_tokens here versus max_tokens in the OpenAI API.
Probe health and scrape metricshealth-and-metrics
curl http://127.0.0.1:30000/health
curl http://127.0.0.1:30000/get_model_info
# start with --enable-metrics, then:
curl http://127.0.0.1:30000/metricsPrometheus metrics are opt-in via --enable-metrics; /get_model_info is the quick way to confirm which checkpoint a port is actually serving.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| vllm | PyPI | You want the most widely deployed open serving engine with the largest body of ops know-how. |
| llama-cpp-python | PyPI | You serve quantized GGUF models on CPU or consumer GPUs without datacenter hardware. |
| transformers | PyPI | You need flexible research-grade inference in Python and throughput is not the constraint. |