TL;DR: SGLang serves an OpenAI-compatible API on localhost:30000 that Cline, Continue.dev, and Aider all speak, and its default-on RadixAttention prefix cache is built for exactly the traffic shape agent tools produce — the same giant system prompt and growing history resent on every single turn. Tool calling needs one flag (--tool-call-parser), the context default inherits vLLM’s crash-at-startup behavior, and the popular AWQ quant path has a loader catch worth knowing before you download 16 GB.

What you’ll be able to do after this guide:

• Serve a tool-calling coding model with one launch_server command, with the right parser for your model family (qwen3_coder for Qwen3-Coder, gpt-oss for gpt-oss, deepseekv31 for DeepSeek)
• Understand why an agent session’s second turn comes back faster on SGLang than on Ollama or llama.cpp — and when that actually matters
• Wire the endpoint into Cline (OpenAI Compatible provider), Continue.dev (openai provider with apiBase), and Aider (openai/ prefix) with configs that survive agent mode

Honest take: SGLang and vLLM are the two production-grade engines, and for a solo developer the difference is smaller than either project’s benchmarks imply — pick vLLM if you want the bigger ecosystem and smoother quant support, pick SGLang if your workload is agent loops hammering the same long prefix, where the radix cache does real, measurable work. One dev, one GPU, no concurrency? Ollama is still less to maintain.

---

The ninth backend, and why it earns a slot

This series has covered Ollama, LM Studio, llama.cpp’s llama-server, vLLM, Docker Model Runner, Jan, and KoboldCpp. SGLang is the last major name missing, and it isn’t a niche one: the project’s README claims production deployments generating trillions of tokens per day across more than 400,000 GPUs, with xAI, NVIDIA, AMD, and Intel among the adopters. It’s Apache 2.0, started at LMSYS (the Chatbot Arena people), and in January 2026 its core contributors formed a startup, RadixArk, to commercialize around it while the open-source project continues — current release v0.5.18, August 24, 2026, three days old as this is written.

So it’s vLLM’s peer, not vLLM’s imitator. The architectural signature that separates them is RadixAttention: SGLang stores KV cache in a radix tree and automatically reuses it across requests that share a prefix, at token granularity, on by default (turn it off with --disable-radix-cache, though it’s hard to imagine why you would here).

Look at what Cline actually sends over the wire and you’ll see why that matters. Every turn of an agent session re-sends the same multi-thousand-token system prompt, the same file context, and the entire conversation so far, plus one new tool result at the end. On Ollama or llama-server, most of that identical prefix gets reprocessed every round trip. On SGLang, turn two onward hits cached KV for everything but the new tail, which cuts time-to-first-token exactly where agent sessions hurt most — the twenty-step loop where the model must reread a 30K-token prefix before emitting each three-line tool call. Agent workloads are the most prefix-heavy traffic a local server will ever see, with input-to-output ratios that community deployment guides put as high as 100:1. This server was shaped for that.

The honest counterweight: vLLM also has prefix caching, Ollama keeps a session’s KV warm between consecutive requests in simpler cases, and none of this changes tokens-per-second once generation starts. RadixAttention’s edge is structural — it survives interleaved requests from multiple tools and parallel subagents sharing one system prompt, which is precisely where per-session caches fall apart.

Step 0 — Platform check

SGLang wants Python 3.10+ and targets NVIDIA CUDA first, with dedicated platform docs for AMD GPUs, Intel Xeon CPUs, Google TPU, NVIDIA Jetson, Ascend NPUs, Intel XPU — and, unusually for this category, an Apple Metal page. We haven’t tested the Metal path and wouldn’t stake a workflow on it yet; on a Mac, LM Studio remains the pragmatic answer. Windows isn’t in the platform list at all, so the realistic route there is WSL2, same as vLLM.

Which means the audience is the same Linux-box-with-an-NVIDIA-card crowd: 16 GB VRAM minimum, and everything below applies unmodified on an RTX 3090- or 4090-class card.

Step 1 — Install and first launch

The official install path is uv:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install --prerelease=allow sglang

That targets CUDA 13; CUDA 12 systems need a documented two-step with pinned torch wheels, and there’s a Docker image (lmsysorg/sglang:v0.5.18) if you’d rather not touch the Python environment at all.

Then launch. Here’s the command we’ll build toward — a 16 GB-and-up card running gpt-oss-20b with working tool calls:

python3 -m sglang.launch_server \
  --model-path openai/gpt-oss-20b \
  --tool-call-parser gpt-oss \
  --context-length 32768 \
  --api-key localkey-123

First launch pulls the weights from Hugging Face, loads, and warms up. The server defaults to 127.0.0.1:30000 — localhost-only out of the box, which is the safe default the llama.cpp server also picked and KoboldCpp didn’t. Add --host 0.0.0.0 only when other machines need in, and keep --api-key on it the moment you do.

Sanity-check it like any OpenAI-compatible endpoint:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer localkey-123" \
  -d '{"model": "openai/gpt-oss-20b",
       "messages": [{"role": "user", "content": "Say ready."}]}'
# → {"id":"...","object":"chat.completion","choices":[{"message":{"role":"assistant","content":"Ready."...

Two quality-of-life notes the other engines make you dig for: SGLang serves interactive API docs at http://localhost:30000/docs (Swagger) while running, and --served-model-name lets you alias the long Hugging Face repo path to something like local-coder so your tool configs stay readable.

Step 2 — One flag decides whether agents work

Cline’s agent loop and Continue.dev’s agent mode both require function calling. On SGLang that’s a single launch flag, --tool-call-parser, which tells the server how to decode the model family’s tool-call wire format into OpenAI-style structured calls. No parser, or the wrong parser, and you get the familiar failure: the model narrates JSON into the message content instead of acting — the same symptom we dissected in the Cline tool-use loop fix.

The parser matrix for coding-relevant families, from the official tool-parser docs as of today:

Model family	--tool-call-parser value
Qwen3-Coder	qwen3_coder
Qwen series (non-Coder)	qwen
gpt-oss	gpt-oss
DeepSeek-V3 / V3.1 / V3.2	deepseekv3 / deepseekv31 / deepseekv32
Llama 3.1–3.3 / Llama 4	llama3 / llama4
GLM series	glm
Kimi K2	kimi_k2
Mistral	mistral

Three caveats straight from the docs. DeepSeek models want an explicit --chat-template pointing at the shipped tool-chat template file. The gpt-oss parser “filters out analysis channel events and only preserves normal text,” which “can cause the content to be empty when explanations are in the analysis channel” — in practice Cline still gets its tool calls, but don’t be surprised by terse assistant text. And forced tool choice (tool_choice: "required") is only fully supported on the Xgrammar grammar backend (--grammar-backend xgrammar); Cline and Continue use auto tool choice, so most people never hit this.

Compare the neighbors: vLLM needs two flags for the same result, Ollama needs zero but will reject the request with a 400 if its template scan decides the model can’t do tools (the “does not support tools” error), and llama-server needs --jinja. SGLang sits in the middle — one explicit flag, no capability gate, and like vLLM it will happily do nothing useful if you forget it.

Step 3 — The context default OOMs loud, same as vLLM

--context-length “defaults to None (will use the value from the model’s config.json instead)” — the docs’ own words, and the same inverted trap we documented for vLLM. Ollama defaults context too small and truncates silently (the context-length fix exists because of it); SGLang defaults to the model’s full advertised maximum and allocates KV budget accordingly. Point it at a 256K-context model on a 24 GB card without the flag and the launch dies instead of the prompt getting quietly clipped.

Two levers, both from the server-arguments docs:

• --context-length 32768 — the working floor for agent tools. Cline’s system prompt plus file context eats 10–20K tokens before the model says a word; 32K fits a serious session, 65536 is comfortable if the weights leave room.
• --mem-fraction-static — the fraction of GPU memory SGLang claims for weights and KV pool. Unset, it’s computed from detected GPU memory and lands around 0.88 as the fallback; the docs say plainly to “use smaller values to address out-of-memory errors,” and 0.7–0.8 is the right neighborhood when the same GPU drives your displays.

There’s also --kv-cache-dtype fp8_e5m2 to halve KV size on supported cards, the same trade we recommended on llama-server — with the standing warning that aggressive KV quantization can degrade tool-call reliability, so change one variable at a time.

Step 4 — Models that fit, and the AWQ catch

SGLang loads safetensors from Hugging Face, with --quantization support listing awq, fp8, gptq, marlin and more. But here’s the honest wrinkle we hit researching this: the exact 24 GB pick from our vLLM guide — the QuantTrio AWQ quant of Qwen3-Coder-30B — has a community-reported loader failure on SGLang. Issue #9838 documents the error verbatim: “vllm is not installed, to use CompressedTensorsW4A16Sparse24 and CompressedTensorsWNA16, please install vllm.” SGLang borrows vLLM’s kernels for those compressed-tensors quant formats, so a bare SGLang install can’t load them. The issue dates to August 2025 and was closed as inactive rather than fixed-and-verified, so treat that path as “may work with vLLM installed alongside, test before you rely on it” — installing a second inference engine to feed the first is nobody’s idea of clean.

Which reshuffles the tier table relative to vLLM:

VRAM	Model (--model-path)	Weights	Parser	Notes
16 GB	openai/gpt-oss-20b	~12–13 GB (native MXFP4)	gpt-oss	The clean pick. 21B MoE, 3.6B active; MXFP4 is the official precision, SGLang had day-0 support, and no borrowed kernels are involved
24 GB	openai/gpt-oss-20b with headroom, or Qwen3-Coder-30B AWQ if the compressed-tensors path works in your install	—	gpt-oss / qwen3_coder	Test the AWQ load before building on it (issue #9838); the extra VRAM otherwise buys you KV space and concurrency, which is what SGLang is for anyway
48 GB+	Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8	~31 GB (official FP8)	qwen3_coder	Qwen’s own FP8 release; the official SGLang cookbook recipe for Qwen3-Coder uses --tool-call-parser qwen3_coder and treats FP8 as the standard precision

If those sizes read as aspirational, the hardware side is runaihome.com’s beat — their best local AI models by VRAM guide covers what fits before you spend anything, and aifoss.dev tracks the open-source serving stack.

Step 5 — Wire in the three tools

Cline: OpenAI Compatible provider

Cline’s local-models documentation covers Ollama, LM Studio, and Atomic Chat only — SGLang isn’t mentioned, so the supported route is the OpenAI Compatible provider, exactly as for vLLM and llama-server:

• Base URL: http://localhost:30000/v1
• API Key: whatever you passed to --api-key (anything non-empty if you didn’t)
• Model ID: the repo path (openai/gpt-oss-20b) — or your --served-model-name alias if you set one
• Model Configuration: set the context window to match your --context-length so Cline’s truncation math is honest

Turn on Use Compact Prompt in Settings → Features per Cline’s own local-model guidance — and note that a smaller system prompt helps twice on SGLang, since the compact prefix both fits better and caches better.

Continue.dev: openai provider with apiBase

Unlike vLLM, Continue has no native SGLang provider (we checked the provider directory today), but none is needed — SGLang’s /v1 speaks the standard dialect, so the generic openai provider with an apiBase override does it:

models:
  - name: gpt-oss 20B (SGLang)
    provider: openai
    model: openai/gpt-oss-20b
    apiBase: http://localhost:30000/v1
    apiKey: localkey-123
    roles: [chat, edit, apply]

The apiBase includes /v1, same as Cline. Agent mode gates on the tool_use capability; Continue autodetects it, and if the agent tools stay greyed out, add capabilities: [tool_use] to the block — capabilities are additive, so per Continue’s docs “you cannot override autodetection — you can only add capabilities,” and nothing breaks by declaring it.

Aider: OpenAI-compatible env vars

export OPENAI_API_BASE=http://localhost:30000/v1
export OPENAI_API_KEY=localkey-123
aider --model openai/openai/gpt-oss-20b

Yes, the doubled openai/ is correct and intentional: the first is Aider’s LiteLLM provider prefix, the second is part of the Hugging Face repo path. This is where --served-model-name local-coder earns its keep — the invocation becomes the saner aider --model openai/local-coder. Aider will warn about unknown context window and costs for any local model; silence it with a .aider.model.metadata.json exactly as in the vLLM guide, swapping in your model name. And since Aider drives edits through search/replace blocks rather than tool calls, it’s the most forgiving of the three if a model’s tool calling is shaky — block-apply failures are their own known issue, unrelated to the backend.

SGLang vs. vLLM vs. the wrappers

	SGLang	vLLM	Ollama	llama.cpp server
Best for	Agent loops, prefix-heavy multi-tool traffic	Concurrent throughput, broadest quant support	One dev, zero fuss	One dev who wants every knob
Default port / host	30000 / localhost-only	8000 / all interfaces	11434 / localhost-only	8080 / localhost-only
Tool calling	One flag + parser	Two flags + parser	Automatic, template-gated 400	--jinja flag
Context default	Model max (OOMs loud)	Model max (OOMs loud)	4K (truncates silent)	4K (truncates silent)
Prefix cache	RadixAttention, default-on, cross-request	Available	Per-session KV reuse	Per-slot cache
Quant formats	FP8/AWQ/GPTQ — compressed-tensors needs vLLM kernels	The reference for AWQ/GPTQ/FP8	GGUF	GGUF
The catch	Smaller ecosystem; the AWQ loader wrinkle	VRAM-greedy, Linux-first	Hides the knobs	You manage everything

When to skip SGLang

Skip it on Windows without WSL2, skip it on a Mac despite the Metal docs page, and skip it if you’re one person running one tool serially — the radix cache still helps there, but not enough to justify managing a Python inference server over ollama pull. It’s also the younger ecosystem of the two production engines: when a new quant format lands, vLLM tends to get the kernels first, as the compressed-tensors story above shows.

Take it seriously if your daily driver is Cline or Claude Code fanning out subagents, if multiple tools share one box, or if you’ve measured your local agent sessions and found time-to-first-token — not generation speed — is where the waiting lives. That’s the prefix problem, and this is the engine that treats it as the main event. No suitable GPU in the house? The same launch_server command on a RunPod instance with --host 0.0.0.0 --api-key gives every tool above a remote backend — swap localhost:30000 for the pod URL and nothing else changes.

FAQ

Does the base URL need /v1? Yes, for all three tools: http://localhost:30000/v1. SGLang also exposes a native /generate endpoint, but the coding tools never touch it.

Is RadixAttention something I have to configure? No — it’s on by default. If you’re benchmarking backends and want a fair single-request comparison, --disable-radix-cache turns it off; for actual agent use, leave it alone.

Why did my Qwen3-Coder AWQ download fail to load with “vllm is not installed”? That quant uses compressed-tensors formats whose kernels SGLang borrows from vLLM (issue #9838). Install vLLM into the same environment, or sidestep it with gpt-oss-20b’s native MXFP4 or Qwen’s official FP8 release.

Can multiple tools hit the same SGLang instance simultaneously? Yes — continuous batching plus the shared radix cache is the pitch. Parallel Cline subagents sharing a system prompt are the best case, since the common prefix is computed once and reused across all of them.

Do I need a different parser for a quantized variant? No. The parser matches the model family’s output format, not the precision — qwen3_coder covers every Qwen3-Coder variant, gpt-oss covers both gpt-oss sizes.

Sources

• SGLang README — sgl-project/sglang (GitHub)
• SGLang releases (v0.5.18, Aug 24 2026)
• SGLang install docs — official docs source (GitHub)
• SGLang quickstart — official docs source (GitHub)
• SGLang server arguments — official docs source (GitHub)
• SGLang tool parser — official docs source (GitHub)
• SGLang Qwen3-Coder cookbook recipe — official docs source (GitHub)
• Fast and Expressive LLM Inference with RadixAttention and SGLang — LMSYS Org
• Qwen3-Coder AWQ compressed-tensors loader issue #9838 — sgl-project/sglang
• gpt-oss day-0 support tracking issue #8833 — sgl-project/sglang
• gpt-oss-20b model card — OpenAI on Hugging Face
• Cline OpenAI Compatible provider docs
• Cline running models locally — official docs
• Continue.dev OpenAI provider (apiBase) docs
• Aider OpenAI-compatible API docs

Last updated August 27, 2026. Pricing and features change frequently; verify current state before purchasing.

Know which coding tool is worth paying for

Hands-on comparisons of AI coding assistants and what each one costs to run — including the local-model path. Sent only when something changes. Unsubscribe anytime.

Email address Subscribe