TL;DR: vLLM serves an OpenAI-compatible API on localhost:8000 that Cline, Continue.dev, and Aider all speak natively — and under concurrent agent load it outperforms Ollama and llama.cpp by design, not by tuning. The cost: tool calling needs two flags (--enable-auto-tool-choice plus the right --tool-call-parser), and vLLM’s memory defaults are inverted from Ollama’s, so your first launch will likely crash instead of quietly truncating context.

What you’ll be able to do after this guide:

• Serve a tool-calling coding model with one vllm serve command, with the correct parser for your model family (qwen3_xml for Qwen3-Coder, hermes for Qwen2.5, llama3_json for Llama)
• Fix the startup out-of-memory error that hits almost everyone who moves from Ollama, by understanding why vLLM preallocates 90% of your GPU and defaults context to the model’s full maximum
• Wire the endpoint into Cline (OpenAI Compatible provider), Continue.dev (native vllm provider), and Aider (openai/ prefix) with configs that survive agent mode

Honest take: For one developer on one GPU, Ollama or llama.cpp remains the saner default — vLLM’s throughput advantage only shows up under concurrency. But if two or more people (or two or more parallel agents) hit the same box, vLLM is the correct answer and nothing in the wrapper category comes close. Set it up once, put it behind an API key, and it behaves like a hosted provider that happens to cost you electricity.

---

Why bother with vLLM when Ollama already works

This is the fourth local-backend engine we’ve covered, after Ollama, LM Studio, and llama.cpp’s llama-server. vLLM is a different animal from all three: it’s the serving engine that inference providers actually run in production, built around PagedAttention and continuous batching so that many simultaneous requests share the GPU efficiently instead of queueing.

That distinction decides who should care:

1. Parallel agents are concurrency. The single-user assumption quietly died in 2026. Cline running three subtasks, Claude Code spawning subagents, you and Continue.dev’s autocomplete hammering the same endpoint — that’s concurrent load, and it’s exactly the case where Ollama and llama-server interleave requests on a single GPU while vLLM batches them through together.
2. Team boxes. A shared workstation or homelab server with a real GPU, serving three developers’ Cline sessions, is vLLM’s home turf. Put --api-key on it and every tool in this guide connects exactly as it would to a paid provider.
3. No GGUF ceiling. vLLM loads safetensors weights straight from Hugging Face — original releases, FP8 checkpoints, AWQ/GPTQ quants. Day-one model support tends to land in vLLM before GGUF conversions stabilize; current stable is v0.27.1 (August 11, 2026), one day behind a major release that added same-week support for the newest open-weight coding models.

The trade is real, though. vLLM assumes Linux and an NVIDIA/AMD data-center-or-desktop GPU, preallocates most of your VRAM at startup whether you use it or not, and has no model library UI at all. If you close the lid of your laptop at night, this is not your tool.

Step 0 — Platform check, honestly

vLLM’s official install targets are Linux with NVIDIA CUDA, AMD ROCm, Intel XPU, and Google TPU, plus CPU paths including Apple Silicon. Windows is not officially supported — the workable routes are WSL2 (full GPU passthrough via the WSL CUDA driver) or Docker Desktop’s Model Runner with the WSL2 backend. A community fork (SystemPanic/vllm-windows) ships native Windows wheels, but it trails upstream and we wouldn’t build a workflow on it. On macOS, vLLM runs but the performance story belongs to MLX-based tools — use LM Studio there instead.

So the realistic audience: a Linux box (or WSL2) with an NVIDIA card, 16 GB VRAM minimum. On a RTX 3090 or 4090 class card, everything below applies unmodified.

Step 1 — Install and first launch

The official quickstart uses uv, and it’s genuinely the least painful path:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

--torch-backend=auto detects your CUDA version and pulls matching PyTorch wheels. AMD cards use --extra-index-url https://wheels.vllm.ai/rocm/ instead.

Then serve a model. This is the full command we’ll build toward — a 24 GB card running Qwen3-Coder with working tool calls:

vllm serve QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --max-model-len 32768 \
  --api-key localkey-123

First launch downloads the weights from Hugging Face (~16 GB here), then compiles and warms up — expect a couple of minutes before you see the Uvicorn line:

INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Sanity-check it like any OpenAI-compatible endpoint:

curl http://localhost:8000/v1/models -H "Authorization: Bearer localkey-123"
# → {"object":"list","data":[{"id":"QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ", ...}]}

Note the model ID is the full Hugging Face repo path — you’ll need it verbatim in every tool config below. The --api-key flag (or VLLM_API_KEY env var) guards the /v1 routes; skip it on a strictly-local setup, but set it the moment the port is reachable from anywhere else.

Step 2 — The two flags that decide whether agents work

Cline and Continue.dev’s agent mode require tool calling (function calling). vLLM supports it well — but not by default, and this is the single biggest difference from Ollama’s behavior. Two flags:

• --enable-auto-tool-choice — per the official docs, this “tells vLLM that you want to enable the model to generate its own tool calls when it deems appropriate.” Without it, tool definitions in the request are ignored and Cline’s agent loop stalls on the first step.
• --tool-call-parser <name> — each model family emits tool calls in its own wire format, and vLLM needs to be told which parser to run over the output. Wrong parser = tool calls come back as plain text in the message content, which is the same “model narrates JSON instead of acting” failure we dissected in the Cline tool-use loop fix.

The parser matrix for coding-relevant models, from vLLM’s tool-calling docs as of today:

Model family	--tool-call-parser value
Qwen3-Coder (30B-A3B, 480B-A35B)	qwen3_xml
Qwen2.5, QwQ-32B	hermes
Llama 3.1 / 3.2 / 4	llama3_json (plus a chat-template file for 3.1)
DeepSeek-V3 family	deepseek_v3 / deepseek_v31
GLM 4.5 / 4.7	glm45 / glm47
Mistral	mistral
Kimi K2	kimi_k2

The docs are also unusually candid about model-side limits: Llama 3’s smaller variants “frequently fail to emit tool calls in the correct format,” and Mistral 7B struggles with parallel calls. The parser can only decode what the model produces — an under-trained model with the right parser still loops. Qwen3-Coder and gpt-oss are the two families we’ve consistently seen drive real agent sessions locally.

This is Ollama’s capability gate turned inside out, by the way. Ollama scans the chat template and rejects requests with a 400 if it decides the model can’t do tools — the error we covered in the “does not support tools” fix. vLLM never refuses; it just silently does nothing useful if you forgot the flags. Pick your failure mode.

Step 3 — Survive the startup OOM

Here’s the trap that catches nearly everyone arriving from Ollama. Ollama’s infamous default is a context that’s too small — 4,096 tokens, silently truncating agent prompts (the subject of our context-length fix). vLLM inverts it: the default context is the model’s full maximum, and vLLM preallocates KV-cache memory for it up front, budgeted against --gpu-memory-utilization (default 0.9 — 90% of VRAM claimed at startup).

Qwen3-Coder’s native maximum is 256K tokens. Weights at ~16 GB plus a 262,144-token KV cache does not fit in the ~21.6 GB that 0.9 × 24 GB allows, so instead of Ollama’s quiet truncation you get a hard crash at startup — an error of the shape (exact wording varies by version):

ValueError: The model's max seq len (262144) is larger than the maximum
number of tokens that can be stored in KV cache. Try increasing
gpu_memory_utilization or decreasing max_model_len when initializing the engine.

The fix is the --max-model-len 32768 you already saw in the launch command. The official memory-conservation docs say it plainly: “You can further reduce memory usage by limiting the context length of the model.” 32K is the same working floor we recommend for every local agent backend — Cline’s system prompt plus file context eats 10-20K before the model says a word. If you have headroom, 65536 is a comfortable ceiling for a 24 GB card with AWQ weights; you’re trading KV space against concurrent request capacity either way.

One more habit from Ollama that must die here: vLLM holds its 90% VRAM claim for the lifetime of the process, so you can’t run it casually alongside a game or a second model the way Ollama’s load-on-demand lets you. Lower --gpu-memory-utilization to 0.7-0.8 if the GPU also drives your displays — and if the card starts falling back to system RAM under other load, that’s a different failure with its own troubleshooter.

Step 4 — Models that fit, without GGUF

vLLM doesn’t load GGUF as its native path — you serve safetensors, and quantization comes as AWQ/GPTQ/FP8 checkpoints or the model’s own native format. Verified picks by VRAM tier:

VRAM	Model (serve argument)	Weights	Parser	Notes
16 GB	openai/gpt-oss-20b	~12-13 GB (native MXFP4)	openai	21B MoE, 3.6B active; OpenAI post-trained it in MXFP4 so this is the official precision, not a lossy conversion. Fits, with modest KV room — keep --max-model-len at 32K
24 GB	QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ	~16 GB (4-bit AWQ)	qwen3_xml	The agent-coding default. Honesty required: 4-bit AWQ measurably degrades this MoE vs. the original — community quant cards say so themselves. It’s still the strongest tool-calling coder at this tier
48 GB+	Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8	~31 GB (official FP8)	qwen3_xml	Qwen’s own FP8 release, near-lossless; the right pick on an RTX 6000-class card or 2×24 GB with --tensor-parallel-size 2

If those sizes read as aspirational, the hardware side of this decision is runaihome.com’s beat — their best local AI models by VRAM guide covers what fits before you spend anything, and aifoss.dev tracks the open-source serving stack side.

Step 5 — Wire in the three tools

Cline: OpenAI Compatible provider

Cline’s local-models documentation covers only Ollama, LM Studio, and Atomic Chat — vLLM isn’t mentioned, so the supported route is the OpenAI Compatible provider, same as for llama-server:

• Base URL: http://localhost:8000/v1
• API Key: whatever you passed to --api-key (anything non-empty if you didn’t)
• Model ID: the full repo path, e.g. QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ
• Model Configuration: set the context window to match your --max-model-len so Cline’s truncation math is honest

Two settings that matter: check Computer Use in the model configuration if you want browser tooling (it’s gated on function-calling support, which you’ve now enabled), and turn on Use Compact Prompt in Settings → Features — Cline’s own local-model docs recommend it, and a smaller system prompt leaves more of your 32K for actual code.

Continue.dev: native vllm provider

Continue (1.3.40 as of today) is the only tool of the three with a first-class vLLM provider — it exists specifically because vLLM’s model-listing response uses results where OpenAI uses data, and the provider absorbs that quirk:

models:
  - name: Qwen3-Coder 30B (vLLM)
    provider: vllm
    model: QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ
    apiBase: http://localhost:8000/v1
    apiKey: localkey-123
    roles: [chat, edit, apply]

Note the apiBase includes /v1 — the exact opposite of Continue’s llama.cpp provider, which wants the bare port. If you keep both backends configured, this asymmetry will bite you exactly once. Agent mode gates on the tool_use capability, which Continue autodetects; if agent tools stay greyed out, add capabilities: [tool_use] to the block (capabilities are additive — they can’t break autodetection).

Continue is also where vLLM earns extra rent: point an autocomplete role at the same server with a small model in router-style setups, and continuous batching means tab-completion spam doesn’t starve your chat requests the way it does on single-slot servers.

Aider: OpenAI-compatible env vars

Aider has no vLLM provider and needs none:

export OPENAI_API_BASE=http://localhost:8000/v1
export OPENAI_API_KEY=localkey-123
aider --model openai/QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ

The openai/ prefix tells Aider’s LiteLLM layer to treat it as a generic OpenAI-compatible endpoint. You’ll get the familiar warning about unknown context window and costs — Aider doesn’t recognize arbitrary local models and falls back to conservative defaults. Silence it with a .aider.model.metadata.json in your project or home directory:

{
  "openai/QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ": {
    "max_input_tokens": 32768,
    "max_output_tokens": 8192,
    "input_cost_per_token": 0,
    "output_cost_per_token": 0
  }
}

Aider’s edit-format loop is less tool-call-dependent than Cline’s — it works via search/replace blocks — so it’s the most forgiving of the three if your model’s tool calling is shaky. If the blocks themselves fail to apply, that’s its own known failure mode, unrelated to vLLM.

The four-backend decision table

	vLLM	Ollama	llama.cpp server	LM Studio
Best for	Concurrent agents, shared boxes, teams	One dev, zero fuss	One dev who wants every knob	One dev who wants a GUI
Model format	safetensors, AWQ/GPTQ/FP8	GGUF	GGUF	GGUF + MLX
Tool calling	Two flags, per-family parser	Automatic, but gated by template scan	--jinja flag	Automatic
Context default	Model max (OOMs loud)	4K (truncates silent)	4K (truncates silent)	4K per-model setting
VRAM behavior	Preallocates ~90% up front	Loads/unloads on demand	Allocates at launch	Loads/unloads via GUI
Windows	WSL2 only	Native	Native	Native
The catch	Linux-first, VRAM-greedy	Wrapper hides the knobs	You manage everything	Closed-source shell

When to skip vLLM

Skip it if you’re one person on a Windows gaming PC — WSL2 works but you’re maintaining a Linux environment to avoid Ollama’s defaults, which we’ve already documented workarounds for. Skip it on a Mac entirely. And local models on any backend still lose to hosted frontier models on long agentic chains — the verdict from our Cursor + Ollama guide hasn’t moved.

Take it seriously if the GPU is shared, the agents are parallel, or the box is headless. And if the card you need doesn’t exist in your house: a RunPod instance running the exact vllm serve command above, plus --api-key, gives every tool in this guide a remote backend that behaves identically — change localhost:8000 to the pod URL and nothing else.

FAQ

Does the base URL need /v1? Cline, Aider, and Continue’s vllm provider: yes, http://localhost:8000/v1. That’s one inconsistency fewer than the llama.cpp setup — but remember Continue’s llama.cpp provider omits it, if you run both.

Why does vLLM crash at startup when Ollama ran the same model fine? Ollama defaulted your context to 4K and truncated quietly; vLLM defaults to the model’s full maximum (256K for Qwen3-Coder) and allocates KV cache for it honestly, which doesn’t fit. Set --max-model-len 32768. Loud beats silent — this crash is the 30-second fix, the truncation was the afternoon-long mystery.

Can I serve GGUF files with vLLM? There’s experimental support, but it’s the wrong tool — GGUF’s home is llama.cpp and its wrappers. On vLLM, use AWQ/GPTQ/FP8 safetensors checkpoints.

Can Cline, Continue, and Aider hit the same vLLM instance simultaneously? Yes — that’s the point. Continuous batching processes concurrent requests together instead of interleaving them. Each additional in-flight request consumes KV-cache space from the same pool, so heavy parallel use is another reason to keep --max-model-len modest.

Do I need a different parser for the AWQ quant vs. the original model? No. The parser matches the model family’s output format, not the quantization. qwen3_xml applies to every Qwen3-Coder variant.

Sources

• vLLM quickstart — official docs (GitHub)
• vLLM tool calling documentation — official docs (GitHub)
• vLLM OpenAI-compatible server — official docs (GitHub)
• vLLM conserving memory — official docs (GitHub)
• vLLM releases (v0.27.1, Aug 11 2026)
• gpt-oss-20b — vLLM recipes
• Qwen3-Coder-30B-A3B-Instruct-AWQ — QuantTrio on Hugging Face
• Cline OpenAI Compatible provider docs
• Cline running models locally — official docs
• Continue.dev vLLM provider docs
• Aider OpenAI-compatible API docs

Last updated August 21, 2026. Pricing and features change frequently; verify current state before purchasing.