TL;DR: llama-swap is a single Go binary that sits in front of llama-server (or vLLM, or any OpenAI-compatible server) and starts, stops, and swaps models automatically based on the model field of each request. One port for every model you own — but the default one-model-at-a-time behavior will thrash if a chat agent and an autocomplete model share it, and the fix is a five-line groups block.

What you’ll be able to do after this guide:

• Install llama-swap (brew, winget, Docker, or a bare binary) and put every GGUF you own behind http://localhost:8080/v1, hot-swapped on demand
• Wire that one endpoint into Cline, Continue.dev, and Aider — including the agent-model + autocomplete-model pairing this series keeps recommending, running concurrently instead of evicting each other
• Set the two config values that separate a working agent setup from a mysteriously stalling one: healthCheckTimeout and a groups block with swap: false

Honest take: If you run exactly one local model, skip this article — llama-swap adds a config file and nothing else. The moment you run two (an agent model for Cline plus a fill-in-the-middle model for Continue autocomplete, or Aider’s architect/editor pair), it’s the missing coordinator for the whole llama-server side of this series: MIT-licensed, zero dependencies, and it deletes the second terminal window you’ve been keeping open.

---

The fifteenth entry in this series isn’t a backend

Fourteen servers in — Ollama, LM Studio, llama-server, vLLM, SGLang, KoboldCpp, LocalAI, Jan, Docker Model Runner, Lemonade, MLX LM, RamaLama, TabbyAPI — this series has a recurring, unsolved annoyance. The bare-metal engines (llama-server, vLLM, TabbyAPI) serve one model per process, so the two-model setup that agent coding actually wants — a 30B-class model for Cline’s tool loops, a 1.5B fill-in-the-middle model for Continue autocomplete — means two terminals, two ports, and two base URLs to keep straight. The app-shaped backends solved it their own way (Ollama auto-loads per request, RamaLama grew a router mode, MLX LM lazy-loads), but on the llama.cpp side you’ve been the model manager.

llama-swap is that manager. The GitHub description is exactly one clause long — “Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc” — and that’s the whole product: a Go proxy, MIT-licensed, 5.6k GitHub stars, that reads the model field of every incoming OpenAI-format request, checks whether the matching upstream server is running, and if not, stops the current one and launches the right one before forwarding. Your tools see a single, permanent endpoint; behind it, server processes come and go.

Version check for this writing: v252, released August 31, 2026 — four days old, in a project that has shipped ten releases since late July. That cadence sounds alarming; in practice the core proxy behavior is stable and the churn is in the web UI and the newer matrix routing engine (more on that below). It runs on Linux, macOS, Windows, and FreeBSD, and because it’s a router rather than an engine, everything the series has established about llama-server — the --jinja flag that decides whether tool calling works, the context-length defaults that break agents — still applies one layer down. llama-swap changes who launches the server, not what the server does.

Step 1 — Install the binary

Package managers carry it on every platform:

# macOS / Linux
brew tap mostlygeek/llama-swap
brew install llama-swap

# Windows
winget install llama-swap

Prebuilt binaries for Linux, macOS, Windows, and FreeBSD sit on the releases page, and there’s a Docker path with batteries included — the unified images bundle llama-server so the container is a complete stack:

docker pull ghcr.io/mostlygeek/llama-swap:unified-cuda
docker run -it --rm --runtime nvidia -p 9292:8080 \
  -v /path/to/models:/models \
  -v /path/to/config.yaml:/etc/llama-swap/config/config.yaml \
  ghcr.io/mostlygeek/llama-swap:unified-cuda

Images ship in unified-cuda, unified-vulkan, cpu, cuda, vulkan, intel, and musa flavors. For this guide we’ll assume the bare binary plus your existing llama-server install, because that’s the setup the rest of the series already built.

Launching is one line:

llama-swap --config config.yaml --listen localhost:8080

The third flag worth knowing is --watch-config, which reloads the config when the file changes — edit, save, and the next request uses the new settings, no restart.

Step 2 — The config that actually works for agents

Everything llama-swap does is driven by one YAML file. The minimum viable version is three lines, but the minimum agent-viable version needs more thought. Here’s the config this guide builds, then the reasoning:

healthCheckTimeout: 500

macros:
  llama-latest: >
    llama-server --host 127.0.0.1 --port ${PORT} --jinja -ngl 99

models:
  qwen3-coder-30b:
    cmd: |
      ${llama-latest}
      -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M
      -c 49152
    name: "Qwen3 Coder 30B (agent)"

  qwen2.5-coder-1.5b:
    cmd: |
      ${llama-latest}
      -hf bartowski/Qwen2.5-Coder-1.5B-Instruct-GGUF:Q4_K_M
      -c 8192
    name: "Qwen2.5 Coder 1.5B (autocomplete)"

groups:
  coding:
    swap: false
    exclusive: true
    members:
      - qwen3-coder-30b
      - qwen2.5-coder-1.5b

Walking through the load-bearing parts:

${PORT} is the whole trick. llama-swap assigns each upstream server its own port automatically (starting at 5800 by default, configurable via startPort) and substitutes it into the command. You never touch upstream ports again; clients only ever see llama-swap’s own listen address. If you followed our earlier llama-server guide, port 8080 now belongs to the proxy instead of the engine — your clients don’t change a thing.

macros keep the config honest. The shared flags — --jinja for tool calling, -ngl 99 for full GPU offload — live in one place. Forgetting --jinja on one model’s command line is exactly the kind of drift that produces the Cline tool-use loop an hour into a session.

The model keys are your API. qwen3-coder-30b isn’t a label; it’s the exact string clients must send in the model field. A request for a name that matches nothing returns an error, not a fallback — if Aider greets you with a model-not-found complaint, that guide applies, and the first thing to check is a typo between config key and client setting. An aliases list per model lets you also answer to names tools hardcode (the docs’ own example: alias a local model as gpt-4o-mini and legacy clients Just Work).

healthCheckTimeout: 500 is not optional for big models. The default is 120 seconds and the minimum is 15: llama-swap holds the incoming request, waits for the upstream’s /health endpoint to go green, then forwards. An 18.6 GB download-size model loading from a busy disk can blow past 120 seconds, and when it does the held request fails — which your agent reports as a mysterious API error. Five hundred seconds costs nothing when loads are fast and saves the session when they aren’t. (The wait isn’t silent either: a sendLoadingState option streams loading progress into the response’s reasoning field, so you can see the swap happening from inside the client.)

Both models fit a 24 GB card together. The 30B MoE at Q4_K_M is an 18.6 GB download and our llama-server guide ran it solo at 65,536 context on an RTX 3090-class card; the 1.5B sidecar adds roughly a gigabyte of weights plus its small KV cache, so the agent model’s context drops to 49,152 here to keep headroom. Check nvidia-smi after both load and tune from there — and if your card is smaller, the runaihome.com VRAM guide maps what each tier can hold, with aifoss.dev tracking the open-model landscape. An RTX 4090 clears it comfortably; renting a 24 GB RunPod instance runs everything here unmodified if you’d rather try before buying.

Which leaves the groups block — the reason this article exists.

The group config that stops swap thrash

llama-swap’s default behavior is in its name: one model at a time, swapped on demand. Send a request for qwen3-coder-30b, it loads. Send one for qwen2.5-coder-1.5b, the 30B is stopped and the 1.5B loads. That default is right for the “I own six models and try them one at a time” workflow.

It is catastrophically wrong for the two-model coding setup. Picture it without the groups block: Cline is mid-task on the 30B, thinking through a tool call. You glance at another file and type a line. Continue’s autocomplete fires a request for the 1.5B. llama-swap dutifully kills the 30B mid-session, loads the 1.5B, serves your ghost text — and then Cline’s next request swaps the whole 18.6 GB back in. Every keystroke-triggered completion buys you a multi-second (or with healthCheckTimeout doing its job, multi-minute) round of musical chairs. The proxy is doing exactly what you configured; you configured thrash.

Here’s the same setup under three management strategies:

	Two bare llama-server terminals	llama-swap, no groups	llama-swap + swap: false group
Ports/base URLs to manage	2	1	1
Chat + autocomplete concurrent	Yes (manually)	No — every model change is a full swap	Yes
Cost of an autocomplete request mid-agent-task	None	Unload 30B → load 1.5B → reload 30B (tens of seconds to minutes each way)	None
VRAM needed	Both models resident	One model at a time	Both models resident
Add a third experimental model	Third terminal, third port	Free — swapped in on demand	Free — swapped outside the group (exclusive: true evicts it when coding resumes)

The fix is the five lines above. In a group with swap: false, all members run simultaneously — the 30B and the 1.5B each keep their own llama-server process and their own VRAM slice, both reachable through the same port, routed by model name. exclusive: true means activating this group unloads anything outside it, so your weekend experiment with some 20 GB roleplay model doesn’t linger in VRAM when Monday’s coding session starts.

Two refinements the docs support:

• persistent: true on a group makes its members immune to eviction by other groups. Put the 1.5B autocomplete model alone in a persistent group and you can keep swapping big agent models freely — ghost text never pays a reload.
• ttl (per model, seconds) auto-unloads after idle time. Sensible on the experiment shelf, counterproductive on your daily agent model — set ttl: 0 (the default: never unload) for anything Cline drives, or a long lunch break turns into a cold reload.

There’s also a newer, expression-based matrix routing engine that can encode fancier concurrency rules than groups. It’s been landing features since late July (v244 added symbolic expression solving) and is exactly the kind of moving surface the operational-limits section of a production setup should avoid for now. Groups are simpler, older, and cover the coding case completely.

Step 3 — Wire in the three tools

Same OpenAI dialect as the rest of the series, one difference: the base URL is now permanent, whatever model is behind it.

Cline (VS Code): API Provider → OpenAI Compatible. Base URL http://127.0.0.1:8080/v1, API key anything non-empty (llama-swap works — the field is Cline’s requirement, not the server’s, unless you configure llama-swap’s optional apiKeys bearer auth), Model ID qwen3-coder-30b — the config key, exactly. Set Cline’s context window to match the -c value (49,152 here), the same discipline the Ollama num_ctx fix drilled.

Continue.dev (~/.continue/config.yaml) — this is where the pairing pays off, chat and autocomplete through one endpoint:

models:
  - name: local-agent
    provider: openai
    model: qwen3-coder-30b
    apiBase: http://127.0.0.1:8080/v1
    apiKey: llama-swap
    roles: [chat, edit, apply]

  - name: local-autocomplete
    provider: openai
    model: qwen2.5-coder-1.5b
    apiBase: http://127.0.0.1:8080/v1
    apiKey: llama-swap
    roles: [autocomplete]

The standing caveat from the Cursor acquisition applies — the extension is frozen at v2.0.0 — but it speaks this API without complaint, and qwen2.5-coder remains the fill-in-the-middle pick from our autocomplete troubleshooter.

Aider (terminal):

export OPENAI_API_BASE=http://127.0.0.1:8080/v1
export OPENAI_API_KEY=llama-swap

aider --model openai/qwen3-coder-30b

The openai/ prefix routes litellm to the generic OpenAI provider; the rest must match the config key. Aider is also where llama-swap’s swapping earns its keep on a single GPU: the project’s own wiki documents an architect-mode recipe pairing a reasoning model (architect) with a coder model (editor), and on one card llama-swap simply swaps between them as Aider alternates requests — slow per turn, but a two-big-model workflow that a bare llama-server can’t do on 24 GB at all. With two GPUs, the wiki’s version pins each model to its own card via per-model env entries (CUDA_VISIBLE_DEVICES=0 and 1) in a concurrent group, and the swap cost disappears.

Smoke-test the whole stack before blaming any client:

curl http://localhost:8080/v1/models
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-coder-30b","messages":[{"role":"user","content":"say ok"}]}'

Expected from the first call (trimmed) — both configured models, whether loaded or not:

{"object":"list","data":[{"id":"qwen3-coder-30b","object":"model", ...},{"id":"qwen2.5-coder-1.5b","object":"model", ...}]}

The second should trigger a visible model load on first call, then answer. The built-in web UI at http://localhost:8080/ui shows what’s loaded, streams logs, and has manual load/unload buttons — genuinely useful when deciding whether a stall is the model loading or the agent looping.

What else the proxy buys you

A few capabilities that no single-engine backend in this series offers, all verified against the current README and config reference:

• Any engine, same port. cmd is an arbitrary command line. Front vLLM for batch throughput and llama-server for low-latency chat in the same config; the README’s advice for Python-based servers (vLLM, TabbyAPI) is to run them via Docker/Podman for clean environment isolation and SIGTERM handling, with a cmdStop entry to stop the container gracefully.
• An Anthropic-dialect door. Alongside the OpenAI surface, llama-swap exposes /v1/messages and /v1/messages/count_tokens — the same API family our Claude Code + Ollama guide targets, so Anthropic-native clients have a route in without a translation shim.
• llama-server passthrough. /infill, /completion, /v1/rerank, and /props pass through, and /upstream/:model_id gives direct access to a specific running server — handy for llama-server’s own web UI.
• Observability. Prometheus metrics at /metrics, a /running endpoint for scripting, /logs streaming, and per-request token metrics in the UI. The SGLang article called ops visibility a differentiator; llama-swap backfills it for the whole llama.cpp fleet.

Where it breaks

It’s a router, not an engine. Token speed, quality, tool-calling fidelity — all unchanged, all still llama.cpp’s (or whatever you put behind it). llama-swap adds a Go proxy hop that’s noise next to inference time; what it actually adds is swap latency, and no config makes an 18.6 GB load instant. The group config above doesn’t make swapping fast — it makes the hot path never swap.

Model names are load-bearing. Every client must send exactly a config key or alias. Rename a key in config.yaml and every tool pointing at the old name breaks at the next request. Keep keys short, stable, and boring.

The default is the trap. Out of the box, two coding tools sharing the proxy will thrash, as above. The failure looks like “my local setup randomly hangs for a minute,” which is why this article leads with the groups block rather than burying it.

Reverse proxies buffer SSE. The README calls out nginx specifically: response buffering must be disabled for streaming endpoints or chat completions stall. Only relevant if you put llama-swap behind another proxy — on localhost, not your problem.

Ten releases in six weeks. The rapid cadence is mostly UI and the matrix engine, and the project has no equivalent of TabbyAPI’s “hobby project” disclaimer — but pin a version if your workflow depends on newer surfaces like matrix routing or the MCP endpoint at /api/mcp.

Performance claim discipline, as always: no tokens-per-second numbers we haven’t measured on hardware you don’t have. The checkable architecture claim is that steady-state throughput through the proxy tracks the bare llama-server numbers on the same hardware, with load cost paid once per swap rather than never or always.

Verdict

llama-swap is the first entry in this series that makes the other entries better instead of competing with them. If your local coding stack is one model on one port, you don’t need it. If it’s the setup this series has been steering toward — a 30B-class agent model plus a small FIM model, maybe a vLLM lane for batch work — it replaces two or three hand-managed terminals with one config file, one port, and one binary that restarts crashed upstreams’ whole lifecycle on demand. Ollama and LM Studio users already get management from their apps and should stay put; llama-server, vLLM, and TabbyAPI users should install this the same afternoon they add a second model. Set healthCheckTimeout: 500, write the groups block before the thrash finds you, and the two-model coding stack finally behaves like one server.

FAQ

Does llama-swap replace Ollama? Functionally it overlaps — Ollama also loads models per request behind one port. The difference is control and reach: llama-swap runs the real llama-server with every flag exposed (no silent context defaults), and it can front vLLM, TabbyAPI, or anything else with an OpenAI-compatible surface. If Ollama’s defaults have never bitten you, stay; if you left Ollama for bare llama-server, this restores the convenience you gave up.

What happens to a request that arrives mid-swap? llama-swap holds it, waits for the new upstream’s health check to pass (up to healthCheckTimeout seconds), then forwards. With sendLoadingState enabled, loading progress streams into the response’s reasoning field so the client shows something instead of nothing.

Can Cline and Continue autocomplete really share one GPU? Yes, with the swap: false group above and enough VRAM for both models — about 20 GB of weights for the 30B Q4_K_M plus the 1.5B sidecar, which is why the agent model’s context drops to 49,152 on a 24 GB card. Without the group, the two tools evict each other on every request; that’s the thrash this article exists to prevent.

Does it work with Claude Code? llama-swap exposes Anthropic-dialect endpoints (/v1/messages, /v1/messages/count_tokens) alongside the OpenAI ones. Our Claude Code local-model guides cover the client-side environment variables; the server side here is the same base-URL swap.

Is there authentication? Optional bearer-token auth via an apiKeys list in the config — off by default, like most of this series. Keep the listen address on 127.0.0.1 unless you’ve turned it on.

Sources

• llama-swap — GitHub repository and README
• llama-swap Configuration reference — project wiki
• llama-swap releases (v252, August 31, 2026)
• llama-swap full config example — docs/config.example.yaml
• Aider + QwQ + Qwen-Coder recipe — llama-swap wiki
• llama.cpp llama-server — GitHub
• Qwen2.5-Coder-1.5B-Instruct GGUF — Hugging Face (bartowski)

Last updated September 4, 2026. llama-swap ships releases weekly; verify current flags and config keys against the repository before building on them.

Recommended Gear

Products linked in this guide:

• RTX 3090 — the used-market 24 GB workhorse that fits the 30B + 1.5B pairing in this guide
• RTX 4090 — same 24 GB capacity, roughly double the inference speed for the impatient

Know which coding tool is worth paying for

Hands-on comparisons of AI coding assistants and what each one costs to run — including the local-model path. Sent only when something changes. Unsubscribe anytime.

Email address Subscribe