TL;DR: LiteLLM is a 58k-star, MIT-licensed gateway that puts every backend this series has covered — plus your cloud API keys — behind one OpenAI-compatible port. It launches nothing and manages no VRAM; what it adds is translation, retries, and the one thing no pure-local setup has: an automatic fallback to a cloud model when your local one fails. The setup trap is a three-character prefix.

What you’ll be able to do after this guide:

• Run the LiteLLM proxy on port 4000 with a config.yaml that names your local Ollama models and one cloud model, wired into Cline, Continue.dev, and Aider through a single base URL
• Avoid the ollama/ vs ollama_chat/ prefix mistake that silently routes chat requests to the wrong Ollama endpoint
• Add a five-line fallback so an agent session survives a crashed local server by failing over to a cloud key instead of dying mid-task

Honest take: If your local stack is one Ollama install and one tool, LiteLLM is a solution looking for a problem — connect directly. It earns its place the day your setup spans local and cloud: one config that holds your Ollama models next to a metered API key, one base URL in every tool, and fallbacks that turn “my GPU box hung” from a lost session into a few cents of cloud spend. Pair it with llama-swap and the two routers cover each other’s blind spots completely.

---

The sixteenth entry manages everything except a GPU

Last week’s llama-swap article broke this series’ pattern: a router instead of an engine, managing which llama-server process holds your VRAM. LiteLLM is the other kind of router, and the two are opposites in a useful way. llama-swap launches, stops, and swaps local server processes; it has no idea what a cloud API is. LiteLLM launches nothing and couldn’t unload a model if you asked — but it speaks to more than 100 providers, translates between API dialects, tracks what cloud calls cost, retries failures, and falls back across models. llama-swap manages your hardware. LiteLLM manages your providers.

The project needs less introduction than anything else this series has covered. Aider users have been running LiteLLM all along without installing it: the embedded library is how Aider talks to every backend, and it’s why Aider model strings carry prefixes like openai/ and ollama_chat/ in the first place. What this guide sets up is the other deliverable of the same codebase — the proxy server (BerriAI calls it an AI gateway), a standalone service your whole toolchain shares.

Version check for this writing: the repo sits at 58.1k GitHub stars, MIT-licensed (an enterprise/ directory carries a separate commercial license — none of it is needed for anything below), with v1.99.1 the latest stable, released September 2, 2026, and v1.101.0 pre-releases already landing. The release cadence is weekly or faster, something the last section returns to. One structural note for anyone who goes source-diving: the docs moved out of the main repo into BerriAI/litellm-docs, and a staged Rust rewrite (litellm-rust, an Axum-based gateway core) is underway — the project’s own README is explicit that Python still owns configuration, routing policy, and spend tracking until the Rust paths reach parity.

Step 1 — Install, config, first request

The proxy is a Python package with its own CLI. The README’s install path:

uv tool install 'litellm[proxy]'

(pipx works the same way; the [proxy] extra is what pulls the server dependencies.)

Everything the proxy does is driven by a config.yaml. Here’s the one this guide builds — two local Ollama models in the roles this series keeps recommending, one cloud escape hatch:

model_list:
  - model_name: coder-local
    litellm_params:
      model: ollama_chat/qwen3-coder:30b
      api_base: http://localhost:11434

  - model_name: autocomplete-local
    litellm_params:
      model: ollama_chat/qwen2.5-coder:1.5b
      api_base: http://localhost:11434

  - model_name: coder-cloud
    litellm_params:
      model: deepseek/deepseek-chat
      api_key: os.environ/DEEPSEEK_API_KEY

litellm_settings:
  drop_params: true
  num_retries: 2

router_settings:
  fallbacks: [{"coder-local": ["coder-cloud"]}]

Launch:

litellm --config config.yaml

The proxy listens on port 4000 by default. Smoke-test it the way you’d test any backend in this series:

curl http://localhost:4000/v1/models

curl http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"coder-local","messages":[{"role":"user","content":"say ok"}]}'

The first call should list coder-local, autocomplete-local, and coder-cloud as model IDs; the second should return a completion generated by your local Ollama install. If it errors instead, work through the connection-refused checklist one layer down — the proxy can only reach an Ollama that’s actually up.

Three parts of that config are load-bearing:

model_name is the public name; litellm_params.model is the routing string. Clients send coder-local; LiteLLM maps it to ollama_chat/qwen3-coder:30b and dispatches to the Ollama API. The split is the point — when you swap the underlying model next month, every tool config stays untouched. It’s the same aliasing idea llama-swap uses, one layer up.

os.environ/DEEPSEEK_API_KEY keeps secrets out of the file. That exact syntax — the literal string os.environ/, then the variable name — is how the config reads keys from the environment. Any provider works here; we picked DeepSeek because V4-Flash’s pricing makes it the cheapest credible fallback tier, but the same line shape holds an Anthropic, OpenAI, or Gemini key.

drop_params: true is cheap insurance. Tools sometimes send OpenAI-specific parameters that a downstream provider rejects with a 400. This flag tells LiteLLM to strip anything the target doesn’t understand instead of failing the request — the kind of mismatch that otherwise surfaces as a mystery error an hour into an agent session.

The three-character trap: ollama/ vs ollama_chat/

LiteLLM has two Ollama prefixes, and they are not interchangeable. ollama/qwen3-coder:30b routes to Ollama’s legacy /api/generate endpoint — raw text completion. ollama_chat/qwen3-coder:30b routes to /api/chat, the endpoint that understands message roles and chat templates. The official provider docs are one line on this: “We recommend using ollama_chat for better responses.”

For coding agents the recommendation is closer to a requirement. Cline and Aider live on multi-turn, role-structured conversations with tool calls; run those through a completion endpoint and you get subtly degraded formatting — mangled diffs, tool-call JSON in the wrong shape — the exact failure family the Cline tool-use loop fix and the “does not support tools” fix exist to debug. The cruel part is that ollama/ doesn’t error. It answers. It just answers worse, and nothing in the response says why. If a model behaves noticeably dumber through LiteLLM than through a direct Ollama connection, check the prefix before anything else. (On tool calling specifically, the docs note LiteLLM falls back to JSON-mode tool calls when a model lacks native support — a useful floor, but the model still has to be one that can follow the schema.)

The other Ollama discipline from this series still applies unchanged: LiteLLM forwards requests, it doesn’t fix Ollama’s context-length defaults. Set your context explicitly at the Ollama layer.

Step 2 — Wire in the three tools

One base URL for everything, and for once one of the tools has a dedicated door.

Cline (VS Code) is the special case: LiteLLM is a named provider in Cline’s API Provider dropdown, sitting alongside Anthropic, OpenAI, and OpenAI Compatible — it’s also the documented path for Cline’s enterprise remote-config setups. Pick LiteLLM, set Base URL http://localhost:4000, API key anything non-empty unless you’ve set a master key (below), Model ID coder-local — the model_name, exactly. If your build’s dropdown lacks the entry, OpenAI Compatible with Base URL http://localhost:4000/v1 behaves identically. Either way, set Cline’s context window to match what the underlying Ollama model actually has — the proxy doesn’t change the model’s limits, only its address.

Continue.dev (~/.continue/config.yaml) — chat and autocomplete through the same port, same shape as the rest of the series:

models:
  - name: gateway-agent
    provider: openai
    model: coder-local
    apiBase: http://localhost:4000/v1
    apiKey: sk-anything
    roles: [chat, edit, apply]

  - name: gateway-autocomplete
    provider: openai
    model: autocomplete-local
    apiBase: http://localhost:4000/v1
    apiKey: sk-anything
    roles: [autocomplete]

The standing caveat from the Cursor acquisition applies — the extension is frozen at v2.0.0 — but an OpenAI-compatible endpoint is exactly what it still speaks fluently.

Aider is where it gets briefly funny: you’re about to point one LiteLLM at another. Aider embeds the LiteLLM library as its provider layer, so hitting the LiteLLM proxy is just another OpenAI-compatible target:

export OPENAI_API_BASE=http://localhost:4000/v1
export OPENAI_API_KEY=sk-anything

aider --model openai/coder-local

The openai/ prefix tells Aider’s embedded LiteLLM “generic OpenAI dialect, at this base” — same recipe as every server in this series. There’s also a purpose-built alternative: the library ships a litellm_proxy/ prefix with its own environment variables (LITELLM_PROXY_API_BASE, LITELLM_PROXY_API_KEY), so aider --model litellm_proxy/coder-local declares the intent explicitly. Both work; the openai/ route has more mileage on it. Either way, a name mismatch between the model string and your model_name entries returns a 400 — the model-not-found guide applies, and the config file is the first place to look.

One more door worth knowing: the proxy exposes an Anthropic-format /v1/messages endpoint and routes it to any configured model, local ones included. Anthropic-native clients — the Claude Code local-model setup among them — get in without a translation shim, hitting the same model_name entries as everyone else.

The five lines that save an agent session

The router_settings block at the end of the config is the reason to pick LiteLLM over every other unifier in this series:

router_settings:
  fallbacks: [{"coder-local": ["coder-cloud"]}]

Read it as: when a request to coder-local fails — after the num_retries: 2 from litellm_settings — send the same request to coder-cloud instead. The client never sees the failure. Cline keeps its tool loop; Aider keeps its diff; what changes is that the completion came from DeepSeek’s API and cost a fraction of a cent instead of nothing.

Local-first developers should think of this as an availability policy, not a privacy leak — but do think of it, because it is one by design: a fallback means your prompt and its full repo context leave the machine whenever the local backend fails. If that’s never acceptable, set the fallback to a second local deployment instead (the syntax is the same — any model_name can back any other), or leave the block out entirely and keep LiteLLM as a pure unifier. If it’s acceptable during a crunch, it converts the classic local-stack failure mode — GPU box wedged at 2 AM, agent session dead — into a blip in the logs and a cloud line item.

Two refinements from the same settings family: context_window_fallbacks reroutes requests that exceed the local model’s context to something bigger (it needs enable_pre_call_checks: true alongside it), and default_fallbacks names a catch-all for any model group without its own rule. A cooldown system (allowed_fails, cooldown_time) stops a flapping backend from being hammered while it’s down.

What each router actually solves

	Direct connection	llama-swap	LiteLLM proxy	LiteLLM → llama-swap
Launches/unloads local servers	You, by hand	Yes	No	llama-swap does
Local + cloud behind one URL	No	No	Yes	Yes
Fallback when a backend dies	No	No	Yes	Yes
API dialects offered	Whatever the engine has	OpenAI + Anthropic	OpenAI + Anthropic (+ translation to 100+ providers)	Both layers
Cloud spend tracking	No	No	Yes	Yes
Runtime footprint	none	one Go binary	a Python service	both

The last column is the endgame stack: tools point at LiteLLM, LiteLLM’s local entries point at llama-swap’s port, llama-swap manages which llama-server actually holds the VRAM. Each layer does the one job the other can’t — provider policy up top, process management below. It’s more moving parts than most setups need, which is why the honest take up top says to start with neither.

What you’re deliberately not setting up

LiteLLM’s README leads with enterprise-gateway features: virtual API keys, per-team budgets, spend dashboards, an admin UI. All of it is real, and all of it requires a Postgres database (DATABASE_URL) plus a master_key in general_settings — which, per the docs, 🚨 must start with sk-. The moment you set a master key, every client config above needs it as the API key.

For a localhost coding stack, skip all of it. No database, no master key, listener on localhost — the same posture as every other backend in this series. The keyless proxy still does everything this article promised: unification, translation, retries, fallbacks. The day you’re provisioning keys for teammates, you’ve outgrown this guide’s scope and the enterprise docs take over.

Hardware, for once, is barely a factor: the proxy is a lightweight Python service that runs comfortably alongside your editor, and the real VRAM question lives a layer down with the models — a used RTX 3090 still being the 24 GB floor this series assumes for the 30B-class agent model, an RTX 4090 the faster version of the same capacity. The runaihome.com VRAM guide maps the tiers below that, aifoss.dev tracks the open-model side, and a rented RunPod GPU behind an Ollama install slots into the same model_list as just another api_base — which is precisely the point of the gateway.

Where it breaks

It manages providers, not processes. LiteLLM will happily route to a model that isn’t loaded, a server that isn’t running, a GPU that’s out of memory — and report the failure. Nothing here starts, stops, or swaps anything. That’s llama-swap’s job, or yours.

The prefix trap doesn’t announce itself. ollama/ where you meant ollama_chat/ produces working-but-worse behavior, not an error. It’s this article’s equivalent of llama-server’s --jinja flag: three characters that decide whether agents behave.

Release velocity cuts both ways. Weekly-or-faster releases, a docs repo that just moved, a Rust core (litellm-rust) being staged in under the Python one — this is the most actively-churning project in the series. The core proxy surface (config schema, /chat/completions, fallbacks) is years stable; the edges are not. Pin your version in anything you’d call production, and treat the Rust gateway as a thing to read about, not deploy, until the project itself says otherwise.

Python, not a static binary. llama-swap is one Go binary; LiteLLM is a Python service with dependencies. uv tool install isolates it well, but it’s a heavier tenant, and cold-start is slower. If all you want is model aliasing on one port with zero cloud involvement, llama-swap alone is the lighter answer.

Fallbacks are a privacy decision. Worth repeating outside the fallback section: a local→cloud fallback ships your context off-machine exactly when you’re least likely to be watching. Configure it deliberately or not at all.

Performance discipline, as always: no latency numbers we haven’t measured on hardware you don’t have. The architecture claim you can check yourself is that the proxy adds one local HTTP hop per request — inference time, which is all that matters at coding-agent scale, still belongs entirely to the backend underneath.

Verdict

LiteLLM is the last piece of the routing story this series started with llama-swap, and the two divide the work cleanly: llama-swap for anyone whose problem is “too many local server processes,” LiteLLM for anyone whose problem is “local and cloud don’t share a config.” One-backend, one-tool setups need neither. But if your reality is Ollama for daily work, a metered cloud key for emergencies, and three tools that each want a base URL — this is the config file that makes them one system, and the five-line fallback is the difference between a local-first stack and a local-only single point of failure. Set ollama_chat/, leave the database off, and the gateway disappears into the plumbing, which is the highest compliment infrastructure gets.

FAQ

Isn’t Aider already using LiteLLM? Why add the proxy? Aider embeds the LiteLLM library, which solves translation for Aider alone. The proxy lifts the same capability out of one tool and shares it across your whole toolchain — Cline, Continue.dev, Claude Code, and Aider all hit one config, one set of aliases, one fallback policy. If Aider is your only tool, the embedded library is genuinely enough.

Does LiteLLM replace llama-swap? No — they don’t overlap. llama-swap launches and swaps local server processes to fit your VRAM; LiteLLM routes across providers and API keys but never touches a process. Stacked (tools → LiteLLM → llama-swap → llama-server), each covers the other’s blind spot.

Do I need the master key and database? Not for a local coding stack. Virtual keys, budgets, and the spend dashboard require Postgres and a master_key (which must start with sk-); the keyless, database-free proxy on localhost does everything in this guide. Add the enterprise layer when you have teammates to provision, not before.

What happens to a request when my local model is down? Without router_settings, the client gets the error after num_retries attempts. With the fallback block, LiteLLM retries locally, then transparently re-sends the request to the fallback model — your agent session continues on the cloud model, and your prompt leaves the machine. That trade is configurable per model group, including falling back to a second local deployment instead.

Can Claude Code use this? The proxy exposes an Anthropic-format /v1/messages endpoint and routes it to any configured model, so Anthropic-native clients have a first-class door. Our Claude Code local-model guide covers the client-side environment variables; the server side is this article’s config with no changes.

Sources

• LiteLLM — GitHub repository and README
• LiteLLM releases (v1.99.1, September 2, 2026)
• LiteLLM proxy configuration reference — config.yaml structure
• LiteLLM Ollama provider — ollama vs ollama_chat
• LiteLLM proxy reliability — fallbacks, retries, cooldowns
• LiteLLM virtual keys — database and master key requirements
• LiteLLM litellm_proxy/ provider prefix
• Aider — OpenAI-compatible APIs
• Cline — LiteLLM provider configuration (enterprise remote config)

Last updated September 5, 2026. LiteLLM ships releases weekly; verify config keys and prefixes against the current docs before building on them.

Recommended Gear

Products linked in this guide:

• RTX 3090 24GB — the used-market 24 GB floor for the 30B-class local agent models this gateway fronts
• RTX 4090 — same capacity, roughly double the speed, for setups where the local lane is primary

Know which coding tool is worth paying for

Hands-on comparisons of AI coding assistants and what each one costs to run — including the local-model path. Sent only when something changes. Unsubscribe anytime.

Email address Subscribe