TL;DR: LocalAI is the aggregator of this series — one MIT-licensed server that fronts llama.cpp, vLLM, MLX, and 60+ other backends behind a single OpenAI-compatible API on localhost:8080. Its model gallery ships coding models with tool calling already wired, which no other backend here does. The catch lives in the context ceiling: bare models default to 512 tokens and even gallery configs stop at 8,192 — both too small for agent sessions until you raise one YAML line.

What you’ll be able to do after this guide:

• Pull a tool-calling coding model with one command (local-ai run qwen3-coder-30b-a3b-instruct) and skip the parser-flag homework that vLLM and SGLang require
• Raise the shipped 8K context to an agent-workable 32K+ — and know why the default silently truncates a Cline session if you don’t
• Wire the endpoint into Cline (OpenAI Compatible provider), Continue.dev (openai provider with apiBase), and Aider (openai/ prefix), with the v4.9.0 auth change accounted for

Honest take: LocalAI is the right pick if you want one always-on server that does chat, embeddings, transcription, and image generation alongside your coding agent — a home-server appliance, not a hot rod. If all you run is one coding model for one tool, Ollama gets you there with fewer moving parts, and if you’re chasing raw agent-loop throughput on an NVIDIA card, vLLM or SGLang are the engines. LocalAI’s edge is breadth and prewired defaults, not speed.

---

The tenth backend is really sixty of them

This series has walked through Ollama, LM Studio, llama.cpp’s llama-server, vLLM, Docker Model Runner, Jan, KoboldCpp, and SGLang. LocalAI predates most of them, and it takes the opposite architectural bet from every one of them.

The others are inference engines. LocalAI orchestrates inference engines. The README’s pitch is “Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required,” and the mechanism is a Go server that manages a stable of backends — llama.cpp, vLLM, MLX, whisper.cpp, diffusers, and enough others that the project counts 60+ — behind one API surface. That surface isn’t just OpenAI-shaped either: the project advertises “drop-in API compatibility: OpenAI, Anthropic, and ElevenLabs APIs across every backend.”

It’s MIT-licensed, and it moves fast: v4.9.0 landed August 20, 2026, eight days before this writing, with v4.8.2 and v4.8.1 earlier the same month. For a coding-tool backend, three v4.9.0 items matter and we’ll hit each: authentication went deny-by-default, the model gallery crossed 1,700 entries, and an opt-in context-compression feature arrived that’s aimed squarely at long agent sessions.

Step 0 — Platform check

This is the least demanding platform story in the series. Docker containers cover Linux, macOS, and Windows, with dedicated images for NVIDIA CUDA, AMD ROCm, Intel GPUs, and Vulkan; macOS gets a native DMG app; Linux gets bare binaries with a documented systemd socket-activation setup that starts the server on the first connection to port 8080. And because the default backend is llama.cpp running GGUF quants, CPU-only genuinely works — slowly, but it works, which is not something you can say for vLLM or SGLang.

The hardware math doesn’t change, though: for the models worth pointing an agent at, you still want 16–24 GB of VRAM, and everything below was written against an RTX 3090-class card. What actually fits in a given VRAM budget is runaihome.com’s territory — their local AI models by VRAM guide is the pre-purchase read.

Step 1 — Install, launch, sanity-check

Docker is the recommended route and the one-liner is boring in the good way:

# CPU-only
docker run -p 8080:8080 --name local-ai -ti localai/localai:latest

# NVIDIA GPU
docker run -p 8080:8080 --gpus all --name local-ai -ti \
  localai/localai:latest-gpu-nvidia-cuda-12

AMD (latest-gpu-hipblas), Intel (latest-gpu-intel), and Vulkan (latest-gpu-vulkan) tags exist too. On bare metal it’s a single binary from GitHub Releases — chmod +x local-ai-* and run it. Either way the server comes up on localhost:8080 with a WebUI at the root and the OpenAI API under /v1.

Then pull a model. This is where LocalAI’s gallery earns its keep — the same registry the WebUI browses is scriptable from the CLI:

local-ai run qwen3-coder-30b-a3b-instruct

That resolves against a gallery of 1,707 models (the v4.9.0 count), downloads the weights — for this entry, the unsloth Q4_K_M GGUF of Qwen3-Coder-30B — and installs a YAML config with templates, stop words, and tool-calling behavior already set. local-ai models list and local-ai models install <name> do the same job non-interactively, and direct references like local-ai run huggingface://<repo>/<file>.gguf or even ollama:// URIs work when the gallery doesn’t have what you want.

Sanity-check the endpoint like every other backend in this series:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-coder-30b-a3b-instruct",
       "messages": [{"role": "user", "content": "Say ready."}]}'
# → {"object":"chat.completion","choices":[{"message":{"role":"assistant","content":"Ready."}}],...}

No API key needed on a fresh localhost install — more on what changed there in a moment.

Step 2 — Tool calling: prewired, with a story behind it

Cline’s agent loop and Continue.dev’s agent mode live and die on function calling, and every backend in this series has made you do something for it — vLLM wants two flags, SGLang wants a parser name, llama-server wants --jinja, Ollama gates you behind its template scan and throws the 400 “does not support tools” error when it disapproves.

LocalAI’s answer is per-backend parsers configured per-model, and the gallery does the configuring for you. On the llama.cpp backend, tool calls are extracted by an incremental C++ parser; on vLLM, the gallery importer “looks up the model family and pre-fills tool_parser: and reasoning_parser:”; on MLX, the parser is auto-detected from the chat template.

The gallery configs are worth reading because they encode real debugging history. The base config that qwen3-coder-30b-a3b-instruct inherits does two non-obvious things, with the project’s own comments explaining why. It sets use_jinja so llama.cpp’s runtime owns templating — “without use_jinja the autoparser falls back to a ‘pure content’ PEG parser that leaks reasoning tags into content” (issue #9985). And it explicitly disables LocalAI’s own grammar layer:

function:
  grammar:
    disable: true
template:
  use_tokenizer_template: true

— because otherwise “the generated grammar overrides [llama.cpp’s native tool pipeline] and the tool-call JSON leaks into content” (issue #10052). That symptom — the model narrating JSON instead of acting — is exactly the failure mode we dissected in the Cline + Ollama tool-loop fix, and here the fix ships in the config. The gpt-oss config does the same dance for the harmony format, delegating to llama.cpp’s jinja runtime so the <|channel|> sections get classified into reasoning_content, content, and tool_calls instead of bleeding channel tags into your diff.

If you bring your own model file instead of using the gallery, all of that is on you: the function: block exposes grammar controls, response_regex extraction, and experimental parallel_calls, and getting it wrong produces the narrated-JSON failure. The practical rule: prefer gallery entries for agent work, and treat hand-rolled configs as the advanced path.

Step 3 — The context ceiling: 512 bare, 8,192 from the gallery

Every backend in this series has a context personality. Ollama and llama-server default small and truncate silently; vLLM and SGLang default to the model’s maximum and crash loudly at startup. LocalAI is in the silent-truncation camp, twice over:

• A bare model config defaults to context_size: 512 — not 4K, 512. Point Cline at that and the system prompt alone overflows it many times over.
• The gallery’s coding-model configs set context_size: 8192 — sane for chat, still short for agents. Cline’s system prompt plus file context runs 10–20K tokens before the first user word; an 8K window means the session is being clipped from turn one, and the model’s confusion looks like tool-calling flakiness rather than what it is.

The fix is one line in the model’s YAML, which lives in your models directory (or is editable in the WebUI’s config editor — note that a context change there requires a restart to take effect, per issue #8647):

context_size: 32768

There’s also context_size: -1, which auto-uses “the model’s full trained context from GGUF metadata (raw max, no VRAM capping; a warning is logged if it may not fit detected VRAM)” — the docs’ own words. For Qwen3-Coder’s native 262,144 tokens on a 24 GB card, -1 is how you re-create vLLM’s out-of-memory startup experience with extra steps. Set an explicit number: 32768 is the working agent floor, 65536 if your VRAM leaves room after weights.

Related and new in v4.9.0: opt-in per-model context compression, which compresses older conversation turns through a local model before inference while preserving “leading system/developer prompts, the newest messages and complete tool-call/result units.” That design — tool-call units survive intact — is aimed at exactly the traffic agent tools generate. It’s brand new and off by default; treat it as an experiment for marathon sessions, not a substitute for a real context window.

Step 4 — Models that fit

Both of the series’ standard picks are in the gallery, preconfigured:

VRAM	Gallery name	Weights file	Notes
16 GB	gpt-oss-20b	ggml-org MXFP4 GGUF	OpenAI’s Apache-2.0 MoE (21B total, 3.6B active); MXFP4 is the model’s native training precision, and the config handles the harmony format for you
24 GB	qwen3-coder-30b-a3b-instruct	unsloth Q4_K_M GGUF	The agentic-coding pick (30.5B total, 3.3B active, 256K native context); Apache 2.0; raise context_size as above
CPU-only / low VRAM	smaller gallery entries	GGUF	Workable for chat and Aider-style diff editing; agent loops get painful below ~10 tokens/s

Because the default backend is llama.cpp, the quant universe is GGUF — same files Ollama and KoboldCpp use. The vLLM backend is there if you want AWQ/FP8 serving under LocalAI’s roof, but at that point run vLLM directly unless you specifically want LocalAI’s gallery and multi-modal endpoints around it. No suitable GPU at all? The same Docker image on a RunPod instance turns every config below into a remote backend — swap localhost:8080 for the pod URL and nothing else changes. The open-source serving stack around all of this is aifoss.dev’s beat.

Step 5 — Wire in the three tools

Cline: OpenAI Compatible provider

• Base URL: http://localhost:8080/v1
• API Key: anything non-empty if you haven’t set one; the real key if you have (below)
• Model ID: the gallery name — qwen3-coder-30b-a3b-instruct
• Model Configuration: set the context window to match your raised context_size, so Cline’s truncation math matches the server’s reality

Cline’s four fields — provider, base URL, key, model ID — and its own docs’ troubleshooting order (“the problem is almost always the Base URL, the key, or the Model ID, in that order”) apply unchanged. Turn on Use Compact Prompt for local models, same as every backend in this series.

Continue.dev: openai provider with apiBase

models:
  - name: Qwen3-Coder 30B (LocalAI)
    provider: openai
    model: qwen3-coder-30b-a3b-instruct
    apiBase: http://localhost:8080/v1
    apiKey: localkey-123
    roles: [chat, edit, apply]

The apiBase includes /v1. Continue autodetects tool support; if agent mode’s tools stay greyed out, add capabilities: [tool_use] — additive, harmless to declare.

Aider: OpenAI-compatible env vars

export OPENAI_API_BASE=http://localhost:8080/v1
export OPENAI_API_KEY=localkey-123
aider --model openai/qwen3-coder-30b-a3b-instruct

The openai/ prefix is Aider’s LiteLLM provider prefix, straight from Aider’s OpenAI-compatible docs — and for once the model name after it is short, because LocalAI’s gallery names are clean aliases rather than full Hugging Face repo paths. Add a .aider.model.metadata.json for context and cost metadata exactly as in the vLLM guide to silence the unknown-model warnings.

The v4.9.0 auth change, in one minute

Until August, LocalAI’s HTTP surface used a protected-prefix allowlist — routes not on the list were public. v4.9.0 inverted it: “a route is public only if its method and path appear in an explicit public registry,” which closed a bypass where “unprefixed aliases such as /moderations, /models, /backends and /mcp/chat/completions fell outside the old protected-prefix list.”

What it means for this setup: a fresh localhost install with no keys configured behaves as before — the curl above works bare. But the moment you set LOCALAI_API_KEY=<key> (single or comma-separated), everything requires it, including /version and generated-asset URLs that used to slip through, so every tool config above needs the real key in its API-key field. Two more things worth knowing: legacy API keys “grant full admin access - there is no role separation,” and if the box serves more than one person, LOCALAI_AUTH=true switches on actual user accounts with roles and per-user keys. If you expose port 8080 beyond localhost without one of these, you’re serving an admin API to your network.

LocalAI vs. the field

	LocalAI	Ollama	llama.cpp server	vLLM / SGLang
Best for	One server, many modalities, prewired agent configs	One dev, zero fuss	One dev, every knob	Throughput and agent loops
Default port	8080	11434	8080	8000 / 30000
Tool calling	Per-backend parsers, gallery-preconfigured	Automatic, template-gated 400	--jinja flag	Parser flags
Context default	512 bare / 8,192 gallery (truncates silent)	4K (truncates silent)	4K (truncates silent)	Model max (OOMs loud)
Beyond LLMs	Whisper, TTS, images, video, embeddings, Anthropic-API compat, MCP agents	LLMs + embeddings	LLMs + embeddings	LLMs
The catch	The most YAML in the series; wrapper adds a layer to debug	Hides the knobs	You manage everything	NVIDIA-first, Python stacks

When to skip LocalAI

Skip it if your entire local-AI life is one coding model feeding one tool — Ollama does that with less surface area, and the Ollama error-fix series exists because even less surface area generates plenty. Skip it for maximum tokens-per-second on a dedicated NVIDIA box; that’s vLLM and SGLang’s contest. And know that the aggregator design cuts both ways: when something misbehaves, you’re debugging LocalAI’s config layer and the backend under it.

Take it seriously if you want a single always-on endpoint that serves your coding agent today and transcription, embeddings for a RAG setup, or image generation tomorrow — or if the gallery’s prewired tool-calling configs are worth more to you than another evening spent reading parser-flag docs. On a homelab box that several tools and family members share, deny-by-default auth plus real user accounts is a genuinely better security story than anything else in this series. That’s the niche: not the fastest backend, the most complete one.

FAQ

Does the base URL need /v1? Yes, for all three tools: http://localhost:8080/v1. The WebUI lives at the root of the same port.

Do I need an API key for localhost use? Not on a fresh install with no keys configured. Once LOCALAI_API_KEY is set, every request needs it — v4.9.0’s deny-by-default change removed the leaky exceptions. Cline requires a non-empty key field either way; any placeholder works when the server has no key.

Why does my agent session act confused after a few turns? Almost certainly context truncation. Gallery configs ship context_size: 8192; raise it to 32768 in the model’s YAML and restart. Bare (non-gallery) model configs default to 512, which no agent survives.

My model narrates JSON tool calls instead of executing them — what broke? A hand-rolled config where LocalAI’s grammar layer fights the backend’s native tool pipeline (issue #10052). Use the gallery entry for your model family, or replicate its function: grammar: disable: true + use_tokenizer_template: true pairing.

Can I use quant formats other than GGUF? Yes — LocalAI can drive vLLM and MLX backends among its 60+, and gallery vLLM entries arrive with tool parsers prefilled. For a dedicated AWQ/FP8 serving box, though, running vLLM directly is fewer layers.

Sources

• LocalAI README — mudler/LocalAI (GitHub)
• LocalAI releases (v4.9.0, Aug 20 2026)
• LocalAI v4.9.0 release notes — auth deny-by-default, context compression, gallery count
• LocalAI quickstart — official docs source (GitHub)
• LocalAI Linux install — official docs source (GitHub)
• LocalAI model configuration (context_size defaults, function block) — official docs source (GitHub)
• LocalAI OpenAI functions / tool calling — official docs source (GitHub)
• LocalAI authentication — official docs source (GitHub)
• LocalAI model gallery index (gpt-oss-20b, qwen3-coder-30b-a3b-instruct entries)
• LocalAI qwen3 gallery base config (grammar disable, use_jinja, issues #9985/#10052)
• LocalAI harmony gallery base config (gpt-oss, context 8192)
• WebUI context_size change requires restart — issue #8647, mudler/LocalAI
• Cline OpenAI Compatible provider docs
• Continue.dev OpenAI provider (apiBase) docs
• Aider OpenAI-compatible API docs

Last updated August 28, 2026. Pricing and features change frequently; verify current state before purchasing.

Know which coding tool is worth paying for

Hands-on comparisons of AI coding assistants and what each one costs to run — including the local-model path. Sent only when something changes. Unsubscribe anytime.

Email address Subscribe