TabbyAPI as a Local Backend for Cline, Continue.dev, and Aider in 2026: EXL3 VRAM Efficiency, the 4K cache_size Trap, and Real API Keys

By AICoderScope Team · 14 min read
tabbyapiexllamav3clinecontinue-devaiderlocal-llmsetup-guide

TL;DR: TabbyAPI is the official OpenAI-compatible server for ExLlamaV3, whose EXL3 quants squeeze more model quality into a 24GB card than any GGUF pipeline in this series. The cost: NVIDIA-only, a cache_size default of 4,096 tokens that lobotomizes coding agents, and — uniquely here — an API key your clients must actually get right.

What you’ll be able to do after this guide:

  • Install TabbyAPI, download an EXL3 quant of Qwen3.8-27B with the built-in downloader, and serve it on port 5000 with a context window agents can actually use
  • Wire the server into Cline (OpenAI Compatible provider), Continue.dev (provider: openai + apiBase), and Aider (openai/ prefix + OPENAI_API_BASE) — with the real generated key, because placeholders bounce off this one
  • Turn on structured tool calling with tool_format, quantize the KV cache to buy context, and know exactly when to walk away (AMD card, Mac, GGUF library)

Honest take: On an NVIDIA card, TabbyAPI is the quality-per-gigabyte king of this series — EXL3 is the only quant format here that credibly holds a dense 27B and an agent-sized context inside 24GB without stepping down the model. But it’s a hobby project by its own README, it speaks only EXL3, and three config keys stand between the quick start and a working Cline session. Tinkerers with a 3090 or 4090 should run it; everyone else keeps Ollama or LM Studio.

The fourteenth backend is the VRAM specialist

Thirteen servers into this series — Ollama, LM Studio, llama-server, vLLM, SGLang, KoboldCpp, LocalAI, Jan, Docker Model Runner, Lemonade, MLX LM, and RamaLama — all but two have been fronts for the same llama.cpp engine and the same GGUF files. TabbyAPI is the third genuine engine change, and the most opinionated: it is “the official API backend server for ExllamaV3,” a FastAPI app that serves turboderp’s ExLlamaV3 inference library over an OpenAI-compatible API, AGPLv3-licensed, with a README that sets expectations in one line: “TabbyAPI is a hobby project made for a small amount of users. It is not meant to run on production servers.”

Why give a hobby project a slot in this series? Because of what the engine underneath does with your VRAM. EXL3 is a quantization format based on Cornell RelaxML’s QTIP work, and turboderp’s headline claim is that “Llama-3.1-70B-EXL3 is coherent at 1.6 bpw” — with the output layer at 3 bpw, that’s a 70B running “in under 16 GB of VRAM.” Treat the low-bpw end as a tech demo, not a daily driver. The practical version of the claim matters more: at the 3–4 bpw range people actually use, EXL3 holds model quality at bitrates where GGUF K-quants have historically fallen apart, which means either a bigger model or a bigger context on the same card.

Our Qwen3.8-27B guide shows why that trade is live. At GGUF Q4_K_M the weights alone run 17–19GB, and on a 24GB card the KV cache — not the model — becomes the squeeze: 24–32K context is the realistic ceiling, and sub-4-bit GGUF quants cost too much quality to be the escape hatch. The same model as a 3.0 bpw EXL3 is arithmetic-simple to size: 27.78B parameters × 3 bits ≈ 10.4GB of quantized layers (EXL3 repos keep the embeddings unquantized, but they sit in system RAM, not VRAM). That’s roughly 7–8GB of freed VRAM on the same card — all of it available as KV cache for the long agent sessions Cline actually runs. Whether 3.0 bpw EXL3 matches Q4_K_M quality on your codebase is something you verify yourself in an afternoon; the format’s design goal, per the ExLlamaV3 README, is exactly that kind of headroom.

Version check for this writing: TabbyAPI publishes no tagged releases — you run rolling main, which last landed commits on September 2, 2026 (tool-parser and reasoning-stream work, as it happens). The engine underneath is exllamav3 1.4.6, released September 2, 2026 on PyPI, requiring PyTorch with CUDA 12.4 or later. Both halves are moving weekly; this is an actively maintained stack, just one that ships from the tip of a branch.

Before you invest: this is an NVIDIA-only guide by hardware fact, not by our choice. ROCm sits on ExLlamaV3’s to-do list, unshipped. AMD GPU? Lemonade is your article. Apple Silicon? MLX LM. And parallel batching via paged attention needs an Ampere GPU or newer — a used RTX 3090 or an RTX 4090 clears the bar, a GTX 1080 Ti does not. The runaihome.com VRAM guide maps what each tier of card buys you; if you’d rather rent before buying, a RunPod 24GB instance runs everything below unmodified.

Step 1 — Install and meet your API keys

Python 3.12 with pip, git, and a CUDA-capable driver are the prerequisites (the wiki warns against Microsoft Store Python on Windows). The start scripts handle the venv and dependencies:

git clone https://github.com/theroyallab/tabbyAPI
cd tabbyAPI
./start.sh          # Windows: .\start.bat

There’s also a Docker path — CUDA images are published to GitHub Container Registry — but the start scripts are the documented mainline, and they double as the updater on later runs.

First launch prints something no other backend in this series prints:

Your API key is: 4f6c...<32 hex chars>
Your admin key is: 9a1e...<32 hex chars>

Both are generated with secrets.token_hex(16) and stored in api_keys.yml next to the config. This is the series’ first real authentication layer: the API key covers generation endpoints, the admin key covers model loading and unloading, and the server accepts them via x-api-key, x-admin-key, or a standard Authorization: Bearer header — which is what makes off-the-shelf OpenAI clients work. Write down the API key now; every client section below needs it. (There is a disable_auth switch in the config, flagged in the sample file with “WARNING: This will make your instance vulnerable!” — with the default bind of 127.0.0.1 and a real key already generated for you, there’s no good reason to touch it.)

Step 2 — Download an EXL3 quant

TabbyAPI only loads EXL3 (and unquantized FP16/BF16) — ExLlamaV2’s exl2 format was dropped from main and lives on a legacy branch, and GGUF is explicitly someone else’s job (YALS, TabbyAPI’s llama.cpp-based sister project). So you need EXL3 weights, and the built-in downloader knows how Hugging Face repos shelve them — one bitrate per branch:

./start.sh download turboderp/Qwen3.8-27B-exl3 --revision 4.0bpw

That drops the weights into models/ under the repo name. turboderp’s own uploads — Qwen3.8-27B-exl3 among them — publish a ladder of revisions from around 2 to 8 bpw; check the repo’s branch list and pick the largest that leaves your context budget intact. On a 24GB card with a 27B dense model, 3.0–4.0 bpw is the sensible window: 4.0 bpw lands near 14GB of layer weights, 3.0 near 10.4GB, and every gigabyte you don’t spend on weights becomes KV cache. Our pick for agent work stays consistent with the Qwen3.8-27B guide: this is the strongest open-weight coder in the 24GB class, and EXL3 is the format that lets it breathe there.

The catch worth naming: the EXL3 catalog is a fraction of the GGUF catalog. turboderp and a handful of community quantizers cover the majors fast, but if your model of choice has no EXL3 branch, your options are converting it yourself (the ExLlamaV3 README puts conversion at “a couple of minutes for smaller models, up to a few hours” for 70B+ on a decent GPU) or staying on a GGUF backend.

Step 3 — The config that actually works for agents

Copy config_sample.yml to config.yml and set five keys:

model:
  model_name: Qwen3.8-27B-exl3   # folder name under models/
  max_seq_len: 32768
  cache_size: 32768               # multiple of 256; THE trap — see below
  cache_mode: Q8                  # or "8,8" — k_bits,v_bits pairs from 2-8
  tool_format: qwen3_coder        # structured tool calls for agent clients

Each line un-breaks something:

cache_size — the 4K default that eats your agent. This is the series-signature trap wearing its fourth costume. The sample config documents cache_size as “Size of the key/value cache to allocate, in tokens (default: 4096)” — and max_seq_len, left unset, resolves to “min(max_position_embeddings, cache_size)”. Follow that arithmetic: out of the box, your 262K-native Qwen serves 4,096 tokens of effective context. A Cline system prompt plus one file read blows past that before the agent does anything, and you get the amnesia spiral we’ve documented since the Ollama num_ctx article — the model re-reads files, forgets the plan, loops. Ollama called it num_ctx, llama-server calls it --ctx-size, RamaLama hid it behind ctx_size: 0, TabbyAPI calls it cache_size. Set both keys, keep them equal (cache_size must be a multiple of 256), and start at 32K.

cache_mode — where EXL3 pays rent. FP16 KV cache is the default and the quality baseline, but this engine lets you name any k_bits,v_bits pair from 2 to 8 — finer-grained than llama.cpp’s fixed menu. Q8 (or 8,8) roughly halves cache VRAM and is the conventional safe point; our Ollama OOM guide covers why coding agents degrade before chat does under aggressive KV quantization, and that logic transfers unchanged. The winning 24GB combination is a 3.0–4.0 bpw model plus Q8 cache at 32K — not FP16 cache at 16K.

tool_format — without it, your agent narrates instead of acting. TabbyAPI revamped tool calling and it “no longer relies on modified Jinja templates” — the server parses model output and returns structured tool_calls — but it only engages when you name a format matching your model family. The wiki’s current list covers qwen3_coder (with aliases including qwen3_5), glm4_5 (aliases include glm4_6 and glm4_7), deepseek_v4, harmony, muse_glimmer, hy_v3, mistral, minimax_m2, and gemma4. For reasoning models, pair it with reasoning: true plus the model’s think tokens — the wiki recommends enabling both together. Skip tool_format and you’re in the failure mode of the Cline + Ollama tool-loop bug: tool-call JSON rendered as chat text, nothing executed. One documented caveat if you’re on European weights: “Older Mistral models tend to have unreliable tool calling support.”

Relaunch ./start.sh and sanity-check — note the key header, which no other backend in this series has needed:

$ curl http://127.0.0.1:5000/v1/models -H "x-api-key: YOUR_API_KEY"
{"object":"list","data":[{"id":"Qwen3.8-27B-exl3","object":"model", ...}]}

Copy that id exactly; it’s the model string for all three clients.

Step 4 — Wire in the three tools

Same OpenAI dialect as the rest of the series, one real difference: the key field finally means something. Where every prior entry accepted ollama or any non-empty placeholder, TabbyAPI verifies. Paste the actual API key from first launch (the admin key also passes generation calls, but the API key is the right scope to hand to clients).

Cline (VS Code): API Provider → OpenAI Compatible. Base URL http://127.0.0.1:5000/v1, API Key = your generated key, Model ID Qwen3.8-27B-exl3. Set Cline’s context window to match your cache_size. A 401 here means the key, not the URL — see the FAQ.

Continue.dev (~/.continue/config.yaml):

models:
  - name: tabby-qwen38
    provider: openai
    model: Qwen3.8-27B-exl3
    apiBase: http://127.0.0.1:5000/v1
    apiKey: YOUR_API_KEY
    roles: [chat, edit, apply]

The standing caveat from the Cursor acquisition applies: the extension is frozen at v2.0.0, stable-but-static. It authenticates and streams against TabbyAPI without complaint.

Aider (terminal):

export OPENAI_API_BASE=http://127.0.0.1:5000/v1
export OPENAI_API_KEY=YOUR_API_KEY   # a placeholder fails here — real key
aider --model openai/Qwen3.8-27B-exl3

The openai/ prefix routes litellm to the OpenAI-compatible provider. Aider’s “Unknown context window size and costs, using sane defaults” warning is cosmetic; the model-not-found guide covers silencing it with .aider.model.settings.yml.

What the engine gives you that llama.cpp doesn’t

Three features worth knowing exist, even if you configure none of them on day one. Speculative decoding is first-class: draft_mode accepts a separate draft model, mtp (multi-token prediction), or ngram — free tokens-per-second for code, which is speculative decoding’s best case, since code is predictable text. Tensor parallelism (tensor_parallel: true) splits one model across multiple GPUs with native or NCCL backends — the dual-3090 crowd’s feature, matching what vLLM offers with less ceremony. And CPU MoE offload (cpu_moe_split_experts) keeps a MoE model’s coldest experts in system RAM with dynamic placement — the config that makes big sparse models like Qwen3.8-Flash-Next plausible on a single card. There’s also inline_model_loading for switching models per-request and use_dummy_models for clients that insist on seeing gpt-3.5-turbo in /v1/models; leave both off until a client forces the issue.

Where it breaks

NVIDIA or nothing. ROCm is on the to-do list, and to-do lists aren’t roadmaps with dates. This is the hardest wall in the series — even KoboldCpp runs on Vulkan.

The EXL3-only diet. Your GGUF library means nothing here, and model coverage depends on quantizers keeping pace with releases. For the majors — Qwen, GLM, Mistral, Gemma — turboderp’s uploads have been prompt. For the long tail, expect to convert or skip.

No releases, by design. Rolling main, a hobby-project disclaimer, and update-by-git-pull. The flip side we verified: commits landed the day before this writing, and the tool-calling and reasoning support tracks new model families fast — the format list above includes parsers for models that shipped this summer.

Auth cuts both ways. The real API key is a genuine security upgrade over the twelve key-less backends before it — and a new failure mode, because every 401 in your agent’s log is now a config error someone has to chase (it’s the FAQ’s first entry for a reason).

Performance claim discipline, as always: no tokens-per-second numbers we haven’t measured on hardware you don’t have. The sizing arithmetic above is checkable, the bpw-quality claims are turboderp’s and labeled as such, and the fair comparison is architecture-level: same model, same card, EXL3 at 3–4 bpw simply leaves more VRAM for context than GGUF Q4_K_M. What that headroom is worth is your codebase’s call.

Verdict

TabbyAPI is the right backend for one clearly-drawn developer: NVIDIA 24GB-class card, comfortable editing a YAML file and reading a wiki, and hungry for either more model or more context than GGUF fits in the same silicon. For that person it’s the best quality-per-gigabyte story in this series, with tool calling that genuinely works for Cline once tool_format is set. It is not the beginner pick — that remains Ollama and LM Studio — and it is not the production pick, by its own README’s word. It’s the enthusiast’s backend, from the developer who has defined local-inference enthusiasm since ExLlama v1. For the FOSS-stack view of these tools, aifoss.dev tracks the open-source side of local AI tooling.

FAQ

Cline shows a 401 / “invalid API key” — what do I check? You’re sending a placeholder, or the wrong key. Open api_keys.yml in the TabbyAPI folder and copy the api_key value into the client. TabbyAPI accepts it as Authorization: Bearer (what OpenAI clients send) or x-api-key. Placeholder keys like ollama — series habit — are exactly what this backend rejects.

Does it run on AMD or Apple Silicon? No. ExLlamaV3 requires CUDA 12.4+; ROCm support is listed as to-do, and there’s no Metal path. AMD users: Lemonade. Mac users: MLX LM.

Can I load my GGUF files? No — EXL3 and unquantized HF weights only. The maintainers point GGUF users to YALS, their llama.cpp-based sister project, or you can stay with llama-server directly.

What happened to exl2 models? “ExLlamaV2 models are no longer supported in the main branch” — a legacy branch keeps the old backend alive, unmaintained. Re-download your daily models as EXL3; the quality-per-bit is better anyway.

Can I use it with Cursor? Same answer as the whole series: Cursor’s BYOK routing goes through Cursor’s servers, so a loopback URL won’t reach it. The Cursor + Ollama guide covers the tunnel workaround, which applies here unchanged — mind that you’d be exposing an authenticated but hobby-grade server to the internet to do it.

Sources

Last updated September 3, 2026. Pricing and features change frequently; verify current state before purchasing.

  • RTX 3090 — the used-market 24GB Ampere card that clears the paged-attention bar and runs the 27B + 32K combination above
  • RTX 4090 — same VRAM, roughly double the memory bandwidth, for when generation speed on a dense 27B starts to grate

Was this article helpful?

Know which coding tool is worth paying for

Hands-on comparisons of AI coding assistants and what each one costs to run — including the local-model path. Sent only when something changes. Unsubscribe anytime.