Jan as a Local Backend for Cline, Continue.dev, and Aider in 2026: The Open-Source Desktop Server With an 8K Context Catch

By AICoderScope Team · 14 min read
janclinecontinue-devaiderlocal-llmsetup-guidetool-calling

TL;DR: Jan is the fully open-source desktop model server — Apache 2.0, 44.1k GitHub stars, llama.cpp underneath — and its OpenAI-compatible API on 127.0.0.1:1337 drives Cline, Continue.dev, and Aider with the same three-line configs as every other backend in this series. Since v0.8.0 a router process loads models on demand, so your coding tool’s first request just works. The catch: an 8,192-token default context that agent workloads outgrow immediately.

What you’ll be able to do after this guide:

  • Turn Jan’s desktop app into a local model server on http://127.0.0.1:1337/v1, with the router (v0.8.0+) loading and unloading models on demand instead of making you babysit a Start button
  • Raise the per-model 8K default context before it silently breaks Cline’s agent loop, and dodge the three config mistakes that make Continue.dev report a dead connection to a perfectly healthy server
  • Wire the endpoint into Cline (OpenAI Compatible provider), Continue.dev (openai provider with apiBase), and Aider (openai/ prefix) — plus keep the server bound to 127.0.0.1, because Jan’s 0.0.0.0 mode shipped a real security hole this June

Honest take: If you want a GUI model manager and open source is a requirement — auditable code, no license terms shifting under you — Jan is now the default pick over LM Studio, and v0.8.0’s router erased most of the UX gap. If you don’t care about the license, LM Studio’s coding-tool integration docs are still a smoother ride, and headless Ollama remains the least-fuss option of all.

The open-source seat in the desktop-app tier

Six engines into this series — Ollama, LM Studio, llama.cpp’s llama-server, vLLM, and Docker Model Runner — a pattern is clear: the desktop-GUI tier belonged to LM Studio, and LM Studio is closed-source. Jan is the answer to exactly that objection. The whole app is Apache 2.0 on GitHub (janhq/jan, 44.1k stars as of today), the engine is llama.cpp like almost everything else here, and the current release is v0.8.4, shipped July 23, 2026.

Jan runs on Windows 10+, macOS 13.6+, and Linux, pulls GGUF models straight from Hugging Face (Llama, Gemma, Qwen, gpt-oss — anything in the format), and on Apple silicon it added MLX models to the Local API Server’s model picker in v0.8.2. It also ships MCP support and optional cloud-provider bridges (OpenAI, Anthropic, Mistral, Groq), which makes it a plausible single control panel for a hybrid stack. None of that matters to Cline, Continue, or Aider, though. What matters to them is one thing: the OpenAI-compatible server on port 1337, and whether it behaves.

Mostly, it behaves. The two releases that changed Jan’s story as a coding backend are v0.8.0 (May 22, 2026), which replaced per-model server processes with a single llama.cpp router that loads and unloads models on demand — no more manually starting a model before your editor can talk to it — and v0.8.2 (June 1, 2026), which raised the default context from 4,096 to 8,192 tokens and added AMD ROCm (HIP) support. One of those numbers is still a trap; we’ll get there.

Step 0 — Enable the server and prove it’s up

In Jan, open Settings → Local API Server. Enable the server and note the defaults, verified against Jan’s docs today: host 127.0.0.1, port 1337, both changeable. The API key field accepts any string — it’s local auth only, and clients pass it as a standard Authorization: Bearer <key> header. Set one even locally (call it jan), because two of the three tools below refuse to save a config without something in the key field anyway.

Then check the endpoint the way every OpenAI-compatible client will see it:

curl http://127.0.0.1:1337/v1/models -H "Authorization: Bearer jan"
# → {"object":"list","data":[{"id":"Qwen3-Coder-30B-A3B-Instruct", ...}]}

Whatever appears in that id field is the exact model ID your coding tools must use — copy it from here, not from memory. Jan’s IDs come from the model name in your library, and a near-miss ID is one of the quietest ways to get a 404 that looks like a connection problem.

The base URL for everything below is http://127.0.0.1:1337/v1 — standard /v1 prefix, unlike Docker Model Runner’s /engines/v1 oddity. Chat completions, models, and streaming all hang off it.

Leave the server address on 127.0.0.1. Jan offers 0.0.0.0 for serving other devices on your network, and until recently that toggle was more dangerous than the UI let on: issue #8453 (reported June 13, 2026) found that binding to 0.0.0.0 silently replaced the Trusted Hosts allowlist with a wildcard — any Host header, any Origin, no DNS-rebinding protection — while the CORS toggle in the UI wasn’t wired to the backend at all. The fix landed in PR #8506; if you have any reason to expose Jan beyond localhost, be on v0.8.4 and set an API key. For a coding backend there is no reason: your editor and your server share a machine.

Step 1 — Pick a model the router can actually serve an agent with

Jan’s Hub pulls from Hugging Face directly, so the model menu is the whole GGUF ecosystem rather than a curated namespace. The picks for agent coding are the same ones that earned their place across this series, because the engine underneath is the same llama.cpp:

HardwareModelWhy
16 GB VRAM / unified memorygpt-oss-20bOpenAI’s 20B-class MoE in MXFP4; the strongest tool-caller at this tier
16 GB, efficiency pickQwen3-Coder-Next3B active parameters, huge native context — our review covers why it punches up
24 GBQwen3-Coder-30B-A3B or Qwen3.8-27BThe 24 GB agent-coding standards; see the Qwen3.8-27B local-backend piece

Q4_K_M remains the quantization sweet spot to start from. For the hardware side — which card, how much memory, whether a 30B-class MoE is realistic on your box — runaihome.com’s best-local-models-by-VRAM guide is the companion read; a used RTX 3090 still covers the whole table.

What’s genuinely different in Jan is what happens after the download. Since v0.8.0 the app runs one llama-server router process (--models-preset router.preset.ini --models-max <N>) that loads and unloads models on demand. When Cline sends its first request naming a model that isn’t in memory yet, the router loads it and answers — you never pre-start anything. The visible cost is that the first request after an idle stretch takes the full model-load time, which some clients read as a hung request. If Cline’s first message of the morning sits on a spinner for twenty seconds and then everything is instant, that’s the router doing its job, not a bug.

Step 2 — Tool calling: auto-detected, and honest about it

Cline’s agent loop and Continue’s agent mode need OpenAI-style function calling, and Jan’s chat-completions endpoint supports it, with streaming and multi-turn. Capability handling is Jan’s quiet strength here: the app auto-detects whether a model supports tool use (shipped back in v0.6.9, with tool-call fixes as recent as v0.7.4’s “changing tools while conversing” bug), and per-model Jinja chat templates are editable in the model settings if a GGUF ships a broken one — the exact knob raw llama-server makes you handle with --jinja and template flags, surfaced as a text box.

The standing caveat from every backend in this series still applies: a correct template doesn’t make a weak model a good agent. A 7B chat model will emit malformed calls under Cline’s pressure and the failure looks like the tool-use loop — the model narrating JSON instead of acting. Stick to the table above. And unlike Ollama, which rejects tool requests against non-tool models with a clean 400 (the “does not support tools” error), a mismatched model in Jan degrades quietly, llama-server-style. If the agent stops acting and starts describing, suspect the model before the plumbing.

Step 3 — The 8K context: better than 4K, still not agent-ready

Jan raised its default per-model context (ctx_len) from 4,096 to 8,192 in v0.8.2. Credit where due — that’s double the default that Ollama, llama-server, and Docker Model Runner still ship. It is also nowhere near enough. Cline’s system prompt plus tool definitions plus a couple of open files clears 8K before the model produces a token, and then you get the same slow-motion failure we dissected for Ollama’s 4K default: the agent forgets the task mid-run, re-reads files, loops. Truncation isn’t an error, so nothing logs.

The fix is per-model. Open the model’s settings (the gear next to it) and raise Context Size — the slider goes up to 131,072 depending on the model. 32K is the working floor for agent use we recommend on every backend; KV-cache memory scales with what you pick, so don’t set 131K on a 16 GB card and expect the model weights to still fit.

One Jan-specific interaction to know: the Fit to Hardware setting auto-caps context based on your available memory. It’s on by design to prevent out-of-memory crashes, but it means the number you set and the number you get can differ — if agent sessions still degrade after you raised the context, check whether the effective value got clamped back down. Under the hood these per-model values live in the router preset (<data-folder>/llamacpp/router.preset.ini); the UI is the supported way to edit them.

Set the same number in your coding tool’s config (below), so its truncation math matches the server’s reality.

Step 4 — Wire in the three tools

Cline: OpenAI Compatible provider

Cline’s provider list has no native Jan entry, so use OpenAI Compatible — the fields below are from Cline’s provider docs, checked today:

  • Base URL: http://127.0.0.1:1337/v1
  • API Key: whatever you set in Jan (e.g. jan)
  • Model ID: the exact id from your /v1/models output
  • Model Configuration: set the context window to the value you gave Jan, and check Computer Use to enable agentic tool use for the model

Continue.dev: openai provider — and the three mistakes that look like an outage

Continue’s config for any OpenAI-compatible server, from its provider docs:

models:
  - name: Qwen3-Coder (Jan)
    provider: openai
    model: Qwen3-Coder-30B-A3B-Instruct
    apiBase: http://127.0.0.1:1337/v1
    apiKey: jan
    roles: [chat, edit, apply]

A real bug report against exactly this setup (continuedev/continue #8246) is worth reading as a checklist of what not to do, because the reporter’s Jan server was healthy the whole time. Their config made three classic mistakes at once: provider: ollama instead of openai (Jan speaks the OpenAI dialect, not Ollama’s), apiBase pointing at the full http://localhost:1337/v1/chat/completions path instead of stopping at /v1 (Continue appends the route itself), and apiKey: Bearer janai (the Bearer prefix is added by the client — putting it in the key field sends Bearer Bearer janai). Any one of these produces a “no response” that looks identical to a dead server. Same triage rule as the LM Studio connection errors: if curl answers and your editor doesn’t, the bug is in the editor’s config, every time.

Aider: OpenAI-compatible env vars

From Aider’s OpenAI-compatible docs, unchanged pattern:

export OPENAI_API_BASE=http://127.0.0.1:1337/v1
export OPENAI_API_KEY=jan
aider --model openai/Qwen3-Coder-30B-A3B-Instruct

The openai/ prefix routes Aider’s LiteLLM layer to the generic OpenAI-compatible path; everything after it must match Jan’s model ID literally. You’ll get the usual warning about unknown context window and costs — silence it with a .aider.model.metadata.json setting max_input_tokens to your configured context and costs to 0. Aider leans on search/replace blocks more than tool calls, which makes it the most forgiving of the three if you’re testing a smaller model — and if the blocks themselves stop applying, that’s a model-quality failure, not Jan’s.

Jan vs LM Studio vs Ollama: the desktop decision

JanLM StudioOllama
Best forGUI + open source as a requirementGUI + smoothest integration docsHeadless simplicity
LicenseApache 2.0, full app on GitHubFreeware, closed-source shellMIT (server)
Model sourceHugging Face GGUF direct + MLX (macOS)Own catalog (GGUF/MLX)Own library + HF
Load behaviorRouter auto-loads on request (v0.8.0+)JIT loadingAuto-loads, keepalive unload
Context default8,192 per model (v0.8.2+)4,096 per model4,096 (truncates silently)
Tool callingAuto-detected, editable Jinja templateAutomaticAutomatic, template-gated 400
The catchYoungest troubleshooting corpus of the three; Fit-to-Hardware can clamp your contextClosed sourceWrapper hides the knobs

The 8,192 default is a real, if small, differentiator: Jan is the only backend in this series whose out-of-box context isn’t the 4K that breaks agents instantly. It still breaks them — just a few tool calls later.

When to skip Jan

Skip it if you’re happy headless — Ollama does the same llama.cpp serving with a larger model library, three years of accumulated GitHub-issue debugging mass, and no Electron-class app running. Skip it if you need concurrent multi-agent throughput; that’s vLLM’s territory, not any desktop app’s. And skip the 0.0.0.0 mode entirely unless you’ve read the security section above and updated past the trusted-hosts fix.

Take it seriously if the LM Studio recommendation in our Cline + LM Studio guide always came with a “but it’s closed source” asterisk for you. Jan is the version of that setup you can read the source of, with a router that finally makes the API server feel unattended, at the price of a slightly rougher documentation trail. If your GPU lives in the cloud instead, a RunPod box runs the same models behind the same OpenAI-compatible pattern; for the broader FOSS tooling around this stack, aifoss.dev tracks it.

FAQ

What’s the base URL for Jan — with or without /v1? http://127.0.0.1:1337/v1 for Cline, Continue, and Aider alike. Continue and Cline append routes like /chat/completions themselves; putting the full route in the base URL is misconfig #2 from the issue above.

Do I need to start a model before my editor can use it? Not since v0.8.0. The router loads the requested model on demand and unloads idle ones. First request after idle pays the load time — a long first spinner is normal.

Cline connects but the agent narrates JSON instead of editing files. Server problem? No — model problem. Jan auto-detects tool-use capability and serves what the chat template produces; a model that wasn’t trained for tool calls degrades silently instead of erroring. Switch to gpt-oss-20b or a Qwen3-Coder variant and check Computer Use is enabled in Cline’s model config.

Why did my agent get dumber twenty minutes into a session? The 8,192-token default context filled up and Jan truncated silently. Raise Context Size in the model’s settings (32K floor for agents), verify Fit to Hardware didn’t clamp it, and mirror the number in your tool’s config.

Is Jan’s API safe to expose on my LAN? Only on v0.8.4+ with an API key set. Before the fix for issue #8453, binding to 0.0.0.0 wildcard-bypassed the Trusted Hosts allowlist and the CORS toggle did nothing. For a coding backend, keep it on 127.0.0.1 and the question disappears.

Sources

Last updated August 23, 2026. Pricing and features change frequently; verify current state before purchasing.

Was this article helpful?