Home / Docs / API overview

API overview

TurboLLM serves both OpenAI- and Anthropic-compatible APIs so any tool can talk to your local models — on the same port, http://localhost:6996. Point an existing client at that URL and it just works.

Quick example

A plain curl to the OpenAI-compatible endpoint:

curl http://localhost:6996/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local","messages":[{"role":"user","content":"hello"}]}'

Two compatible APIs, one port

Everything is served from http://localhost:6996. Pick whichever API your tool already speaks — there's nothing extra to configure to expose both.

OpenAI-compatible →

Drop-in for anything that talks to the OpenAI API. Point your client's base URL at http://localhost:6996/v1.

Anthropic-compatible →

For tools built around the Anthropic Messages API. Same host and port.

Integrations →

Ready-made setups for popular tools so you can skip the wiring.

Text classification →

Laya and Jev models answer yes/no, pick-one and scale questions about a text with /v1/systemone. One place to find them, one playground.

Jev: System One, classify & rerank →

Ask a Jev (NLI) model structured questions about content with /v1/systemone, or label premise/hypothesis pairs and rerank documents with /v1/classify and /v1/rerank.

Laya: System One on any OS →

The same /v1/systemone answered by Laya, a small decision model that runs on CPU or GPU on Windows, macOS and Linux, beside your chat model.

Building a real chat app? There's a stateful API too

The gateway above is stateless — you send the full message history on every request. For building an actual chat product, TurboLLM will also serve a versioned, tenant-aware External Chat API with persistence, history, editing, and resumable generation built in, plus a pluggable storage interface so you're never locked into SQLite.

External Chat API → Experimental

Stateful chats and messages, SSE streaming with automatic resumption, and a documented ChatStore interface you can implement over your own database. Shipped in v1.11.5 — manual setup via config.json.

The gateway loads models for you

Name any model in your API request and TurboLLM loads it on the fly. It reads the model field, fuzzy-matches it against your library, and loads it if it isn't already running — keeping up to four models hot in an LRU pool.

That means an agent hopping between a coding model, a vision model, and an embedder just names each one in its requests. No pre-wiring, no manual load step.

LRU pool

Up to four models stay resident at once. When a fifth is requested, the least-recently-used model is evicted to make room.

API-key auth

You can require an API key when sharing TurboLLM over a LAN. Set it in config or via the TURBOLLM_API_KEY environment variable.

  1. Turn on the requirement

    Enable Require API key under Settings → Network.

  2. Create a key

    Go to Developer → API Keys → Create. A key is required once Require API key is on.

  3. Send it with requests

    Pass the key from your client, or set TURBOLLM_API_KEY in the environment.

Structured output

Constrain any response to a GBNF grammar via the response_format parameter, so the model's output matches the shape your application expects.

Where to go next

Read the detail pages for the exact endpoints and payloads: OpenAI-compatible and Anthropic-compatible. Building a chat product rather than integrating an existing client? Read the External Chat API guide instead. If you'd rather not wire things up by hand, start from Integrations for ready-made tool setups.

Previous← Routines