Home / Docs / API overview

API overview

TurboLLM serves both OpenAI- and Anthropic-compatible APIs so any tool can talk to your local models — on the same port, http://localhost:6996. Point an existing client at that URL and it just works.

Quick example

A plain curl to the OpenAI-compatible endpoint:

curl http://localhost:6996/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local","messages":[{"role":"user","content":"hello"}]}'

Two compatible APIs, one port

Everything is served from http://localhost:6996. Pick whichever API your tool already speaks — there's nothing extra to configure to expose both.

OpenAI-compatible →

Drop-in for anything that talks to the OpenAI API. Point your client's base URL at http://localhost:6996/v1.

Anthropic-compatible →

For tools built around the Anthropic Messages API. Same host and port.

Integrations →

Ready-made setups for popular tools so you can skip the wiring.

Building a real chat app? There's a stateful API too

The gateway above is stateless — you send the full message history on every request. For building an actual chat product, TurboLLM will also serve a versioned, tenant-aware External Chat API with persistence, history, editing, and resumable generation built in, plus a pluggable storage interface so you're never locked into SQLite.

External Chat API → Coming soon

Stateful chats and messages, SSE streaming with automatic resumption, and a documented ChatStore interface you can implement over your own database.

The gateway loads models for you

Name any model in your API request and TurboLLM loads it on the fly. It reads the model field, fuzzy-matches it against your library, and loads it if it isn't already running — keeping up to four models hot in an LRU pool.

That means an agent hopping between a coding model, a vision model, and an embedder just names each one in its requests. No pre-wiring, no manual load step.

LRU pool

Up to four models stay resident at once. When a fifth is requested, the least-recently-used model is evicted to make room.

API-key auth

You can require an API key when sharing TurboLLM over a LAN. Set it in config or via the TURBOLLM_API_KEY environment variable.

  1. Turn on the requirement

    Enable Require API key under Settings → Network.

  2. Create a key

    Go to Developer → API Keys → Create. A key is required once Require API key is on.

  3. Send it with requests

    Pass the key from your client, or set TURBOLLM_API_KEY in the environment.

Structured output

Constrain any response to a GBNF grammar via the response_format parameter, so the model's output matches the shape your application expects.

Where to go next

Read the detail pages for the exact endpoints and payloads: OpenAI-compatible and Anthropic-compatible. Building a chat product rather than integrating an existing client? Read the External Chat API guide instead. If you'd rather not wire things up by hand, start from Integrations for ready-made tool setups.

Previous← Routines