OpenAI-compatible API

TurboLLM speaks the OpenAI wire format. Point any OpenAI-compatible client, SDK, or tool at the local gateway and it works unchanged — chat completions, embeddings, and model listing all use the endpoints and request shapes you already know.

Base URL

The gateway serves the OpenAI-compatible API on your local machine:

$http://localhost:6996/v1

Set this as the base URL (base_url / baseURL) in any OpenAI SDK or as the endpoint in tools that accept a custom OpenAI-compatible server.

Authentication

An API key is optional. Requests need an Authorization: Bearer <key> header only when Require API key is turned on. When it is off, you can send any placeholder key or omit the header entirely.

Placeholder keys

OpenAI SDKs require a non-empty API key field to initialize. If you haven't enabled Require API key, pass any string (for example "local") — it isn't validated.

Supported endpoints

Method	Path	Purpose
GET	`/v1/models`	List available models
GET	`/v1/models/{model}`	Retrieve a single model
POST	`/v1/chat/completions`	Chat completions (streaming and non-streaming)
POST	`/v1/embeddings`	Generate embeddings
GET	`/v1/health`	Gateway health check

Model loading

The gateway loads the model named in the request's model field on demand. The name is fuzzy-matched, so you don't have to type the exact filename. The generic name local also works as a stand-in for the active model.

Your first call

Here's the same minimal "hello" chat completion in three clients. Each one points at http://localhost:6996/v1.

curl http://localhost:6996/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer local" \
  -d '{
    "model": "local",
    "messages": [
      { "role": "user", "content": "Hello" }
    ]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:6996/v1",
    api_key="local",
)

resp = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Hello"}],
)

print(resp.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:6996/v1",
  apiKey: "local",
});

const resp = await client.chat.completions.create({
  model: "local",
  messages: [{ role: "user", content: "Hello" }],
});

console.log(resp.choices[0].message.content);

Streaming

Set stream: true to receive a standard OpenAI Server-Sent Events (SSE) stream. Tokens arrive as data: chunks and the stream ends with data: [DONE] — exactly as the OpenAI SDKs expect, so streaming helpers work without changes.

curl http://localhost:6996/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{ "role": "user", "content": "Hello" }],
    "stream": true
  }'

Structured output

The response_format field supports JSON mode, constraining the model to emit valid JSON. Power users can go further and enforce a GBNF grammar for tighter control over the output shape.

Embeddings

POST /v1/embeddings works with embedding models (bert-family GGUF). These are served from a dedicated pool slot, so embedding requests don't contend with your chat model.

curl http://localhost:6996/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "input": "Hello"
  }'

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:6996/v1",
    api_key="local",
)

resp = client.embeddings.create(
    model="local",
    input="Hello",
)

print(resp.data[0].embedding)

Previous← API overview NextAnthropic-compatible →