Liter LLM

Liter-LLM Universal LLM Client

Liter-LLM is a universal LLM API client with a Rust core and native bindings for Python, TypeScript/Node.js, Go, Java, C#, Ruby, PHP, Elixir, WebAssembly, and C (FFI). One unified interface reaches 165 providers (OpenAI, Anthropic, Google Gemini, Groq, Mistral, Cohere, AWS Bedrock, Azure, and many more).

Capability map

• Modalities — chat completions, streaming, tool/function calling, structured outputs, embeddings, image generation, audio (speech, transcription), moderation, web search, OCR, reranking, batch, and file operations.
• Provider routing — provider/model prefix selects the backend (openai/gpt-4o, anthropic/claude-sonnet-4-20250514, gemini/gemini-2.0-flash). Set model_hint to skip the prefix.
• Middleware — response caching, rate limiting (RPM/TPM), cost tracking, budget enforcement (hard / soft), fallback chains, circuit-breaker cooldown, and background health checks.
• Search providers (12) and OCR providers (4) — first-class search and ocr methods routed by the same prefix convention.
• Proxy server — liter-llm api exposes an OpenAI-compatible gateway with 22 REST endpoints, virtual API keys, budgets, and SSE streaming.
• MCP server — liter-llm mcp exposes 22 tools mirroring the proxy endpoints, for MCP-compatible clients (Claude Code, Claude Desktop).
• 14 language bindings — same surface, language-native naming (snake_case for Python/Rust/Ruby/Go/Elixir/PHP, camelCase for TS/Node/WASM/C#/Java).

When to use the MCP server vs the SDK vs the proxy

Rule of thumb: reach for the MCP server inside an agent session; the SDK when building software; the proxy when centralizing access for multiple consumers.

Installation

CLI (proxy + MCP server)

# Homebrew (macOS / Linux)
brew install xberg-io/tap/liter-llm

# or run it without a persistent install (the CLI proxy package self-installs the binary)
npx @xberg-io/liter-llm-cli --help
uvx --from liter-llm-cli liter-llm --help

# or download a prebuilt binary from the latest GitHub release:
#   https://github.com/xberg-io/liter-llm/releases/latest

# or build from source
cargo install liter-llm-cli

# or Docker (35MB image)
docker pull ghcr.io/xberg-io/liter-llm

Language bindings

Quick start (Python, async)

import asyncio
import os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest

async def main() -> None:
    client = create_client(api_key=os.environ["OPENAI_API_KEY"])
    request = ChatCompletionRequest.from_json(
        '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"Hello!"}]}'
    )
    response = await client.chat(request)
    print(response.choices[0].message.content)

asyncio.run(main())

Provider routing

The prefix before / in the request's model selects the provider:

ChatCompletionRequest.from_json('{"model":"openai/gpt-4o","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"anthropic/claude-sonnet-4-20250514","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"gemini/gemini-2.0-flash","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"groq/llama3-70b","messages":[...]}')

API keys come from environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, GROQ_API_KEY, MISTRAL_API_KEY, CO_API_KEY, and AWS credentials for Bedrock.

Configuration

Pass basic options to create_client(...) (api_key, base_url, timeout_secs, max_retries, model_hint), or set the full surface in a liter-llm.toml (SDK) / liter-llm-proxy.toml (proxy) — both auto-discover from the cwd upward.

# liter-llm.toml
api_key = "${OPENAI_API_KEY}"
model_hint = "openai"
timeout_secs = 120
max_retries = 5

[cache]
max_entries = 512
ttl_seconds = 600

[budget]
global_limit = 50.0
enforcement = "hard"

[rate_limit]
rpm = 60
tpm = 100000

Proxy server

liter-llm api --config liter-llm-proxy.toml

22 OpenAI-compatible endpoints, model routing by prefix, virtual API keys, per-key RPM/TPM limits, cost tracking, budget enforcement, response caching, SSE streaming, and an OpenAPI 3.1 spec at /openapi.json.

MCP server

liter-llm mcp --transport stdio   # for Claude Code / Claude Desktop
liter-llm mcp --transport http --port 3001

This plugin auto-registers the stdio server via scripts/mcp-launch.sh. The 22 tools mirror the proxy endpoints: chat, embed, generate_image, speech, transcribe, moderate, rerank, search, ocr, list_models; file ops (create_file, list_files, retrieve_file, delete_file, file_content); batch ops (create_batch, list_batches, retrieve_batch, cancel_batch); and Responses API (create_response, retrieve_response, cancel_response).

Common pitfalls

1. Provider prefix is required — use "provider/model" unless model_hint is set, or routing fails.
2. API keys are SecretString — never logged or serialized. Read from env vars; never hardcode.
3. Python methods are async — await inside an async context.
4. Naming conventions differ — camelCase for TS/Node/WASM/C#/Java, snake_case elsewhere.
5. Streaming chunks may have null content — null-check chunk.choices[0].delta.content before use.
6. Budget modes — hard rejects over-budget requests; soft logs and allows.

Additional resources

• Upstream docs: https://docs.liter-llm.xberg.io
• GitHub: https://github.com/xberg-io/liter-llm