Liter-LLM Universal LLM Client
Liter-LLM is a universal LLM API client with a Rust core and native bindings for
Python, TypeScript/Node.js, Go, Java, C#, Ruby, PHP, Elixir, WebAssembly, and C
(FFI). One unified interface reaches 174 providers (OpenAI, Anthropic, Google
Gemini, Groq, Mistral, Cohere, AWS Bedrock, Azure, and many more).
Capability map
- Modalities — chat completions, streaming, tool/function calling,
structured outputs, embeddings, image generation, audio (speech,
transcription), moderation, web search, OCR, reranking, batch,
and file operations.
- Provider routing —
provider/model prefix selects the backend
(openai/gpt-4o, anthropic/claude-sonnet-4-20250514,
gemini/gemini-2.0-flash). Set a full provider/model as model_hint to choose the client provider.
- Middleware — response caching, rate limiting (RPM/TPM), cost tracking,
budget enforcement (
hard / soft), fallback chains, circuit-breaker
cooldown, and background health checks.
- Search providers (12) and OCR providers (4) — first-class
search
and ocr methods routed by the same prefix convention.
- Proxy server —
liter-llm api exposes an OpenAI-compatible gateway with
22 REST endpoints, virtual API keys, budgets, and SSE streaming.
- MCP server —
liter-llm mcp exposes 22 tools mirroring the proxy
endpoints, for MCP-compatible clients (Claude Code, Claude Desktop).
- 14 language bindings — same surface, language-native naming (snake_case
for Python/Rust/Ruby/Go/Elixir/PHP, camelCase for TS/Node/WASM/C#/Java).
When to use the MCP server vs the SDK vs the proxy
| Use the … |
When you want to … |
Entry point |
| MCP server |
Let the agent call LLM APIs directly as tools, with no glue code |
liter-llm mcp --transport stdio (this plugin auto-registers it) |
| SDK / binding |
Write application code that calls LLMs in a specific language |
pip install liter-llm, cargo add liter-llm, etc. |
| Proxy |
Give many apps/teams a shared OpenAI-compatible endpoint with keys, budgets, and rate limits |
liter-llm api --config liter-llm-proxy.toml |
Rule of thumb: reach for the MCP server inside an agent session; the SDK when
building software; the proxy when centralizing access for multiple consumers.
Installation
CLI (proxy + MCP server)
# Homebrew (macOS / Linux)
brew install xberg-io/tap/liter-llm
# or run it without a persistent install (the CLI proxy package self-installs the binary)
pnpm dlx @xberg-io/liter-llm-cli@2.2.3 --help
# or download a prebuilt binary from the latest GitHub release:
# https://github.com/xberg-io/liter-llm/releases/latest
# or build from source
cargo install liter-llm-cli
# or Docker (35MB image)
docker pull ghcr.io/xberg-io/liter-llm
Language bindings
| Language |
Install |
| Python |
pip install liter-llm |
| Node.js |
pnpm add @xberg-io/liter-llm |
| Rust |
cargo add liter-llm |
| Go |
go get github.com/xberg-io/liter-llm/packages/go/v2 |
| Ruby |
gem install liter_llm |
| PHP |
composer require xberg-io/liter-llm |
| C# |
dotnet add package XbergIo.LiterLlm |
| WASM |
pnpm add @xberg-io/liter-llm-wasm |
Quick start (Python, async)
import asyncio
import os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest
async def main() -> None:
client = create_client(api_key=os.environ["OPENAI_API_KEY"])
request = ChatCompletionRequest.from_json(
'{"model":"openai/gpt-4o","messages":[{"role":"user","content":"Hello!"}]}'
)
response = await client.chat(request)
print(response.choices[0].message.content)
asyncio.run(main())
Provider routing
The prefix before / in the request's model selects the provider:
ChatCompletionRequest.from_json('{"model":"openai/gpt-4o","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"anthropic/claude-sonnet-4-20250514","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"gemini/gemini-2.0-flash","messages":[...]}')
ChatCompletionRequest.from_json('{"model":"groq/llama3-70b","messages":[...]}')
Client construction resolves credentials for its selected provider from environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY,
GEMINI_API_KEY, GROQ_API_KEY, MISTRAL_API_KEY, COHERE_API_KEY, and AWS
credentials for Bedrock. Create a separate client for each provider with matching
model_hint and credentials. Request prefixes change routing but reuse the client
API key; they do not select a new environment key per request.
Configuration
Pass basic options to create_client(...) (api_key, base_url,
timeout_secs, connect_timeout_secs, max_retries, model_hint), or set the full surface in a
liter-llm.toml (SDK) / liter-llm-proxy.toml (proxy) — both auto-discover
from the cwd upward.
# liter-llm.toml
api_key = "${OPENAI_API_KEY}"
model_hint = "openai/gpt-4o"
timeout_secs = 120
max_retries = 5
[cache]
max_entries = 512
ttl_seconds = 600
[budget]
global_limit = 50.0
enforcement = "hard"
[rate_limit]
rpm = 60
tpm = 100000
| Option |
Default |
Description |
api_key |
required |
Provider key, wrapped in SecretString (never logged). |
base_url |
from registry |
Override the provider base URL. |
model_hint |
none |
Full provider/model used to select the client provider and credentials. |
timeout_secs |
60 |
Request timeout in seconds. |
connect_timeout_secs |
10 |
Connect timeout in seconds. A connection that does not open in time is retried up to max_retries times. Zero is an error on every target. wasm32 ignores any other value. |
max_retries |
3 |
Retries on 429/5xx with exponential backoff. |
cache |
none |
max_entries, ttl_seconds. |
budget |
none |
global_limit, model_limits, enforcement. |
rate_limit |
none |
rpm, tpm. |
cost_tracking |
false |
Per-request cost tracking. |
tracing |
false |
OpenTelemetry spans. |
Proxy server
liter-llm api --config liter-llm-proxy.toml
22 OpenAI-compatible endpoints, model routing by prefix, virtual API keys,
per-key RPM/TPM limits, cost tracking, budget enforcement, response caching, SSE
streaming, and an OpenAPI 3.1 spec at /openapi.json.
MCP server
liter-llm mcp --transport stdio # for Claude Code / Claude Desktop
liter-llm mcp --transport http --port 3001
This plugin auto-registers the stdio server via scripts/mcp-launch.sh. The 22
tools mirror the proxy endpoints: chat, embed, generate_image, speech,
transcribe, moderate, rerank, search, ocr, list_models; file ops
(create_file, list_files, retrieve_file, delete_file, file_content);
batch ops (create_batch, list_batches, retrieve_batch, cancel_batch);
and Responses API (create_response, retrieve_response, cancel_response).
Common pitfalls
- Provider routing — use
"provider/model" to select a request backend. Bare
model names fall back to the configured client provider (OpenAI by default).
- API keys are SecretString — never logged or serialized. Read from env
vars; never hardcode.
- Python methods are async —
await inside an async context.
- Naming conventions differ — camelCase for TS/Node/WASM/C#/Java,
snake_case elsewhere.
- Streaming chunks may have null content — null-check
chunk.choices[0].delta.content before use.
- Budget modes —
hard rejects over-budget requests; soft logs and
allows.
Additional resources
1---2description: Universal LLM API client for 174 providers with native bindings for 14 languages. Use when writing code that calls LLM APIs via liter-llm in Python, TypeScript, Rust, Go, Java, C#, Ruby, PHP, Elixir, WASM, or C, when running the OpenAI-compatible proxy, or when calling LLMs through the MCP server. Covers chat, streaming, tool calling, embeddings, image generation, speech, transcription, moderation, web search, OCR, reranking, provider routing, middleware, and configuration.3license: MIT4metadata:5 source: https://skillmd.com/skills/xberg-io/liter-llm-36 author: xberg-io7 version: "0.1.0"8 repository: https://github.com/xberg-io/liter-llm9name: liter-llm-310---1112<!--13AI-RULEZ :: GENERATED FILE — DO NOT EDIT14Content-Hash: blake3:3e0dcb446dcc37cd4709da048b254902f8d7f825b553308c5d2507441989556215Source-Hash: blake3:9b218e20701c9606ae4a1a4a7e4d1d6fd683a9d8f706fd47515ef28cd6ea07c716Schema-Version: v117-->1819# Liter-LLM Universal LLM Client2021Liter-LLM is a universal LLM API client with a Rust core and native bindings for22Python, TypeScript/Node.js, Go, Java, C#, Ruby, PHP, Elixir, WebAssembly, and C23(FFI). One unified interface reaches 174 providers (OpenAI, Anthropic, Google24Gemini, Groq, Mistral, Cohere, AWS Bedrock, Azure, and many more).2526## Capability map2728- **Modalities** — chat completions, streaming, tool/function calling,29 structured outputs, embeddings, image generation, audio (speech,30 transcription), moderation, web search, OCR, reranking, batch,31 and file operations.32- **Provider routing** — `provider/model` prefix selects the backend33 (`openai/gpt-4o`, `anthropic/claude-sonnet-4-20250514`,34 `gemini/gemini-2.0-flash`). Set a full `provider/model` as `model_hint` to choose the client provider.35- **Middleware** — response caching, rate limiting (RPM/TPM), cost tracking,36 budget enforcement (`hard` / `soft`), fallback chains, circuit-breaker37 cooldown, and background health checks.38- **Search providers (12)** and **OCR providers (4)** — first-class `search`39 and `ocr` methods routed by the same prefix convention.40- **Proxy server** — `liter-llm api` exposes an OpenAI-compatible gateway with41 22 REST endpoints, virtual API keys, budgets, and SSE streaming.42- **MCP server** — `liter-llm mcp` exposes 22 tools mirroring the proxy43 endpoints, for MCP-compatible clients (Claude Code, Claude Desktop).44- **14 language bindings** — same surface, language-native naming (snake_case45 for Python/Rust/Ruby/Go/Elixir/PHP, camelCase for TS/Node/WASM/C#/Java).4647## When to use the MCP server vs the SDK vs the proxy4849| Use the … | When you want to … | Entry point |50|-----------|--------------------|-------------|51| **MCP server** | Let the agent call LLM APIs directly as tools, with no glue code | `liter-llm mcp --transport stdio` (this plugin auto-registers it) |52| **SDK / binding** | Write application code that calls LLMs in a specific language | `pip install liter-llm`, `cargo add liter-llm`, etc. |53| **Proxy** | Give many apps/teams a shared OpenAI-compatible endpoint with keys, budgets, and rate limits | `liter-llm api --config liter-llm-proxy.toml` |5455Rule of thumb: reach for the MCP server inside an agent session; the SDK when56building software; the proxy when centralizing access for multiple consumers.5758## Installation5960### CLI (proxy + MCP server)6162```bash63# Homebrew (macOS / Linux)64brew install xberg-io/tap/liter-llm6566# or run it without a persistent install (the CLI proxy package self-installs the binary)67pnpm dlx @xberg-io/liter-llm-cli@2.2.3 --help6869# or download a prebuilt binary from the latest GitHub release:70# https://github.com/xberg-io/liter-llm/releases/latest7172# or build from source73cargo install liter-llm-cli7475# or Docker (35MB image)76docker pull ghcr.io/xberg-io/liter-llm77```7879### Language bindings8081| Language | Install |82|----------|---------|83| Python | `pip install liter-llm` |84| Node.js | `pnpm add @xberg-io/liter-llm` |85| Rust | `cargo add liter-llm` |86| Go | `go get github.com/xberg-io/liter-llm/packages/go/v2` |87| Ruby | `gem install liter_llm` |88| PHP | `composer require xberg-io/liter-llm` |89| C# | `dotnet add package XbergIo.LiterLlm` |90| WASM | `pnpm add @xberg-io/liter-llm-wasm` |9192## Quick start (Python, async)9394```python95import asyncio96import os97from liter_llm import create_client98from liter_llm._internal_bindings import ChatCompletionRequest99100async def main() -> None:101 client = create_client(api_key=os.environ["OPENAI_API_KEY"])102 request = ChatCompletionRequest.from_json(103 '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"Hello!"}]}'104 )105 response = await client.chat(request)106 print(response.choices[0].message.content)107108asyncio.run(main())109```110111## Provider routing112113The prefix before `/` in the request's `model` selects the provider:114115```python116ChatCompletionRequest.from_json('{"model":"openai/gpt-4o","messages":[...]}')117ChatCompletionRequest.from_json('{"model":"anthropic/claude-sonnet-4-20250514","messages":[...]}')118ChatCompletionRequest.from_json('{"model":"gemini/gemini-2.0-flash","messages":[...]}')119ChatCompletionRequest.from_json('{"model":"groq/llama3-70b","messages":[...]}')120```121122Client construction resolves credentials for its selected provider from environment variables: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`,123`GEMINI_API_KEY`, `GROQ_API_KEY`, `MISTRAL_API_KEY`, `COHERE_API_KEY`, and AWS124credentials for Bedrock. Create a separate client for each provider with matching125`model_hint` and credentials. Request prefixes change routing but reuse the client126API key; they do not select a new environment key per request.127128## Configuration129130Pass basic options to `create_client(...)` (`api_key`, `base_url`,131`timeout_secs`, `connect_timeout_secs`, `max_retries`, `model_hint`), or set the full surface in a132`liter-llm.toml` (SDK) / `liter-llm-proxy.toml` (proxy) — both auto-discover133from the cwd upward.134135```toml136# liter-llm.toml137api_key = "${OPENAI_API_KEY}"138model_hint = "openai/gpt-4o"139timeout_secs = 120140max_retries = 5141142[cache]143max_entries = 512144ttl_seconds = 600145146[budget]147global_limit = 50.0148enforcement = "hard"149150[rate_limit]151rpm = 60152tpm = 100000153```154155| Option | Default | Description |156|--------|---------|-------------|157| `api_key` | required | Provider key, wrapped in `SecretString` (never logged). |158| `base_url` | from registry | Override the provider base URL. |159| `model_hint` | none | Full provider/model used to select the client provider and credentials. |160| `timeout_secs` | 60 | Request timeout in seconds. |161| `connect_timeout_secs` | 10 | Connect timeout in seconds. A connection that does not open in time is retried up to `max_retries` times. Zero is an error on every target. `wasm32` ignores any other value. |162| `max_retries` | 3 | Retries on 429/5xx with exponential backoff. |163| `cache` | none | `max_entries`, `ttl_seconds`. |164| `budget` | none | `global_limit`, `model_limits`, `enforcement`. |165| `rate_limit` | none | `rpm`, `tpm`. |166| `cost_tracking` | false | Per-request cost tracking. |167| `tracing` | false | OpenTelemetry spans. |168169## Proxy server170171```bash172liter-llm api --config liter-llm-proxy.toml173```17417522 OpenAI-compatible endpoints, model routing by prefix, virtual API keys,176per-key RPM/TPM limits, cost tracking, budget enforcement, response caching, SSE177streaming, and an OpenAPI 3.1 spec at `/openapi.json`.178179## MCP server180181```bash182liter-llm mcp --transport stdio # for Claude Code / Claude Desktop183liter-llm mcp --transport http --port 3001184```185186This plugin auto-registers the stdio server via `scripts/mcp-launch.sh`. The 22187tools mirror the proxy endpoints: `chat`, `embed`, `generate_image`, `speech`,188`transcribe`, `moderate`, `rerank`, `search`, `ocr`, `list_models`; file ops189(`create_file`, `list_files`, `retrieve_file`, `delete_file`, `file_content`);190batch ops (`create_batch`, `list_batches`, `retrieve_batch`, `cancel_batch`);191and Responses API (`create_response`, `retrieve_response`, `cancel_response`).192193## Common pitfalls1941951. **Provider routing** — use `"provider/model"` to select a request backend. Bare196 model names fall back to the configured client provider (OpenAI by default).1972. **API keys are SecretString** — never logged or serialized. Read from env198 vars; never hardcode.1993. **Python methods are async** — `await` inside an async context.2004. **Naming conventions differ** — camelCase for TS/Node/WASM/C#/Java,201 snake_case elsewhere.2025. **Streaming chunks may have null content** — null-check203 `chunk.choices[0].delta.content` before use.2046. **Budget modes** — `hard` rejects over-budget requests; `soft` logs and205 allows.206207## Additional resources208209- Upstream docs: <https://docs.liter-llm.xberg.io>210- GitHub: <https://github.com/xberg-io/liter-llm>