caliber-ai-org LLM Provider
Critical
All providers MUST implement the LLMProvider interface from src/llm/types.ts with three methods:
- call(options: LLMCallOptions): Promise — single non-streaming call returning text
- stream(options: LLMStreamOptions, callbacks: LLMStreamCallbacks): Promise — streaming call invoking callbacks
- listModels?(): Promise<string[]> — optional; list available models from the API
Initialize client in constructor and store defaultModel from config. Example:
this.client = new YourSDK({ apiKey: config.apiKey }). Never lazy-initialize on first call — providers are instantiated once and cached in src/llm/index.ts.For EVERY response in call() and stream(), invoke trackUsage(model, usage) from src/llm/usage.js before returning/ending. This is mandatory — it captures token metrics for CLI telemetry and cost analysis. If the API doesn't return usage data, estimate via estimateTokens(text), which assumes ~4 chars per token.
Both call() and stream() must respect the model parameter using pattern:
options.model || this.defaultModel. Never hardcode model names. Callers supply model overrides via LLMCallOptions.model.Error handling: catch all errors, preserve error messages unchanged. The retry logic in src/llm/index.ts handles transient errors (ECONNRESET, socket hang up, 529 overload). For seat-based providers (Cursor, Claude CLI), wrap stderr via parseSeatBasedError() for user-friendly messages.
Always update ProviderType union (Step 2), DEFAULT_MODELS (Step 4), and createProvider() switch case (Step 5) in lock-step. Missing any one breaks the build or causes runtime Unknown provider error.
Instructions
Step 1: Create provider class file
Verify directory exists: ls -la src/llm/. Create src/llm/your-provider.ts. Match existing provider patterns (src/llm/anthropic.ts, src/llm/openai-compat.ts).
Minimal structure:
import type { LLMProvider, LLMCallOptions, LLMStreamOptions, LLMStreamCallbacks, LLMConfig, TokenUsage } from './types.js';
import { trackUsage } from './usage.js';
import { estimateTokens } from './utils.js';
export class YourProviderProvider implements LLMProvider {
private client: YourSDKType;
private defaultModel: string;
constructor(config: LLMConfig) {
if (!config.apiKey) throw new Error('API key required');
this.client = new YourSDK({ apiKey: config.apiKey, ...(config.baseUrl && { baseURL: config.baseUrl }) });
this.defaultModel = config.model;
}
async call(options: LLMCallOptions): Promise<string> {
const model = options.model || this.defaultModel;
const response = await this.client.messages.create({ model, max_tokens: options.maxTokens || 4096, system: options.system, messages: [{ role: 'user', content: options.prompt }] });
trackUsage(model, { inputTokens: response.usage?.input_tokens || 0, outputTokens: response.usage?.output_tokens || 0 });
return response.content?.[0]?.text || '';
}
async stream(options: LLMStreamOptions, callbacks: LLMStreamCallbacks): Promise<void> {
const model = options.model || this.defaultModel;
const messages = [...(options.messages || []), { role: 'user' as const, content: options.prompt }];
try {
const stream = await this.client.stream({ model, max_tokens: options.maxTokens || 10240, system: options.system, messages });
let stopReason: string | undefined, usage: TokenUsage | undefined;
for await (const chunk of stream) {
if (chunk.delta?.text) callbacks.onText(chunk.delta.text);
if (chunk.delta?.stop_reason) stopReason = chunk.delta.stop_reason;
if (chunk.usage) usage = { inputTokens: chunk.usage.input_tokens, outputTokens: chunk.usage.output_tokens };
}
if (usage) trackUsage(model, usage);
callbacks.onEnd({ stopReason, usage });
} catch (error) { callbacks.onError(error instanceof Error ? error : new Error(String(error))); }
}
}
Verify: File exports the class; imports match existing providers.
Step 2: Add to ProviderType union
Edit src/llm/types.ts line 1. Add your provider in kebab-case:
export type ProviderType = 'anthropic' | 'vertex' | 'openai' | 'cursor' | 'claude-cli' | 'your-provider';
Verify: npx tsc --noEmit shows no ProviderType errors.
Step 3: Add config fields
If your provider needs fields beyond apiKey, model, baseUrl, extend LLMConfig in src/llm/types.ts:
export interface LLMConfig {
provider: ProviderType;
model: string;
fastModel?: string;
apiKey?: string;
baseUrl?: string;
yourProviderSecret?: string;
}
Step 4: Update config.ts
Edit src/llm/config.ts:
Line 9: Add to DEFAULT_MODELS:
export const DEFAULT_MODELS: Record<ProviderType, string> = {
anthropic: 'claude-sonnet-4-6',
vertex: 'claude-sonnet-4-6',
openai: 'gpt-5.4-mini',
cursor: 'sonnet-4.6',
'claude-cli': 'default',
'your-provider': 'your-provider/default-model',
};
Line 17: Add to MODEL_CONTEXT_WINDOWS if known:
export const MODEL_CONTEXT_WINDOWS: Record<string, number> = {
'your-provider/model-name': 128_000,
};
Line 59: In resolveFromEnv(), add env detection before final return null:
if (process.env.YOUR_PROVIDER_API_KEY) {
return {
provider: 'your-provider',
apiKey: process.env.YOUR_PROVIDER_API_KEY,
model: process.env.CALIBER_MODEL || DEFAULT_MODELS['your-provider'],
baseUrl: process.env.YOUR_PROVIDER_BASE_URL,
};
}
Line 115: In readConfigFile() validation, add 'your-provider' to includes list.
Verify: npm run test -- src/llm/__tests__/ -t config confirms env var detection works.
Step 5: Register in factory
Edit src/llm/index.ts. Add import (line ~4):
import { YourProviderProvider } from './your-provider.js';
In createProvider() switch (line ~24), add before default case:
case 'your-provider':
return new YourProviderProvider(config);
Verify: npx tsc --noEmit passes; no type errors on switch cases.
Step 6: Write tests
Create src/llm/__tests__/your-provider.test.ts:
import { describe, it, expect, beforeEach } from 'vitest';
import { YourProviderProvider } from '../your-provider.js';
describe('YourProviderProvider', () => {
let provider: YourProviderProvider;
beforeEach(() => {
provider = new YourProviderProvider({ provider: 'your-provider', model: 'test', apiKey: 'test' });
});
it('implements LLMProvider interface', () => {
expect(typeof provider.call).toBe('function');
expect(typeof provider.stream).toBe('function');
});
it('call() returns string', async () => {
const result = await provider.call({ system: 'helpful', prompt: 'hi' });
expect(typeof result).toBe('string');
});
it('stream() invokes callbacks', async () => {
const texts: string[] = [];
let ended = false;
await provider.stream({ system: 'helpful', prompt: 'hi' }, {
onText: (t) => texts.push(t),
onEnd: () => { ended = true; },
onError: () => {},
});
expect(ended).toBe(true);
});
});
Verify: npm run test -- src/llm/__tests__/your-provider.test.ts passes.
Step 7: Integration test
Run factory tests with your provider env var:
YOUR_PROVIDER_API_KEY=test npm run test -- src/llm/__tests__/index.test.ts
Verify: getProvider() instantiates your provider; llmCall() dispatches correctly.
Examples
Example 1: Local LM Studio server
User says: "I need caliber to use my local LM Studio instance."
Actions: Create src/llm/lm-studio.ts extending OpenAICompatProvider. Add 'lm-studio' to ProviderType. In config.ts:
if (process.env.LM_STUDIO_BASE_URL) {
return { provider: 'lm-studio', apiKey: '', model: 'local', baseUrl: process.env.LM_STUDIO_BASE_URL };
}
Register in createProvider() case. User: export LM_STUDIO_BASE_URL=http://localhost:8000/v1. Result: caliber uses local LM Studio; tokens estimated via estimateTokens().
Example 2: Ollama (seat-based)
User says: "Ollama is auto-detected; no API key needed."
Actions: Create src/llm/ollama.ts extending OpenAICompatProvider. Add 'ollama' to ProviderType and SEAT_BASED_PROVIDERS. In config.ts:
if (process.env.OLLAMA_HOST) {
return { provider: 'ollama', model: 'mistral', baseUrl: process.env.OLLAMA_HOST || 'http://localhost:11434/v1' };
}
Result: Offline per-machine LLM without API keys.
Common Issues
Unknown provider: your-provider
- Cause: ProviderType updated but createProvider() case missing.
- Fix: Add case in src/llm/index.ts and import the class.
Cannot find module './your-provider.js'
- Cause: File named your_provider.ts (underscore) not your-provider.ts (kebab).
- Fix: Rename file to use kebab-case.
API key is required for YourProvider
- Cause: Env var YOUR_PROVIDER_API_KEY not set; resolveFromEnv() didn't detect it.
- Fix: Verify env var name in config.ts matches. Test:
YOUR_PROVIDER_API_KEY=test npm run test -- src/llm/__tests__/index.test.ts.
LLM response did not include usage tokens
- Cause: Provider API doesn't return usage (local models).
- Fix: Estimate tokens:
trackUsage(model, { inputTokens: estimateTokens(options.system + options.prompt), outputTokens: estimateTokens(response.text) });
Stream callbacks never fire; onEnd not called
- Cause: Async iterator not fully consumed before method returns.
- Fix: Ensure iteration completes before onEnd():
for await (const chunk of stream) { } callbacks.onEnd({ stopReason, usage });
My model parameter is ignored
- Cause: call()/stream() doesn't use
options.model || this.defaultModel. - Fix: Replace hardcoded model:
const model = options.model || this.defaultModel; const response = await this.client.create({ model, ... });
trackUsage() is never called
- Cause: Forgot to call trackUsage() in call() or stream().
- Fix: Add after every response:
trackUsage(model, { inputTokens: ..., outputTokens: ... });
Type error: Provider doesn't implement LLMProvider
- Cause: Missing method or wrong signature.
- Fix: Verify all required methods exist with exact signatures from src/llm/types.ts. Copy-paste from anthropic.ts as reference.