AI Provider Codegen Rules (@ax-llm/ax)
Use this skill to generate AI provider setup, configuration, and chat code. Prefer short, modern, copyable patterns. Do not write tutorial prose unless the user explicitly asks for explanation.
Quick Setup
import { ai } from '@ax-llm/ax';
const openai = ai({ name: 'openai', apiKey: 'sk-...' });
const claude = ai({ name: 'anthropic', apiKey: 'sk-ant-...' });
const gemini = ai({ name: 'google-gemini', apiKey: 'AIza...' });
const azure = ai({ name: 'azure-openai', apiKey: 'your-key', resourceName: 'your-resource', deploymentName: 'gpt-5-4-mini' });
const deepseek = ai({ name: 'deepseek', apiKey: 'sk-...' });
const mistral = ai({ name: 'mistral', apiKey: 'your-key' });
const cohere = ai({ name: 'cohere', apiKey: 'your-key' });
const custom = ai({
name: 'openai-compatible',
apiKey: process.env.PROVIDER_API_KEY,
apiURL: 'https://example.com/v1',
config: { model: 'provider/model-name' },
});
const reka = ai({ name: 'reka', apiKey: 'your-key' });
const grok = ai({ name: 'grok', apiKey: 'your-key' });
name selects a deployment profile; config.model selects a model inside that
deployment. Never infer provider behavior from a model ID. For example,
name: 'together' with a deepseek-ai/... model uses Together's profile rules,
not DeepSeek's native request shape.
Use axAIProfiles() to discover the complete named catalog and
axGetAIProfile(name) to inspect endpoint requirements, authentication,
operations, capabilities, model rules, sources, and review dates. Use
name: 'openai-compatible' plus apiURL for an unlisted custom endpoint;
unknown names are errors.
Profile-only branded classes were removed in the major-version migration. Use
ai({ name: ... }) for Azure OpenAI, Cohere, DeepSeek, DeepSeek Responses,
Mistral, Reka, and Grok. Retained low-level classes represent genuine
transports/runtimes only; legacy model enum and catalog exports remain usable.
For Amazon Bedrock's OpenAI Chat Completions deployment profile, verified GPT-6
Sol/Luna, GPT-6 Astra, GPT-6.1 Sol, and GPT-5.6 Sol IDs (including geographic
prefixes) support native JSON Schema and mapped thinking budgets. These models
accept the Standard service tier; unsupported Flex/Priority requests fail locally.
Astra and GPT-6.1 Sol require reasoning and reject function tools on this profile.
Sol/Luna and GPT-5.6 Sol require explicit thinkingTokenBudget: 'none' (or
config.reasoningEffort: 'none') for function tools. Use name: 'openai-responses'
with the same Bedrock base URL when reasoning and tools are both required.
For complex signatures, AxGen uses native schema output by default, so output alone does not
need to disable reasoning. GPT-OSS and unknown-model defaults are unchanged.
Typesafe / Jev (TypeScript)
Use the ax-typesafe skill for Jev question design, rich criteria, native scoring, transport settings, and hybrid examples. The two interfaces serve different output contracts:
| Need | Interface |
|---|---|
| Required boolean/class/rubric score outputs through Ax signatures | ai({ name: 'typesafe', apiKey, trueThreshold: 0.9 }) and ax(...).forward() |
| Explicit provider request shapes or structured state/criteria | typesafe({ apiKey }).systemOne({ state, questions }) |
Use .withEvidence() on boolean/class fields or f.score(levels).withEvidence()
to retain probabilities, native scores, and confidence through normal AxGen
outputs. The string forms use (evidence). See the
decision evidence guide for
result shapes, provenance, and reranking with both Jev and OpenAI Decisions.
Both default to jev-latest. Set the adapter's model with config.model; set
native defaults with model. Native listModels() retrieves the provider's
catalog without changing configured Ax model aliases.
Boolean conversion uses noul >= trueThreshold; the default is 0.5, and the
finite threshold must be in [0, 1]. It applies to all booleans on that provider
instance, is never sent to the server, and does not affect native probabilities.
Choice preserves the selected label without an automatic confidence cutoff.
Value descriptions from boolean(true "...", false "...") and
class "support, billing"(support "...", billing "...") become native criteria.
The same annotations render as readable descriptions for conventional providers.
The signature skill
covers the string and fluent .describeValues(...) forms.
Numeric outputs, including bounded numbers, are unsupported; scoring uses
explicit score rubrics in signatures. Freeform text, optional/array/nested outputs, media,
tools, and sampling controls are also unsupported. Ax sends its normal prompt
as state. Use a separate generative program for prose or tools.
Typesafe-only balancers retain schema-required generation. Mixed pools select Typesafe only when the actual request already contains a supported schema. Incompatible requests are excluded from fallbacks even with degradation enabled. Usage and raw adapter answers remain in the existing usage and chat-log APIs. Python, Java, C++, Go, and Rust also implement these interfaces. Each generated package includes a Typesafe/Jev skill with its native API syntax.
Renewable Credentials
Use credentialProvider for expiring bearer tokens. The callback runs for each
request attempt and receives { profile, operation, method, url }. Fresh
headers override static authentication headers.
const vertex = ai({
name: 'vertex-ai',
apiURL: process.env.VERTEX_AI_API_URL!,
config: { model: 'google/gemma-4-26b-a4b-it-maas' },
credentialProvider: async ({ operation, url }) => ({
Authorization: `Bearer ${await tokenSource.fresh({ operation, url })}`,
}),
});
- Authentication-required profiles accept
apiKeyorcredentialProvider. - The hook covers chat, stream, embeddings, Responses, transcription, speech, and retry attempts. A hook error stops before transport.
- Do not expect an automatic replay after a completed 401 or 403; generation requests are not safely idempotent.
- Keep ADC or cloud-SDK dependencies in the host. Ax core only owns the neutral callback contract.
Vertex capability rules are model-aware. Documented Gemini MaaS IDs prefer
native schema. The exact google/gemma-4-26b-a4b-it-maas rule prefers
json_object, excludes native schema, defaults thinking to max, writes
chat_template_kwargs.enable_thinking, and extracts/replays
reasoning_content. Unknown Vertex models remain conservative.
WebLLM is browser-only and requires a host-created WebLLM engine. The host
loads or reloads models with WebLLM APIs such as CreateMLCEngine(...); Ax
only forwards chat requests to that loaded engine. Do not present WebLLM as a
portable AxIR provider or a server-side default.
import { ai, AxAIWebLLMModel } from '@ax-llm/ax';
const engine = await CreateMLCEngine(AxAIWebLLMModel.Llama32_3B_Instruct);
const llm = ai({
name: 'webllm',
engine,
config: {
model: AxAIWebLLMModel.Llama32_3B_Instruct,
stream: false,
supportsFunctions: false,
},
});
Chrome Built-in AI (TypeScript/JavaScript only)
The chrome-ai provider calls Chrome's browser LanguageModel API and runs
Gemini Nano locally without an API key. It is available only to the
TypeScript/JavaScript package inside Chrome, not to Node.js or the generated
Python, Java, C++, Go, and Rust packages. It has a browser-only AxIR exemption;
do not add it to portable provider catalogs or migration backlogs.
import { ai, ax } from '@ax-llm/ax';
const llm = ai({ name: 'chrome-ai' });
const program = ax('question -> answer');
// Call from a user interaction in a supported Chrome browser.
const result = await program.forward(llm, { question: 'What is 2+2?' });
The default uses Chrome's sampling settings. Chrome extensions can specify
temperature and topK together; when only one is supplied, Ax obtains the
other from LanguageModel.params(). An incomplete pair without defaults fails
before session creation. Web pages do not support these legacy numeric settings.
Streaming chunks contain successive text and are passed through unchanged.
Ax releases each session on completion, cancellation, or failure. The adapter
supports native JSON Schema output and text conversations, with no tools or
embeddings. For a runnable browser demo, build Ax and open
src/examples/chrome-ai-chat.html through a local web server.
Model Presets
import { ai, AxAIGoogleGeminiModel } from '@ax-llm/ax';
const gemini = ai({
name: 'google-gemini',
apiKey: process.env.GOOGLE_APIKEY!,
config: { model: AxAIGoogleGeminiModel.Gemini38Flash },
models: [
{ key: 'tiny', model: AxAIGoogleGeminiModel.Gemini35FlashLite, description: 'Fast + cheap', modelConfig: { maxTokens: 1024 } },
{ key: 'simple', model: AxAIGoogleGeminiModel.Gemini38Flash, description: 'Balanced' },
],
});
await gemini.chat({ model: 'tiny', chatPrompt: [{ role: 'user', content: 'Hi' }] });
config.model names the provider's default model. Use preset keys such as
'tiny' or 'simple' in per-call model overrides; preset settings belong in
modelConfig.
Model Catalog
import { axGetSupportedAIModels } from '@ax-llm/ax';
const providers = axGetSupportedAIModels();
const openai = providers.find((provider) => provider.name === 'openai');
console.log(openai?.models[0]?.promptTokenCostPer1M);
console.log(openai?.capabilities.serviceTiers);
const reasoningModel = openai?.models.find(
(model) => model.capabilities.thinkingLevels.includes('high')
);
console.log(reasoningModel?.capabilities.thinkingLevels);
const textProviders = axGetSupportedAIModels({ type: 'text' });
const embeddingProviders = axGetSupportedAIModels({ type: 'embeddings' });
Use axGetSupportedAIModels() to build provider/model selectors before creating an ai(...) instance. It returns bundled static metadata: provider names, display names, default models, raw AxModelInfo pricing/details, model type ('text', 'embeddings', 'code', 'audio', or 'image'), operation availability, provider-training data use, and normalized capabilities for thinking, thoughts, structured outputs, audio, image output, temperature, top-p, portable thinking levels, and verified service tiers. Provider capabilities describe the default deployment profile; each static model also carries its resolved capabilities.thinkingLevels and capabilities.serviceTiers. Portable thinking levels can collapse onto the same provider-native value. serviceTiers lists verified explicit tiers; serviceTier: 'auto' remains available as the provider-delegated policy.
Provider groups and models are sorted cheapest to most expensive based on bundled input + output token pricing; unpriced models sort last. Dynamic profiles remain useful even when models is empty because their provider-level capability metadata is still returned.
Filter with { type: 'all' | 'text' | 'embeddings' | 'code' | 'audio' | 'image' } or an array of those values. The 'text' filter includes code-capable models; use 'code' to show only code-first models.
Dynamic providers such as Azure OpenAI deployments are marked with isDynamic: true and may have an empty or static-limited model list.
The same AxIR-backed catalog is public in every generated package: Python
get_supported_ai_models(), Java Ax.getSupportedAIModels(), C++
get_supported_ai_models(), Go GetSupportedAIModels(...), and Rust
get_supported_ai_models() (with get_supported_ai_models_with_options(...)
for filters). It includes the named OpenAI-compatible profiles accepted by
ai(...), including azure-openai, openrouter, together, fireworks, and
the explicit openai-compatible fallback. These entries are usually dynamic,
so inspect provider-level capabilities even when models is empty.
Portable Inference Service Tiers
Use the shared serviceTier option with auto, standard, flex, or
priority. Set it per call, as an instance option, or on a model-key preset;
the narrower setting wins. auto delegates to the provider and is omitted
when that provider has no explicit auto value. Ax rejects unsupported explicit
tiers before transport.
import { ai, AxAIGoogleGeminiModel } from '@ax-llm/ax';
const gemini = ai({
name: 'google-gemini',
apiKey: process.env.GOOGLE_APIKEY!,
config: { model: AxAIGoogleGeminiModel.Gemini35Flash },
});
const response = await gemini.chat(
{ chatPrompt: [{ role: 'user', content: 'Run this when capacity is free.' }] },
{ serviceTier: 'flex' },
);
Inspect ai.getFeatures(model).serviceTiers for the verified tiers on a named
profile or exact model. Named OpenAI, Gemini, Azure, Bedrock, Mistral, Groq,
OpenRouter, DeepInfra, Cerebras, Grok, Databricks, and Fireworks profiles map
the portable value to their provider dialect. Exact modelInfo.supported
metadata can opt a conservative custom deployment into a tier.
The applied provider value is normalized into
response.modelUsage.tokens.serviceTier; aliases such as default,
on_demand, and performance become standard or priority. Add
serviceTierPricing.flex or .priority to AxModelInfo when tier pricing
differs, and cost estimates will use the tier that actually served the request.
Gemini service tiers remain GenerateContent-only: Vertex AI and Gemini Live
reject explicit tiers. Anthropic speed: 'fast' remains a separate API.
Routing And Balancing
Choose the primitive by responsibility:
AxMultiServiceRoutercombines model lists and dispatches the model key the caller already selected. It does not select a model.AxProviderRouterselects a provider by request capability and may degrade unsupported media through configured processors. When the selected provider supports images natively, each image remains an image object with its payload, MIME type, detail level, cache and optimization hints, alt text, and ordering with surrounding text intact.AxBalancerwithout a strategy orders equivalent services once with a comparator, retries transient provider failures, and fails over in that order.AxBalancerwithstrategy.type: 'adaptive'selects among services exposing the same logical model aliases using learned provider reliability, successful latency, and estimated cost.
In TypeScript, capabilities.requiresAudioOutput or
modelConfig.audio.output.enabled requires a service with
media.audio.output.supported. Audio input and output are separate requirements.
Balancer output formats come only from children that support output; compatibility
reports count requested input and output independently.
Generated provider routers expose the same independent counts through
get_compatibility_report (Python, C++, Rust), GetCompatibilityReport (Go),
and getCompatibilityReport (Java). Reports include ranked provider names,
scores, missing/supported capabilities, and a summary for request-compatible
providers.
Adaptive balancing is operational routing, not semantic prompt-to-model routing. Every provider model behind an alias must be an acceptable substitute for that application. Keep quality evaluation and content-aware model selection outside the balancer.
import { AxBalancer, AxInMemoryBalancerStatsStore } from '@ax-llm/ax';
const statsStore = new AxInMemoryBalancerStatsStore();
const routeKeys = new Map<string, string>([
[openai.getId(), 'openai-primary'],
[anthropic.getId(), 'anthropic-primary'],
]);
const llm = AxBalancer.create([openai, anthropic] as const, {
strategy: {
type: 'adaptive',
deadlineMs: 6_000,
badOutcomeCost: 0.02,
expectedTokens: { promptTokens: 1_200, completionTokens: 300 },
namespace: 'support-v1',
routeKey: (service) => {
const key = routeKeys.get(service.getId());
if (!key) throw new Error('Missing stable route key.');
return key;
},
slice: ({ options }) =>
options?.customLabels?.workflow ?? 'default-workflow',
statsStore,
onRoutingEvent: (event) => telemetry.emit('llm.route', event),
},
});
The score is estimated request cost plus badOutcomeCost times the probability of provider failure or missing deadlineMs. badOutcomeCost and estimated cost must use the same currency or unit. By default, cost uses expectedTokens, the route's concrete model mapping, and getEstimatedCost(); missing catalog pricing contributes zero, while estimateCost can supply application pricing. Failures use an EWMA; successful latency is modeled in log space with a Normal-Inverse-Gamma posterior, and Thompson sampling supplies the deadline risk. Capability filtering still runs before ranking.
Rules:
- Reuse the in-memory store across balancers in one process. For multiple processes, implement
AxBalancerStatsStorewith Redis or an application database; itsobserve()operation must be atomic. - A custom store requires stable, unique
routeKeyvalues. Stats are partitioned bynamespace,slice, logical model, and route. statsStoreis decision state.onRoutingEventis best-effort telemetry and must not be used as the authoritative routing state.- Routing events contain scores, route metadata, and sanitized failure categories, never prompts, responses, or raw provider errors.
- Adaptive mode attempts each candidate once. Provider-client retries remain controlled by
AxAIServiceOptions.retry. - Streaming can fail over before the first emitted chunk. A mid-stream transient failure is learned but cannot be replayed after partial output; caller cancellation is not recorded.
- Adaptive selection applies to chat. Embedding, transcription, and speech keep existing balancer behavior.
See the adaptive balancer example for complete provider setup.
Generated Runtime Cancellation
Generated Python, Java, C++, and Rust packages expose a reusable AxCancellationToken; Go keeps its idiomatic context.Context API. Pass the token/context to chat, streaming, embeddings, transcription, speech, or AxGen/AxAgent/AxFlow forwarding. The same cancellation propagates through provider routers, multi-service routing, balancers, retries, and cancellation-aware custom transports.
Cancellation is thread-safe, one-shot, and first-reason-wins. It stops before a new transport attempt, wakes retry backoff, and terminates an active stream at the next blocking-I/O boundary. The surfaced error is always the non-retryable AxAIServiceAbortedError, so routers and balancers must not retry or fail over. Existing methods remain available; use the added cancellation variants/options when cancellation is needed. Realtime WebSocket turns retain their separate lifecycle.
Chat
const res = await llm.chat({
chatPrompt: [
{ role: 'system', content: 'You are concise.' },
{ role: 'user', content: 'Write a haiku about the ocean.' },
],
});
if (!('results' in res)) throw new Error('Expected a non-streaming chat response');
console.log(res.results[0]?.content);
Meta Muse
Use meta for the Responses API (the recommended default), meta-chat for
Chat Completions, or meta-messages for Anthropic-compatible Messages. All
three use MODEL_API_KEY; muse-spark-1.3 is the default model.
import { ai, AxAIMetaModel } from '@ax-llm/ax';
const muse = ai({
name: 'meta',
apiKey: process.env.MODEL_API_KEY!,
config: { model: AxAIMetaModel.MuseSpark13 },
});
const result = await muse.chat({
chatPrompt: [{ role: 'user', content: 'Summarize the attached brief.' }],
});
thinkingTokenBudget: 'highest' maps to Meta xhigh; none is rejected.
Responses reasoning IDs, summaries, encrypted content, and message phases
are retained in Ax chat memory so stateless multi-turn replay works.
Streaming Ax programs also retain the final-answer phase. Responses failure
events raise errors; token-limited streams are not accepted as completed
program output.
Generated Responses clients retain parallel tool calls and their provider call
IDs, and completion events do not repeat text or arguments already streamed.
Assistant turns containing only generated images or encrypted reasoning can
be replayed without adding placeholder text.
All three Spark profiles support function-based structured output. When Ax
selects its internal __axOutput tool, the profiles require that sole tool
using the protocol's unnamed required choice. Explicit caller-named choices
remain unsupported, including a caller naming __axOutput directly.
Use muse-image-1.0 through the same chat() method. Text and reference images
belong in normal chat content, and generated images arrive in
result.results[0].images. Muse Image rejects ordinary function tools. Use
transcribe() for Muse Voice batch audio and streaming chat() with PCM16 for
realtime transcription; Ax adds no Meta-specific service methods.
Muse Glimmer is self-hosted. Point Ax's vllm, llama-cpp, ollama, or
lm-studio profile at the server you operate. Ax does not download, serve, or
manage Glimmer weights.
Batch Audio
Use ai.transcribe(...) for batch speech-to-text and ai.speak(...) for batch text-to-speech. These are separate from conversational .chat() audio config.
const transcript = await llm.transcribe({
audio: { data: base64Wav, format: 'wav' },
model: 'gpt-4o-mini-transcribe',
language: 'en',
});
const speech = await llm.speak({
text: transcript.text,
model: 'gpt-4o-mini-tts',
voice: 'alloy',
format: 'mp3',
});
console.log(transcript.text);
console.log(speech.data);
Gemini's defaults are gemini-3.8-flash-tts for speak() and the dedicated
gemini-3.5-transcribe model for transcribe(). Gemini takes no output
format, so the returned mime type sets the label: 3.8 TTS returns WAV, while
earlier TTS models return raw 24 kHz PCM, reported as pcm16 with its
sampleRate and channels.
const gemini = ai({ name: 'google-gemini', apiKey: process.env.GOOGLE_APIKEY! });
const speech = await gemini.speak({ text: 'Hello from Ax.', voice: 'Kore' });
const heard = await gemini.transcribe({
audio: { data: speech.data, format: speech.format },
});
Providers without the requested audio endpoint throw AxMediaNotSupportedError. Use speech forward options for signature audio artifacts and modelConfig.audio for conversational chat audio.
Common Options
stream(boolean): enable SSE; true by defaultthinkingTokenBudget:'minimal'|'low'|'medium'|'high'|'highest'|'none'showThoughts: include thoughts in outputfunctionCallMode:'auto'|'native'|'prompt'debug,logger,tracer,rateLimiter,timeouttimeout,fetchandcorsProxygiven to onechat()orembed()call override the service's owntimeoutis in milliseconds and bounds the wait for the response headers: a request whose response has not started in time fails withAxAIServiceTimeoutError, and a stream that has started runs on
Sampling Parameters
Each provider starts from its own defaults: temperature: 0 for OpenAI Chat,
Anthropic and Gemini, and temperature: 0.7 with topP: 1 for
openai-responses. Model info lists the sampling parameters a model rejects
(notSupported: temperature, topP, topK, presencePenalty,
frequencyPenalty):
- A provider default for such a parameter is never sent.
- An explicit value is sent when the model accepts it for that request, and
otherwise dropped with a one-time
console.warnnaming the setting and the model. Explicit values come from the AI'sconfig, a model key'smodelConfig, or the request'smodelConfig. - Models whose info sets
supported.temperatureOnestill take an explicittemperature: 1: every OpenAI model below, and the Anthropic models that deprecated sampling. - GPT-5.1 to 5.4 take sampling while they don't reason, which is their
default. GPT-5.5 and 5.6 take it only with reasoning effort
none(thinkingTokenBudget: 'none', orreasoningEffort: 'none'in the config). gpt-5, gpt-5-mini, gpt-5-nano and the o-series never take it; the o-series still getmaxTokensandn. - A profile without model info (
azure-openai,openai-compatible, ...) uses OpenAI's info for an exact o-series model name. - Anthropic: Opus 4.7 and later, Opus 5, Fable 5 and Sonnet 5 deprecated
sampling, so they take only
temperature: 1(notopPortopK). The other Claude models take every value with thinking off; while thinking, onlytemperature: 1,topPof 0.95 or above, and notopK. They rejecttemperatureandtopPtogether (notSupported.temperatureWithTopP): an explicittopPgoes alone in place of the default temperature, and an explicittemperaturewins over it with a warning. The defaulttemperature: 0is sent only without thinking, as before. Claude on Vertex keeps that older rule for explicit values too, and warns about a drop. - Gemini: the server-managed Flash models ignore
temperature,topPandtopK; the Gemini API rejectspresencePenaltyandfrequencyPenalty. Gemini 3 returns one candidate, sonabove 1 is dropped (AxGen then gets one sample), and Ax raises a temperature below 1 to 1, as Google recommends, with a one-time warning for an explicit value. Vertex keeps its request shape;presencePenaltyis never sent to Gemini and is warned about.
import { ai, AxAIOpenAIModel } from '@ax-llm/ax';
const llm = ai({ name: 'openai', apiKey, config: { model: AxAIOpenAIModel.GPT6Luna } });
// GPT-6 Luna takes temperature with reasoning off, so this one is sent.
await gen.forward(llm, values, {
thinkingTokenBudget: 'none',
modelConfig: { temperature: 0.2 },
});
A profile without a base URL of its own (openai-compatible, databricks,
amazon-bedrock, vertex-ai, ...) needs apiURL: ai(...) throws
<Name> requires apiURL without it.
Global Runtime Defaults
Use axGlobals when the app wants one live default for AI requests, generator runs, flows, or metrics:
import { ai, axGlobals, axCreateDefaultColorLogger } from '@ax-llm/ax';
import { metrics, trace } from '@opentelemetry/api';
axGlobals.rateLimiter = async (next, info) => next();
axGlobals.tracer = trace.getTracer('my-app');
axGlobals.meter = metrics.getMeter('my-app');
axGlobals.debug = true;
axGlobals.logger = axCreateDefaultColorLogger();
axGlobals.customLabels = { service: 'api' };
axGlobals.onUsage = (event) => usageQueue.enqueue(event);
const llm = ai({ name: 'openai', apiKey: process.env.OPENAI_APIKEY! });
Rules:
axGlobals.rateLimiter,tracer,meter,logger,debug,abortSignal, andcustomLabelsare live runtime defaults; each operation snapshots them at its start, even if the AI instance already exists.- Precedence is: per-call options, then explicit AI/service options, then current
axGlobals, then built-in defaults. - The limiter receives
nextplus operation, provider, model, streaming state, and previous service usage. It wraps chat and embedding provider execution, including streaming and retries; its errors propagate. It may delay, reject, skip, or invokenextmultiple times. - Tracer, meter, and usage-observer failures are fail-open. Limiter failures are fail-closed.
- Runtime-hook telemetry contains metadata and usage only, never prompts, outputs, tool arguments, or tool results.
- External meter instruments are independent of balancer-local
getMetrics()snapshots. AdaptAxMeterto OpenTelemetry at the application boundary; generated packages do not require an OpenTelemetry dependency. customLabelsmerge from globals to service to call options; later sources override earlier keys.abortSignalvalues are merged, so either a global shutdown signal or a local request signal can cancel the request.axGlobals.onUsagereceives one immutable normalized event for each completed chat or embedding call that reports token usage. A fully consumed stream emits once.- Usage observers are best-effort and fail-open. Ax does not await them; synchronously enqueue events and persist or aggregate them out of band.
Clear process-wide hooks during shutdown or test teardown:
axGlobals.rateLimiter = undefined;
axGlobals.tracer = undefined;
axGlobals.meter = undefined;
Use usageContext for multi-tenant and request attribution:
const llm = ai({
name: 'openai',
apiKey: process.env.OPENAI_APIKEY!,
options: {
usageContext: {
tenantId: 'tenant-42',
feature: 'support-chat',
attributes: { environment: 'production' },
},
},
});
await llm.chat(request, {
usageContext: {
userId: user.id,
requestId: requestId,
runId: runId,
},
});
Per-call context overrides service defaults, while attributes are shallow-merged. Events include normalized tokens, provider/model, available session and remote IDs, and a streaming flag. They do not estimate currency cost; calculate that downstream against a versioned pricing table.
DeepSeek Notes
import { ai, AxAIDeepSeekModel } from '@ax-llm/ax';
const deepseek = ai({
name: 'deepseek',
apiKey: process.env.DEEPSEEK_APIKEY!,
config: { model: AxAIDeepSeekModel.DeepSeekV4Flash },
});
DeepSeek's current API models are deepseek-v4-flash and deepseek-v4-pro.
Legacy model enum/catalog values remain available for source compatibility, but
profile rules are applied only to model IDs verified for the selected deployment.
DeepSeek V4 supports thinking mode. When thinkingTokenBudget is omitted, Ax
selects its logical max level and sends thinking: { type: "enabled" } with
reasoning_effort: "max". Set thinkingTokenBudget: "none" explicitly to
disable it. DeepSeek's API exposes low, medium, high, and max:
Ax maps minimal and low to low, preserves medium, maps high to high,
and maps highest to max. DeepSeek V4
thinking models support tools, but reject the tool_choice request parameter,
so Ax omits auto and Ax-generated __axOutput tool choices for deepseek-v4-pro,
deepseek-v4-flash, and deepseek-reasoner while still sending tool
definitions. An explicitly forced caller tool choice fails before the request.
DeepSeek does not support native JSON
schema structured outputs. Ax therefore uses validated json_object for a
single required string or code output, including an AxAgent actor's
javascriptCode field, without exposing a provider tool. Richer structured
outputs use the synthetic __axOutput function when function calling is
available, or validated json_object when it is not.
DeepSeek Chat returns thinking traces as reasoning_content. During a tool
loop, Ax preserves that field on the assistant tool-call message and sends it
back on the following request together with non-null content and the original
tool calls. This compatibility mode is declared by the DeepSeek deployment profile;
the official OpenAI Chat adapter does not emit or expose reasoning_content.
The same logical default is declared independently for verified DeepSeek V4
rules in the Together, Fireworks, and OpenRouter profiles, then mapped to each
deployment's own request dialect. A custom openai-compatible endpoint never
inherits it from a DeepSeek-looking model ID.
Other verified deployment rules follow the same policy. Grok 4.6 maps logical
max to xhigh, while Grok 4.5 and 4.3 map it to high. Groq GPT-OSS and
Cerebras GPT-OSS map it to high; Groq Qwen 3.6 maps it to its documented
reasoning-enabled default; Cerebras Gemma 4 and DeepInfra DeepSeek R1 map it
to high. An explicit none is sent only for model/deployment combinations
that document disabling reasoning. Grok 4.6/4.5 and GPT-OSS on Groq or Cerebras
reject none before network I/O because those APIs do not support disabling
reasoning for those models.
Hugging Face Router remains conservative: routing policies such as :fastest
may choose a different inference provider without changing the base model ID,
so Ax does not attach one provider's reasoning contract to that dynamic route.
Extended Thinking
import { ai, AxAIAnthropicModel } from '@ax-llm/ax';
const claude = ai({
name: 'anthropic',
apiKey: process.env.ANTHROPIC_APIKEY!,
config: { model: AxAIAnthropicModel.Claude55Opus },
});
const res = await claude.chat(
{ chatPrompt: [{ role: 'user', content: 'Solve step by step...' }] },
{ thinkingTokenBudget: 'medium', showThoughts: true },
);
if (!('results' in res)) throw new Error('Expected a non-streaming chat response');
console.log(res.results[0]?.thought);
console.log(res.results[0]?.content);
Gemini 4 Hydrogen
Select AxAIGoogleGeminiModel.Gemini4Hydrogen (gemini-4-hydrogen) with
name: 'google-gemini'. Hydrogen supports native structured output, tools,
streaming, and multimodal input. Logical thinkingTokenBudget values low,
medium, high, and highest map to low, medium, high, and xhigh.
none and minimal map to low; none also hides returned thoughts.
Omitting the level preserves Google's high default. Numeric thinking budgets
are rejected. Native config.thinking.thinkingLevel: 'xhigh' is also supported;
Gemini 3 models clamp xhigh to high.
Hydrogen accepts maxTokens alongside a thinking level. Ax supports a single
response segment of up to 65,536 output tokens and rejects larger output caps;
automatic continuation for longer decodes is not implemented. The 1,048,576-token
context limit includes both input and output. Sampling parameters are omitted,
penalties and multiple candidates are not sent, and explicit contextCache
is unsupported. Google's implicit caching still applies automatically.
Run npm run tsx src/examples/typescript/generation/gemini-hydrogen.ts with
GOOGLE_APIKEY. Python, Java, C++, Go, and Rust also support Hydrogen through
the shared Core implementation and include runnable generation examples. Use
model: "gemini-4-hydrogen" and the native package
thinkingTokenBudget / thinking_token_budget option.
See Google's Hydrogen guide
and pricing.
Budget Levels
| Level | Anthropic (tokens) | Gemini 2.5 (tokens) | Gemini 3 level |
|---|---|---|---|
'none' |
disabled; lowest effort where thinking is always on | 0 on Flash/Lite; minimum on Pro | lowest supported, thoughts hidden |
'minimal' |
1,024 | 200 | minimal, or low when minimal is unsupported |
'low' |
5,000 | 800 | low |
'medium' |
10,000 | 5,000 | medium, or the nearest image/legacy level |
'high' |
20,000 | 10,000 | high |
'highest' |
32,000 | 24,500 | high |
Gemini 3 uses thinkingLevel; Gemini 2.5 and older models use numeric
thinkingBudget. Ax selects the wire field after resolving a named model
preset to its real model. Gemini 3.8 Flash, Gemini 3.7 Flash, and Gemini 3.1 Pro
clamp minimal to low; image and legacy Gemini 3 models clamp to their
documented two-level sets. Numeric Gemini 3 budgets fail locally. none always hides returned
thoughts, even when the model must still perform its minimum amount of thinking.
Gemini 3.8 Live accepts no thinking settings, so Ax sends none. Gemini 3.8 Live
Extended Thinking requires a level: Ax sends medium when none is requested and
clamps minimal to low.
The native google-gemini deployment profile and its aliases use these Gemini
rules, including when configured for Vertex with projectId and region. The
separate OpenAI-compatible vertex-ai profile keeps its own request rules and
does not inherit native Gemini fields from a Gemini-looking model ID.
For GPT-5.6 and GPT-6 (Astra, Sol, Luna, GPT-6.1 Sol), these map to none,
low, low, medium, high, and a top rung that depends on the API surface: xhigh on
Chat Completions, which rejects max, and max on the Responses API, which is
the only place it is served. These models default an omitted effort to
medium, so none is sent explicitly; GPT-6 Astra and GPT-6.1 Sol refuse
none and Ax throws.
Earlier OpenAI models retain their existing mapping.
Anthropic document citations
Use a document chat part with source, optional title and context, and
citations: { enabled: true }. The same value without type works in an AxGen
file input or f.file() field. Source forms are:
{ type: 'text', data: 'plain text' }{ type: 'base64', mediaType: 'application/pdf', data: 'base64 PDF' }{ type: 'url', url: 'https://example.com/source.pdf' }{ type: 'file', fileId: 'uploaded file id' }(first-party Files API){ type: 'content', content: [{ type: 'text', text: 'one citable block' }] }
Existing PDF file parts also accept title, context, citations, and cache.
Enable citations on every document in the request, including earlier messages,
or on none. Native JSON-schema output (output_config.format, including
.useStructured()) is rejected locally when citations are enabled. Use a text
signature and cite the document in the output field description. Ax does not
silently weaken an explicitly requested structured-output contract.
response.results[i].textBlocks preserves each output block's text and citations.
Streaming updates are complete snapshots for the given block index; replace
that block rather than appending its snapshot text. Ordinary result.content
retains its existing text-delta behavior. Citation source offsets point into the
input document, while a text block identifies the output passage they support.
Document citations have no url; inspect type, documentIndex, documentTitle,
snippet, and the corresponding character, page, or content-block range.
After forward() or streamingForward(), use program.getChatLog()[i].textBlocks
for output passage associations, scoped to that round trip. Blocks also carry
resultIndex. getUsage()[i].citations retains a flat source list; document indices
are request-local and must not be treated as global document identifiers.
Routers preserve native document inputs and exclude incompatible providers,
even when media degradation is enabled.
Run src/examples/typescript/generation/anthropic-document-citations.ts for
non-streaming and streaming calls. Python, Java, C++, Go, and Rust support the
same sources and validation. Generated chat results use text_blocks and their
generation logs use text_blocks with result_index, plus usage.citations.
Read these per-request logs using get_chat_log() (Python/C++/Rust),
getChatLog() (Java), or ChatLog (Go). Citation location fields retain the
camelCase names shown above. Public document citation examples are available
under each language's generation/ directory.
Anthropic Model-Specific Behavior
- Opus 5.5, Fable 5.1, Fable 5, Opus 5, Opus 4.8, 4.7, and 4.6 plus Sonnet 5, Sonnet 5.5, and Haiku 5.5:
adaptive thinking, no manual
budget_tokens, and notemperature/topP/topK. When thoughts are requested, Ax asks Anthropic for summarized display; when they are hidden, Ax explicitly requestsdisplay: 'omitted'. - Opus 5.5, Fable 5.1, and Fable 5 always think, so
thinkingTokenBudget: 'none'sends the lowest effort with thoughts hidden instead of disabling it. - Opus 5 and Sonnet 5 think b
…(truncated)