bx-ai: Audio (Speech, Streaming, Transcription)
BoxLang is the AI-native software productivity platform for building, modernizing and running applications, with developers and AI agents working together.
To let an agent edit and measure audio files (trim, mix, master, analyze, local speech), install
bx-audio: it registers tools such asaudioTrim@bxaudioin the tool registry. See thebx-audioskill.speak@bxaiuses a cloud provider,audioSpeak@bxaudioruns a local voice.
Text to Speech: aiSpeak()
aiSpeak( text="", params={}, options={} )
Returns depend on the call:
| Call | Returns |
|---|---|
aiSpeak( "Hello" ) |
AiSpeechResponse |
aiSpeak( "Hello", {}, { outputFile: "/tmp/a.mp3" } ) |
the saved file path (string) |
aiSpeak() (no text) |
fluent AiSpeechRequest builder |
response = aiSpeak( "Welcome to BoxLang!", { voice: "nova" }, { provider: "openai", outputFormat: "mp3" } )
response.saveToFile( "/tmp/welcome.mp3" ) // returns absolute path, creates parent dirs
response.getSize() // bytes
response.getMimeType() // "audio/mpeg"
AiSpeechResponse methods: saveToFile(), getBase64(), getMimeType(), toDataURI(), toStruct(), hasAudio(), getSize().
Where things go:
params: provider body fields:model,voice,speed,instructions, plus provider-native fields.options:provider,apiKey,outputFile,outputFormat,timeout(default 30), logging flags.voice,speedandinstructionsare also accepted here.outputFormatdefaults tomp3.
Gender keywords
voice: "male" or voice: "female" (or .male() / .female()) is resolved to a concrete voice through audio.voiceGenderMap for the active provider. If the map has an empty value (Mistral), the provider default voice is used.
aiSpeak( "Hi there", { voice: "female" }, { provider: "elevenlabs" } )
Fluent Builder
aiSpeak()
.text( "Your order has shipped." )
.provider( "cartesia" )
.female()
.asWav()
.outputFile( "/tmp/order.wav" )
.speak()
Real methods on AiSpeechRequest: text, model, provider, apiKey, voice, male, female, speed, instructions, outputFormat, asMP3, asWav, asFlac, asOpus, asPCM, outputFile, timeout, withParams, withOptions, withLogging. Terminators: speak() and stream( callback ). Static factory: AiSpeechRequest::of( text ).
asFlac() and asOpus() exist, but Cartesia throws InvalidArgument for them. Check the provider docs before using them.
Streaming Speech: aiSpeakStream()
aiSpeakStream( text, callback, params={}, options={} )
The callback receives one struct per event, discriminated by type:
| Event | Shape |
|---|---|
| audio | { type: "audio", data: binary, format, sampleRate, sequence } |
| timestamps | { type: "timestamps", words: [], start: [], end: [] } (providers that support them, e.g. Cartesia SSE) |
| done | { type: "done", chunks, bytes } (always last) |
Return an explicit false from the callback to stop early (barge-in, client disconnect). Any other return value, including nothing, continues.
summary = aiSpeakStream(
"Thanks for calling BoxLang support.",
( event ) => {
if ( event.type == "audio" ) {
if ( !socket.send( event.data ) ) return false // client gone: stop generating
}
},
{ voice: "female" },
{ provider: "cartesia", outputFormat: "pcm" }
)
// summary: { provider, model, audioFormat, sampleRate, chunks, bytes, completed }
Fluent: aiSpeak().text( "Hi" ).provider( "cartesia" ).asPCM().stream( ( e ) => handle( e ) ).
options.timeoutis an idle timeout in seconds (longest wait for headers or between received bytes), not a total duration cap. Default 30.- Empty text throws
InvalidArgument. - Providers without the
speechStreamcapability throwUnsupportedCapability. Grok does NOT support streaming (its streaming API is a WebSocket, not implemented).
Custom providers
Implement IAiSpeechStreamService (struct function speakStream( AiSpeechRequest speechRequest, function callback ), returning the summary struct) and include "speechStream" in the provider's CAPABILITIES. Plain TTS needs IAiSpeechService (speak( speechRequest ), returns AiSpeechResponse) and "speech". SpeechStreamHelper (streamBinary, streamSSE) is the shared transport used by the built-in providers.
Choosing a Format
| Use case | outputFormat |
|---|---|
Browser <audio> tag, saved files |
mp3 |
| Voice agents, Web Audio, lowest latency | pcm |
| Telephony (8 kHz) | mulaw or alaw (Cartesia) |
| Whole-file playback without compression | wav |
Streaming to a browser over HTTP
Pattern: the callback writes each audio chunk to the HTTP response and flushes; when the write fails (listener closed the tab), return false so the provider stops generating. Two modes: raw mp3 bytes for <audio src="speak.bxm?...">, or newline-delimited JSON events (base64 PCM plus timestamps) for a Web Audio player. See the runnable MiniServer demo in the bx-ai repo: examples/http-streaming-speech/ (speak.bxm, speak-events.bxm, index.html). Also examples/advanced/15-audio-tts-example.bxs and 19-audio-streaming-example.bxs.
Cartesia Provider
- Sonic TTS (default model
sonic-3.6) and Ink-Whisper STT (ink-whisper). Capabilities:speech,speechStream,transcription. - Formats:
mp3,wav,pcm,mulaw,alaw.flacandopusthrowInvalidArgument. - Default sample rates: pcm 24000, mp3 and wav 44100, mulaw/alaw 8000. Override with
params.sample_rate(mp3 also acceptsparams.bit_rate). - Params:
speed,volume,emotion(folded intogeneration_config), and pass-throughlanguage,locale,accent,normalization,pronunciation_dict_id,add_timestamps,add_phoneme_timestamps,context_id. - Streaming transport: mp3 and wav use
/tts/bytes(no timestamps); pcm, mulaw and alaw use SSE with word timestamps.params.transport("bytes"or"sse") forces one; SSE with mp3/wav throwsInvalidArgument. translate()throwsUnsupportedCapability.
aiSpeakStream(
"Hello", ( e ) => { if ( e.type == "timestamps" ) println( e.words ) },
{ add_timestamps: true, sample_rate: 24000, emotion: "happy" },
{ provider: "cartesia", outputFormat: "pcm" }
)
Speech to Text: aiTranscribe() and aiTranslate()
aiTranscribe( audio="", params={}, options={} )
aiTranslate( audio="", params={}, options={} ) // speech in, English text out
audio is a local file path, a URL, or binary data. Default returnFormat is "text" (a string). options.returnFormat: "response" returns an AiTranscriptionResponse (getText, getLanguage, getWords, getSegments, getDuration, getModel, getProvider, getWordCount, getFormattedDuration, hasWords, hasSegments, toStruct). Called with no audio, both return a fluent AiTranscriptionRequest.
text = aiTranscribe( "/tmp/call.mp3", { language: "es" }, { provider: "groq" } )
res = aiTranscribe( "/tmp/call.wav", {}, { provider: "cartesia", returnFormat: "response" } )
println( res.getText() & " (" & res.getLanguage() & ")" )
aiTranscribe().file( "/tmp/a.mp3" ).provider( "openai" ).language( "en" ).withWordTimestamps().transcribe()
aiTranslate().file( "/tmp/es.mp3" ).provider( "openai" ).translate()
Transcription builder methods: file, url, data, model, provider, apiKey, language, inputFormat, timeout, withWordTimestamps, withSegmentTimestamps, withTimestamps, diarize, asJSON, asText, asVerboseJSON, asSRT, asVTT, withParams, withOptions, withLogging, terminators transcribe() and translate().
Audio Capability Table
From each service's CAPABILITIES array:
| Provider | speech | speechStream | transcription | translation |
|---|---|---|---|---|
openai |
yes | yes | yes | yes |
cartesia |
yes | yes | yes | throws |
elevenlabs |
yes | yes | yes | no (delegates to transcribe) |
mistral |
yes | yes | yes | no (delegates to transcribe) |
gemini |
yes | yes | yes | no (prompt-based) |
grok |
yes | no | no | no |
groq |
no | no | yes | yes |
Only translation listed in CAPABILITIES counts as native translation. ElevenLabs, Mistral and Gemini implement translate() but do not translate through a dedicated endpoint.
API keys
aiService() resolves the key in order: explicit apiKey option, then the <PROVIDER>_API_KEY environment or system setting (CARTESIA_API_KEY, ELEVENLABS_API_KEY, OPENAI_API_KEY, GROQ_API_KEY, ...), then module settings. audio.defaultApiKey is only used when no convention variable exists for the resolved provider.
Module Settings (audio)
// boxlang.json > modules > bxai > settings
audio : {
defaultProvider : "openai",
defaultApiKey : "",
defaultVoice : "",
defaultOutputFormat : "mp3",
defaultSpeechModel : "",
defaultTranscriptionModel : "",
voiceGenderMap : {
openai : { male: "ash", female: "nova" },
grok : { male: "rex", female: "eve" },
gemini : { male: "Fenrir", female: "Aoede" },
elevenlabs : { male: "CwhRBWXzGAHq8TQ4Fs17", female: "EXAVITQu4vr4xnSDxMaL" }
// mistral (empty, provider default) and cartesia (voice UUIDs) also ship
}
}
Override a single entry, e.g. audio.voiceGenderMap.openai.male = "echo".
Events
Announced via BoxAnnounce:
| Event | Payload |
|---|---|
beforeAISpeech / afterAISpeech |
speechRequest, service, (result on after); stream: true when from aiSpeakStream() (result is the summary struct) |
beforeAITranscription / afterAITranscription |
transcriptionRequest, service, (result on after) |
beforeAITranslation / afterAITranslation |
transcriptionRequest, service, (result on after) |
Common Pitfalls
- ❌ Do NOT hardcode API keys in source
- ✅ Use
<PROVIDER>_API_KEYenvironment variables or module settings
- ✅ Use
- ❌ Do NOT expose a speech-streaming endpoint without authentication and rate limits: every request spends provider credits (the demo endpoints are demo only)
- ✅ Cap text length and whitelist providers and formats server-side
- ❌ Do NOT ignore client disconnects in the stream callback
- ✅ Return an explicit
falsewhen the write fails so the provider stops generating
- ✅ Return an explicit
- ❌ Do NOT use
mp3for voice agents or telephony- ✅ Use
outputFormat: "pcm"(voice agents) ormulaw/alaw(telephony, Cartesia)
- ✅ Use
- ❌ Do NOT call
aiSpeakStream()with Grok or Groq: they lack thespeechStreamcapability and throwUnsupportedCapability - ❌ Do NOT use
flacoropuswith Cartesia, oraiTranslate()with Cartesia - ❌ Do NOT treat
timeoutas a total duration limit for streams: it is an idle timeout - ✅ Handle only
type == "audio"events for playback;timestampsanddonecarry no audio bytes - ✅
aiSpeak()withoptions.outputFilereturns a path string, not a response object