Text-to-Speech Chunking and Parallel Synthesis Strategy

Text-to-Speech Chunking and Parallel Synthesis Strategy

Overview

This document describes the intelligent chunking and parallel synthesis strategy used in the TTS feature to minimize audio playback latency and ensure seamless transitions between chunks.

Version Implemented: 0.234.204

Goals

1. Fast Initial Playback: Audio should start playing within 1-2 seconds
2. Seamless Transitions: No pauses or gaps between chunks during playback
3. Efficient Synthesis: Use parallel API calls to minimize total synthesis time
4. Scalability: Handle messages of any length from short phrases to long responses

Chunking Strategy

Progressive Word-Based Chunks

Text is split into progressively larger chunks to balance fast initial playback with efficient synthesis:

Chunk 1: 10 words   (plays immediately, fast synthesis)
Chunk 2: 15 words   (synthesized with chunk 1, queued for seamless transition)
Chunk 3: 20 words   (synthesized in Group 1)
Chunk 4: 25 words   (synthesized in Group 1)
Chunk 5: 30 words   (synthesized in Group 1)
Chunk 6+: 40 words each (synthesized in subsequent groups)

Why Progressive Sizing?

• Small initial chunks (10, 15 words) ensure fast synthesis and quick audio start
• Growing chunk sizes (20, 25, 30 words) provide smooth transitions while reducing total API calls
• Large final chunks (40 words) efficiently handle remaining content without excessive API requests

Parallel Synthesis Strategy

Phase 1: Initial Parallel Synthesis (Chunks 1 & 2)

// Synthesize chunks 1 and 2 simultaneously
const [chunk1Audio, chunk2Audio] = await Promise.all([
    synthesizeChunk(chunk1Text),
    synthesizeChunk(chunk2Text)
]);

// Start playing chunk 1 immediately
playAudio(chunk1Audio);

// Queue chunk 2 (already synthesized, ready to play)
audioQueue.push(chunk2Audio);

Timeline:

• t=0s: Start synthesis of chunks 1 & 2 in parallel
• t=2s: Both chunks complete synthesis
• t=2s: Chunk 1 starts playing
• t=5s: Chunk 1 finishes, chunk 2 plays immediately (no pause!)

Phase 2: Group-Based Parallel Synthesis (Chunks 3+)

While chunks 1 and 2 are playing, remaining chunks are synthesized in groups of 5:

Group 1 (Chunks 3-7):

// While chunks 1 & 2 play, synthesize chunks 3-7 in parallel
await Promise.all([
    synthesizeChunk(chunk3),  // 20 words
    synthesizeChunk(chunk4),  // 25 words
    synthesizeChunk(chunk5),  // 30 words
    synthesizeChunk(chunk6),  // 40 words
    synthesizeChunk(chunk7)   // 40 words
]);

Group 2 (Chunks 8-12):

• After Group 1 completes, synthesize next 5 chunks in parallel
• Process continues until all chunks are synthesized

Group N:

• Final group may have fewer than 5 chunks
• All remaining chunks synthesized in parallel

Playback Flow

Example: 100-word message

Word Distribution:
├─ Chunk 1: Words 1-10     (10 words)
├─ Chunk 2: Words 11-25    (15 words)
├─ Chunk 3: Words 26-45    (20 words)
├─ Chunk 4: Words 46-70    (25 words)
└─ Chunk 5: Words 71-100   (30 words)

Synthesis Timeline:
t=0s:   Start synthesizing chunks 1 & 2 in parallel
t=2s:   Chunks 1 & 2 complete, chunk 1 starts playing
        Start synthesizing chunks 3-5 in parallel
t=3s:   Chunk 1 finishes, chunk 2 plays immediately
t=4s:   Chunks 3-5 complete synthesis
t=5s:   Chunk 2 finishes, chunk 3 plays immediately
t=7s:   Chunk 3 finishes, chunk 4 plays immediately
t=10s:  Chunk 4 finishes, chunk 5 plays immediately
t=13s:  Chunk 5 finishes, playback complete

Result: No pauses between chunks, seamless playback!

Implementation Details

Code Location

• File: static/js/chat/chat-tts.js
• Function: playTTS() - Main orchestration
• Function: queueChunksInGroups() - Group-based parallel synthesis

Key Components

1. Text Chunking

const words = plainText.split(/\s+/);
const chunks = [];
let index = 0;

// Progressive chunks: 10, 15, 20, 25, 30 words
if (words.length > index) {
    chunks.push(words.slice(index, index + 10).join(' '));
    index += 10;
}
if (words.length > index) {
    chunks.push(words.slice(index, index + 15).join(' '));
    index += 15;
}
// ... continues for 20, 25, 30

// Remaining in 40-word chunks
while (index < words.length) {
    chunks.push(words.slice(index, index + 40).join(' '));
    index += 40;
}

2. Initial Parallel Synthesis

const firstChunk = chunks.shift();
const secondChunk = chunks.shift();

const [firstAudio, secondAudio] = await Promise.all([
    synthesizeChunk(firstChunk, messageId),
    synthesizeChunk(secondChunk, messageId)
]);

// Play first, queue second
currentPlayingAudio = firstAudio;
audioQueue.push({ audio: secondAudio, url: secondAudio.src });
await currentPlayingAudio.play();

3. Group-Based Queue Processing

while (chunks.length > 0) {
    const groupSize = Math.min(5, chunks.length);
    const groupChunks = chunks.splice(0, groupSize);
    
    // Synthesize all in parallel
    const results = await Promise.all(
        groupChunks.map(text => synthesizeChunk(text, messageId))
    );
    
    // Add to queue in order
    results.forEach(audio => audioQueue.push(audio));
}

4. Continuous Playback

currentPlayingAudio.onended = () => {
    playNextChunk(messageId);  // Automatically plays next queued chunk
};

Performance Metrics

Typical Performance (Azure Speech Service)

• Single chunk synthesis: ~2 seconds
• 5 chunks in parallel: ~2-3 seconds (not 10 seconds!)
• Audio start latency: 2 seconds (time to synthesize chunk 1)
• Chunk transition delay: 0ms (next chunk already synthesized and queued)

Comparison: Sequential vs Parallel

Sequential Synthesis (Old Approach):

Chunk 1: 2s synthesis → play
Chunk 2: 2s synthesis → play (user experiences pause)
Chunk 3: 2s synthesis → play (user experiences pause)
Total: 6s before all chunks synthesized

Parallel Synthesis (Current Approach):

Chunks 1 & 2: 2s synthesis in parallel
Chunks 3-7:   2s synthesis in parallel (while 1 & 2 play)
Total: 4s before all chunks synthesized
No pauses during playback

Edge Cases Handled

1. Very Short Messages (<10 words)

• Single chunk created
• No grouping needed
• Plays immediately after synthesis

2. Messages with Odd Word Counts

• Final chunk may be smaller than 40 words
• Last group may have fewer than 5 chunks
• All handled gracefully with Math.min() and conditionals

3. Synthesis Failures

• Each chunk synthesis wrapped in try-catch
• Failed chunks logged but don't block other chunks
• Playback continues with available chunks

4. Race Conditions

• isQueueing flag prevents overlapping queue operations
• Queue checked with retry logic (100ms intervals, up to 10 retries)
• Ensures chunk 2 is available when chunk 1 ends

Debugging and Monitoring

Console Logging

The implementation includes comprehensive debug logging:

[TTS] Split into 5 chunks: [10, 15, 20, 25, 30] words
[TTS] Synthesizing chunks 1 and 2 in parallel...
[TTS] Chunk 2 pre-queued, ready to play after chunk 1
[TTS] Playing chunk 1 immediately
[TTS] Group 1: Synthesizing 3 chunks in parallel
[TTS] Starting synthesis for chunk 3 (20 words)
[TTS] Starting synthesis for chunk 4 (25 words)
[TTS] Starting synthesis for chunk 5 (30 words)
[TTS] Chunk 3 queued (20 words), queue size: 1
[TTS] Group 1 complete, 0 chunks remaining
[TTS] All 1 groups complete, total queue size: 3

Backend Logging

Backend route includes debug_print statements:

• Request received with text length
• Azure Speech SDK configuration
• Synthesis completion time
• Audio data size
• Error conditions

Future Optimizations

Potential Improvements

1. Dynamic chunk sizing based on average synthesis time
2. Adaptive grouping based on network latency
3. Pre-synthesis caching for common phrases
4. Streaming synthesis if Azure Speech SDK supports it
5. Predictive synthesis for anticipated follow-up messages

Monitoring Considerations

• Track synthesis time per chunk size
• Monitor pause duration between chunks
• Log user interruptions/stops
• Measure total playback completion rate

Related Files

• Frontend: static/js/chat/chat-tts.js
• Backend: route_backend_tts.py
• Feature Documentation: TEXT_TO_SPEECH.md
• Configuration: config.py (VERSION)

Version History

• 0.234.199: Initial parallel synthesis (all chunks at once)
• 0.234.200: First attempt at dual-chunk synthesis (chunks 1+2)
• 0.234.201: Staged synthesis (1, then 2, then 3+4, etc.)
• 0.234.202: Unlimited staging with pairs
• 0.234.203: Progressive chunking with group-based synthesis
• 0.234.204: Final optimization - chunks 1+2 parallel, then groups of 5