# VoiceMode Architecture

> Understanding how VoiceMode components work together to enable voice conversations.

- Skill: `tools-only/voicemode-architecture` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add tools-only/voicemode-architecture`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/voicemode-architecture/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: tools-only (https://skillmd.com/u/tools-only)
- Updated: 2026-09-29
- Page: https://skillmd.com/skills/tools-only/voicemode-architecture

---

# VoiceMode Architecture

Understanding how VoiceMode components work together to enable voice conversations.

## System Overview

VoiceMode is built as a Model Context Protocol (MCP) server that provides voice capabilities to AI assistants. It follows a modular architecture with clear separation between voice services, audio processing, and client interfaces.

```
┌─────────────────────────────────────────────┐
│             MCP Client (Claude)             │
└─────────────────┬───────────────────────────┘
                  │ MCP Protocol
┌─────────────────┴───────────────────────────┐
│           VoiceMode MCP Server              │
├──────────────────────────────────────────────┤
│              Core Components                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  │
│  │  Tools   │  │ Providers│  │  Config  │  │
│  └──────────┘  └──────────┘  └──────────┘  │
├──────────────────────────────────────────────┤
│            Voice Services                    │
│  ┌──────────────────┐  ┌──────────────────┐  │
│  │     Whisper      │  │      Kokoro      │  │
│  │      (STT)       │  │       (TTS)      │  │
│  └──────────────────┘  └──────────────────┘  │
└──────────────────────────────────────────────┘
```

## Core Components

### MCP Server

The FastMCP-based server (`server.py`) is the entry point that:
- Exposes tools, resources, and prompts via MCP protocol
- Handles stdio transport for communication
- Manages service lifecycle and health checks
- Auto-imports all tools from the tools directory

### Tools System

Tools are the primary interface for voice interactions:

**converse**: Main voice conversation tool
- Handles audio recording and playback
- Manages TTS/STT service selection
- Implements silence detection and VAD
- Uses local microphone for audio capture

**Service tools**: Installation and management
- `whisper_install`, `kokoro_install`
- Service start/stop/status operations
- Model and configuration management

### Provider System

The provider system (`providers.py`) implements service discovery and failover:

1. **Discovery**: Automatically finds running services
2. **Health Checks**: Validates service availability
3. **Failover**: Falls back to alternative services
4. **Load Balancing**: Distributes requests across providers

Provider selection priority:
1. User-specified URL (environment variable)
2. Local services (auto-discovered)
3. Cloud services (OpenAI)

### Configuration Layer

Multi-layered configuration system (`config.py`):

1. **Environment Variables**: Highest priority
2. **Project Config**: `.voicemode.env` in working directory
3. **User Config**: `~/.voicemode/voicemode.env`
4. **Defaults**: Built-in sensible defaults

## Voice Services

### Whisper (Speech-to-Text)

Local STT service using OpenAI's Whisper model:
- Runs on port 2022 by default
- Provides OpenAI-compatible API
- Supports multiple model sizes
- Hardware acceleration (Metal, CUDA)

### Kokoro (Text-to-Speech)

Local TTS service with natural voices:
- Runs on port 8880 by default
- OpenAI-compatible API
- Multiple languages and voices
- Efficient caching system

## Audio Pipeline

### Recording Flow

```
Microphone → Audio Capture → VAD → Silence Detection → STT Service → Text
```

1. **Audio Capture**: PyAudio for microphone input
2. **VAD**: WebRTC VAD filters non-speech
3. **Silence Detection**: Determines recording end
4. **STT Processing**: Converts audio to text

### Playback Flow

```
Text → TTS Service → Audio Stream → Format Conversion → Speaker
```

1. **TTS Generation**: Creates audio from text
2. **Streaming**: Chunks for real-time playback
3. **Format Conversion**: FFmpeg handles formats
4. **Playback**: PyAudio for speaker output

## Service Architecture

### Service Lifecycle

1. **Installation**: Download binaries, create configs
2. **Registration**: systemd/launchd service files
3. **Startup**: Health checks, port binding
4. **Discovery**: Auto-detection by VoiceMode
5. **Monitoring**: Status checks, log rotation

### Service Communication

All services expose OpenAI-compatible APIs:
- Unified interface for TTS/STT
- Standard authentication (API keys)
- Consistent error handling
- Format negotiation

## Audio Transport

### Local Microphone

Direct microphone/speaker access using PyAudio:
- Low latency audio I/O
- No network overhead
- Privacy-focused (all processing local)
- WebRTC VAD for voice activity detection

## Security Model

### API Key Management

- Never stored in code
- Environment variable priority
- Secure MCP transport
- Optional local-only mode

### Audio Privacy

- Local processing option
- No cloud storage
- User-controlled recording

## Performance Optimization

### Caching Strategy

- Model caching (Whisper/Kokoro)
- Audio format caching
- Provider health caching
- Configuration caching

### Resource Management

- Lazy service loading
- Connection pooling
- Memory limits (systemd)
- CPU throttling

## Error Handling

### Graceful Degradation

1. Primary service fails
2. Attempt fallback service
3. Use cloud service if available
4. Return informative error

### Recovery Mechanisms

- Automatic service restart
- Connection retry logic
- Circuit breaker pattern
- Health check recovery

## Extension Points

### Adding New Tools

1. Create tool in `tools/` directory
2. Implement with FastMCP decorators
3. Auto-imported by server
4. Available via MCP

### Custom Providers

1. Implement provider interface
2. Add discovery logic
3. Register in provider system
4. Configure endpoints

### Service Integration

1. Create service installer
2. Add systemd/launchd templates
3. Implement health checks
4. Update CLI commands

## Deployment Patterns

### Development

- Local services
- Debug logging
- Hot reload
- Mock providers

### Production

- Service supervision
- Log rotation
- Health monitoring
- Failover configuration

### Containerized

- Docker compose setup
- Service orchestration
- Volume management
- Network isolation

## Future Architecture

### Planned Enhancements

- Plugin system for tools
- Webhook support
- Multi-language support
- GPU cluster support

### Scalability Path

- Distributed services
- Queue-based processing
- Caching layers
- Load balancing