# Desktop Voice Assistant

> Desktop Voice Assistant (Jarvis)

- Skill: `lucadominguez/desktop-voice-assistant` (Agent Skill)
- Install (CLI): `npx skillmds@latest add lucadominguez/desktop-voice-assistant`
- Raw SKILL.md: https://api.skillmd.com/api/skills/lucadominguez/desktop-voice-assistant/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: lucadominguez (https://skillmd.com/u/lucadominguez)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/lucadominguez/desktop-voice-assistant

---

# Desktop Voice Assistant (Jarvis)

Build a wake-word-triggered voice assistant that runs on login, greets via TTS, checks email for relevance, and accepts spoken commands — all powered by Hermes in WSL.

## Architecture

```
┌──────────────────────┐     ┌───────────────────────┐
│  Windows (host)      │     │  WSL (subsystem)      │
│                       │     │                       │
│  jarvis.py ──────────┼────▶│  hermes (AI agent)    │
│  ├─ speech_recognition│    │  fetch_emails.py       │
│  ├─ edge-tts (TTS)   │     │  himalaya OR google-   │
│  ├─ pygame (playback)│     │  workspace API         │
│  └─ wake-word loop   │     │                       │
└──────────────────────┘     └───────────────────────┘
```

**Why this split:** WSL often lacks PulseAudio bridging, so the microphone and speakers are inaccessible from Linux. Windows Python has native audio I/O. Hermes runs in WSL where it's installed and configured. The bridge is `wsl hermes chat -q "..."` called from the Windows process.

**Two variants exist:**
- **Full Jarvis** (template `templates/jarvis.py`) — wake-word voice assistant with email digest, Hermes AI backend, and persistent conversation loop
- **Morning Coach** (template `templates/coach.py`) — standalone motivational routine, no Hermes dependency, simple question-and-response flow, exits after 2–3 minutes. Good for login-only use.

## Components

### 1. Voice Loop (Windows-side Python)

Runs on the Windows host. Two TTS stacks available:

**Recommended: edge-tts + pygame** (neural voices, Jarvis-quality)

```powershell
pip install edge-tts pygame speechrecognition pyaudio
```

edge-tts calls Microsoft's free neural TTS API — produces natural, charismatic voices. `en-GB-RyanNeural` is the most Jarvis-like (deep British male). `en-US-GuyNeural` for deep American. `en-GB-SoniaNeural` for warm British female.

```python
import asyncio, edge_tts, pygame, tempfile, os, time

pygame.mixer.init()

async def generate_speech(text: str) -> str:
    """Convert text to MP3 via edge-tts. Returns temp file path."""
    tmp = tempfile.NamedTemporaryFile(suffix=".mp3", delete=False)
    tmp.close()
    communicate = edge_tts.Communicate(text, "en-GB-RyanNeural", rate="+10%")
    await communicate.save(tmp.name)
    return tmp.name

def speak(text: str):
    audio_path = asyncio.run(generate_speech(text))
    pygame.mixer.music.load(audio_path)
    pygame.mixer.music.play()
    while pygame.mixer.music.get_busy():
        time.sleep(0.05)
    pygame.mixer.music.unload()
    os.unlink(audio_path)
```

Pitfall: edge-tts requires internet (calls Microsoft's API). Otherwise it's free and has no rate limits in practice. The async `asyncio.run()` wrapper is needed because `Communicate.save()` is async — calling `asyncio.run()` each time is fine for a sequential conversation flow.

**Fallback: pyttsx3** (SAPI5, offline, robotic)

```powershell
pip install speechrecognition pyaudio pyttsx3
```

Only two voices available by default on most Windows installs: Microsoft David (male) and Microsoft Zira (female). Both sound robotic compared to edge-tts. To check available voices:

```python
import pyttsx3
e = pyttsx3.init()
for v in e.getProperty('voices'):
    print(v.id, v.name, v.languages)
```

Use pyttsx3 only when offline operation is required or edge-tts fails.

### 2. Hermes Bridge

Call Hermes from Windows Python via `wsl`:

```python
import subprocess

def call_hermes(prompt: str) -> str:
    result = subprocess.run(
        ["wsl", "hermes", "chat", "-q", prompt, "--quiet"],
        capture_output=True, text=True, timeout=120, encoding="utf-8"
    )
    return result.stdout.strip()
```

Each call starts a fresh session — no conversation continuity between commands. For follow-up context, include it in the prompt. If session continuity is needed, use `hermes --resume <session-id> -q "..."` and track the session ID.

**WSL cold-start:** First `wsl` call after reboot has 5–15 second delay. Mitigate by keeping WSL alive with `wsl --exec dbus-launch true` in a background process, or by calling `wsl echo ready` once at script startup before the conversation loop begins.

### Voice Input: Reliable Listening

The single-attempt `listen()` pattern silently fails when the user pauses, speaks too softly, or background noise spikes. Use a retry pattern with escalating timeouts:

```python
def listen(recognizer, mic, timeout: float = 12) -> str | None:
    """Listen with retry. Returns transcribed text or None."""
    for attempt in range(2):
        try:
            with mic as source:
                audio = recognizer.listen(source, timeout=timeout, phrase_time_limit=10)
            text = recognizer.recognize_google(audio).strip()
            if text:
                print(f"[Heard] {text}")
                return text
        except sr.WaitTimeoutError:
            if attempt == 0:
                timeout += 6   # more time on second attempt
                continue
        except sr.UnknownValueError:
            if attempt == 0:
                timeout += 4
                continue
        except sr.RequestError as e:
            print(f"[STT error: {e}]")
            return None
    return None
```

Key details:
- `adjust_for_ambient_noise(source, duration=1.5)` once before the loop — not inside it
- `energy_threshold` around 3500–4000 is a good starting point
- `dynamic_energy_threshold = True` adapts to changing room noise
- Two attempts with increasing timeout handles the common "user paused before speaking" case
- Always print what was heard so the user can see if STT got it wrong

### Conversational Callback Pattern

When the assistant asks a question and the user dodges ("nothing", "I don't know", "tired", "maybe"), push back instead of accepting the non-answer:

```python
CALL_OUTS = {
    "nothing": "Come on. There is always something. Name one thing you'd be proud of by tonight.",
    "dont know": "Then make one up. One thing that would make today a win. Go.",
    "tired": "Tired is a feeling, not a fact. What if you gave it twenty minutes and THEN decided?",
    "not sure": "Take a guess. First thing that comes to mind.",
    "maybe": "Maybe is not a plan. Commit to something. Right here, right now.",
}

def detect_callout(text: str) -> str | None:
    lower = text.lower().strip()
    for keyword, response in CALL_OUTS.items():
        if keyword in lower:
            return response
    if len(lower) < 4 and lower not in ["yes", "no", "yeah", "yep", "nope"]:
        return "That was pretty short. Give me a real answer."
    return None
```

Loop this up to 3–4 retries before giving up and providing a default goal/response.

### 3. Email Digest Pipeline

Two backends available:

**Option A: himalaya (IMAP, simpler)** — recommended for email-only use. No Google Cloud project needed. Setup:
1. Enable 2FA on Gmail → generate App Password at https://myaccount.google.com/apppasswords
2. Install himalaya: `cargo install himalaya --locked` or use pre-built binary
3. Configure `~/.config/himalaya/config.toml` with Gmail IMAP/SMTP + App Password
4. Use `templates/fetch_emails_himalaya.py` — fetches envelopes via `himalaya envelope list --output json`, filters for unread by checking for absence of `"seen"` flag

himalaya config for Gmail:
```toml
[accounts.default]
email = "you@gmail.com"
display-name = "Your Name"
default = true

backend.type = "imap"
backend.host = "imap.gmail.com"
backend.port = 993
backend.encryption.type = "tls"
backend.login = "you@gmail.com"
backend.auth.type = "password"
backend.auth.cmd = "cat ~/.config/himalaya/app-password"

message.send.backend.type = "smtp"
message.send.backend.host = "smtp.gmail.com"
message.send.backend.port = 587
message.send.backend.encryption.type = "start-tls"
message.send.backend.login = "you@gmail.com"
message.send.backend.auth.type = "password"
message.send.backend.auth.cmd = "cat ~/.config/himalaya/app-password"

folder.aliases.sent = "[Gmail]/Sent Mail"
folder.aliases.drafts = "[Gmail]/Drafts"
folder.aliases.trash = "[Gmail]/Trash"
```

Store the App Password in `~/.config/himalaya/app-password` (chmod 600) so config.toml doesn't contain secrets directly.

**Option B: google-workspace (OAuth, full Google API)** — required for Calendar, Drive, Docs, or if the Google account doesn't support App Passwords. Setup involves creating a Google Cloud project, enabling APIs, and OAuth consent flow. See the `google-workspace` skill for full setup. Use `templates/fetch_emails.py` with the Gmail API backend.

Both templates produce the same JSON output shape, so the Jarvis voice loop works with either backend unchanged.

### Relevance Filtering

The filtering prompt is the critical piece. It must define what "relevant" means clearly:

```
You are Jarvis. Here are my recent unread emails: [JSON]

Analyze and tell me ONLY about ones that are personally relevant:
- Requires my response or action
- Pertains to my real life (not marketing, newsletters, spam)
- Could affect me personally (bills, appointments, security)
- From a real person I'd care about

Skip: marketing, newsletters, promotions, social media, spam,
receipts, generic updates. Respond in natural Jarvis tone. Be concise.
```

### 4. Windows Startup Integration

Place a `.bat` file in the Startup folder:

```
C:\Users\<user>\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\
```

Content:
```bat
@echo off
start wt --title "J.A.R.V.I.S." -d "C:\Users\<user>\Desktop\AI" cmd /k "python jarvis.py"
```

This launches Windows Terminal on login. The `cmd /k` keeps the window open after the script exits so the user can see the final output.

Alternative: use a `.vbs` script for silent/minimized launch if visual output isn't needed.

## Setup Checklist

1. Install Windows Python dependencies: `pip install edge-tts pygame speechrecognition pyaudio`
2. Choose email backend:
   - **himalaya** (simpler): Enable 2FA on Gmail, generate App Password, install himalaya (`cargo install himalaya --locked`), configure `~/.config/himalaya/config.toml`, use `templates/fetch_emails_himalaya.py`
   - **google-workspace** (OAuth): Load the `google-workspace` skill and follow its setup flow, use `templates/fetch_emails.py`
3. Customize the chosen fetch script → `~/jarvis/fetch_emails.py` in WSL. Update `WSL_FETCH_EMAILS` path in jarvis.py accordingly.
4. Customize `templates/jarvis.py` (or `templates/coach.py` for coach-only) → destination on Windows
5. Place a startup `.bat` in the Windows Startup folder (see Startup Integration above)
6. Test: `python jarvis.py` → say wake word → give a command

## Support Files

- `templates/jarvis.py` — Full voice assistant (edge-tts TTS, wake-word loop, Hermes bridge, email digest). **Updated to edge-tts+pygame stack.**
- `templates/jarvis.bat` — Windows Startup folder launcher script (auto-starts Jarvis on login via Windows Terminal).
- `templates/coach.py` — Standalone morning coach (no Hermes needed). edge-tts voice, question-and-response flow, exits after 2–3 min.
- `templates/fetch_emails.py` — google-workspace backend email fetcher (OAuth required)
- `templates/fetch_emails_himalaya.py` — himalaya backend email fetcher (App Password, simpler)
- `references/wsl-windows-commands.md` — WSL → Windows command execution pitfalls and workarounds
- `references/himalaya-install-pitfalls.md` — himalaya installation (rustup/cargo path, Gmail 2FA prerequisite)

## Caveats

- **WSL must be running.** The `wsl` command auto-starts WSL if it's not running, but first launch has a cold-start delay of 5-15 seconds. Call `wsl echo ready` at script startup to warm WSL before the conversation loop begins.
- **edge-tts requires internet.** Calls Microsoft's free neural TTS API. No API key needed, no rate limits in practice. Offline fallback: pyttsx3 (see TTS section above).
- **Google STT requires internet.** `recognize_google()` calls Google's free speech API. Offline fallback: install Vosk (`pip install vosk`) and download a model.
- **No conversation continuity.** Each `hermes chat -q` is a fresh session. Hermes's persistent memory still applies (preferences, facts), but chat context doesn't carry over.
- **ANSI escape codes in Hermes output.** `hermes chat -q` may include terminal formatting in stdout even with `--quiet`. Always strip ANSI escape sequences before speaking: `re.sub(r'\\x1b\\[[0-9;]*m', '', output)`.
- **WSL → Windows command execution.** When calling Windows commands from WSL bash, backslashes in paths are eaten by bash, and quotes are mangled when passing through `cmd.exe /c`. The reliable workaround is to write a small `.bat` file on the Windows filesystem and call that instead. See `references/wsl-windows-commands.md` for the full pattern and examples from real debugging sessions.