Desktop Voice Assistant

Desktop Voice Assistant (Jarvis)

Build a wake-word-triggered voice assistant that runs on login, greets via TTS, checks email for relevance, and accepts spoken commands — all powered by Hermes in WSL.

Architecture

┌──────────────────────┐     ┌───────────────────────┐
│  Windows (host)      │     │  WSL (subsystem)      │
│                       │     │                       │
│  jarvis.py ──────────┼────▶│  hermes (AI agent)    │
│  ├─ speech_recognition│    │  fetch_emails.py       │
│  ├─ edge-tts (TTS)   │     │  himalaya OR google-   │
│  ├─ pygame (playback)│     │  workspace API         │
│  └─ wake-word loop   │     │                       │
└──────────────────────┘     └───────────────────────┘

Why this split: WSL often lacks PulseAudio bridging, so the microphone and speakers are inaccessible from Linux. Windows Python has native audio I/O. Hermes runs in WSL where it's installed and configured. The bridge is wsl hermes chat -q "..." called from the Windows process.

Two variants exist:

• Full Jarvis (template templates/jarvis.py) — wake-word voice assistant with email digest, Hermes AI backend, and persistent conversation loop
• Morning Coach (template templates/coach.py) — standalone motivational routine, no Hermes dependency, simple question-and-response flow, exits after 2–3 minutes. Good for login-only use.

Components

1. Voice Loop (Windows-side Python)

Runs on the Windows host. Two TTS stacks available:

Recommended: edge-tts + pygame (neural voices, Jarvis-quality)

pip install edge-tts pygame speechrecognition pyaudio

edge-tts calls Microsoft's free neural TTS API — produces natural, charismatic voices. en-GB-RyanNeural is the most Jarvis-like (deep British male). en-US-GuyNeural for deep American. en-GB-SoniaNeural for warm British female.

import asyncio, edge_tts, pygame, tempfile, os, time

pygame.mixer.init()

async def generate_speech(text: str) -> str:
    """Convert text to MP3 via edge-tts. Returns temp file path."""
    tmp = tempfile.NamedTemporaryFile(suffix=".mp3", delete=False)
    tmp.close()
    communicate = edge_tts.Communicate(text, "en-GB-RyanNeural", rate="+10%")
    await communicate.save(tmp.name)
    return tmp.name

def speak(text: str):
    audio_path = asyncio.run(generate_speech(text))
    pygame.mixer.music.load(audio_path)
    pygame.mixer.music.play()
    while pygame.mixer.music.get_busy():
        time.sleep(0.05)
    pygame.mixer.music.unload()
    os.unlink(audio_path)

Pitfall: edge-tts requires internet (calls Microsoft's API). Otherwise it's free and has no rate limits in practice. The async asyncio.run() wrapper is needed because Communicate.save() is async — calling asyncio.run() each time is fine for a sequential conversation flow.

Fallback: pyttsx3 (SAPI5, offline, robotic)

pip install speechrecognition pyaudio pyttsx3

Only two voices available by default on most Windows installs: Microsoft David (male) and Microsoft Zira (female). Both sound robotic compared to edge-tts. To check available voices:

import pyttsx3
e = pyttsx3.init()
for v in e.getProperty('voices'):
    print(v.id, v.name, v.languages)

Use pyttsx3 only when offline operation is required or edge-tts fails.

2. Hermes Bridge

Call Hermes from Windows Python via wsl:

import subprocess

def call_hermes(prompt: str) -> str:
    result = subprocess.run(
        ["wsl", "hermes", "chat", "-q", prompt, "--quiet"],
        capture_output=True, text=True, timeout=120, encoding="utf-8"
    )
    return result.stdout.strip()

Each call starts a fresh session — no conversation continuity between commands. For follow-up context, include it in the prompt. If session continuity is needed, use hermes --resume <session-id> -q "..." and track the session ID.

WSL cold-start: First wsl call after reboot has 5–15 second delay. Mitigate by keeping WSL alive with wsl --exec dbus-launch true in a background process, or by calling wsl echo ready once at script startup before the conversation loop begins.

Voice Input: Reliable Listening

The single-attempt listen() pattern silently fails when the user pauses, speaks too softly, or background noise spikes. Use a retry pattern with escalating timeouts:

def listen(recognizer, mic, timeout: float = 12) -> str | None:
    """Listen with retry. Returns transcribed text or None."""
    for attempt in range(2):
        try:
            with mic as source:
                audio = recognizer.listen(source, timeout=timeout, phrase_time_limit=10)
            text = recognizer.recognize_google(audio).strip()
            if text:
                print(f"[Heard] {text}")
                return text
        except sr.WaitTimeoutError:
            if attempt == 0:
                timeout += 6   # more time on second attempt
                continue
        except sr.UnknownValueError:
            if attempt == 0:
                timeout += 4
                continue
        except sr.RequestError as e:
            print(f"[STT error: {e}]")
            return None
    return None

Key details:

• adjust_for_ambient_noise(source, duration=1.5) once before the loop — not inside it
• energy_threshold around 3500–4000 is a good starting point
• dynamic_energy_threshold = True adapts to changing room noise
• Two attempts with increasing timeout handles the common "user paused before speaking" case
• Always print what was heard so the user can see if STT got it wrong

Conversational Callback Pattern

When the assistant asks a question and the user dodges ("nothing", "I don't know", "tired", "maybe"), push back instead of accepting the non-answer:

CALL_OUTS = {
    "nothing": "Come on. There is always something. Name one thing you'd be proud of by tonight.",
    "dont know": "Then make one up. One thing that would make today a win. Go.",
    "tired": "Tired is a feeling, not a fact. What if you gave it twenty minutes and THEN decided?",
    "not sure": "Take a guess. First thing that comes to mind.",
    "maybe": "Maybe is not a plan. Commit to something. Right here, right now.",
}

def detect_callout(text: str) -> str | None:
    lower = text.lower().strip()
    for keyword, response in CALL_OUTS.items():
        if keyword in lower:
            return response
    if len(lower) < 4 and lower not in ["yes", "no", "yeah", "yep", "nope"]:
        return "That was pretty short. Give me a real answer."
    return None

Loop this up to 3–4 retries before giving up and providing a default goal/response.

3. Email Digest Pipeline

Two backends available:

Option A: himalaya (IMAP, simpler) — recommended for email-only use. No Google Cloud project needed. Setup:

1. Enable 2FA on Gmail → generate App Password at https://myaccount.google.com/apppasswords
2. Install himalaya: cargo install himalaya --locked or use pre-built binary
3. Configure ~/.config/himalaya/config.toml with Gmail IMAP/SMTP + App Password
4. Use templates/fetch_emails_himalaya.py — fetches envelopes via himalaya envelope list --output json, filters for unread by checking for absence of "seen" flag

himalaya config for Gmail:

[accounts.default]
email = "you@gmail.com"
display-name = "Your Name"
default = true

backend.type = "imap"
backend.host = "imap.gmail.com"
backend.port = 993
backend.encryption.type = "tls"
backend.login = "you@gmail.com"
backend.auth.type = "password"
backend.auth.cmd = "cat ~/.config/himalaya/app-password"

message.send.backend.type = "smtp"
message.send.backend.host = "smtp.gmail.com"
message.send.backend.port = 587
message.send.backend.encryption.type = "start-tls"
message.send.backend.login = "you@gmail.com"
message.send.backend.auth.type = "password"
message.send.backend.auth.cmd = "cat ~/.config/himalaya/app-password"

folder.aliases.sent = "[Gmail]/Sent Mail"
folder.aliases.drafts = "[Gmail]/Drafts"
folder.aliases.trash = "[Gmail]/Trash"

Store the App Password in ~/.config/himalaya/app-password (chmod 600) so config.toml doesn't contain secrets directly.

Option B: google-workspace (OAuth, full Google API) — required for Calendar, Drive, Docs, or if the Google account doesn't support App Passwords. Setup involves creating a Google Cloud project, enabling APIs, and OAuth consent flow. See the google-workspace skill for full setup. Use templates/fetch_emails.py with the Gmail API backend.

Both templates produce the same JSON output shape, so the Jarvis voice loop works with either backend unchanged.

Relevance Filtering

The filtering prompt is the critical piece. It must define what "relevant" means clearly:

You are Jarvis. Here are my recent unread emails: [JSON]

Analyze and tell me ONLY about ones that are personally relevant:
- Requires my response or action
- Pertains to my real life (not marketing, newsletters, spam)
- Could affect me personally (bills, appointments, security)
- From a real person I'd care about

Skip: marketing, newsletters, promotions, social media, spam,
receipts, generic updates. Respond in natural Jarvis tone. Be concise.

4. Windows Startup Integration

Place a .bat file in the Startup folder:

C:\Users\<user>\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\

Content:

@echo off
start wt --title "J.A.R.V.I.S." -d "C:\Users\<user>\Desktop\AI" cmd /k "python jarvis.py"

This launches Windows Terminal on login. The cmd /k keeps the window open after the script exits so the user can see the final output.

Alternative: use a .vbs script for silent/minimized launch if visual output isn't needed.

Setup Checklist

1. Install Windows Python dependencies: pip install edge-tts pygame speechrecognition pyaudio
2. Choose email backend:
   • himalaya (simpler): Enable 2FA on Gmail, generate App Password, install himalaya (cargo install himalaya --locked), configure ~/.config/himalaya/config.toml, use templates/fetch_emails_himalaya.py
   • google-workspace (OAuth): Load the google-workspace skill and follow its setup flow, use templates/fetch_emails.py
3. Customize the chosen fetch script → ~/jarvis/fetch_emails.py in WSL. Update WSL_FETCH_EMAILS path in jarvis.py accordingly.
4. Customize templates/jarvis.py (or templates/coach.py for coach-only) → destination on Windows
5. Place a startup .bat in the Windows Startup folder (see Startup Integration above)
6. Test: python jarvis.py → say wake word → give a command

Support Files

• templates/jarvis.py — Full voice assistant (edge-tts TTS, wake-word loop, Hermes bridge, email digest). Updated to edge-tts+pygame stack.
• templates/jarvis.bat — Windows Startup folder launcher script (auto-starts Jarvis on login via Windows Terminal).
• templates/coach.py — Standalone morning coach (no Hermes needed). edge-tts voice, question-and-response flow, exits after 2–3 min.
• templates/fetch_emails.py — google-workspace backend email fetcher (OAuth required)
• templates/fetch_emails_himalaya.py — himalaya backend email fetcher (App Password, simpler)
• references/wsl-windows-commands.md — WSL → Windows command execution pitfalls and workarounds
• references/himalaya-install-pitfalls.md — himalaya installation (rustup/cargo path, Gmail 2FA prerequisite)

Caveats

• WSL must be running. The wsl command auto-starts WSL if it's not running, but first launch has a cold-start delay of 5-15 seconds. Call wsl echo ready at script startup to warm WSL before the conversation loop begins.
• edge-tts requires internet. Calls Microsoft's free neural TTS API. No API key needed, no rate limits in practice. Offline fallback: pyttsx3 (see TTS section above).
• Google STT requires internet. recognize_google() calls Google's free speech API. Offline fallback: install Vosk (pip install vosk) and download a model.
• No conversation continuity. Each hermes chat -q is a fresh session. Hermes's persistent memory still applies (preferences, facts), but chat context doesn't carry over.
• ANSI escape codes in Hermes output. hermes chat -q may include terminal formatting in stdout even with --quiet. Always strip ANSI escape sequences before speaking: re.sub(r'\\x1b\\[[0-9;]*m', '', output).
• WSL → Windows command execution. When calling Windows commands from WSL bash, backslashes in paths are eaten by bash, and quotes are mangled when passing through cmd.exe /c. The reliable workaround is to write a small .bat file on the Windows filesystem and call that instead. See references/wsl-windows-commands.md for the full pattern and examples from real debugging sessions.