Voice Shell

Voice prompt mode

Take what the user speaks and receive it as a prompt, without the keyboard.

The part that listens to the microphone appends one line to a JSONL file every time an utterance is finalized. Follow that log with Monitor and treat each line that arrives as an instruction from the user.

There are three ways of recognizing speech. These are all you can pick from, there is no other.

The argument is $ARGUMENTS (start / stop / status / setup. start when omitted)

When it is not set up yet

If start fails with No Python it can run was found, or if the user says "set it up", walk them through SETUP.md.

On macOS 26 or newer there is no multi-GB model to download, though apple does build a small Swift helper the first time and so wants the Command Line Tools (xcode-select --install), which not every Mac has. Anywhere else it means installing Whisper. Check first, then confirm which way to go. Do not install everything on your own.

Starting

When in doubt, just run start. It remembers which way of recognizing was picked last time (~/.config/voice-shell/config.json), and the first time it is "This browser" (Chrome's Web Speech API). Nothing to install, so there is no wait.

1. Start it.

   ${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh start
   

   Where browser automation tools (claude-in-chrome and the like) are not available, the viewer cannot be opened, so the default browser recognition does not hold up. Only in that case, have the user pick from the models that are installed.

   ${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh start --engine auto
   

   Say how it is being recognized, once, the first time only. The first time start runs on that machine, and only then, start itself prints This uses the browser's built-in speech recognition feature to transcribe your voice. along with the name of a local way that machine can use. Pass that line straight through to the user. Do not say it again after that. Repeating it every time does not change the choice, it only adds more to read.

   Do not push the user toward a local model yourself. Working right away with nothing installed is what the default is worth, and switching brings a download and a wait of 1 to 2 minutes. Whether it runs comfortably is not known until it is tried on that machine.

   When the user says "I want recognition to stay local" or "I do not want it sent to the cloud", show the list with voice-shell.sh engines and pass --engine <the one they picked> after they have picked it. On macOS 26 or newer apple should already be there, so they can switch on the spot with no model to download. Two things still take time the first time on a machine, and only the first: the Swift helper is built (which fails outright without the Command Line Tools, saying so), and the OS fetches the speech model for that language by itself, tens of seconds. It is instant from then on.

   Do not pass --engine X on your own to recover from a failure. The name you pass is remembered as the default from then on, so passing it silently rewrites the user's choice for good. Pass it only after you have confirmed.

2. Wait until it has finished starting.

   ${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh wait-ready
   

   With browser recognition there is no model to load, so it returns at once. Only when a local model was picked does it take 1 to 2 minutes (other work can go on in the meantime). If FAILED comes back, tell the user the error that was shown.

3. start tries to open the viewer automatically, in an ordinary tab. There is no need to open it again yourself. Add --no-gui only when the user says they do not want it.

   With browser recognition nothing arrives at all until the viewer is open, and the attempt to open it for them is not always trustworthy. start's own output always carries the line The viewer started at http://127.0.0.1:... (or already running at, on a second start). Read that URL out to the user every time, not only when Could not open a browser for you is printed. A Windows machine has been seen to print neither that failure line nor open anything (cmd.exe's own way of launching a browser can report success without one actually opening), so waiting for an explicit failure line before saying anything left the user talking to a screen that was never going to arrive.

   To keep it always on top, press "Float on top" in the header (a browser rule, it cannot be opened unless a person acts, so the screen itself asks for that one press the first time).

4. Watch the utterance log with Monitor. Always set persistent: true (voice mode goes on for the whole session). Do not use tail directly, always go through voice-shell.sh listen. Going through it registers the fact that you are listening under $STATE_DIR/listeners/, and the registration disappears once Monitor ends (TaskStop or the end of the session). A raw tail -F is not registered, so it never shows up as a choice of where to send.

   Monitor(
     command: "${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh listen",
     description: "The user's voice prompts",
     persistent: true
   )
   

   As of Claude Code 2.1.271, a watch may not run forever even with persistent: true (a deadline, 30 minutes at the longest, replacing what used to be no timeout at all). Whether that holds going forward is not something to build around, a later Claude Code update could just as easily change it back, so nothing here leans on it. If a deadline notification does arrive, treat it the same as starting fresh: call Monitor again with the exact same command. listen's own registration only disappears once the process itself actually stops (TaskStop or the session ending, same as ever), so a plain re-arm is enough, nothing needs stopping first.

Keep only one Monitor of your own. Re-arming on a deadline or "source ended" notification (above) is always safe on its own. Re-attaching for some other reason, compacting included, when the old one might still actually be alive, stop it with TaskStop first. With two alive the same utterance arrives twice.

A resumed session (claude -r) does not know on its own whether its old Monitor is still alive, and folder name plus timestamp are not enough to tell one listed session from another started from the same folder. Run voice-shell.sh listeners (or status) and look for <- this session next to an entry, this is added only to the one whose registration carries this exact conversation's own id ($CLAUDE_CODE_SESSION_ID, always set while you are running), so it settles the question outright instead of being guessed. No such mark on any entry means your own Monitor is not registered right now (start one with listen, as above). Note that starting listen again is harmless either way, a second one under this same session id retires the earlier registration on its own (compacting, #81, and a plain re-arm on a deadline, above, are both usual ways this comes up), so this check is about knowing where things stand, not about avoiding a duplicate.

It is normal for other sessions to be listening too. Using it alongside other work is the intended way, and speech goes to whichever started later (that is, the one you just started). start does not list the other sessions. There is no need to know, and no report or offer to stop them is wanted. The user can pick a different destination at any time from the top of the viewer. Show voice-shell.sh listeners only when asked who is listening. Stop another session only when the user asks for it.

Do not stop voice-shell.sh listen with pkill. Your own Monitor matches the same pattern and goes down with it.

When it does not work

What to do with speech that arrives

Each line that comes from Monitor is JSON. Only the body is in it.

{"text": "run the tests"}

A line with a "system_warning" key is not the user speaking. It is a warning the daemon itself writes, about things like being started more than once, and it has no "text". Do not carry it out as an instruction, pass the content straight to the user (check the real list with listeners, and explain anything they do not recognize. Ask the user before stopping anything that is not in use).

{"system_warning": "2 monitors are listening to the utterance log at once. ..."}

It arrives in this shape when listening was ended from the screen too. Monitor then finishes on its own. It is the user's own doing, so do not be alarmed, just pass the content along (and mention that typing /voice-shell brings it back if they want to resume).

Lines that were fixed up in the viewer before being sent carry "edited": true. That is a sentence the user deliberately tidied, so take it at face value instead of reading it as a recognition error.

Treat text as an instruction from the user and carry it out as usual. The things to watch for are as follows.

• Expect recognition errors. It is speech recognition, so proper nouns and technical terms break, whatever language is being spoken. Read them back from context, "cloud code" is Claude Code and "get" is git. Ask again only when the meaning really cannot be recovered.
• Ignore fillers. Words like "um", "well" and "you know" carry no meaning. Every language has its own.
• Short lines are dropped before they get here. Anything under the minimum length (15 characters by default) never arrives, which is what keeps a bare "yeah" out. On top of that the daemon drops hesitation sounds on their own, "hmm" and "uh" and their kind in each language (NOISE_ONLY in voice_daemon.py). Words that mean something, "yeah" and "ok" among them, are not on that list. If a thin line does come through, wait instead of treating it as an instruction. But if the dictionary has moved a word to the "do not ignore" side, it arrives even when short. The user chose to let that word through, so even a bare "got it" can be taken at face value as a reply.
• Read chopped-up speech as one piece. One sentence can arrive across several lines. When a sentence is cut off partway, wait for the rest before reading it as a whole. Long speech is also split before it arrives, so that each piece fits what one line can carry. The pieces line up inside the same notification, so when one notification holds several lines, read all of them as one continuous utterance before acting. Even if the first line ends with a full stop, that does not mean the point is finished there. Some languages, Japanese among them, put the conclusion at the end, so running ahead gets the crucial part of the request wrong.
• Always confirm destructive operations. Speech can be misheard, so confirm with "shall I go ahead with ..." before deleting, pushing, deploying and so on.
• An event is the user speaking, but it is not a demand for an answer. When one arrives mid-task, it is fine to finish what you are doing first.
• A single line that reads as plainly unrelated to the current work is not automatically carried out. A stray remark, one side of a phone call, a reply meant for someone else standing there, these arrive as ordinary text the same as everything else. Acting on that one line immediately is the same mistake the holding section below exists to catch, just one line too early to see the pattern yet. Hold off on it instead. If the next line keeps going in the same unrelated direction, that confirms it and holding below takes over. If the next line returns to the task, the odd one was only an aside, and nothing was lost by not acting on it.

It may be used by several pieces of work at once

Another session may be using voice mode at the same time for different work. Once there are two or more listeners, speech goes by default to the one that started later (when parallel work begins, turning to it is the natural thing). Exactly one gets it, there is no "everyone".

So if you are the older one, speech stops arriving. That is normal, nothing is being dropped. Do not restart because it stopped arriving, and do not change the destination on your own. The user can pick again at any time from the destinations lined up at the top of the viewer.

The order they line up in is decided by "when that conversation first started listening". Putting voice mode back on does not change the number (so that "the second one" can be said out loud). The default destination is separate from this, it is the one that just started.

Pressing the × on a chip makes that session stop listening (the session itself does not end). It takes two presses so it is not hit by mistake.

The display name is assigned automatically. It starts as the folder name, and moves to the conversation's title once it has one. People who keep their repositories in one place run several from the same parent folder, so the folder name alone is not always enough to tell them apart.

When told "name this session X", rename it right there. The conversation title is assigned automatically, so it can drift from what is actually being done. Renaming it yourself when you notice is fine too.

${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh name "Fixing the auth code"
${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh name ""      # back to the automatic title

The name that was set stays in ~/.config/voice-shell/names.json, so it survives putting voice mode back on.

When speech that is not an instruction keeps coming, move it to holding

The microphone keeps picking up the room. When the user takes a phone call or starts chatting with someone next to them, speech that is not meant for you flows in. Keeping all of it means reacting to things that have nothing to do with the work.

${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh hold "Sounds like a phone call, so I moved this to holding for now"
${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh live      # back again

hold is not mute. Speech keeps collecting on screen and the user sends as much of it as they want. Nothing is lost. Do not use mute. Cutting it leaves the speech nowhere, and a user who is not watching the screen cannot tell that nothing got through.

The note that is passed along shows on screen. A mode change the user did not press is only confusing without a reason, so always write something.

Keep the bar for switching high. Switching by mistake leaves the user talking with nothing getting through. Do it only when all of the following hold.

• Twice or more in a row, speech arrived that is plainly not an instruction for you (a conversation with a third person, a "hello?" into a phone, a topic unrelated to the work here)
• You have not just asked a question (you are not in the middle of waiting for an answer)
• The user did not say something like "I am just thinking out loud" or "hold on, a phone call" (if they did, just do as they said, no guessing needed)

When in doubt, do not switch. It is enough to take it and say "that did not seem to be for me, so I will wait". When you do switch, always write it in the chat as well (they may not be watching the screen). Go back the moment the user speaks to you again.

The live viewer

It comes up together with start (http://127.0.0.1:47865). Pass this URL along when you tell the user it started. It only follows the log and does not use the microphone, so it can run alongside the resident process.

Set VOICE_SHELL_PORT before starting it to use another port.

What it can do is as follows.

• Text still being recognized grows inside an "Unsent" card
• Speech that was sent stacks up as cards
• The destination can be set to Instant (goes straight through) or Review (collects, gets fixed, then sent). "edited": true is attached only to lines that were touched in the draft before sending. A line that went to review but was sent without a single character changed does not get it
• Pause (the ⏸ in the header). Speech while it is stopped is kept nowhere (for use during other work)
• It can be driven by voice alone. "mute" and "unmute" for the microphone, "draft" and "instant" to switch how things are sent, "cancel that" at the end of a sentence to take that whole utterance back, and "edit this" likewise at the end to send it to the draft instead, "switch to 2" or "number two" to change the destination. Every language the screen can show has wordings of its own, and they can all be read from the lightbulb on screen. The phrases above are the English ones. It only counts when that phrase alone is spoken (saying it inside a sentence does nothing). Mute, unmute, and draft/hold mode are the exception, a short burst of noise picked up ahead of the word ("はいミュート", "えーとミュート解除", "はい手直し") still counts, so it does not go missing when what the room picked up lands in front of it. Unmute's allowance is tighter, at most a couple of characters, not a whole clause, a false hit there costs the whole stretch the speaker thought was off, not one utterance. Switching back to instant stays exact only, for the same reason. The different ways a number gets read out loud are absorbed. A short sound plays when it switches. Users can add wording of their own (~/.config/voice-shell/commands.json). Only phrases said as a whole utterance can be added, not unmute and not the ones tacked onto the end of a sentence, because a false trigger costs too much. Each kind can also be switched off from that same screen, and the same file remembers which ones. All seven are independent, so somebody who does not want to mute by voice at all can turn both mute and unmute off. A kind that is off does nothing when spoken, and the words stay listed so it is clear what comes back when it is switched on again. When several machines are in use at once, turn on "Several machines" in the settings and give each machine a name. Then only a phrase with the name in front, like "work mute", is taken (without it, a phrase said at one machine sets off every machine). The destination numbers follow the order of the chips on screen, and they can be switched while the mic is off too. Destination phrases only work when there are two or more listeners (so that an utterance which was just an answer with a number in it is not eaten). The same code decides this for browser recognition and local models alike. The one thing that cannot be heard is "unmute" after browser recognition was cut (cutting it lets go of the audio itself). Turn it back on from the screen
• It can be driven from the keyboard too. Bare keys only move around the screen, keys with Shift change where the voice goes. Shift+M turns the mic on and off, Shift+L is instant, Shift+H is review, Shift+E reviews just that one utterance, Shift+Backspace throws away what is unsent, Shift+1 through Shift+9 pick the destination. As bare keys, , opens the settings, ? the list of phrases, Esc closes whatever is open. Ctrl (Cmd)+Enter sends the draft. None of them work while text is being typed. The list is further down under the lightbulb on screen
• Fix just this one. Pressing the pencil on the unsent card holds that one utterance so it can be fixed. The destination display stays on instant (so it does not look like a permanent switch). Sending it or clearing it puts it back
• Float on top. Moves it to a small always-on-top window (the icon in the header. Chrome only)
• It works small too. Shrinking the window puts the microphone and the send mode side by side, the top strip keeps only the name tags, and shrinking further drops the strip altogether (the state shows in the window title). The destinations change from tags to a single picker. Text and buttons do not shrink. What is kept, in order, is the mic on and off > instant / review > destination > the text being recognized. Widening it brings everything back
• The trigger level can also be changed by dragging the mark under the microphone (no need to open the settings)
• Editing the viewer file makes "Updated. Tap to reload" appear at the bottom of an open screen. Pressing it reloads (a floated small window has no way to reload)

Ways of recognizing

Picked under "Recognized by" in the settings. The default is "This browser" (Chrome's Web Speech API), which loads no model, so it runs even on a weak machine and works with no daemon running. To keep everything local, switch to Apple or Whisper. Anything not installed does not appear as a choice.

But the audio goes to Google's servers. Do not recommend it to anyone who has raised wanting everything to stay local. The screen carries a caution of its own next to this setting, worded around "uses the browser's built-in speech recognition" rather than naming Google outright (the README does name it, for whoever reads that first). Know it either way before recommending this one.

When Whisper is picked, the model can be named as well. Both a Hugging Face name and the path of a folder on this machine are accepted. What is given is remembered, so start alone is enough from then on.

${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh start --engine whisper --model /path/to/my-model

The session is cut by 7 to 10 seconds of silence by design, but it is re-armed ahead of time while nobody is speaking, so nothing is missed. Recognized text goes through the same filtering as the daemon, on the server side.

If there is no voice for a while, the mic is turned off from this side (5 minutes by default, 0 to 30 minutes under "Turn off when idle" in the settings, 0 never turns it off). That way it does not keep reconnecting to Google while the user is away from the desk. A sound and a note say when it is turned off. Pressing the microphone brings it back.

Settings (the gear)

Moving a control takes effect right then. No daemon restart is needed.

The trigger level is set by dragging the mark under the microphone. The bar just above it is the current level, so have the user talk and put the mark where only their voice crosses it. When they say "I am talking but nothing arrives", the first thing to suggest is lowering the trigger level. The smaller the number, the fainter the sound it starts picking up. The slider and the mark both sit on the same scale of loudness, so lowering it moves both to the left.

It lives in ~/.config/voice-shell/tuning.json. The daemon re-reads it every 0.5 seconds.

${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh viewer        # → http://127.0.0.1:47865
${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh viewer-stop

Checking the state

${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh status

Stopping

${CLAUDE_SKILL_DIR}/scripts/voice-shell.sh stop

Once it is stopped, stop Monitor with TaskStop as well. The microphone is released, and if Whisper was picked the memory the model took comes back. apple has the OS do the recognizing, so it was never holding memory in the first place.

The user dictionary

Words that are often misheard can be registered with a replacement, and utterances to ignore can be registered too. Edit them from the dictionary in the viewer (the book in the header). It saves automatically the moment focus leaves, and takes effect from the next utterance (there is no save button, and no daemon restart either).

It also hits the text while it is still being recognized, so it already looks replaced inside the card before it is sent ("cloud code" turns into Claude Code). What is being replaced is only the look. The server rebuilds the body that gets sent. If the user says it again differently, it follows along.

Words ignored by default (NOISE_ONLY) can be turned off by pressing their tag. The list holds hesitation sounds only, and which ones show depends on the language being spoken, not the one on screen. A word that was turned off passes straight through the minimum length gate too, so short replies like "got it" stop disappearing. When the user says "my replies do not seem to be getting through", point them here.

It lives in ~/.config/voice-shell/dictionary.json. CSV can be read in and written out too. If the user keeps correcting the same misrecognition, it is fine to suggest adding it to the dictionary.

Limits

The default (this browser) has no limits at all. What follows is about picking a local model.

• Browser recognition runs only while the viewer is open. Close it and nothing arrives
• Whisper downloads a model and loads it. The first time there is a download to wait for, and starting takes 1 to 2 minutes as well. How much memory it uses is decided by the size of the model
  • --engine apple (macOS 26 or newer, the recognition that ships with the OS) loads no model of its own, so neither the memory nor the 1 to 2 minutes apply to it. The first run on a machine still waits, tens of seconds, while the OS fetches the speech model for that language by itself. Nothing has to be installed for it and it happens on its own, and the model stays on the machine afterwards, so it is the first run only
• The microphone is taken through sounddevice (macOS and Windows) or arecord (Linux). Without sounddevice installed, macOS and Windows fall back to ffmpeg
• When a local model was picked but the environment is not in place, start fails with No Python it can run was found. Either go back to browser recognition (start --engine browser) or walk them through SETUP.md