# User Emotion Detection > Maps a detected user emotion — from facial expression (emotion.detected) OR speech (speech_emotion.detected) — into a mood signal logged via the Mood skill, then picks one response route (music / checkin / action). This is about the USER's emotion (input), NOT the device's own expression — that's emotion/SKILL.md. - Skill: `autonomous-ai/user-emotion-detection` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add autonomous-ai/user-emotion-detection` - Raw SKILL.md: https://api.skillmd.com/api/skills/autonomous-ai/user-emotion-detection/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: autonomous-ai (https://skillmd.com/u/autonomous-ai) - Updated: 2026-09-21 - Page: https://skillmd.com/skills/autonomous-ai/user-emotion-detection --- # User Emotion Detection ## Strict Trigger Runs **only** when this turn's input has one of: - `[emotion] Emotion detected: ` — facial expression from camera. - `[speech_emotion] Speech emotion detected: ` — vocal expression from microphone. Both are first-class triggers and route through the same logic below. The only differences are: 1. The mood signal `source` field — `"camera"` for `[emotion]`, `"voice"` for `[speech_emotion]`. 2. The label vocabulary — face uses `Fear/Surprise/Disgust`, voice uses `Fearful/Surprised/Disgusted`. The mapping table below covers both; they bucket identically. For any other event (`[sensing:presence.*]`, `[activity]`, `[sensing:sound]`, `[sensing:light.*]`, …) → **STOP. No tool calls.** Don't infer emotion from older turns — every signal must come from a real `[emotion]` or `[speech_emotion]` event in THIS turn. > **EXECUTE, don't plan.** Before any reply (including `NO_REPLY`) you MUST embed a `[HW:/mood/log:{...}]` marker in the reply for this event (it fires the POST async via the runtime — see "What this skill produces" below). Writing *"I will post a mood signal"* in `thinking` is **not** enough — if no `[HW:/mood/log:...]` marker appears in the reply text this turn, the skill failed. Every `emotion.detected` / `speech_emotion.detected` event = at least one mood signal log. No exceptions, no skips, no "already logged recently" reasoning. (`curl` POST is the documented fallback only when the HW marker would break the body regex; do not use it as the default.) > **Logging has no spoken narration.** Only the selected response route supplies speech or `NO_REPLY`. Never narrate mood logging, mapping, skill selection or workflow steps. Keep analysis in a separate reasoning channel; if none is available, omit analysis entirely. > **When thinking is enabled, keep channels separate.** Use only the provider's > native thinking/reasoning channel for route comparisons, cooldown checks, > uncertainty, style selection, and marker construction. Enabling thinking does > not permit a reasoning summary in assistant text. Every ordinary text/content > block can be spoken, including text before a tool call and text in the final > response. Do not put `` tags or a heading such as `Thinking:` into text > as a substitute for the native channel. If that channel is unavailable, omit > the analysis. Call required tools directly with no text introduction; after > their results, emit only the required markers and the user-facing sentence. > **EMOTION TURN OUTPUT CONTRACT — highest priority.** For a triggered > `[emotion]` / `[speech_emotion]` turn, send no assistant-text preamble before > or between skill/tool calls. Your entire final assistant message > must be exactly one of these two shapes: > > ```text > [required mood signal and selected-route HW markers] NO_REPLY > [required mood signal and selected-route HW markers] > ``` > > Nothing may appear before, between, or after those parts. In particular, do > **not** output the detected label or confidence; describe the cue as weak; > name a bucket, route, rule, source, prefix, marker, log, decision, or mood; > state what you are about to do; draft a line; or count its words/sentences. > Those are private scratch work, never user-facing text. Do the writes via HW > markers, then start immediately with the first word the user should hear. > Bad: `Emotion: Anger. Weak camera cue. Route = checkin. Let me compose...` > Good: `[HW:...] That sounds rough — I'm right here.` After reading the selected route's reference, emit its required markers and the spoken sentence, then end the turn. Do not reread the reference to refine the wording. Silently check that removing the markers leaves only the sentence or `NO_REPLY`; delete all planning prose, including `An emotion event`, `Let me check`, `Let me route`, and `Let me combine them all`. Never announce this check. Also omit `Route falls to checkin`, `music cooldown active`, `Weak sad cue`, `comfort/invite tone`, and `Mood signal must be logged`. These are routing notes, not part of a checkin, even when followed by a valid spoken sentence. Do not repeat or summarize native thinking in the final text. When the input explicitly says `weak camera cue` or `weak voice cue`, do not assert the detected feeling or a visible expression as fact. In a positive checkin, use a neutral invitation instead of assuming happiness or a smile. Keep the signal log and routing rules unchanged; uncertainty affects phrasing, not whether to log the event. ## What this skill does On every `[emotion]` or `[speech_emotion]` event, turn the detected user emotion into a mood signal. Log it via the Mood skill, then route a response (music / checkin / action) based on the pre-fetched context. This skill does NOT: - Fire `[HW:/emotion:…]` markers. Emotion expression is `emotion/SKILL.md`'s job, driven by conversation context — not auto-mapped from a sensor reading. - Require a spoken reply. Whether to speak is decided by the normal reply rules (SOUL + sensing SKILL), not by this skill. - Write to the wellbeing log. Wellbeing is for physical activity (drink/break/celebrate/sedentary); emotions live in the mood log. ## Trigger Either of: ``` [emotion] Emotion detected: . ← camera (face) [speech_emotion] Speech emotion detected: . ← microphone (voice) ``` Face FER labels: `Happy`, `Sad`, `Angry`, `Fear`, `Surprise`, `Disgust`, `Neutral`. Voice emotion2vec labels: `Happy`, `Sad`, `Angry`, `Fearful`, `Surprised`, `Disgusted`, `Neutral` (plus `Other`, `` — dropped upstream). Both formats end with the same `.` anchor; the same regex parses either one. ## Emotion → mood (for the signal log) Both label vocabularies map to the same mood values — voice variants (`Fearful`, `Surprised`, `Disgusted`) bucket identically to their face counterparts (`Fear`, `Surprise`, `Disgust`). | Detected emotion (face OR voice) | `mood` value to log | |---|---| | `Happy` | `happy` | | `Sad` | `sad` | | `Angry` | `frustrated` | | `Fear` / `Fearful` | `stressed` | | `Surprise` / `Surprised` | `excited` | | `Disgust` / `Disgusted` | `frustrated` | | `Neutral` | `normal` | ## What this skill produces A single `kind=signal` row in the mood log, emitted as an HW marker at the start of your spoken reply (the runtime fires the POST async, no tool turn): ``` [HW:/mood/log:{"kind":"signal","source":"","trigger":"","mood":"","user":""}] ``` **`source` is decided by the event prefix on THIS turn:** - `[emotion]` → `"camera"` - `[speech_emotion]` → `"voice"` Don't override based on prior turns or the `recent_signals` block; the prefix is authoritative. `mapped_mood` comes straight from the `[emotion_context: ...]` block — do NOT look it up from the table on the fly. Every detected emotion in the mapping table gets logged (including `Neutral` → `normal`) — Mood needs the recency for decision synthesis. Use `"unknown"` when the context tag is missing. **Do NOT use `curl` exec for this signal log** — see `mood/SKILL.md`'s "What to write" section for the rationale (HW marker is single-trip, curl burns a tool turn). If no `[HW:/mood/log:...]` marker appears in the reply this turn, the skill failed. ## Combined with mood + music-suggestion The backend injects this turn with an `[emotion_context: {...JSON...}]` block that pre-computes everything the three skills need (this skill is the router, mood logs the decision, music-suggestion fires only when this router picks the `music` route). **Do NOT fire any read tool calls** — the data is already in the message. Pre-fetched fields (use directly): - `mapped_mood` — already maps this turn's `` per the table above. This is the value to log as the signal mood. **You no longer need to look it up yourself.** - `recent_signals`, `prior_decision`, `is_decision_stale` — feed `mood/SKILL.md`'s decision rules and this skill's routing table. - `audio_playing`, `last_suggestion_age_min`, `audio_recent`, `music_pattern_for_hour`, `suggestion_worthy` — feed this skill's routing table (see **Response routing** below) and `music-suggestion/SKILL.md`'s genre pick. Single combined plan, not three sequential workflows: - **Decide locally** — apply mood decision rules from `mood/SKILL.md`; pick a route from the routing table below; if the route is `music`, evaluate genre from `music-suggestion/SKILL.md`. - **Writes (inline markers in the same reply)** — emit the mood signal marker (this skill), the mood decision marker when required by the mood rules, and on `music` or `checkin` route, the music-suggestion log marker (the shared cooldown channel). Use the marker shapes in the respective skills; do not also POST these logs through a shell. Preserve the selected route's speech or `NO_REPLY` after the markers. ### Fallback (only if `[emotion_context: ...]` is missing) If the message has no context block (pre-fetch failed), fall back to the read batch from `mood/SKILL.md` and `music-suggestion/SKILL.md` (concurrent GETs in one bash via `& ... wait`). Reply: routing decides the spoken reply (see next section). Never narrate the mapping, logging, or routing decision. ## Response routing (this skill is the router) After logging the mood signal, pick **exactly one** response route. Read straight from `[emotion_context: ...]` — no extra tool calls. Apply top-to-bottom, first match wins: | # | Condition | Route | What happens | |---|---|---|---| | 1 | `audio_playing == true` | **action** | LED-only ambient ack, no spoken reply. Emit `[HW:/emotion:{"emotion":"caring","intensity":0.4}]` + `NO_REPLY`. Music is already covering — don't talk over it. | | 2 | `suggestion_worthy == true` AND (`is_decision_stale == false` OR fresh decision synthesized this turn) AND `last_suggestion_age_min ∉ [0, 7)` | **music** | See `music-suggestion/SKILL.md` for genre + phrasing + log marker. | | 3 | anything else (cooldown active, mood not worthy, stale decision with no fresh synthesis, mapped_mood normal/frustrated, etc.) | **checkin** | See `reference/checkin.md` for phrasing + log marker. One soft open-ended line. | Rules: - **One route per turn.** Don't double-fire (e.g. music + checkin both). Pick the first matching row. - **Cooldown only gates music, not checkin.** When `last_suggestion_age_min ∈ [0, 7)` the music branch is blocked (row #2 fails its third clause) and the event falls through to checkin (row #3). The agent still asks — it just doesn't suggest music back-to-back. The only `NO_REPLY` path is row #1 (active audio). - **Output ownership:** `music` → produced by `music-suggestion/SKILL.md`. `checkin` → produced by `reference/checkin.md` (this skill). `action` → emitted inline by this router (the `[HW:/emotion:...]` marker in row #1). - **Cooldown is shared** between music and checkin: both log via `music-suggestion/log` so `last_suggestion_age_min` reflects either channel. - **Never narrate the routing decision in the spoken reply.** The reply is read aloud verbatim — the row you picked and why is scratch, not speech. Device-observed leak, 2026-08-24: *"A speech-emotion happy cue, fresh decision, audio idle, cooldown clear — routes to a music suggestion (speak only, unknown user). Sounds like a good mood — want some upbeat feel-good tunes?"* Only the last sentence was the reply; everything before it was the routing table thought out loud. Start the reply at the first word the user should hear. - `Neutral` is filtered upstream at HAL and never reaches this skill in practice; no special case needed here. ## Voice cue is weaker than camera cue Speech emotion (`[speech_emotion]`) is noisier than facial expression on short utterances — emotion2vec flips between `sad / fearful / angry` within the same affective state. The hedge `(weak voice cue; ...; treat as uncertain, ...)` is baked into the message for that reason. Practical rules: - **Don't relax the cooldown.** Music suggestions still gate on `last_suggestion_age_min ∉ [0, 7)` regardless of source. A fresh voice signal does NOT reset the cooldown that a recent camera-driven music suggestion left. - **Prefer Comfort/Invite phrasing on voice-only negative reads.** When the router falls through to checkin (row #3) and the trigger is `[speech_emotion]` with `bucket=negative`, lean toward Comfort/Invite rather than Ask — probing a maybe-misclassified utterance feels worse than acknowledging it. - **Cross-modal reinforcement still applies.** If `recent_signals` shows the same `mapped_mood` from `source="camera"` in the last ~10 min and now voice fires the same mood, treat it as a confirmation — the Mood skill's decision synthesis already handles this; no extra logic here. - **No skip-on-low-confidence.** Don't read `confidence=...` out of the hedge text and pre-filter; HAL already enforced `confidence >= SPEECH_EMOTION_CONFIDENCE_THRESHOLD` before sending. Anything that reaches this skill is worth logging.