Sound Design & Audio
Sound is the channel that reaches an eye that's looking elsewhere — and the first thing users mute when it's abused. Design for restraint: silent by default, every cue earns its place, a tiny coherent palette, off the UI thread, always respectful of mute/DND. The worked identity here is the Harbor (a maritime operator console for an AI agent fleet), but the principles are general.
When to Use
- Deciding whether a given event should make a sound at all (most shouldn't).
- Designing a coherent set of UI cues (confirm/error/notify/transition/arrival/departure) as earcons or auditory icons.
- Giving a product a sonic identity (a motif family, not random samples).
- Synthesizing the actual sounds (which method for which cue) and mastering them for short-cue UI.
- Wiring playback/mixing in a native app, especially Rust/gpui.
NOT for
- Music composition / scoring for media (different craft).
- Speech synthesis / TTS / voice UX.
- Implementing DSP primitives or audio codecs from scratch.
Decision Points
flowchart TD
A[An event happened] --> B{Async — happening while the user is NOT looking at the app?}
B -->|No| Z[Stay silent — let the visual + motion carry it]
B -->|Yes| C{Consequential + would missing it be a real problem?}
C -->|No| Z
C -->|Yes| D{Does it resemble a real-world sound?}
D -->|Yes, has a referent| E[Auditory icon]
D -->|No, abstract| F[Earcon — a motive, composable into families]
E --> G{Synth or sample?}
F --> G
G -->|Procedural, parametric, tiny| H[Synthesize in-app: fundsp]
G -->|Rich/foley/one-off| I[Sampled asset → rodio/kira]
H --> J[Play OFF the UI thread; respect mute/DND/volume]
I --> J
Core Rules
- Silent by default. Audio is opt-in and earns its way back on one cue at a time.
- A hard vocabulary ceiling (~6 cues). A seventh proposal kills one first; overlapping motives destroy learnability.
- Every sound earns its place — only async, consequential, would-be-missed events. Synchronous feedback the user is staring at almost never needs sound.
- One cue per event class, not per event — coalesce bursts (twelve agents finishing ≠ twelve pings).
- A coherent palette, not a sample grab-bag — a motif family (e.g. arrival/departure = same instrument, contour inverted).
- Respect the user and the OS — honor system mute/DND, expose a volume + master off, never startle, never sound-only (audio augments a visual signal).
- Off the UI thread — playback/synthesis on an audio thread; never block render.
Failure Modes
Anti-Pattern: "The everything-clicks UI"
Symptom: every hover/toggle/press makes a sound; users mute the app within an hour.
Detection: play_sound( appears on synchronous, user-is-looking interactions, or >6 distinct cues exist.
Fix: delete synchronous-feedback sounds; reserve audio for async/consequential/missed.
Anti-Pattern: "Startle / fatigue"
Symptom: a cue is sharp, loud, or sits in the 2–5kHz fatigue band and fires dozens of times a day.
Detection: the cue's spectral peak is in the ear's most sensitive band; no low-pass; high peak loudness.
Fix: mid-range carrier, gentle envelope, round the top off, target conservative LUFS; test an 8-hour day.
Anti-Pattern: "Plays while muted/DND"
Symptom: a notification dings during a screenshare or with the system muted.
Detection: playback path doesn't check system mute / Do-Not-Disturb / app setting.
Fix: gate every play on the mute/DND/volume state; default-respect the OS.
Anti-Pattern: "Blocks the UI thread"
Symptom: a frame hitches when a sound plays.
Detection: decode/playback happens on the render/event thread.
Fix: dedicated audio thread + a voice pool; pre-decode/cache assets.
Anti-Pattern: "Sound-only signal"
Symptom: the only indication an agent failed is a sound — missed by deaf/HoH users or anyone muted.
Detection: an event has a cue but no visual counterpart.
Fix: audio augments a flag/badge/toast; never the sole carrier.
Worked Example: the Harbor "approve & land" and "agent failed" cues
- Should they sound?
agent-failed — async, consequential, easily missed → yes. approve & land — the user just clicked it and is watching → borderline; sound only if the land completes later, async → a quiet confirm on completion, not on click.
- Family: both are earcons in the maritime palette.
arrival/departure are the matched pair (warm bell, rising vs. falling contour); agent-failed = departure flavored darker (lower, a touch dissonant); confirm = a short, dry, resolved two-note.
- Synthesize (
04): confirm — FM bell, ~880Hz, 180ms exp decay, consonant resolve, slight plate tail. agent-failed — same bell, pitched down a minor third, falling contour, a hair of inharmonicity. Master both to a conservative short-cue LUFS, de-essed, 48k/24-bit.
- Wire (
03): synthesize procedurally with fundsp (parametric, tiny) or ship as cached ogg; play through a kira/rodio voice pool on the audio thread; gate on mute/DND + the app's audio setting.
- Never sound-only:
agent-failed always co-fires the red flag on the Quay + an inbox entry; the sound is the augment that reaches you when you're in another window.
Quality Gates
Fork Guidance
Fork by lane: principles (01 — when/why/what, psychoacoustics, accessibility) · identity (02 — the concrete sound map + motif family) · engineering (03 — the Rust audio stack, threading, mixing) · production (04 — synthesis methods, tools, mastering, sourcing/licensing). Keep the vocabulary-ceiling decision in the parent.
Reference Map
references/01-ui-sound-design-principles.md — when an event earns a sound, the functional vocabulary, earcon vs. auditory icon, psychoacoustics (frequency/duration/envelope/masking), accessibility + ethics.references/02-the-harbor-sonic-identity.md — a concrete on-brand sound map (sonar/bell/foghorn/flag-whoosh) keyed to fleet events (spawn/board/steer/hop/dispatch/approve/fail/flag-change), as a motif family.references/03-audio-in-rust.md — the Rust audio stack (rodio/cpal/kira/fundsp/symphonia/oddio), playback architecture, voice pool, latency, off-thread mixing, integrating with a gpui loop.references/04-audio-engineering-and-production.md — synthesis methods (subtractive/FM/granular/physical-modeling), tools (SuperCollider/Sonic Pi/Plugdata/Ableton), mastering short cues (LUFS, transient shaping), formats, sourcing + licensing.
Sibling skills: pairs with rust-gpui-motion (sync cues to transitions) and the capstone build-coop-ide-gpui (the fleet events that fire these cues).
Skill Bundle Index
Every file in this skill, and when to open it. Auto-generated; run scripts/index_references.py --fix.
references/
references/01-ui-sound-design-principles.md — UI Sound Design Principles — > Scope: audible feedback for Harbor, a native Rust gpui operator console for an AI agent fleet.references/02-the-harbor-sonic-identity.md — The Harbor Sonic Identity — > A maritime motif kit for the Harbor operator console — soft sonar pings, water laps, ship's bells, distant foghorns, signal-flag whooshes,references/03-audio-in-rust.md — Audio in Rust — Playing and Synthesizing Sound in a Native gpui App — > Scope: the Harbor operator console is a native Rust gpui desktop app.references/04-audio-engineering-and-production.md — Audio Engineering & Production — Making the Actual Sounds — > Scope: the craft layer beneath Harbor's sound design.
1---2name: sound-design-and-audio3description: UI sound design + audio engineering for apps, with a worked maritime/agent-fleet sonic identity and the Rust audio stack to ship it. Use for: deciding whether an event should make a sound, designing earcons/auditory-icons, building a coherent sound palette, synthesizing cues, and wiring playback in a native (gpui/Rust) app. Trigger on: UI sound, earcon, auditory icon, sound design, audio engineering, sonification, notification sound, rodio/cpal/kira/fundsp/oddio, synthesis (subtractive/FM/granular), foley, mastering, LUFS, "make it make sound", sonic identity, audio cue. NOT for: music composition for film/games, speech/TTS, or DSP/codec implementation.4---5
6# Sound Design & Audio
7
8Sound is the channel that reaches an eye that's looking elsewhere — and the first thing users mute when it's abused. Design for **restraint**: silent by default, every cue earns its place, a tiny coherent palette, off the UI thread, always respectful of mute/DND. The worked identity here is the **Harbor** (a maritime operator console for an AI agent fleet), but the principles are general.
9
10## When to Use
11
12- Deciding whether a given event should make a sound at all (most shouldn't).
13- Designing a coherent set of UI cues (confirm/error/notify/transition/arrival/departure) as earcons or auditory icons.
14- Giving a product a *sonic identity* (a motif family, not random samples).
15- Synthesizing the actual sounds (which method for which cue) and mastering them for short-cue UI.
16- Wiring playback/mixing in a native app, especially Rust/gpui.
17
18## NOT for
19
20- Music composition / scoring for media (different craft).
21- Speech synthesis / TTS / voice UX.
22- Implementing DSP primitives or audio codecs from scratch.
23
24## Decision Points
25
26```mermaid
27flowchart TD
28 A[An event happened] --> B{Async — happening while the user is NOT looking at the app?}
29 B -->|No| Z[Stay silent — let the visual + motion carry it]
30 B -->|Yes| C{Consequential + would missing it be a real problem?}
31 C -->|No| Z
32 C -->|Yes| D{Does it resemble a real-world sound?}
33 D -->|Yes, has a referent| E[Auditory icon]
34 D -->|No, abstract| F[Earcon — a motive, composable into families]
35 E --> G{Synth or sample?}
36 F --> G
37 G -->|Procedural, parametric, tiny| H[Synthesize in-app: fundsp]
38 G -->|Rich/foley/one-off| I[Sampled asset → rodio/kira]
39 H --> J[Play OFF the UI thread; respect mute/DND/volume]
40 I --> J
41```
42
43## Core Rules
44
45- **Silent by default.** Audio is opt-in and earns its way back on one cue at a time.
46- **A hard vocabulary ceiling (~6 cues).** A seventh proposal kills one first; overlapping motives destroy learnability.
47- **Every sound earns its place** — only async, consequential, would-be-missed events. Synchronous feedback the user is staring at almost never needs sound.
48- **One cue per event *class*, not per event** — coalesce bursts (twelve agents finishing ≠ twelve pings).
49- **A coherent palette, not a sample grab-bag** — a motif family (e.g. arrival/departure = same instrument, contour inverted).
50- **Respect the user and the OS** — honor system mute/DND, expose a volume + master off, never startle, never sound-only (audio augments a visual signal).
51- **Off the UI thread** — playback/synthesis on an audio thread; never block render.
52
53## Failure Modes
54
55### Anti-Pattern: "The everything-clicks UI"
56**Symptom**: every hover/toggle/press makes a sound; users mute the app within an hour.
57**Detection**: `play_sound(` appears on synchronous, user-is-looking interactions, or >6 distinct cues exist.
58**Fix**: delete synchronous-feedback sounds; reserve audio for async/consequential/missed.
59
60### Anti-Pattern: "Startle / fatigue"
61**Symptom**: a cue is sharp, loud, or sits in the 2–5kHz fatigue band and fires dozens of times a day.
62**Detection**: the cue's spectral peak is in the ear's most sensitive band; no low-pass; high peak loudness.
63**Fix**: mid-range carrier, gentle envelope, round the top off, target conservative LUFS; test an 8-hour day.
64
65### Anti-Pattern: "Plays while muted/DND"
66**Symptom**: a notification dings during a screenshare or with the system muted.
67**Detection**: playback path doesn't check system mute / Do-Not-Disturb / app setting.
68**Fix**: gate every play on the mute/DND/volume state; default-respect the OS.
69
70### Anti-Pattern: "Blocks the UI thread"
71**Symptom**: a frame hitches when a sound plays.
72**Detection**: decode/playback happens on the render/event thread.
73**Fix**: dedicated audio thread + a voice pool; pre-decode/cache assets.
74
75### Anti-Pattern: "Sound-only signal"
76**Symptom**: the only indication an agent failed is a sound — missed by deaf/HoH users or anyone muted.
77**Detection**: an event has a cue but no visual counterpart.
78**Fix**: audio augments a flag/badge/toast; never the sole carrier.
79
80## Worked Example: the Harbor "approve & land" and "agent failed" cues
81
821. **Should they sound?** `agent-failed` — async, consequential, easily missed → **yes**. `approve & land` — the user just clicked it and is watching → borderline; sound only if the land completes *later*, async → a quiet `confirm` on completion, not on click.
832. **Family**: both are earcons in the maritime palette. `arrival`/`departure` are the matched pair (warm bell, rising vs. falling contour); `agent-failed` = `departure` flavored darker (lower, a touch dissonant); `confirm` = a short, dry, *resolved* two-note.
843. **Synthesize** (`04`): `confirm` — FM bell, ~880Hz, 180ms exp decay, consonant resolve, slight plate tail. `agent-failed` — same bell, pitched down a minor third, falling contour, a hair of inharmonicity. Master both to a conservative short-cue LUFS, de-essed, 48k/24-bit.
854. **Wire** (`03`): synthesize procedurally with `fundsp` (parametric, tiny) or ship as cached ogg; play through a `kira`/`rodio` voice pool on the audio thread; gate on mute/DND + the app's audio setting.
865. **Never sound-only**: `agent-failed` always co-fires the red flag on the Quay + an inbox entry; the sound is the *augment* that reaches you when you're in another window.
87
88## Quality Gates
89
90- [ ] Default-silent; audio opt-in with a master off + volume.
91- [ ] ≤ ~6 cues; each maps to one event *class*; bursts coalesce.
92- [ ] Cues form a coherent family (shared motive, parametric variation).
93- [ ] Every play gated on system mute/DND + app setting; never startles.
94- [ ] No cue in the 2–5kHz fatigue band for repeated events; conservative LUFS; survives an 8-hour day.
95- [ ] Playback/synthesis off the UI thread (audio thread + voice pool).
96- [ ] No sound-only signals — every cue augments a visual.
97
98## Fork Guidance
99
100Fork by lane: **principles** (`01` — when/why/what, psychoacoustics, accessibility) · **identity** (`02` — the concrete sound map + motif family) · **engineering** (`03` — the Rust audio stack, threading, mixing) · **production** (`04` — synthesis methods, tools, mastering, sourcing/licensing). Keep the vocabulary-ceiling decision in the parent.
101
102## Reference Map
103
104- `references/01-ui-sound-design-principles.md` — when an event earns a sound, the functional vocabulary, earcon vs. auditory icon, psychoacoustics (frequency/duration/envelope/masking), accessibility + ethics.
105- `references/02-the-harbor-sonic-identity.md` — a concrete on-brand sound map (sonar/bell/foghorn/flag-whoosh) keyed to fleet events (spawn/board/steer/hop/dispatch/approve/fail/flag-change), as a motif family.
106- `references/03-audio-in-rust.md` — the Rust audio stack (rodio/cpal/kira/fundsp/symphonia/oddio), playback architecture, voice pool, latency, off-thread mixing, integrating with a gpui loop.
107- `references/04-audio-engineering-and-production.md` — synthesis methods (subtractive/FM/granular/physical-modeling), tools (SuperCollider/Sonic Pi/Plugdata/Ableton), mastering short cues (LUFS, transient shaping), formats, sourcing + licensing.
108
109**Sibling skills:** pairs with `rust-gpui-motion` (sync cues to transitions) and the capstone `build-coop-ide-gpui` (the fleet events that fire these cues).
110
111<!-- BEGIN BUNDLE INDEX (auto: index_references.py) -->
112
113## Skill Bundle Index
114
115*Every file in this skill, and when to open it. Auto-generated; run `scripts/index_references.py --fix`.*
116
117**`references/`**
118- [`references/01-ui-sound-design-principles.md`](references/01-ui-sound-design-principles.md) — UI Sound Design Principles — > Scope: audible feedback for **Harbor**, a native Rust **gpui** operator console for an AI agent fleet.
119- [`references/02-the-harbor-sonic-identity.md`](references/02-the-harbor-sonic-identity.md) — The Harbor Sonic Identity — > A maritime motif kit for the Harbor operator console — soft sonar pings, water laps, ship's bells, distant foghorns, signal-flag whooshes,
120- [`references/03-audio-in-rust.md`](references/03-audio-in-rust.md) — Audio in Rust — Playing and Synthesizing Sound in a Native gpui App — > Scope: the Harbor operator console is a native Rust [gpui](https://www.gpui.rs/) desktop app.
121- [`references/04-audio-engineering-and-production.md`](references/04-audio-engineering-and-production.md) — Audio Engineering & Production — Making the Actual Sounds — > Scope: the craft layer beneath Harbor's sound *design*.
122
123<!-- END BUNDLE INDEX -->
curiositech