Agent Screencast

Agent Screencast

Record narrated, captioned screen recordings of web applications. The pipeline uses agent-browser for browser automation, a TTS provider for voice synthesis (edge-tts by default, Gemini 3.1 Flash TTS optional), and ffmpeg for assembly.

Prerequisites

Before starting, verify the tools are available:

which uv            || echo "MISSING: curl -LsSf https://astral.sh/uv/install.sh | sh"
which agent-browser || echo "MISSING: npm i -g agent-browser && agent-browser install"
which ffmpeg        || echo "MISSING: brew install ffmpeg"

No installation step is needed — uv run handles Python dependencies automatically via inline script metadata.

If using the optional Gemini TTS provider (see Voice provider below), also ensure an API key is exported:

[ -n "$GEMINI_API_KEY" ] || [ -n "$GOOGLE_API_KEY" ] || \
  echo "MISSING: export GEMINI_API_KEY=... (get one at https://aistudio.google.com/apikey)"

Voice provider

Two options, selectable per recording:

Gemini 3.1 Flash TTS paid pricing: $20/1M audio output tokens at 25 tokens/second of audio → ~$0.0005/s (≈ $0.03/min). A 90-second PR demo runs ~$0.045. Free tier is $0 but Google may use submitted text and audio to improve their models — don't use it for scripts that reference unreleased features, customer names, or sensitive text.

Select the provider either in the script JSON:

{ "tts_provider": "gemini", "voice": "Kore", ... }

or via CLI flag (overrides the script):

uv run <skill-dir>/agent-screencast.py script.json --tts-provider gemini -o out.mp4

When gemini is selected, the pipeline fails fast if no API key is in the environment and prints a cost preflight before synthesis. Subtitles for Gemini are split by sentence and distributed across each segment (Gemini doesn't expose word-level timestamps the way edge-tts does).

How It Works

The pipeline has three passes:

1. Research pass — Visit pages, snapshot them, understand what's on screen
2. Dry-run validation — Replay the script without recording, taking screenshots per segment to verify actions work (login succeeds, pages load, clicks land)
3. Script + execute — Generate audio, record browser synced to audio timing, assemble final MP4

The insight: generate all narration audio first, then record the browser with precise wait durations matching each segment's audio length. This avoids dead air and ensures narration matches what's actually visible.

Workflow

Step 1: Research the target pages

Use agent-browser to visit and snapshot each page/state you want to demo:

agent-browser open http://localhost:4321
agent-browser wait --load networkidle
agent-browser snapshot -i

The snapshot gives you the accessibility tree with interactive element refs (@e1, @e2, etc.) that you'll use in the segment script actions.

Take multiple snapshots if the demo spans several pages or states. Note which refs correspond to which elements — refs are invalidated after navigation, so each page gets its own set.

For pages that need clicking or interaction to reveal UI, do that during research:

agent-browser click @e5
agent-browser wait --load networkidle
agent-browser snapshot -i   # fresh refs for the new state

Step 2: Write the segment script

Create a JSON file describing each segment of the video. Each segment has narration text and browser actions that will play during that narration.

{
  "title": "Feature demo",
  "base_url": "http://localhost:4321",
  "tts_provider": "edge",
  "voice": "en-US-GuyNeural",
  "segments": [
    {
      "id": "intro",
      "narration": "Here's the new dashboard. Let's explore the key metrics.",
      "actions": [
        { "cmd": "open", "arg": "http://localhost:4321/dashboard" },
        { "cmd": "wait", "arg": "--load networkidle" },
        { "cmd": "wait", "arg": "500" }
      ],
      "caption_overlay": "Dashboard Overview"
    },
    {
      "id": "interact",
      "narration": "Clicking the revenue card opens a detailed breakdown by region.",
      "actions": [
        { "cmd": "click", "arg": "@e3" },
        { "cmd": "wait", "arg": "--load networkidle" },
        { "cmd": "scroll", "arg": "down 300" },
        { "cmd": "wait", "arg": "500" }
      ],
      "caption_overlay": null
    }
  ]
}

Segment fields

Available actions

Writing robust actions

Element refs (@e1, @e2) are fragile — they change when the DOM changes (cookie banners, modals, dynamic content). Prefer these approaches in order:

1. eval with DOM queries (most robust) — Use CSS selectors or attribute lookups that survive DOM changes:

   { "cmd": "eval", "arg": "document.querySelector('input[name=\"email\"]').focus()" }
   

2. find with semantic locators — More stable than refs, but syntax must match exactly (find text "Sign In" click, not find text "Sign in" click):

   { "cmd": "find", "arg": "text \"Sign In\" click" }
   

3. @e refs — Only use for elements that are stable (no preceding dynamic content like cookie banners). Always re-snapshot after navigation.

For form filling, use eval with the native value setter pattern to work with reactive frameworks (Vue, React):

{
  "cmd": "eval",
  "arg": "(() => { const el = document.querySelector('input[name=\"email\"]'); const s = Object.getOwnPropertyDescriptor(window.HTMLInputElement.prototype, 'value').set; s.call(el, 'user@test.com'); el.dispatchEvent(new Event('input', { bubbles: true })); return 'ok'; })()"
}

For button clicks, match by text content to avoid ref fragility:

{
  "cmd": "eval",
  "arg": "document.querySelectorAll('button').forEach(b => { if (b.textContent.trim() === 'Sign In') b.click() }); 'ok'"
}

Writing good narration

• Reference what the viewer will actually see: "Watch the quantities update" not "The system recalculates values"
• Use present tense throughout
• Keep segments between 5-15 seconds of speech (~15-40 words each)
• Total video: 30-90 seconds for PR demos, up to 3 minutes for walkthroughs
• Don't narrate obvious UI actions ("I'm clicking the button") — describe why or what happens

Voice options

edge (default) — run uv run edge-tts --list-voices for the full list:

gemini — 30 prebuilt voices on gemini-3.1-flash-tts-preview:

Others: Fenrir, Aoede, Callirrhoe, Autonoe, Enceladus, Iapetus, Umbriel, Algieba, Despina, Erinome, Algenib, Rasalgethi, Laomedeia, Achernar, Alnilam, Schedar, Gacrux, Pulcherrima, Achird, Zubenelgenubi, Vindemiatrix, Sadachbia, Sadaltager, Sulafat, Orus. Gemini also accepts natural-language direction inside the narration text itself ("Say cheerfully: ...") for emotion and pacing control.

Step 3: Validate with dry-run

Always dry-run before recording. This replays all actions without recording or generating audio, taking a screenshot after each segment. It catches broken selectors, failed logins, and navigation issues before you waste time on a full recording.

uv run <skill-dir>/agent-screencast.py <path-to-script.json> --dry-run --session-dir ./session

This outputs:

• Per-segment pass/fail status
• Screenshots in ./session/dry-run/ (one per segment)
• Current URL after each segment (to verify navigation worked)

Review the screenshots (use the Read tool to view PNGs) before proceeding. If any segment fails, fix the script and re-run --dry-run until all pass.

Step 4: Run the pipeline

Save the script JSON, then run the self-contained script (uv resolves deps automatically):

uv run <skill-dir>/agent-screencast.py <path-to-script.json> -o output.mp4 --session-dir ./session

Where <skill-dir> is the path to this skill's directory (the folder containing this SKILL.md).

This executes all three phases automatically:

1. Narration — generates MP3 + SRT subtitles per segment via edge-tts
2. Recording — opens headless browser, replays actions synced to audio durations
3. Assembly — merges video + audio + subtitles into final MP4 via ffmpeg

CLI options

Re-running parts of the pipeline

If the recording looks wrong but audio is fine:

uv run <skill-dir>/agent-screencast.py script.json -o output.mp4 --session-dir ./session --skip-narration

If you just need to re-assemble (e.g. after tweaking subtitles):

uv run <skill-dir>/agent-screencast.py script.json -o output.mp4 --session-dir ./session --skip-narration --skip-recording

Step 5: Review and iterate

Play the output video. Common issues and fixes:

Example: Full workflow

Here's what a typical session looks like end-to-end:

# 1. Research
agent-browser open http://localhost:3000
agent-browser wait --load networkidle
agent-browser snapshot -i
# See: @e1 [link] "Dashboard", @e2 [button] "New Project", ...

agent-browser click @e1
agent-browser wait --load networkidle
agent-browser snapshot -i
# See: @e1 [heading] "Dashboard", @e3 [card] "Revenue", ...

# 2. Write script based on what you saw (save as demo-script.json)
#    Prefer eval with CSS selectors over @e refs for robustness

# 3. Validate with dry-run (catches broken selectors, failed logins, etc.)
uv run <skill-dir>/agent-screencast.py demo-script.json --dry-run --session-dir ./session
# Review screenshots in ./session/dry-run/*.png

# 4. Record (only after dry-run passes)
uv run <skill-dir>/agent-screencast.py demo-script.json -o feature-demo.mp4 --session-dir ./session

# 5. Output: feature-demo.mp4 with narration + subtitles

Git

The pipeline generates large binary intermediates (audio, video, screenshots). Add these to the project's .gitignore before running:

# agent-screencast intermediates and output
session/
*.mp4
*.webm
*.mp3
*.wav
*.srt

Subtitle note

If ffmpeg has libass support, subtitles are burned into the video. Otherwise they're embedded as a soft subtitle track (toggle-able in video players). To get burned-in subtitles: brew install ffmpeg with libass support.