Agentic video understanding
Hireable understanding layer. The model takes a goal and decides what to watch, at what speed, and through which modality (frames, audio, transcript), fetching only the moments needed. Vendor claims: up to ~66% lower cost and ~88% fewer tokens vs static fixed-FPS ingest, with higher accuracy.
What this is / is not
Is: goal → watch only what you need → timestamps + quotes + confidence.
Is not: a video editor. Do not cut, overlay, caption-burn, render, schedule, post, email, or write CRM from this skill. Hand cuts to Overlap, FFmpeg, or net-new-video-editor. Approvals stay with the calling lane.
When to use
- Pre-call / sales-call mining: buyer objection, next step, competitive mention
- Shortform scoring: find a 3-second standalone hook and in/out points
- Longform / X research: named-person + contrast moments in podcast or YouTube tape
- Talent review: bar evidence in a Loom or trial recording
- Client audit: every mention of a keyword across a discovery recording
Skip when the job is already a clean transcript and you only need text search.
Inputs
| Field |
Required |
Notes |
source |
yes |
URL or local media path the runtime can read |
goal |
yes |
One sentence retrieval goal |
keywords |
no |
Extra strings to bias retrieval |
max_moments |
no |
Default 5 |
modality |
no |
auto (default), frames, audio, or transcript |
Process
- Restate the goal as 1–3 retrieval queries. Done when each query is falsifiable (you would know if a moment matched).
- Call Gemini agentic video understanding (Gemini API or AI Studio) with
source, queries, max_moments, and modality preference. Prefer the agentic path over fixed-FPS full ingest when available. Done when the API returns candidate windows or an explicit empty set.
- Normalize moments into the output schema below. Flag paraphrase vs verbatim. Drop fabricated timestamps. Done when every kept moment has
t_start, t_end, modality, quote, why, confidence.
- Stop and hand off to the caller. Do not cut, overlay, schedule, publish, email, or CRM-write.
Output schema
Markdown for humans, optional JSON for machines:
{
"goal": "",
"source": "",
"moments": [
{
"t_start": "MM:SS",
"t_end": "MM:SS",
"modality": "frames|audio|transcript",
"quote": "",
"verbatim": true,
"why": "",
"confidence": 0.0
}
],
"empty_reason": null,
"tokens_note": "agentic path used|fallback static ingest"
}
Hard gates
- No full fixed-FPS ingest when the agentic path is available
- No invented timestamps or quotes
- No dumping full transcripts or client PII into public artifacts
- No cut / render / overlay / schedule / publish / send from this skill
Setup
- Gemini API key or Google AI Studio access: https://ai.studio
- See Google’s developer guide for agentic video understanding in Gemini
- Env:
GEMINI_API_KEY (or the project’s existing Google AI credential)
Caller one-liners
- Pre-call:
goal="exact next-step commitment and any pricing pushback"
- Shortform:
goal="best 3-second standalone hook; return in/out for one clip"
- Talent:
goal="evidence they hit the role bar on X; max 5 moments"
- Audit:
goal="every mention of Reddit, AEO, or budget"
Completion
Done when the caller has the schema above (or a documented empty set) and this skill has performed no side effects beyond the Gemini read.
1---2name: agentic-video-understanding3description: >-4 Use when an agent must extract moments, quotes, objections, hooks, or evidence5 from long video or audio cheaper than full-frame ingest — sales calls,6 podcasts, YouTube episodes, Loom trials, discovery recordings. Goal-directed7 watch via Gemini agentic video understanding (frames, audio, or transcript).8 Not for cutting, overlays, rendering, scheduling, or publishing.9metadata:10 source: https://skillmd.com/skills/ericosiu/agentic-video-understanding11---1213# Agentic video understanding1415Hireable understanding layer. The model takes a goal and decides what to watch, at what speed, and through which modality (frames, audio, transcript), fetching only the moments needed. Vendor claims: up to ~66% lower cost and ~88% fewer tokens vs static fixed-FPS ingest, with higher accuracy.1617## What this is / is not1819**Is:** goal → watch only what you need → timestamps + quotes + confidence.2021**Is not:** a video editor. Do not cut, overlay, caption-burn, render, schedule, post, email, or write CRM from this skill. Hand cuts to Overlap, FFmpeg, or `net-new-video-editor`. Approvals stay with the calling lane.2223## When to use2425- Pre-call / sales-call mining: buyer objection, next step, competitive mention26- Shortform scoring: find a 3-second standalone hook and in/out points27- Longform / X research: named-person + contrast moments in podcast or YouTube tape28- Talent review: bar evidence in a Loom or trial recording29- Client audit: every mention of a keyword across a discovery recording3031Skip when the job is already a clean transcript and you only need text search.3233## Inputs3435| Field | Required | Notes |36|-------|----------|-------|37| `source` | yes | URL or local media path the runtime can read |38| `goal` | yes | One sentence retrieval goal |39| `keywords` | no | Extra strings to bias retrieval |40| `max_moments` | no | Default 5 |41| `modality` | no | `auto` (default), `frames`, `audio`, or `transcript` |4243## Process44451. **Restate the goal** as 1–3 retrieval queries. Done when each query is falsifiable (you would know if a moment matched).462. **Call Gemini agentic video understanding** (Gemini API or AI Studio) with `source`, queries, `max_moments`, and modality preference. Prefer the agentic path over fixed-FPS full ingest when available. Done when the API returns candidate windows or an explicit empty set.473. **Normalize moments** into the output schema below. Flag paraphrase vs verbatim. Drop fabricated timestamps. Done when every kept moment has `t_start`, `t_end`, `modality`, `quote`, `why`, `confidence`.484. **Stop and hand off** to the caller. Do not cut, overlay, schedule, publish, email, or CRM-write.4950## Output schema5152Markdown for humans, optional JSON for machines:5354```json55{56 "goal": "",57 "source": "",58 "moments": [59 {60 "t_start": "MM:SS",61 "t_end": "MM:SS",62 "modality": "frames|audio|transcript",63 "quote": "",64 "verbatim": true,65 "why": "",66 "confidence": 0.067 }68 ],69 "empty_reason": null,70 "tokens_note": "agentic path used|fallback static ingest"71}72```7374## Hard gates7576- No full fixed-FPS ingest when the agentic path is available77- No invented timestamps or quotes78- No dumping full transcripts or client PII into public artifacts79- No cut / render / overlay / schedule / publish / send from this skill8081## Setup8283- Gemini API key or Google AI Studio access: https://ai.studio84- See Google’s developer guide for agentic video understanding in Gemini85- Env: `GEMINI_API_KEY` (or the project’s existing Google AI credential)8687## Caller one-liners8889- Pre-call: `goal="exact next-step commitment and any pricing pushback"`90- Shortform: `goal="best 3-second standalone hook; return in/out for one clip"`91- Talent: `goal="evidence they hit the role bar on X; max 5 moments"`92- Audit: `goal="every mention of Reddit, AEO, or budget"`9394## Completion9596Done when the caller has the schema above (or a documented empty set) and this skill has performed no side effects beyond the Gemini read.