Video research
Gemini reads a YouTube URL directly. There is no download, no transcript step, and no frame extraction. Point it at the URL and ask.
For one video, call the API. For many, use the gameref pipeline in
itsryanarmani/scripts, which caches per video and pools the findings.
One video
TOKEN=$(python3 -c "
import json,os,urllib.request,urllib.parse
d=json.load(open(os.path.expanduser('~/.config/gcloud/adc_personal.json')))
b=urllib.parse.urlencode({'client_id':d['client_id'],'client_secret':d['client_secret'],
'refresh_token':d['refresh_token'],'grant_type':'refresh_token'}).encode()
print(json.load(urllib.request.urlopen('https://oauth2.googleapis.com/token',b))['access_token'])")
curl -s -X POST "https://aiplatform.googleapis.com/v1/projects/YOUR_PROJECT_ID/locations/global/publishers/google/models/gemini-3.8-flash:generateContent" \
-H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' -d '{
"contents": [{"role": "user", "parts": [
{"fileData": {"fileUri": "https://www.youtube.com/watch?v=VIDEO_ID", "mimeType": "video/*"}},
{"text": "YOUR QUESTION"}
]}]}'
Many videos
gameref search "QUERY" "ANOTHER QUERY" --per-query 12 --min-seconds 180 --max-seconds 1800
gameref analyse videos.json --limit 34 --clip-seconds 600
gameref merge reports --out findings.json
The merge stage counts how many videos saw each finding. That column is the output that matters: one sighting is a lead, twenty is a requirement. It is the reason to watch many videos instead of asking a model once.
Four traps, each of which cost a run
AI Studio and Vertex are different billing pools. A key for
generativelanguage.googleapis.com can return 429 prepayment credits are depleted while Vertex on the same Google account works normally. Check Vertex
before concluding you are out of quota.
Vertex needs an explicit role on each contents entry. Without it you get
Please use a valid role: user, model, which does not name the missing field.
AI Studio defaults it, so a body copied from AI Studio docs fails here.
Vertex wants camelCase and this one fails silently. Use fileData,
fileUri, mimeType. Send AI Studio's file_data / file_uri spelling and the
request returns 200 with the video dropped — the model answers from the text
prompt alone and produces a confident, plausible paragraph about a video it
never watched. Nothing in the response says so. Always check
usageMetadata.promptTokensDetails for a VIDEO modality entry before trusting
an answer.
videoMetadata is a sibling of fileData, not a child. Nesting it returns
400 Unknown name "videoMetadata" at contents[0].parts[0].file_data.
{"fileData": {"fileUri": "...", "mimeType": "video/*"},
"videoMetadata": {"startOffset": "60s", "endOffset": "660s"}}
Cost
Roughly 260 tokens per second of video, plus audio separately. Ten minutes is about 25,000 tokens. A five-hour longplay is millions and mostly repeats what its first ten minutes showed, so clip with offsets rather than sending whole streams.
Ask for structure, not prose
Pass a responseSchema with responseMimeType: application/json. A paragraph
has to be re-read by a human before it can be used; a schema gives rows you can
count, sort and turn into issues.
Include a confidence field with two values, one for what the model watched and one for what it reasoned from a UI element or a result. Without it, observation and inference blend into a single confident list, and the guesses are indistinguishable from the facts. Say explicitly in the prompt that a short honest list beats a long guessed one.
Ask what the video shows, not what the model knows. A question like "what are this genre's mechanics" gets answered from training data whether or not the video loaded. "What mechanics can you see happening on screen, with timestamps" cannot be answered without watching.