Media Tag

Media Tag

Context: $ARGUMENTS

Four open-source vision-language families, four different jobs:

• CLIP / SigLIP — "is this image closer to prompt A or prompt B?" Zero-shot classification, similarity scoring, semantic search index.
• BLIP-2 — pure image captioning. One image in, one sentence out.
• LLaVA — full vision-language model. Captioning, VQA, dense description, video-frame narration.

All commercial-safe.

Quick start

• Caption a single image (sentence): → Step 3 (tag.py caption --model blip2)
• Describe in detail or ask a question (VLM): → Step 4 (tag.py describe --model llava)
• Zero-shot classify against your own label list: → Step 5 (tag.py classify --model clip)
• Build and query a CLIP search index over a folder: → Step 6 (tag.py search)
• Bulk-tag a folder to CSV: → Step 7 (tag.py tag-batch)
• Per-scene description of a video: → Step 8 (tag.py video-describe)
• Pre-download weights: → Step 2 (tag.py install <model>)

When to use

• Auto-alt-text for a photo library or CMS.
• Content moderation / safety (classify against a forbidden-label list).
• Semantic search over a local image folder ("find the red car in the rain").
• Cataloguing / DAM metadata enrichment (auto-tag with a controlled vocabulary).
• Quick video content summary — one description per second or per scene cut.
• Caption → search index → retrieval pipelines for RAG over images.
• Do NOT use for facial recognition of named individuals (these models weren't trained for that; accuracy and ethics both terrible).
• Do NOT use for OCR — use media-ocr-ai. Do NOT use for object-detection bboxes — use cv-opencv (YOLO) or cv-mediapipe.

Step 1 — Pick a model

Full per-model matrix (benchmark accuracy, caption quality, context length, speed) in references/models.md.

Step 2 — Install / pre-download

PEP 723 header in scripts/tag.py declares all dependencies. uv run manages an ephemeral venv.

uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py install clip
uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py install siglip
uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py install blip2
uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py install llava

Per-model notes:

• CLIP — via open_clip (preferred for variety) or transformers (openai/clip-vit-large-patch14). open_clip exposes a huge zoo (ViT-B-32, ViT-L-14, ViT-H-14, ViT-bigG-14, with OpenAI / LAION / DataComp pretrain tags).
• SigLIP — google/siglip-base-patch16-384, -large-patch16-384, -so400m-patch14-384. All Apache 2.0. Use transformers.
• BLIP-2 — Salesforce/blip2-opt-2.7b (BSD-3), -opt-6.7b, -flan-t5-xl. Use transformers.
• LLaVA — llava-hf/llava-1.5-7b-hf, llava-hf/llava-v1.6-mistral-7b-hf (LLaVA-NeXT), llava-hf/llava-onevision-qwen2-7b-ov-hf (includes video support, recommended). All Apache 2.0 as repackaged by llava-hf. Use transformers.

Step 3 — Single-image caption (BLIP-2 or LLaVA)

uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py caption \
  --model blip2 --in photo.jpg --out-text caption.txt

Output: one sentence, WCAG-alt-text-style. Good for cataloguing.

For richer captions:

uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py caption \
  --model llava --in photo.jpg --out-text caption.txt

LLaVA tends to produce 2–4 sentences describing scene, subject, mood. Pass --prompt to steer:

--prompt "Describe this image in one short sentence suitable for alt text."

Step 4 — VLM description + VQA (LLaVA)

uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py describe \
  --model llava --in photo.jpg \
  --prompt "What objects are in this image and what is their color?" \
  --out answer.txt

Arbitrary VQA via --prompt. Useful prompts are cached in references/prompts.md — read it when drafting prompts for object detection, scene analysis, or video narration.

Step 5 — Zero-shot classification (CLIP or SigLIP)

uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py classify \
  --model clip --in photo.jpg \
  --labels "a photo of a cat,a photo of a dog,a photo of a car,a photo of a bicycle" \
  --top-k 2

Output (JSON on stdout):

{"top": [
  {"label": "a photo of a cat", "score": 0.87},
  {"label": "a photo of a dog", "score": 0.09}
]}

Pro tip — use full-sentence prompts ("a photo of X"), not bare words ("X"). CLIP / SigLIP are heavily biased by phrasing; "a photo of a cat" outperforms "cat" by 5–10 points on ImageNet-class prompts.

For SigLIP:

--model siglip

Same interface; different pretrain + sigmoid head. Higher-quality scores without softmax competition between labels — each score is independent probability.

Step 6 — CLIP semantic search index

# 1. build embedding index of a folder
uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py search \
  --model clip --index photos/ --index-out photos_clip.npz

# 2. query
uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py search \
  --model clip --index photos_clip.npz \
  --query "a red car in the rain" --top-k 10

Index is a .npz with files (list of paths) + embeddings (float32 L2-normalized array). Query is a text embedding dot-producted against the index.

For scale beyond ~50 k images, swap the brute-force dot-product for FAISS or Qdrant — the reference doc references/models.md has a short recipe, or integrate into your own pipeline.

Step 7 — Bulk-tag a folder to CSV

# tags.txt contains one candidate label per line
uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py tag-batch \
  --model clip --in-dir photos/ --tags-file tags.txt \
  --out-csv tags.csv --threshold 0.25 --top-k 5

Writes filename,label,score rows for every image × top-k labels above threshold. Good for automated catalog metadata.

Step 8 — Video scene description

uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py video-describe \
  --model llava --in clip.mp4 \
  --sample-fps 1 --prompt "Describe this frame in one sentence." \
  --out clip_scenes.jsonl

Samples the video at N fps, runs LLaVA on each frame, writes one JSONL row per sample: {"t": 3.0, "text": "..."}. Downstream, merge adjacent identical descriptions for scene-level summary, or pair with media-scenedetect for true scene-boundary sampling.

Gotcha: per-frame LLaVA is expensive. For a 5-minute video at --sample-fps 1 on a 3090 that's ~5 minutes of inference. Use --sample-fps 0.2 (one frame per 5s) or pre-segment with media-scenedetect and run once per scene.

Available scripts

• scripts/tag.py — subcommands: caption, classify, describe, search, tag-batch, video-describe, install. Each supports --dry-run, --verbose, --device {auto,cpu,cuda,mps}.

Reference docs

• Read references/models.md when picking a specific CLIP / SigLIP / BLIP-2 / LLaVA variant or tuning for VRAM, latency, or benchmark score.
• Read references/prompts.md when crafting LLaVA prompts for object detection, scene analysis, caption style, or video narration.
• Read references/LICENSES.md before shipping model output commercially or redistributing weights — includes a DROPPED-models list (Florence-2, CogVLM, Gemini, GPT-4V, Claude Vision) and why.

Gotchas

• "a photo of X" beats "X". CLIP / SigLIP classification accuracy jumps 5–10 points when you use full-sentence prompts. The codebase of OpenAI CLIP's eval harness ships 80 prompt templates ("a photo of a small X", "a photo of a large X", etc.) that average to a further 1–3 point gain. See references/prompts.md.
• Normalize embeddings before dot-product search. open_clip returns unnormalized features; you must L2-normalize before cosine similarity. The script handles this; if you build your own index, call features /= features.norm(dim=-1, keepdim=True).
• SigLIP scores are independent probabilities, not softmax-normalized. Two labels can both score 0.9; that's expected. Interpret as "likelihood this is X" per label, not "pick one of the labels".
• BLIP-2 can hallucinate. If the image is abstract / text-heavy / low-quality, the caption will confidently invent details. Cross-check with a second model (LLaVA) for anything user-facing.
• LLaVA OneVision (llava-onevision) handles video natively via frame-grid input — the older llava-1.5 and llava-1.6-mistral are image-only. Use OneVision if you're processing a video directly; otherwise sample frames and process individually.
• Float16 on CUDA is usually fine. On CPU or MPS, fall back to float32 for stability. Script auto-picks.
• Florence-2 is INTENTIONALLY SKIPPED. Microsoft dual-licensed it — code is MIT, but the CC-BY-4.0 on the published weights has ambiguity around fine-tuned derivatives. See references/LICENSES.md for the details. If/when a clean Apache variant ships, add it back.
• CLIP "a photo of a person" is still not face recognition. You cannot get named-individual recognition from CLIP/SigLIP/BLIP-2/LLaVA reliably, and you should not try — use a purpose-built recognizer (cv-opencv FaceRecognizerSF) and only with consent.
• No prompts, no interactive input. Scripts are fully non-interactive. All parameters via flags.

Examples

Example 1 — Alt text for an image

uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py caption \
  --model blip2 --in hero.jpg \
  --prompt "Describe this image in one short sentence for alt text."

Example 2 — Classify photo against custom labels

uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py classify \
  --model siglip --in street.jpg \
  --labels "a photo of a pedestrian,a photo of a cyclist,a photo of a car,a photo of a bus,a photo of a truck" \
  --top-k 3

Example 3 — Semantic search the last-year's camera roll

uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py search \
  --model clip --index ~/Pictures/2025/ --index-out 2025.npz

uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py search \
  --index 2025.npz --query "sunset at the beach" --top-k 20

Example 4 — Bulk-tag a folder with a controlled vocabulary

tags.txt:

a photo of a person
a photo of food
a photo of a landscape
a photo of architecture
a photo of an animal
a photo of a vehicle

uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py tag-batch \
  --model clip --in-dir photos/ --tags-file tags.txt \
  --out-csv photos_tags.csv --top-k 2 --threshold 0.20

Example 5 — Describe a video clip scene-by-scene

# optional: first find scene boundaries via media-scenedetect
scenedetect -i clip.mp4 detect-adaptive list-scenes

# then describe each scene's mid-frame
uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py video-describe \
  --model llava --in clip.mp4 --sample-fps 0.5 \
  --prompt "Describe this scene in 1-2 sentences." \
  --out scenes.jsonl

Example 6 — Detailed visual QA

uv run ${CLAUDE_SKILL_DIR}/scripts/tag.py describe \
  --model llava --in kitchen.jpg \
  --prompt "List all appliances you can see and estimate their color."

Troubleshooting

OSError: google/siglip-base-patch16-384 is not a local folder

Cause: No network on first run, or HF Hub outage. Solution: Pre-download with tag.py install siglip when online; weights cache to ~/.cache/huggingface/hub/.

Search returns nonsense for perfectly reasonable queries

Cause: Embeddings not L2-normalized before cosine similarity. Solution: The script normalizes by default. If building your own index, features /= features.norm(dim=-1, keepdim=True) before any dot-product or FAISS index insert.

LLaVA OOM on a 24 GB GPU

Cause: Default torch.float16 + full attention is still tight for 7 B on large inputs. Solution: Use --device cpu (slow but works), or pick a smaller variant (llava-1.5-7b is smaller than -13b), or enable 4-bit via bitsandbytes (see references/models.md).

CLIP classification scores all look the same

Cause: Label prompts are too similar (cosine similarity saturates) or too dissimilar from ImageNet training distribution. Solution: Rewrite prompts as "a photo of X" / "an image showing X". For domain-specific labels, an ensemble of 5-10 prompt templates per class averaged is the standard trick.

Video-describe runs for hours

Cause: --sample-fps 1 is one LLaVA inference per second. For a 30-min video that's 1 800 inferences. Solution: Reduce to --sample-fps 0.2 (one per 5 s), or pre-segment with media-scenedetect and run once per scene midpoint.

BLIP-2 captions are boring / repetitive

Cause: Default greedy decoding. Solution: The script uses num_beams=5 by default. Crank to 10 or enable do_sample=True with temperature 0.7 via --beams 10 --temperature 0.7 (if wired in). Or switch to LLaVA for more varied language.

SigLIP scores don't sum to 1

Cause: SigLIP uses sigmoid, not softmax. Each label probability is independent. Solution: Treat as independent yes/no probabilities. Sum > 1 or < 1 is expected.