Qwen Mm Plugins Omni Av

Qwen-MM-Plugins Omni-AV

You have qwen-mm-plugins-omni-av MCP tools. They call the Qwen-Omni model, which reads the video frames and the embedded audio track together, so one call can reason over both. Prefer these tools over manual ffmpeg/ffprobe scripting.

Check the qwen-mm-plugins-omni-av tools in your tool list for full schemas and parameters.

When to Use Which Tool

• Transcribe speech, plain text → omni_asr (one continuous string, no timestamps).
• Transcribe with timestamps → omni_asr_timestamped (granularity = sentence or word; also returns SRT).
• Who said what → omni_multi_speaker_asr (diarization: speaker labels + timestamps + SRT; pass num_speakers if known).
• Describe the content over time → omni_av_caption (splits into spans, one description + start/end per span).
• Find WHEN something happens → omni_av_grounding (natural-language query → matching time segments). Temporal localization.
• Count how many times an event/object/action occurs → omni_av_counting (target → total + per-occurrence timestamps).
• Analyze / caption a music track → omni_music_caption (whole-track tags — genre / moods / instruments / key / time signature / vocal profile — plus a dense English caption for music generation; audio-only, no timestamps).

Every tool takes a local audio/video file_path (or an http/OSS URL) and supports dry_run=true to preview the request without calling the API. The AV tools (caption/grounding/counting) accept fps and max_pixels to trade temporal/spatial detail against token cost.

Relationship to other capabilities (do NOT overlap)

• omni_asr* vs core transcribe_audio: core's transcribe_audio uses the dedicated qwen3-asr-flash service (fast, chunks long files, 27 languages). The omni_asr* tools here use the Omni model — pick them when you want Omni's understanding (multi-speaker diarization, controllable word/sentence granularity, or transcription fused with visual context). For a straight, long-file transcription, core's transcribe_audio is cheaper.
• omni_av_grounding (temporal, WHEN) vs core grounding (spatial, WHERE): this tool locates a span in time; core's grounding draws a bounding box in a single image. They are different axes — don't substitute one for the other.
• Long videos (30 min+): for whole-video QA over long content, use the qwen-mm-plugins-video-memory skill (hierarchical graph memory) instead of feeding the entire file to these per-call tools. These tools target clips up to a few minutes.

Tips

• Cost/latency: sending video samples frames — raise fps only for fast or frequent events; keep max_pixels at the default (≈448²) unless fine visual detail matters. ASR-family tools send only the audio track (the video's audio is extracted first), so they are cheaper on video input.
• Timestamps are in seconds from the start of the media.
• Language: pass language as a hint (e.g. zh, en) for the ASR/caption tools when known; otherwise the model auto-detects.
• Requires DASHSCOPE_API_KEY. Default model qwen3.5-omni-plus; override per call with model.