Qwen-MM-Plugins Core
You have qwen-mm-plugins-core MCP tools available. Use them to read and visualize any file (images, videos, documents, code, data, 3D models, notebooks, etc.) and to call vision-language APIs. Prefer these MCP tools over manual scripting.
Check the qwen-mm-plugins-core tools in your tool list for full schemas and parameters.
When to Use Which Tool
Native reading (feeds content directly to you):
- See a file (PDF, Office, CSV, code, notebook, 3D, ...) →
visualize
- Read an image with dynamic resolution →
read_image
- Read a video (extract frames) →
read_video
- Get media metadata (duration, resolution, fps, codecs, bitrate, audio/video/subtitle tracks, chapters) →
media_info
- Save specific frame(s) of a video to file (for image_search) →
save_view (pass times=[...])
Producing / annotating (writes an image file):
- Crop a rectangular region from an image →
crop
- Draw bounding boxes on an image (e.g. to visualize
grounding output) → draw_bbox
External API calls (DashScope):
- Call an external VLM about images/videos →
vision_chat
- Transcribe speech from audio/video →
transcribe_audio
- Detect/locate objects in an image →
grounding
- Extract text from an image →
ocr
- Segment objects in an image →
segmentation
- Reverse image search to identify an entity from a frame →
image_search
- Search the web for facts →
web_search
- Read a web page in depth →
web_extractor
Visualize — Supported Formats
| Category |
Extensions |
Notes |
| Documents |
.pdf, .svg |
Built-in (pypdfium2 + resvg) |
| Office |
.docx, .pptx, .vsdx |
Needs libreoffice |
| Data |
.csv, .xlsx |
Text table + chart image |
| Code |
.js, .ts, .py, .go, .rs, .md, ... |
Returns text (markdown code block) |
| Plain text |
.txt, .text, .log |
Returns text (fenced block) |
| Web pages |
.html, .htm, .mhtml |
Screenshot; needs playwright |
| Diagrams |
.drawio |
XML → SVG rendering |
| Subtitles |
.srt, .vtt |
Returns text |
| 3D Models |
.obj, .stl, .glb, .gltf, .fbx, .ply, .step, .stp |
Built-in; blender for best quality |
| GIS/Geo |
.geojson, .kml, .shp |
Built-in |
| Notebooks |
.ipynb |
Text cells + embedded images |
| LaTeX |
.tex |
Compiles to PDF; falls back to source on failure |
| Images/Videos |
.jpg, .mp4, ... |
Delegates to read_image/read_video |
Use pages for page ranges, budget for resolution, max_pages to cap output.
Tips
Resolution budgets: small for preview, normal (~1024) default, large for fine detail.
Video strategy: fps=0 auto-selects. Skim with fps=1, budget="normal" in 5-min chunks, then detail with fps=2, budget="large" on interesting segments. Use start_time/end_time for windowing. Before any editing/clipping task, run media_info first to learn the source properties (codecs, tracks, fps, rotation).
Vision chat: pass images/videos + text prompt. Default model qwen3.7-plus. Use dry_run=true to inspect payloads. Details in references/vision_chat.md.
Video knowledge & search (any video question that needs external knowledge — identify a specific thing, OR a fact about what's shown that you can't confirm from the frames alone): watch with read_video — for a long video do a low-fps overview first (~32 frames, fps≈32/duration) then zoom, and don't run ffmpeg/montage yourself — grab a frame with save_view (times=[...]), and you MUST confirm with image_search (reverse-search the frame) and/or web_search before answering — never commit from frames alone. Details in references/video_search.md.
ASR: accepts audio or video, auto-chunks long files. Formats: srt (default), text, json. Needs DASHSCOPE_API_KEY.
Segmentation: needs a SAM3 server (SAM3_SERVER_URL). To stand one up, run references/launch_sam3_server.py (multi-GPU HTTP server; see its header for prerequisites).
1---2name: qwen-mm-plugins-core3description: Vision-Language MCP tools for reading/visualizing any file (images, videos, documents, code, data, 3D models, etc.) and calling VL APIs (caption, ASR, search, etc.).4---5
6# Qwen-MM-Plugins Core
7
8You have `qwen-mm-plugins-core` MCP tools available. Use them to read and visualize any file (images, videos, documents, code, data, 3D models, notebooks, etc.) and to call vision-language APIs. Prefer these MCP tools over manual scripting.
9
10Check the `qwen-mm-plugins-core` tools in your tool list for full schemas and parameters.
11
12## When to Use Which Tool
13
14Native reading (feeds content directly to you):
15- **See a file** (PDF, Office, CSV, code, notebook, 3D, ...) → `visualize`
16- **Read an image** with dynamic resolution → `read_image`
17- **Read a video** (extract frames) → `read_video`
18- **Get media metadata** (duration, resolution, fps, codecs, bitrate, audio/video/subtitle tracks, chapters) → `media_info`
19- **Save specific frame(s)** of a video to file (for image_search) → `save_view` (pass `times=[...]`)
20
21Producing / annotating (writes an image file):
22- **Crop a rectangular region** from an image → `crop`
23- **Draw bounding boxes** on an image (e.g. to visualize `grounding` output) → `draw_bbox`
24
25External API calls (DashScope):
26- **Call an external VLM** about images/videos → `vision_chat`
27- **Transcribe speech** from audio/video → `transcribe_audio`
28- **Detect/locate objects** in an image → `grounding`
29- **Extract text** from an image → `ocr`
30- **Segment objects** in an image → `segmentation`
31- **Reverse image search** to identify an entity from a frame → `image_search`
32- **Search the web** for facts → `web_search`
33- **Read a web page** in depth → `web_extractor`
34
35## Visualize — Supported Formats
36
37| Category | Extensions | Notes |
38|----------|-----------|-------|
39| Documents | `.pdf`, `.svg` | Built-in (pypdfium2 + resvg) |
40| Office | `.docx`, `.pptx`, `.vsdx` | Needs `libreoffice` |
41| Data | `.csv`, `.xlsx` | Text table + chart image |
42| Code | `.js`, `.ts`, `.py`, `.go`, `.rs`, `.md`, ... | Returns text (markdown code block) |
43| Plain text | `.txt`, `.text`, `.log` | Returns text (fenced block) |
44| Web pages | `.html`, `.htm`, `.mhtml` | Screenshot; needs `playwright` |
45| Diagrams | `.drawio` | XML → SVG rendering |
46| Subtitles | `.srt`, `.vtt` | Returns text |
47| 3D Models | `.obj`, `.stl`, `.glb`, `.gltf`, `.fbx`, `.ply`, `.step`, `.stp` | Built-in; `blender` for best quality |
48| GIS/Geo | `.geojson`, `.kml`, `.shp` | Built-in |
49| Notebooks | `.ipynb` | Text cells + embedded images |
50| LaTeX | `.tex` | Compiles to PDF; falls back to source on failure |
51| Images/Videos | `.jpg`, `.mp4`, ... | Delegates to `read_image`/`read_video` |
52
53Use `pages` for page ranges, `budget` for resolution, `max_pages` to cap output.
54
55## Tips
56
57**Resolution budgets**: `small` for preview, `normal` (~1024) default, `large` for fine detail.
58
59**Video strategy**: `fps=0` auto-selects. Skim with `fps=1, budget="normal"` in 5-min chunks, then detail with `fps=2, budget="large"` on interesting segments. Use `start_time`/`end_time` for windowing. Before any editing/clipping task, run `media_info` first to learn the source properties (codecs, tracks, fps, rotation).
60
61**Vision chat**: pass `images`/`videos` + `text` prompt. Default model `qwen3.7-plus`. Use `dry_run=true` to inspect payloads. Details in `references/vision_chat.md`.
62
63**Video knowledge & search** (any video question that needs external knowledge — identify a specific thing, OR a fact about what's shown that you can't confirm from the frames alone): watch with `read_video` — for a long video do a low-fps overview first (~32 frames, `fps≈32/duration`) then zoom, and don't run `ffmpeg`/montage yourself — grab a frame with `save_view` (`times=[...]`), and you MUST confirm with `image_search` (reverse-search the frame) and/or `web_search` before answering — never commit from frames alone. Details in `references/video_search.md`.
64
65**ASR**: accepts audio or video, auto-chunks long files. Formats: `srt` (default), `text`, `json`. Needs `DASHSCOPE_API_KEY`.
66
67**Segmentation**: needs a SAM3 server (`SAM3_SERVER_URL`). To stand one up, run `references/launch_sam3_server.py` (multi-GPU HTTP server; see its header for prerequisites).
theheavenlyd3mon