# Cross Modal Alignment And Dialogue Grounding

> Use this skill when the user wants questions about 'who said it', 'which sound matches this moment', 'connect the voice to the gesture', 'use both the video and the audio', or 'ground the answer in dialogue or subtitle timing.' Trigger it whenever neither the video alone nor the audio alone is enough to answer reliably.

- Skill: `dingxingdi/cross-modal-alignment-and-dialogue-grounding-2` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add dingxingdi/cross-modal-alignment-and-dialogue-grounding-2`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dingxingdi/cross-modal-alignment-and-dialogue-grounding-2/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: dingxingdi (https://skillmd.com/u/dingxingdi)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/dingxingdi/cross-modal-alignment-and-dialogue-grounding-2

---


# Skill: cross-modal-alignment-and-dialogue-grounding
## 1. Capability Definition & Real Case
* **Professional Definition**: The capability to temporally align visual events with speech, sound, subtitles, or speaker identity, so that the answer depends on synchronized multimodal evidence rather than on vision-only or audio-only cues.
* **Dimension Hierarchy**: Multimodal and Generative Interpretation->Cross-Modal Grounding->cross-modal-alignment-and-dialogue-grounding
### Real Case

**[Case 1]**
* **Initial Environment**: A long kitchen-and-living-room family video with clear speech, object interactions, and repeated refrigerator visits. The needed clue spans one spoken instruction and a later visual storage action.
* **Real Question**: Which shelf in the refrigerator, counting from the top, does Cary's family usually put the wine they bought on?
* **Real Trajectory**: Locate the spoken instruction about where yogurt should go, normalize 'second floor' as the second shelf from the top, find the later refrigerator placement of the wine, and infer the wine shelf by aligning the later visual placement with the earlier verbal reference.
* **Real Answer**: Third.
* **Why this demonstrates the capability**: The answer emerges only after synchronizing dialogue content with later visual evidence. This tests whether the agent can align two modalities over time instead of treating speech and video as unrelated streams.

---
## Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially:

1. **Phase 1: Environment Exploration**
   Read the exploration guidelines to discover raw knowledge seeds:
   `references/EXPLORATION.md`

2. **Phase 2: Trajectory Selection**
   Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
   `references/SELECTION.md`

3. **Phase 3: Data Synthesis**
   Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
   `references/SYNTHESIS.md`