Vision Language Response Judgment

Skill: vision-language-response-judgment

1. Capability Definition & Real Case

• Professional Definition: This capability is the ability to generate, inspect, and judge evaluation instances where the target output is an evaluative artifact rather than a task solution, and where the evaluator must ground its verdict in explicit evidence, criteria, or safety policy. In this capability, the Agent is not solving the original user problem; it is assessing the quality, correctness, safety, or robustness of another response, trajectory, or artifact and producing a score, preference, label, ranking, or critique.
• Dimension Hierarchy: Open-ended Response Evaluation->Multimodal Response Evaluation->vision-language-response-judgment

Real Case

[Case 1]

• Initial Environment: An evaluation environment contains a bar chart and a user question asking what percentage of workers are not working from home. Two candidate answers are available. One answer correctly reads the chart and states the right percentage with a brief explanation. The other answer sounds fluent but misreads the visual values and hallucinates an unsupported comparison.
• Real Question: Which response should the evaluator prefer for this image-based question?
• Real Trajectory: Inspect the visual input, extract the relevant visual evidence, then read both candidate answers. Check whether each answer refers to the correct chart elements, avoids hallucinated details, and answers the exact user question. Return a verdict grounded in visual-text alignment.
• Real Answer: The evaluator should prefer the answer that correctly reads the chart and avoids unsupported visual claims.
• Why this demonstrates the capability: This case demonstrates multimodal judgment rather than text-only answer evaluation. The evaluator must determine whether the language is faithful to the image and whether any visually grounded inference is correct. It therefore targets visual evidence tracking, hallucination resistance, and multimodal pairwise judgment.

Pipeline Execution Instructions

To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:

1. Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds: references/EXPLORATION.md

2. Phase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: references/SELECTION.md

3. Phase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: references/SYNTHESIS.md