How do I install the Multi Turn Preference Evaluation skill?

Run npx skillmds@latest add dingxingdi/multi-turn-preference-evaluation-2 in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Multi Turn Preference Evaluation safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Multi Turn Preference Evaluation?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Multi Turn Preference Evaluation free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Multi Turn Preference Evaluation?

dingxingdi (@dingxingdi) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Multi Turn Preference Evaluation by dingxingdi

Name: Multi Turn Preference Evaluation
Author: dingxingdi

Multi Turn Preference Evaluation

Use this skill when a user wants data where the evaluator must judge which assistant handled a back-and-forth conversation better, especially when people say things like 'compare two chats', 'see who handled the follow-up better', 'test memory across turns', 'check whether the answer stayed on track', or 'make the second question matter'. Trigger it for pairwise judging of multi-turn dialogues where later turns depend on earlier turns, and where the decision should reflect instruction following, coherence, recall, and usefulness across the full exchange. Example triggers in plain language include: 'give me judge data for two-step conversations', 'make questions where the follow-up exposes weak memory', 'compare which answer handles the second part better', and 'test whether the model keeps the context straight across turns'.

dingxingdi dbbbe8c 5 files · 16.4 KB Updated Sep 17, 2026

File contents

Skill: multi-turn-preference-evaluation

1. Capability Definition & Real Case

Professional Definition: This capability is the ability to generate, inspect, and judge evaluation instances where the target output is an evaluative artifact rather than a task solution, and where the evaluator must ground its verdict in explicit evidence, criteria, or safety policy. In this capability, the Agent is not solving the original user problem; it is assessing the quality, correctness, safety, or robustness of another response, trajectory, or artifact and producing a score, preference, label, ranking, or critique.
Dimension Hierarchy: Open-ended Response Evaluation->Contextual Text Response Evaluation->multi-turn-preference-evaluation

Real Case

[Case 1]

Initial Environment: An evaluation environment contains one user prompt about the effect of the Federal Reserve buying bonds in the secondary market, followed by a second user prompt asking for three everyday-life examples. Two assistant transcripts are available side by side. Assistant A answers the first turn correctly but becomes repetitive and vague on the follow-up. Assistant B answers the first turn correctly and then gives concrete examples about interest rates, inflation, and employment.
Real Question: Which assistant handled the conversation better overall?
Real Trajectory: Read the first-turn question and both first-turn answers. Read the second-turn request and both follow-up answers. Compare whether each assistant preserved context, addressed the second-turn instruction, and remained helpful across the full dialogue. Return a pairwise verdict with a short explanation.
Real Answer: Assistant B is better overall.
Why this demonstrates the capability: This case isolates a core multi-turn judging failure mode: both assistants look acceptable on the opening turn, but only one assistant correctly adapts when the user changes the task from factual recall to applied explanation. The evaluator must integrate evidence across turns rather than scoring each turn in isolation. It therefore tests dialogue-state tracking, follow-up sensitivity, and preference judgment under conversational dependency.

Pipeline Execution Instructions

To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:

Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds: references/EXPLORATION.md
Phase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: references/SELECTION.md
Phase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: references/SYNTHESIS.md

---name: multi-turn-preference-evaluation-2description: Use this skill when a user wants data where the evaluator must judge which assistant handled a back-and-forth conversation better, especially when people say things like 'compare two chats', 'see who handled the follow-up better', 'test memory across turns', 'check whether the answer stayed on track', or 'make the second question matter'. Trigger it for pairwise judging of multi-turn dialogues where later turns depend on earlier turns, and where the decision should reflect instruction following, coherence, recall, and usefulness across the full exchange. Example triggers in plain language include: 'give me judge data for two-step conversations', 'make questions where the follow-up exposes weak memory', 'compare which answer handles the second part better', and 'test whether the model keeps the context straight across turns'.---# Skill: multi-turn-preference-evaluation## 1. Capability Definition & Real Case* **Professional Definition**: This capability is the ability to generate, inspect, and judge evaluation instances where the target output is an evaluative artifact rather than a task solution, and where the evaluator must ground its verdict in explicit evidence, criteria, or safety policy. In this capability, the Agent is not solving the original user problem; it is assessing the quality, correctness, safety, or robustness of another response, trajectory, or artifact and producing a score, preference, label, ranking, or critique.* **Dimension Hierarchy**: Open-ended Response Evaluation->Contextual Text Response Evaluation->multi-turn-preference-evaluation### Real Case**[Case 1]*** **Initial Environment**: An evaluation environment contains one user prompt about the effect of the Federal Reserve buying bonds in the secondary market, followed by a second user prompt asking for three everyday-life examples. Two assistant transcripts are available side by side. Assistant A answers the first turn correctly but becomes repetitive and vague on the follow-up. Assistant B answers the first turn correctly and then gives concrete examples about interest rates, inflation, and employment.* **Real Question**: Which assistant handled the conversation better overall?* **Real Trajectory**: Read the first-turn question and both first-turn answers. Read the second-turn request and both follow-up answers. Compare whether each assistant preserved context, addressed the second-turn instruction, and remained helpful across the full dialogue. Return a pairwise verdict with a short explanation.* **Real Answer**: Assistant B is better overall.* **Why this demonstrates the capability**: This case isolates a core multi-turn judging failure mode: both assistants look acceptable on the opening turn, but only one assistant correctly adapts when the user changes the task from factual recall to applied explanation. The evaluator must integrate evidence across turns rather than scoring each turn in isolation. It therefore tests dialogue-state tracking, follow-up sensitivity, and preference judgment under conversational dependency.## Pipeline Execution InstructionsTo synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially:1. **Phase 1: Environment Exploration** Read the exploration guidelines to discover raw knowledge seeds: `references/EXPLORATION.md`2. **Phase 2: Trajectory Selection** Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: `references/SELECTION.md`3. **Phase 3: Data Synthesis** Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: `references/SYNTHESIS.md`

Frequently asked questions