# Constitution AI Feedback Eval

> This evaluation probes how different instructional guidelines (constitutions) shape AI-generated medical dialogues across specific socio-communicative dimensions like empathy, information gathering, and decision-making. It measures human preference for dialogue quality under varying constitutional constraints. Use when the user wants to benchmark on Custom AI-generated medical dialogues, or asks about evaluating this task. Reports Bradley-Terry preference rate.

- Skill: `qhjqhj00/constitution-ai-feedback-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/constitution-ai-feedback-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/constitution-ai-feedback-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/constitution-ai-feedback-eval

---


# constitution-ai-feedback-eval

> Evaluating the role of `Constitutions' for learning from AI feedback — Redgate et al. (2024) (arXiv:2411.10168, 2024)

## What this evaluates

This evaluation probes how different instructional guidelines (constitutions) shape AI-generated medical dialogues across specific socio-communicative dimensions like empathy, information gathering, and decision-making. It measures human preference for dialogue quality under varying constitutional constraints.

## Datasets

- **Custom AI-generated medical dialogues** — total ?; splits: test (-1)

## Metrics

- `Bradley-Terry preference rate` **(primary)** — range: [0, 1]
  - Estimated from pairwise human preferences using a Bradley-Terry model. Represents the relative likelihood that a dialogue generated under a specific constitution is preferred over others for a given evaluation dimension.

## Input / output format

**Input**: AI-generated medical dialogue transcripts conditioned on different constitutional guidelines.

**Output**: Human pairwise preference ratings across six communication dimensions (Fostering Relationship, Gathering Information, Providing Information, Decision Making, Responding to Emotions, Enabling treatment behaviour).

## Scoring recipe

```python
# predictions: list of human pairwise choices (winner_constitution, loser_constitution) per dimension
# gold: constitution labels
counts = defaultdict(int)
for winner, loser in predictions:
    counts[winner] += 1
    counts[loser] += 1
# Fit Bradley-Terry model to estimate log-odds preference parameters
bt_params = fit_bradley_terry(counts)
# Convert to preference rate for reporting
return {dim: sigmoid(bt_params[dim]) for dim in dimensions}
```

## Common pitfalls

- Error bars represent 95% confidence intervals not adjusted for multiple comparisons.
- Holistic constitution selection showed no clear pattern despite significant dimension-specific preferences.
- Participants often disliked verbose or overly emotive responses even when rating them as more empathetic.

## Evidence (verbatim from paper)

> In Figure 2, we show the rate at which the conversations generated according to each constitution are preferred to the others for each dimension of evaluation, alongside the estimated parameters for a Bradley-Terry model.

## Citation

```bibtex
@misc{redgate2024constitutions,
  title={Evaluating the role of `Constitutions' for learning from AI feedback},
  author={Redgate et al. (2024)},
  year={2024},
  note={arXiv:2411.10168}
}
```

- arXiv: 2411.10168