Inverse Constitutional AI Eval

inverse-constitutional-ai-eval

Inverse Constitutional AI: Compressing Preferences into Principles — Findeis et al. (2024) (arXiv:2406.06560, 2024)

What this evaluates

Tests a framework's ability to compress pairwise preference data into interpretable natural language principles (constitutions) and use them to reconstruct original annotations. It probes the model's adaptability to aligned, unaligned, individual, and demographic group preferences, as well as its capacity for bias detection.

Datasets

• Synthetic data — total 90; splits: (unstated)
• AlpacaEval — total 648; splits: train (65), test (65)
• Chatbot Arena Conversations — total 33000; splits: (unstated)
• PRISM — total 8011; splits: (unstated)

Metrics

• agreement (primary) — range: [0, 1]
  • Fraction of correctly reconstructed preferences out of the total number of preferences in the test set.
• accuracy — range: [0, 1]
  • Number of correctly reconstructed relevant data points divided by the total number of relevant data points for a given principle.
• relevance — range: [0, 1]
  • Fraction of data points where a specific principle applies out of the total data points evaluated.

Input / output format

Input: Pairwise preference instances (prompt + two responses) along with a generated constitution (set of natural language principles) to guide annotation.

Output: Binary selection of the preferred response, or a score indicating agreement with the original annotation.

Scoring recipe

def compute_agreement(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_relevance(principle, data_points):
    applies = sum(1 for dp in data_points if principle_applies(dp))
    return applies / len(data_points)

Common pitfalls

• Small sample sizes for individual/group preferences prevent standard train/test splits.
• Default annotators already achieve high agreement on aligned data, leaving little room for improvement.
• Constitution transferability across different LLM families incurs performance loss.

Evidence (verbatim from paper)

We find that constitutional annotators reconstruct original annotations better in the orthogonal and unaligned datasets, and keep high agreement in the aligned case... Results, shown in [Table 1], reveal biases regarding verbosity, style, and assertiveness... showing relevance (fraction of data points where the principle applies) and accuracy (correctly reconstructed relevant data points).

Citation

@misc{findeis2024inverse,
  title={Inverse Constitutional AI: Compressing Preferences into Principles},
  author={Findeis et al. (2024)},
  year={2024},
  note={arXiv:2406.06560}
}

• arXiv: 2406.06560