Ravel Eval

ravel-eval

RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations — Huang et al. (2024) (arXiv:2402.17700, 2024)

What this evaluates

Evaluates interpretability methods' ability to disentangle polysemantic language model representations by isolating causal attributes through activation interventions on residual stream features.

Datasets

• RAVEL — total ?; splits: test (-1), Entity (-1), Context (-1); repo https://github.com/explanare/ravel

Metrics

• Disentanglescore (primary) — range: [0, 1]
  • Composite metric combining causal attribution and isolation scores to measure how well a learned feature subspace isolates a target attribute from others.
• Causescore — range: percent
  • Measures the causal effect of intervening on a target feature subspace on a specific attribute. Defined as the percentage of times intervening on one attribute's feature changes another attribute.
• Iso — range: [0, 1]
  • Measures the isolation of a learned feature, indicating how independent it is from other attributes during intervention.

Input / output format

Input: Residual stream representation vectors N^L_tE at the last token of an entity across Transformer layers L in {1, ..., 32}.

Output: A targeted feature subspace F_A (or intervention vector) intended to isolate a specific attribute from the representation.

Scoring recipe

# For each layer L and entity representation N^L_tE:
# 1. Extract learned feature subspace F_A for target attribute.
# 2. Perform interchange intervention: replace N^L_tE with F_A.
# 3. Compute Cause score: measure causal effect on target attribute.
# 4. Compute Iso score: measure independence from other attributes.
# 5. Compute Disentanglescore = f(Cause, Iso).
# Return max Disentanglescore across all layers on test split.

Common pitfalls

• Methods that do not use counterfactual supervision (e.g., PCA, Sparse Autoencoder) struggle significantly compared to intervention-trained methods.
• Some attribute pairs (e.g., country-language, latitude-longitude) are inherently difficult to disentangle due to unavoidable ripple effects, regardless of the method used.
• Evaluations are limited to the last token of the entity in the residual stream; ignoring earlier tokens or other representation sites may miss important disentanglement signals.

Evidence (verbatim from paper)

We evaluate each method on every representation N^L_tE and report the highest disentanglement score on test splits in Table[2]. A Causescore of 0.62 for column Continent, row Timezone (bottom left corner), means that, when intervening on the Continent feature, the same subspace changes Timezone 62% of the time.

Citation

@misc{huang2024ravel,
  title={RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2402.17700}
}

• arXiv: 2402.17700