Intervention Scoring

intervention_scoring

Automatically Interpreting Millions of Features in Large Language Models — Gonçalo Paulo et al. (arXiv:2410.13928, 2024)

What this evaluates

Evaluates the fidelity of natural language explanations for sparse autoencoder (SAE) features by measuring how well an explanation predicts the downstream effects of directly intervening on the feature's activation, rather than just correlating with input contexts.

Datasets

• Gemma 2 9B SAE features — total 500; splits: test (500)

Metrics

• intervention_scoring (primary) — range: [-1, 1]
  • Measures the correlation between the predicted effect of a feature intervention (derived from its natural language explanation) and the actual observed change in activation patterns when the feature is directly manipulated. Higher scores indicate the explanation accurately captures the feature's causal role.
• fuzzing — range: [-1, 1]
  • Measures how well an explanation helps predict which tokens have non-zero activation in a given context. Correlates closely with simulation scoring due to SAE sparsity.
• detection — range: [0, 1]
  • Binary or continuous score assessing whether a given context correctly activates the target feature versus non-activating contexts.

Input / output format

Input: Per feature: a natural language interpretation, a set of activating contexts, and a set of non-activating contexts (typically 100 of each).

Output: A scalar score (e.g., Spearman correlation or accuracy) quantifying how well the explanation matches the feature's behavior under intervention or context activation.

Scoring recipe

def intervention_scoring(feature, explanation, contexts):
    predicted_effects = []
    actual_effects = []
    for ctx in contexts:
        pred = llm_predict_effect(explanation, ctx)
        predicted_effects.append(pred)
        base_act = get_feature_activation(ctx)
        intervened_act = intervene_feature(ctx, feature, strength=high)
        actual = intervened_act - base_act
        actual_effects.append(actual)
    score = spearman_corr(predicted_effects, actual_effects)
    return score

Common pitfalls

• Relying solely on top-activating examples for evaluation misses features whose behavior is better explained by downstream effects rather than input contexts.
• Different scoring methods (fuzzing, detection, simulation, intervention) have distinct failure modes and low inter-correlation; using a single metric yields incomplete fidelity assessment.
• Evaluation on a small subset of the activation distribution fails to capture how interpretations generalize across quantiles.

Evidence (verbatim from paper)

Intervention scoring proposes to measure how well a given interpretation can predict the effect of interventions on the corresponding feature. Here we compare correlational interpretations generated with our pipeline and scored with fuzzing, to a set of interventional interpretations scored with intervention scoring. Our hypothesis is that some features will have low correlational scores because their behavior is better explained by their downstream effects than by the contexts where they are active.

Citation

@misc{paulo2024automaticallyinterpreting,
  title={Automatically Interpreting Millions of Features in Large Language Models},
  author={Gonçalo Paulo et al.},
  year={2024},
  note={arXiv:2410.13928}
}

• arXiv: 2410.13928