Almanacs Eval

almanacs-eval

ALMANACS: A Simulatability Benchmark for Language Model Explainability — Mills et al. (2023) (arXiv:2312.12747, 2023)

What this evaluates

Evaluates whether language model explanations (e.g., weights, qualitative descriptions) enable a second predictor model to accurately simulate and predict the behavior of a synthetic linear model across safety-relevant scenarios. The benchmark specifically probes simulatability and robustness to distributional shift between training and test variable values.

Datasets

• ALMANACS Synthetic Dataset — total ?; splits: train (-1), test (-1); repo https://github.com/edmundmills/ALMANACS

Metrics

• probability (primary) — range: [0, 1]
  • The predictor outputs a float between 0 and 1 representing the likelihood of a 'Yes' answer. Performance is evaluated by comparing this predicted probability against the true probability generated by the synthetic linear model.

Input / output format

Input: Template-based scenario questions with five variable placeholders, accompanied by either WEIGHTS (numerical weights and variable-to-score mappings) or QUALITATIVE (natural language descriptions of variable influence and ranking) explanations.

Output: A JSON object containing a 'reasoning' string and a 'probability' float representing the model's estimated likelihood of a 'Yes' answer.

Scoring recipe

def score(predictions, gold):
    # predictions: list of predicted probabilities from predictor
    # gold: list of true probabilities from synthetic model
    # Evaluate how closely predicted probabilities match the ground truth
    mse = sum((p - g) ** 2 for p, g in zip(predictions, gold)) / len(predictions)
    return mse

Common pitfalls

• The synthetic model relies on all-distilroberta-v1 embeddings and per-placeholder PCA, which may not capture the semantic complexity of real-world LLM reasoning.
• Predictors must estimate missing variable scores from limited examples, introducing compounding errors when multiplied by weights.
• The benchmark explicitly tests distributional shift between train and test variables, meaning standard in-distribution fine-tuning will not generalize.

Evidence (verbatim from paper)

Given a sample S with scores {s_1, s_2, ..., s_5} for the values filled in the template, the prediction y which corresponds to the probability of a 'yes' answer is computed as: y = σ(∑ w_i s_i) where σ denotes the sigmoid function.

Citation

@misc{mills2023almanacs,
  title={ALMANACS: A Simulatability Benchmark for Language Model Explainability},
  author={Mills et al. (2023)},
  year={2023},
  note={arXiv:2312.12747}
}

• arXiv: 2312.12747