Superni Performance Prediction Eval

superni-performance-prediction-eval

Third-Party Language Model Performance Prediction from Instruction — Nadkarni et al. (2024) (arXiv:2403.12413, 2024)

What this evaluates

Evaluates the ability of a predictor model to estimate the performance of instruction-following language models on unseen tasks, using only the task instruction as input. It probes the fundamental challenge of third-party model transparency and controllability at the task level.

Datasets

• SuperNI — total ?; splits: test (-1); repo https://github.com/rahuln/instr-perf-pred

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error between the predicted performance score and the true performance score across all test tasks. Computed as sqrt(mean((predicted - true)^2)).

Input / output format

Input: Task instruction (string), optionally accompanied by two positive demonstrations (string) for few-shot prompting.

Output: A single floating-point number representing the predicted performance score (e.g., Exact Match or ROUGE-L percentage, or cross-entropy loss).

Scoring recipe

def compute_rmse(predictions, ground_truth):
    errors = [(p - g) ** 2 for p, g in zip(predictions, ground_truth)]
    return math.sqrt(sum(errors) / len(errors))

# Protocol: Train predictor on instruction-metric pairs, evaluate on 10 random splits of SuperNI test set, report mean RMSE ± std.

Common pitfalls

• Confusing the target metric being predicted (Exact Match, ROUGE-L, or Loss) with the evaluation metric used to assess the predictor (RMSE).
• Assuming that increasing the size of the predictor model or adding diverse training tasks (e.g., BIG-bench) will significantly improve prediction accuracy, as domain shift and lack of learnable signal limit gains.
• Expecting the predictor to generalize across different instruction formats or model families without retraining, as performance predictability varies significantly by model and metric.

Evidence (verbatim from paper)

Table 1 shows our main results of predicting performance on SuperNI test set instructions, with mean and standard deviation (subscript) RMSE values across all train-test splits of the SuperNI test tasks. We explore using both Exact Match and ROUGE-L as the target metrics to be predicted for each task, with base and large RoBERTa models as the PP models as well as LLaMA-13B (upper bound) and the simple mean baseline (lower bound).

Citation

@misc{nadkarni2024instrperfpred,
  title={Third-Party Language Model Performance Prediction from Instruction},
  author={Nadkarni et al. (2024)},
  year={2024},
  note={arXiv:2403.12413}
}

• arXiv: 2403.12413