Curll Eval

curll-eval

CurLL: A Developmental Framework to Evaluate Continual Learning in Language Models — Kalyan et al. (2025) (arXiv:2510.13008, 2025)

What this evaluates

Evaluates continual learning capabilities in language models by measuring skill retention, forward/backward transfer, and catastrophic forgetting across a developmental skill graph spanning ages 5–10. It probes how sequential, joint, and independent training affect performance on instruction, context-question-answer, and context-sentence-question-answer tasks.

Datasets

• CurLL — total ?; splits: test (-1)

Metrics

• LLM rating score (1-5) (primary) — range: other
  • An external LLM rates the model's generation on a scale of 1 to 5 based on correctness and stage-specific weightage. Scores are summed across IR, CQA, and CSQA task types to produce a final evaluation metric.

Input / output format

Input: Instruction/question formatted via a standard chat template, optionally with context paragraphs.

Output: Model-generated text completion following the instruction.

Scoring recipe

def compute_curll_score(predictions, prompts, rubric):
    total_score = 0
    for pred, prompt in zip(predictions, prompts):
        # External LLM rates generation on 1-5 scale based on correctness & stage weightage
        rating = llm_judge(prompt, pred, rubric)
        total_score += rating
    return total_score

Common pitfalls

• Relies on an external LLM for scoring rather than exact-match or deterministic metrics, introducing potential judge bias or inconsistency.
• The stage-specific weightage in the rating rubric is not detailed in the main text, making exact replication difficult without consulting the appendix.
• Forgetting is measured as performance difference between joint and continual training, which can be confounded by vocabulary and format distribution shifts across developmental stages.

Evidence (verbatim from paper)

To evaluate the trained models, the instances from test set are passed through the chat template and the model is asked to complete the generation post instruction. These inferences along with the prompt is passed to an LLM to rate on a scale of 1-5. This is followed for all three types of test sets. Each model is evaluated on test sets of all stages. The prompt and rubrics of evaluation are mentioned in Appendix[E]. The main objective of the rating is to evaluate the correctness of the model inference with some weightage to the stage on which the model is being evaluated. The summation of scores across test set types (IR, CQA, CSQA) is presented in Figure[4].

Citation

@misc{kalyan2025curll,
  title={CurLL: A Developmental Framework to Evaluate Continual Learning in Language Models},
  author={Kalyan et al. (2025)},
  year={2025},
  note={arXiv:2510.13008}
}

• arXiv: 2510.13008