Chain Of Instructions Eval

chain-of-instructions-eval

Chain-of-Instructions: Compositional Instruction Tuning on Large Language Models — Hayati et al. (2024) (arXiv:2402.11532, 2024)

What this evaluates

Evaluates LLMs on multi-step compositional instruction following, generalization to hard single-step tasks, and multilingual summarization. It probes the model's ability to chain subtask outputs as inputs for subsequent steps and maintain coherence across multiple instructions.

Datasets

• CoI2-test — total ?; splits: test (-1)
• CoI3-test — total ?; splits: test (-1)
• BIG-Bench Hard (BBH) — total ?; splits: test (-1)
• Multilingual Summarization — total 300; splits: test (300)

Metrics

• Rouge-L (primary) — range: [0, 100]
  • Standard ROUGE-L metric measuring the longest common subsequence between the reference and generated text. The paper reports scores as percentages (0-100). Variants include Rouge-L (all), Rouge-L (src), and Rouge-L (tgt) for multilingual tasks.
• #valid outputs — range: count
  • Count of correctly generated summaries in the source and target languages, filtering out cases where the model fails to produce a valid summary.
• Human preference rate — range: percent
  • Percentage of human annotators who prefer the CoI-tuned model's output over the baseline output for a given instance.

Input / output format

Input: Compositional instructions specifying chained subtasks (e.g., 'Simplify the given sentence by paraphrasing it. and then Determine if the paraphrased sentence has proper punctuation with True or False.') followed by the source input text.

Output: Multi-step generation where intermediate outputs are explicitly labeled and chained (e.g., 'Task 1 output and task 2 input: [paraphrased text]. Task 2 output: [True/False]'). For downstream tasks, bilingual summaries in source and target languages.

Scoring recipe

def compute_metrics(predictions, references, human_choices):
    # ROUGE-L calculation (reported as percentage)
    rouge_l = rouge_l_score(references, predictions) * 100
    
    # Valid output counting for multilingual tasks
    valid_src = sum(1 for p in predictions if has_valid_summary(p, lang='src'))
    valid_tgt = sum(1 for p in predictions if has_valid_summary(p, lang='tgt'))
    
    # Human preference rate
    prefer_coI = sum(1 for choice in human_choices if choice == 'CoI') / len(human_choices)
    
    return {'Rouge-L': rouge_l, '#valid_src': valid_src, '#valid_tgt': valid_tgt, 'Prefer CoI': prefer_coI}

Common pitfalls

• Long 3-step instructions (CoI3) drastically increase token length, causing hallucinations and poor generalization if the model lacks shorter-chain training data.
• ROUGE-L alone fails to capture multi-step correctness or language-specific formatting, requiring supplementary valid-output counting and human evaluation.
• Performance is highly sensitive to training data language coverage; models trained on English-to-Punjabi/German/Catalan struggle with Spanish-to-English tasks.

Evidence (verbatim from paper)

We evaluate the performance of the models using four metrics below. • Rouge-L (all) the Rouge score of the summary of the whole generated output. We do this because sometimes the model may not generate a summary in the target language or there is no clear boundary for the summary in the source language and target language. • Rouge-L (src) the Rouge score only from the summary in the source language. • Rouge-L (tgt) the Rouge score only from the summary in the target language. • #valid outputs how many valid summaries in the source (#valid src outputs) and target languages (#valid tgt outputs) are generated because sometimes the model may not generate them properly.

Citation

@misc{hayati2024chainofinstructions,
  title={Chain-of-Instructions: Compositional Instruction Tuning on Large Language Models},
  author={Hayati et al. (2024)},
  year={2024},
  note={arXiv:2402.11532}
}

• arXiv: 2402.11532