Mti Bench Eval

mti-bench-eval

Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once? — Son et al. (2024) (arXiv:2402.11597, 2024)

What this evaluates

Evaluates whether large language models can process multiple distinct instructions simultaneously within a single inference call, compared to sequential or batched approaches. It probes reasoning consistency, format adherence, and inference efficiency across a diverse set of 28 NLP tasks.

Datasets

• MTI Bench — total 5000; splits: test (-1); repo https://github.com/guijinSON/MTI-Bench

Metrics

• exact match (EM) (primary) — range: [0, 1]
  • Exact match between the model's generated output inside the designated HTML tag and the ground truth answer. Scores are averaged across all sub-tasks and instances.

Input / output format

Input: A single prompt containing multiple distinct instructions (sub-tasks) to be solved concurrently, formatted with clear task delimiters.

Output: Model generates responses for each sub-task enclosed in specific HTML tags (e.g., answer), with one tag per sub-task in the order presented.

Scoring recipe

def compute_em(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_clean = extract_html_tag_content(pred)
        if pred_clean == gold:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Strict exact-match scoring penalizes minor formatting variations (e.g., extra spaces, different number formats) even if the semantic answer is correct.
• Models often fail to strictly follow the HTML tag formatting instruction, outputting text outside the tags or mismatching tag names, which breaks automated parsing.
• Confusing Multi-Task Inference (multiple sub-tasks per instance) with Batch Prompting (multiple independent instances per call), leading to incorrect baseline comparisons.

Evidence (verbatim from paper)

Therefore, we prompted LLMs to return their outputs within an HTML tag (e.g., output), which is then assessed via exact match (EM).

Citation

@misc{son2024multi,
  title={Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?},
  author={Son et al. (2024)},
  year={2024},
  note={arXiv:2402.11597}
}

• arXiv: 2402.11597