Chatgpt Multitask Eval

chatgpt-multitask-eval

A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity — Bang et al. (2023) (arXiv:2302.04023, 2023)

What this evaluates

Evaluates ChatGPT's zero-shot multitask capabilities across summarization, machine translation, sentiment analysis, question answering, dialogue, and misinformation detection. It probes the model's generalization, reasoning, multilingual understanding, and task-specific performance without fine-tuning.

Datasets

• CNN/DM — total ?; splits: test (-1)
• SAMSum — total ?; splits: test (-1)
• FLoRes-200 — total ?; splits: test (-1)
• NusaX — total ?; splits: test (-1)
• bAbI — total ?; splits: test (-1)
• EntailmentBank — total ?; splits: test (-1)
• CLUTRR — total ?; splits: test (-1)
• StepGame — total ?; splits: test (-1)
• Pep-3k — total ?; splits: test (-1)
• COVID-Social — total ?; splits: test (-1)
• COVID-Scientific — total ?; splits: test (-1)
• MultiWOZ2.2 — total ?; splits: test (-1)
• OpenDialKG — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels or answers out of total samples.
• ROUGE-1 — range: [0, 1]
  • Unigram overlap between generated and reference text, typically measured as F1 score.
• ChrF++ — range: [0, 1]
  • Character n-gram F-score used for machine translation quality assessment.
• Macro F1 — range: [0, 1]
  • Unweighted mean of F1 scores across all classes.
• JGA — range: [0, 1]
  • Joint Goal Accuracy: percentage of dialogue turns where all slot values and intent are predicted correctly.
• BLEU — range: [0, 1]
  • Geometric mean of modified n-gram precisions with brevity penalty.
• Inform Rate — range: [0, 1]
  • Percentage of turns where the system successfully provides requested information.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve for binary classification.
• ROUGE-L — range: [0, 1]
  • Longest Common Subsequence overlap between generated and reference text.
• FeQA — range: [0, 1]
  • Factuality QA score measuring factual consistency in generated responses.

Input / output format

Input: Zero-shot text prompts, dialogue turns, or parallel sentences provided as natural language instructions or context.

Output: Model-generated text responses, translations, or classifications.

Scoring recipe

def evaluate(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    rouge1 = rouge_score(golds, predictions, rouge_types=['rouge1'])
    bleu = corpus_bleu(golds, predictions)
    inform = sum(1 for p in predictions if p['informative']) / len(predictions)
    return {'accuracy': acc, 'rouge1': rouge1, 'bleu': bleu, 'inform_rate': inform}

Common pitfalls

• Evaluates on small subsets (30-200 samples) rather than full test sets, risking high variance and unrepresentative results.
• Zero-shot evaluation without task-specific fine-tuning makes direct comparison to fully-fine-tuned SOTA models inherently unfair.
• Automatic metrics like BLEU and ROUGE penalize ChatGPT's longer, more fluent responses compared to concise golden answers.

Evidence (verbatim from paper)

DST is mediocre while ChatGPT successfully leverages all information provided while answering the questions with a 71.1% inform rate and 5.65 BLEU score.

Citation

@misc{bang2023chatgptmultitask,
  title={A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity},
  author={Bang et al. (2023)},
  year={2023},
  note={arXiv:2302.04023}
}

• arXiv: 2302.04023