Chatgpt Nlp Eval

chatgpt-nlp-eval

Pushing the Limits of ChatGPT on NLP Tasks — Sun et al. (2023) (arXiv:2306.09719, 2023)

What this evaluates

Evaluates ChatGPT's few-shot generation and reasoning capabilities across multiple NLP tasks, including question answering, commonsense reasoning, natural language inference, and sentiment analysis. It probes the model's ability to follow task-specific formalizations, leverage retrieved demonstrations, and mitigate hallucination through self-verification.

Datasets

• SQuADv2 — total ?; splits: test (-1)
• TQA — total ?; splits: test (-1)
• MRQA-OOD — total ?; splits: test (-1)
• CSQA — total ?; splits: test (-1)
• StrategyQA — total ?; splits: test (-1)
• RTE — total ?; splits: test (-1)
• CommitmentBank — total ?; splits: test (-1)
• SST-2 — total ?; splits: test (-1)
• IMDB — total ?; splits: test (-1)
• Yelp — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of instances where the model's generated output exactly matches the ground truth label or answer substring.

Input / output format

Input: Task-specific prompts: for QA, context split into indexed sentences concatenated with a question; for NLI, premise and hypothesis; for commonsense, question with multiple-choice options; for sentiment, input text. Demonstrations are retrieved and appended.

Output: Model generates text following task formalization: e.g., '# (index) answer' for QA; 'yes' or 'no' for each NLI relation; selected option letter/text for others; sentiment keyword for sentiment analysis.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return correct / len(golds)

Common pitfalls

• ChatGPT's output must strictly adhere to the prompt's expected format (e.g., index + substring for QA); parsing failures occur if the model deviates.
• Performance heavily depends on the demonstration retrieval method; using a fine-tuned retriever vs. random/SimCSE causes significant accuracy drops.
• Hallucination in reasoning tasks can produce plausible but incorrect answers, requiring the self-verification step to correct before final scoring.

Evidence (verbatim from paper)

We use accuracy in three-class (e.g., entail, contrast, neural) as the evaluation metric.

Citation

@misc{sun2023pushing,
  title={Pushing the Limits of ChatGPT on NLP Tasks},
  author={Sun et al. (2023)},
  year={2023},
  note={arXiv:2306.09719}
}

• arXiv: 2306.09719