qhjqhj00 hc3-human-eval
How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection — Guo et al. (2023) (arXiv:2301.07597, 2023)
What this evaluates
Evaluates the ability to distinguish AI-generated responses from human expert answers and assesses perceived helpfulness across multiple domains. It probes linguistic realism, factual reliability, and stylistic alignment with human communication.
Datasets
- HC3 — total 40000; splits: reddit_eli5 (-1), open_qa (-1), wiki_csai (-1), medical (-1), finance (-1), baike (-1), nlpcc_dbqa (-1), medicine (-1), psychology (-1), law (-1); repo https://github.com/Hello-SimpleAI/chatgpt-comparison-detection
Metrics
detection accuracy(primary) — range: [0, 1]- The proportion of instances where human testers correctly identify the ChatGPT-generated answer.
helpfulness proportion— range: [0, 1]- The proportion of instances where human testers prefer the ChatGPT-generated answer over the human answer.
Input / output format
Input: A question paired with either a single answer (human or ChatGPT) or a pair of answers (one human, one ChatGPT).
Output: Binary choice: 'ChatGPT' or 'Human' for detection tasks; 'ChatGPT' or 'Human' for helpfulness preference.
Scoring recipe
def compute_detection_accuracy(predictions, gold):
correct = sum(p == g for p, g in zip(predictions, gold))
return correct / len(predictions)
def compute_helpfulness_proportion(predictions, gold):
return sum(predictions) / len(predictions)
Common pitfalls
- Helpfulness is highly subjective and influenced by tester personality and domain-specific expectations.
- Expert testers familiar with ChatGPT patterns achieve significantly higher detection accuracy than amateurs.
- Pairwise comparison tasks yield higher detection rates than single-text identification due to contrast effects.
Evidence (verbatim from paper)
For all Turing tests, we report the proportion that ChatGPT-generated answer is correctly detected by testers. For the helpfulness test, we report the proportion that ChatGPT-generated answer is considered to be more helpful.
Citation
@misc{guo2023hc3,
title={How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection},
author={Guo et al. (2023)},
year={2023},
note={arXiv:2301.07597}
}
- arXiv: 2301.07597