Essay Quality Eval

essay-quality-eval

AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays — Herbold et al. (2023) (arXiv:2304.14276, 2023)

What this evaluates

Evaluates the quality and linguistic characteristics of argumentative essays generated by different AI models compared to human-written texts. It probes logical structure, vocabulary richness, syntactic complexity, and stylistic markers through expert human annotation.

Datasets

• Student Essay Dataset (90 topics) — total 270; splits: test (270); repo https://github.com/sherbold/chatgpt-student-essay-study

Metrics

• Mean Rating Score (primary) — range: [1, 5]
  • Average score assigned by expert teachers across six criteria: Topic and completeness, Logic and composition, Expressiveness and composition, Language mastery, Complexity, and Vocabulary and text linking.
• Cohen's d — range: other
  • Effect size measure for pairwise comparisons between essay sources, calculated as the difference in means divided by the pooled standard deviation.
• Wilcoxon p-value — range: [0, 1]
  • P-value from the Wilcoxon signed-rank test adjusted for multiple comparisons using the Holm-Bonferroni method to assess statistical significance of rating differences.

Input / output format

Input: Argumentative essay text on one of 90 predefined topics, labeled by source (Human, ChatGPT-3, or ChatGPT-4).

Output: Numerical rating per criterion (1–5 scale implied by reported means and SDs), aggregated per essay and criterion.

Scoring recipe

ratings = {essay_id: {criterion: [score_1, ..., score_N] for criterion in CRITERIA} for essay_id in ESSAYS}
alpha = cronbach_alpha(ratings)
means = {c: mean(scores) for c, scores in ratings.items()}
for pair in [(Human, GPT3), (Human, GPT4), (GPT3, GPT4)]:
    p_val, d = wilcoxon_signed_rank_test(ratings[pair[0]], ratings[pair[1]])
    p_val_adj = holm_bonferroni_adjust(p_val)
    report(p_val_adj, d)

Common pitfalls

• The rating scale is not explicitly defined in the text but is inferred to be 1–5 based on reported means (e.g., 3.90, 5.03) and standard deviations (~1).
• Statistical tests are paired (Wilcoxon signed-rank) because each topic contains essays from all three sources, requiring within-topic comparisons rather than independent group tests.
• Multiple comparisons are corrected using the Holm-Bonferroni method, which raises the significance threshold compared to unadjusted p-values.

Evidence (verbatim from paper)

The statistical analysis of the ratings reported in Table 4 shows that differences between the human-written essays and the ones generated by both ChatGPT models are significant. The effect sizes for human versus ChatGPT-3 essays are between 0.52 and 1.15, i.e., a medium ( $d \in [0.5, 0.8)$ ) to large ( $d \in [0.8, 1.2)$ ) effect. ... P-values of the Wilcoxon signed-rank tests adjusted for multiple comparisons using the Holm-Bonferroni method. Effect sizes measured with Cohen's $d$ reported for significant results.

Citation

@misc{herbold2023aiwriteessay,
  title={AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays},
  author={Herbold et al. (2023)},
  year={2023},
  note={arXiv:2304.14276}
}

• arXiv: 2304.14276