# Essay Quality Eval

> Evaluates the quality and linguistic characteristics of argumentative essays generated by different AI models compared to human-written texts. It probes logical structure, vocabulary richness, syntactic complexity, and stylistic markers through expert human annotation. Use when the user wants to benchmark on Student Essay Dataset (90 topics), or asks about evaluating this task. Reports Mean Rating Score.

- Skill: `qhjqhj00/essay-quality-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/essay-quality-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/essay-quality-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/essay-quality-eval

---


# essay-quality-eval

> AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays — Herbold et al. (2023) (arXiv:2304.14276, 2023)

## What this evaluates

Evaluates the quality and linguistic characteristics of argumentative essays generated by different AI models compared to human-written texts. It probes logical structure, vocabulary richness, syntactic complexity, and stylistic markers through expert human annotation.

## Datasets

- **Student Essay Dataset (90 topics)** — total 270; splits: test (270); repo https://github.com/sherbold/chatgpt-student-essay-study

## Metrics

- `Mean Rating Score` **(primary)** — range: [1, 5]
  - Average score assigned by expert teachers across six criteria: Topic and completeness, Logic and composition, Expressiveness and composition, Language mastery, Complexity, and Vocabulary and text linking.
- `Cohen's d` — range: other
  - Effect size measure for pairwise comparisons between essay sources, calculated as the difference in means divided by the pooled standard deviation.
- `Wilcoxon p-value` — range: [0, 1]
  - P-value from the Wilcoxon signed-rank test adjusted for multiple comparisons using the Holm-Bonferroni method to assess statistical significance of rating differences.

## Input / output format

**Input**: Argumentative essay text on one of 90 predefined topics, labeled by source (Human, ChatGPT-3, or ChatGPT-4).

**Output**: Numerical rating per criterion (1–5 scale implied by reported means and SDs), aggregated per essay and criterion.

## Scoring recipe

```python
ratings = {essay_id: {criterion: [score_1, ..., score_N] for criterion in CRITERIA} for essay_id in ESSAYS}
alpha = cronbach_alpha(ratings)
means = {c: mean(scores) for c, scores in ratings.items()}
for pair in [(Human, GPT3), (Human, GPT4), (GPT3, GPT4)]:
    p_val, d = wilcoxon_signed_rank_test(ratings[pair[0]], ratings[pair[1]])
    p_val_adj = holm_bonferroni_adjust(p_val)
    report(p_val_adj, d)
```

## Common pitfalls

- The rating scale is not explicitly defined in the text but is inferred to be 1–5 based on reported means (e.g., 3.90, 5.03) and standard deviations (~1).
- Statistical tests are paired (Wilcoxon signed-rank) because each topic contains essays from all three sources, requiring within-topic comparisons rather than independent group tests.
- Multiple comparisons are corrected using the Holm-Bonferroni method, which raises the significance threshold compared to unadjusted p-values.

## Evidence (verbatim from paper)

> The statistical analysis of the ratings reported in Table 4 shows that differences between the human-written essays and the ones generated by both ChatGPT models are significant. The effect sizes for human versus ChatGPT-3 essays are between 0.52 and 1.15, i.e., a medium ( $d \in [0.5, 0.8)$ ) to large ( $d \in [0.8, 1.2)$ ) effect. ... P-values of the Wilcoxon signed-rank tests adjusted for multiple comparisons using the Holm-Bonferroni method. Effect sizes measured with Cohen's  $d$  reported for significant results.

## Citation

```bibtex
@misc{herbold2023aiwriteessay,
  title={AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays},
  author={Herbold et al. (2023)},
  year={2023},
  note={arXiv:2304.14276}
}
```

- arXiv: 2304.14276