# Selfcheckgpt Eval

> Evaluates a model's ability to detect hallucinated versus factual content in generated text using zero-resource consistency metrics across stochastic samples. It probes whether factual knowledge yields coherent, consistent outputs while hallucinated content exhibits divergence across multiple generations. Use when the user wants to benchmark on SelfCheckGPT dataset, or asks about evaluating this task. Reports AUC-PR.

- Skill: `qhjqhj00/selfcheckgpt-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/selfcheckgpt-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/selfcheckgpt-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/selfcheckgpt-eval

---


# selfcheckgpt-eval

> SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models — Manakul et al. (2023) (arXiv:2303.08896, 2023)

## What this evaluates

Evaluates a model's ability to detect hallucinated versus factual content in generated text using zero-resource consistency metrics across stochastic samples. It probes whether factual knowledge yields coherent, consistent outputs while hallucinated content exhibits divergence across multiple generations.

## Datasets

- **SelfCheckGPT dataset** — total ?; splits: (unstated); repo https://github.com/potsawee/selfcheckgpt

## Metrics

- `AUC-PR` **(primary)** — range: [0, 1]
  - Area Under the Precision-Recall curve for binary classification of factual vs. non-factual sentences. Computed by grouping major- and minor-inaccurate labels into a non-factual class.
- `Pearson correlation` — range: [-1, 1]
  - Pearson correlation coefficient measuring linear correlation between predicted passage-level factuality scores and human judgments.
- `Spearman correlation` — range: [-1, 1]
  - Spearman's rank correlation coefficient measuring monotonic relationship between predicted passage-level factuality scores and human judgments.

## Input / output format

**Input**: A target sentence or passage generated by an LLM, plus N=20 stochastic samples generated from the same prompt (temperature=1.0).

**Output**: Binary factuality label (factual vs. non-factual) per sentence, or a continuous factuality score per sentence/passage.

## Scoring recipe

```python
def compute_auc_pr(predictions, gold_labels):
    precisions, recalls, _ = precision_recall_curve(gold_labels, predictions)
    return auc(recalls, precisions)

def compute_passage_score(sentence_scores):
    return mean(sentence_scores)

def compute_correlation(predictions, gold_scores):
    pearson_r, _ = pearsonr(predictions, gold_scores)
    spearman_r, _ = spearmanr(predictions, gold_scores)
    return pearson_r, spearman_r
```

## Common pitfalls

- Using only top-5 token probabilities to compute entropy underestimates model uncertainty compared to using the full probability distribution.
- Proxy LLMs may perform near random baseline due to distribution mismatch with the target LLM's generation style, even for common tokens.
- Averaging sentence-level scores for passage-level ranking can introduce noise because human judgments on factuality are subjective.

## Evidence (verbatim from paper)

> Table 2 show the performance of our approaches, where the following observations can be made: 1) LLM’s probabilities p correlate well with factuality. Our results show that probability measures (from the LLM generating the texts) are strong baselines for assessing factuality. Factual sentences can be identified with an AUC-PR of 53.97, significantly better than the random baseline of 27.04, with the AUC-PR for hallucination detection also increasing from 72.96 to 83.21.

## Citation

```bibtex
@misc{manakul2023selfcheckgpt,
  title={SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models},
  author={Manakul et al. (2023)},
  year={2023},
  note={arXiv:2303.08896}
}
```

- arXiv: 2303.08896