How do I install the Climatecheck Eval skill?

Run npx skillmds add qhjqhj00/climatecheck-eval in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Climatecheck Eval safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: makes network calls. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Climatecheck Eval?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Climatecheck Eval free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Climatecheck Eval?

qhjqhj00 (@qhjqhj00) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Climatecheck Eval by qhjqhj00 — Agent Skill

Name: Climatecheck Eval
Rating: 4.2 (5 reviews)
Author: qhjqhj00

Climatecheck Eval

Tests the ability to retrieve relevant scholarly abstracts for climate change claims from social media and classify the relationship between claims and abstracts as supporting, refuting, or inconclusive. Use when the user wants to benchmark on ClimateCheck, or asks about evaluating this task. Reports Recall@10.

qhjqhj00 558c181 2.5 KB Updated Sep 8, 2026 3 repo stars

File contents

climatecheck-eval

NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

What this evaluates

Datasets

ClimateCheck — total 435; splits: train (-1), test (-1); HF rabuahmad/climatecheck; repo https://sdproc.org/2025/climatecheck.html

Metrics

Recall@10 (primary) — range: [0, 1]
- Proportion of claims where the correct abstract is ranked within the top 10 retrieved results.

Input / output format

Input: Social media claim (atomic tweet-styled statement).

Output: Top-10 ranked scholarly abstracts (Subtask I) and relation classification (supports/refutes/not enough information) for retrieved pairs (Subtask II).

Scoring recipe

retrieved_abstracts = model.retrieve_top_k(claim, k=10)
recall_at_10 = 1 if gold_abstract in retrieved_abstracts else 0
pred_relations = model.classify_relations(claim, retrieved_abstracts)
f1_score = f1_score(gold_relations, pred_relations)

Common pitfalls

Subtask I evaluates retrieval (Recall@10, Bpref) while Subtask II evaluates classification (F1) on the retrieved set, which can be conflated in leaderboard reporting.
The dataset is small (435 claims), making aggregate metrics highly sensitive to individual claim difficulty and annotation noise.

Evidence (verbatim from paper)

Subtask I of ClimateCheck focused on retrieving the top 10 relevant abstracts per claim, evaluated using Recall@K as well as Binary Preference (Bpref), while subtask II asked participants to classify the relation of each claim-abstract pair they retrieved, evaluated using F1-score with additional scaling based on successful retrieval. The top team achieved a Recall@10 score of 0.66, Bpref of 0.49 (subtask I), and and F1 of 0.73 (subtask II).

Citation

@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}

arXiv: 2509.22141

---name: climatecheck-evaldescription: Tests the ability to retrieve relevant scholarly abstracts for climate change claims from social media and classify the relationship between claims and abstracts as supporting, refuting, or inconclusive. Use when the user wants to benchmark on ClimateCheck, or asks about evaluating this task. Reports Recall@10.---# climatecheck-eval> NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)## What this evaluatesTests the ability to retrieve relevant scholarly abstracts for climate change claims from social media and classify the relationship between claims and abstracts as supporting, refuting, or inconclusive.## Datasets- **ClimateCheck** — total 435; splits: train (-1), test (-1); HF `rabuahmad/climatecheck`; repo https://sdproc.org/2025/climatecheck.html## Metrics- `Recall@10` **(primary)** — range: [0, 1] - Proportion of claims where the correct abstract is ranked within the top 10 retrieved results.## Input / output format**Input**: Social media claim (atomic tweet-styled statement).**Output**: Top-10 ranked scholarly abstracts (Subtask I) and relation classification (supports/refutes/not enough information) for retrieved pairs (Subtask II).## Scoring recipe```pythonretrieved_abstracts = model.retrieve_top_k(claim, k=10)recall_at_10 = 1 if gold_abstract in retrieved_abstracts else 0pred_relations = model.classify_relations(claim, retrieved_abstracts)f1_score = f1_score(gold_relations, pred_relations)```## Common pitfalls- Subtask I evaluates retrieval (Recall@10, Bpref) while Subtask II evaluates classification (F1) on the retrieved set, which can be conflated in leaderboard reporting.- The dataset is small (435 claims), making aggregate metrics highly sensitive to individual claim difficulty and annotation noise.## Evidence (verbatim from paper)> Subtask I of ClimateCheck focused on retrieving the top 10 relevant abstracts per claim, evaluated using Recall@K as well as Binary Preference (Bpref), while subtask II asked participants to classify the relation of each claim-abstract pair they retrieved, evaluated using F1-score with additional scaling based on successful retrieval. The top team achieved a Recall@10 score of 0.66, Bpref of 0.49 (subtask I), and and F1 of 0.73 (subtask II).## Citation```bibtex@misc{abuhmad2025nfdi4ds, title={NFDI4DS Shared Tasks for Scholarly Document Processing}, author={Abu Ahmad et al. (2025)}, year={2025}, note={arXiv:2509.22141}}```- arXiv: 2509.22141

Frequently asked questions