Climate Finance Bench Eval

climate-finance-bench-eval

Climate Finance Bench — Mankour et al. (2025) (arXiv:2505.22752, 2025)

What this evaluates

Evaluates Retrieval-Augmented Generation (RAG) systems on climate-finance question answering using expert-validated Q&A pairs from corporate sustainability reports. It measures answer correctness across different retrieval strategies and LLMs, while also quantifying the environmental footprint (GHG emissions) of each configuration.

Datasets

• Climate Finance Bench — total 330; splits: test (330); repo https://github.com/Pladifes/climate_finance_bench

Metrics

• accuracy (primary) — range: percent
  • Percentage of answers labeled as 'correct' by human annotators or an LLM-as-a-Judge grader. Labels are correct, incomplete, or incorrect, with exactness in numeric values and appropriate textual evidence as key factors.
• ghg_emissions_per_query — range: other
  • Estimated greenhouse gas emissions per query in grams of CO2 equivalent, calculated via CodeCarbon for local models and EcoLogits for API models.

Input / output format

Input: A question derived from corporate sustainability reports, paired with retrieved context passages from a specified RAG configuration (minimal or hybrid).

Output: A natural language answer generated by the LLM.

Scoring recipe

def compute_accuracy(predictions, golds, questions, judge="Claude"):
    correct = 0
    for pred, gold, q in zip(predictions, golds, questions):
        label = judge.evaluate(pred, gold, q) # returns "correct", "incomplete", or "incorrect"
        if label == "correct":
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Retrieval quality is the dominant performance bottleneck, not LLM capacity or prompt strategy, so optimizing the retriever yields larger gains than switching to larger models.
• Docling's HTML conversion without post-processing can degrade retrieval performance due to parsing noise, despite preserving layout.
• Emissions estimates for API models have high uncertainty (±4-5 g CO2eq), so comparisons should account for confidence intervals rather than point estimates alone.

Evidence (verbatim from paper)

Figure 1: Accuracy breakdown (correct, incomplete, incorrect) for the Minimal RAG configuration across five LLMs.

Citation

@misc{mankour2025climatefinancebench,
  title={Climate Finance Bench},
  author={Mankour et al. (2025)},
  year={2025},
  note={arXiv:2505.22752}
}

• arXiv: 2505.22752