Crews Ews Budget Eval

crews-ews-budget-eval

AI for Climate Finance: Agentic Retrieval and Multi-Step Reasoning for Early Warning System Investments — Vaghefi et al. (2025) (arXiv:2504.05104, 2025)

What this evaluates

Evaluates an AI system's ability to extract and classify budget allocations for Early Warning System (EWS) investments from heterogeneous financial PDF reports. It probes multi-label classification, numerical budget extraction with tolerance, and evidence retrieval/mapping in climate finance contexts.

Datasets

• MDB Evidence Set — total 500; splits: test (500)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions. A prediction is a true positive only if the pillar label is correct AND the predicted budget amount falls within ±5% of the gold amount relative to the document's total budget ($|\hat{b}{d,p} - b{d,p}| \leq 0.05 B_d^{\text{tot}}$).
• Precision — range: [0, 1]
  • Ratio of true positives to all positive predictions (TP / (TP + FP)), computed under the same ±5% budget tolerance rule.
• Recall — range: [0, 1]
  • Ratio of true positives to all actual positives (TP / (TP + FN)), computed under the same ±5% budget tolerance rule.
• F1 — range: [0, 1]
  • Harmonic mean of Precision and Recall: $2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$.
• Recall@5 — range: [0, 1]
  • Fraction of gold evidence segments found within the top-5 ranked retrieval results.
• Total Amount Percentage Error — range: percent
  • Percentage difference between the sum of predicted pillar amounts ($\hat{B}_d^{\text{tot}}$) and the gold total budget ($B_d^{\text{tot}}$).

Input / output format

Input: Heterogeneous PDF financial reports (CREWS-Fund documents or MDB project documents) containing budget allocations, narrative text, and multi-layout financial tables.

Output: Structured JSON output containing predicted budget allocations per EWS pillar (5 pillars), binary pillar indicators, retrieved evidence segments, and evidence-to-pillar mappings.

Scoring recipe

def compute_metrics(preds, gold):
    tp, fp, fn = 0, 0, 0
    for p in gold['pillars']:
        if gold['labels'][p] == 1:
            if preds['labels'][p] == 1 and abs(preds['budgets'][p] - gold['budgets'][p]) <= 0.05 * gold['total']:
                tp += 1
            else:
                fn += 1
        elif preds['labels'][p] == 1:
            fp += 1
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return {'precision': prec, 'recall': rec, 'f1': f1}

Common pitfalls

• Ignoring the ±5% budget tolerance rule when calculating TPs; the paper explicitly requires numerical fidelity alongside correct labeling.
• Treating the task as single-label classification instead of multi-label over the five fixed EWS pillars.
• Confusing document-level total budget ($B_d^{\text{tot}}$) with pillar-level amounts when applying the tolerance threshold.

Evidence (verbatim from paper)

A prediction for pillar p in document d is counted as a true positive (TP) only if both conditions hold: (a) Correct label. The model assigns the pillar label that is truly present, i.e., y_{d,p} = 1 and \hat{y}{d,p} = 1. (b) Budget fidelity. The predicted allocation is numerically faithful, i.e., |\hat{b}{d,p} - b_{d,p}| \leq 0.05 B_d^{\text{tot}}, a ±5% tolerance around the gold amount for that pillar. Table 1 reports the scores: the agent attains 0.87 accuracy, 0.89 precision, 0.83 recall...

Citation

@misc{vaghefi2025aiforclimatefinance,
  title={AI for Climate Finance: Agentic Retrieval and Multi-Step Reasoning for Early Warning System Investments},
  author={Vaghefi et al. (2025)},
  year={2025},
  note={arXiv:2504.05104}
}

• arXiv: 2504.05104