Sustainableqa Eval

sustainableqa-eval

SustainableQA: A Comprehensive Question Answering Dataset for Corporate Sustainability and EU Taxonomy Reporting — Abdallah et al. (2025) (arXiv:2508.03000, 2025)

What this evaluates

Evaluates language models' ability to extract precise factual answers and generate semantically accurate responses from complex corporate sustainability and EU Taxonomy reports. It also benchmarks retrieval systems' capacity to locate relevant regulatory and financial passages in domain-specific, long-form documents.

Datasets

• SustainableQA — total 195287; splits: train (-1), val (-1), test (-1)

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • 1 if the predicted answer span exactly matches the gold span character-for-character, else 0.
• F1-score — range: [0, 1]
  • Harmonic mean of token-level precision and recall over the predicted and gold answer spans.
• BERTScore — range: [0, 1]
  • Cosine similarity between contextual embeddings of predicted and gold text, averaged across tokens.
• ROUGE-L — range: [0, 1]
  • F1 score based on the longest common subsequence between predicted and gold text.
• METEOR — range: [0, 1]
  • Weighted harmonic mean of unigram precision and recall, incorporating stemming and paraphrase matching.
• BLEU — range: [0, 1]
  • Geometric mean of n-gram precisions with a brevity penalty to discourage overly short predictions.

Input / output format

Input: Question, optionally accompanied by retrieved context passages or few-shot examples depending on the prompting strategy.

Output: Extracted text span (for factoid questions) or generated natural language answer (for non-factoid and tabular questions).

Scoring recipe

def compute_metrics(predictions, golds, question_type):
    if question_type == 'factoid':
        em = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(predictions)
        prec = token_overlap(predictions, golds, mode='precision')
        rec = token_overlap(predictions, golds, mode='recall')
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        return {'EM': em, 'F1': f1, 'Precision': prec, 'Recall': rec}
    else:
        return {
            'BERTScore': bertscore(predictions, golds),
            'ROUGE-L': rouge_l(predictions, golds),
            'METEOR': meteor(predictions, golds),
            'BLEU': bleu(predictions, golds)
        }

Common pitfalls

• Dense retrievers (DPR, Contriever) underperform BM25 because they are trained on short texts (~100 words) and lack exposure to specialized sustainability terminology, causing a structural and domain mismatch when encoding long passages.
• Exact Match (EM) drops sharply with multi-span answers, but F1 remains stable, meaning models often capture relevant information without aligning perfectly with the gold span boundaries.
• Zero-shot prompting without context yields near-zero EM for factoid questions; providing domain-specific context is strictly necessary for viable performance.

Evidence (verbatim from paper)

For factoid questions, we employed span-based evaluation metrics including Exact Match (EM), Precision, Recall, and F1-score. For non-factoid and tabular questions, we utilized BERTScore, BLEU, ROUGE-L, and METEOR to evaluate semantic similarity, lexical overlap, and text quality.

Citation

@misc{abdallah2025sustainableqa,
  title={SustainableQA: A Comprehensive Question Answering Dataset for Corporate Sustainability and EU Taxonomy Reporting},
  author={Abdallah et al. (2025)},
  year={2025},
  note={arXiv:2508.03000}
}

• arXiv: 2508.03000