Finreflectkg Eval

finreflectkg-eval

FinReflectKG: Agentic Construction and Evaluation of Financial Knowledge Graphs — Arun et al. (2025) (arXiv:2508.17906, 2025)

What this evaluates

Evaluates the quality, schema compliance, diversity, and factual grounding of automatically extracted financial knowledge graph triples from SEC 10-K filings. It measures rule-based compliance, entity and relation coverage, semantic diversity via entropy, and comparative quality using an LLM-as-a-Judge framework.

Datasets

• S&P 100 SEC 10-K Filings (2024) — total 100; splits: test (-1)

Metrics

• CheckRules (primary) — range: [0, 1]
  • CR(t) = (1/R) * sum(phi_i(t)) for i=1 to R, where phi_i(t) is 1 if triple t complies with rule i, else 0. R=4 rules: Subject Reference, Entity Length Constraint, Entity Schema Compliance, Relationship Schema Compliance.
• Entity Coverage Ratio (ECR) — range: [0, 1]
  • Proportion of unique entities relative to total extracted elements, measuring extraction completeness.
• Shannon Entropy — range: other
  • H(X) = -sum(p_i * log2(p_i)) over normalized frequency distributions of entities, types, or relationships to quantify semantic diversity.
• LLM-as-a-Judge Score — range: percent
  • Comparative scores for Precision, Faithfulness, Comprehensiveness, and Relevance. Evaluated via Qwen3-32B (temp=0.1) with 3 independent votes and a 4th tie-breaker vote when consensus is not reached.

Input / output format

Input: Extracted triples (subject, relation, object) per document chunk, source text chunk, and predefined entity/relation schema.

Output: Per-triple compliance flags, aggregate coverage ratios, entropy values, and LLM-as-a-Judge comparative scores (percentages).

Scoring recipe

def compute_checkrules(triples, rules):
    scores = []
    for t in triples:
        compliant = [1 if check_rule(t, r) else 0 for r in rules]
        scores.append(sum(compliant) / len(rules))
    return scores

def compute_llm_judge(triples, source_text, judge_model='Qwen3-32B', temp=0.1):
    votes = []
    for _ in range(3):
        votes.append(judge_model.generate(prompt=f'Rate precision, faithfulness, etc. for {triples} given {source_text}', temperature=temp))
    if len(set(votes)) > 1:
        votes.append(judge_model.generate(prompt='Tie-breaker for...', temperature=temp))
    return majority_vote(votes)

Common pitfalls

• Abstract references like 'the company' or 'we' are explicitly flagged as non-compliant in CheckRules, requiring canonicalization to specific names or tickers.
• Lower Shannon/Rényi entropy in the reflection method is intentional to reduce redundancy and improve graph navigability, not a metric failure.
• LLM-as-a-Judge evaluation deliberately avoids Chain-of-Thought prompting to prevent overthinking, and requires a 4th tie-breaker vote when initial 3 votes disagree.

Evidence (verbatim from paper)

To address these systematic issues, CheckRules evaluates each extracted triple against a set of rules: Subject Reference, Entity Length Constraint, Entity Schema Compliance, Relationship Schema Compliance. Each extracted triple is individually evaluated against these rules. For a triple t with R rules, the CheckRules score is: CR(t) = (1/R) sum(phi_i(t)) where phi_i(t) in {0,1} indicates compliance.

Citation

@misc{arun2025finreflectkg,
  title={FinReflectKG: Agentic Construction and Evaluation of Financial Knowledge Graphs},
  author={Arun et al. (2025)},
  year={2025},
  note={arXiv:2508.17906}
}

• arXiv: 2508.17906