Autograph R1 Eval

autograph-r1-eval

AutoGraph-R1: End-to-End Reinforcement Learning for Knowledge Graph Construction — Tsang et al. (2025) (arXiv:2510.15339, 2025)

What this evaluates

This benchmark evaluates the functional utility of reinforcement learning-optimized knowledge graphs in end-to-end retrieval-augmented generation pipelines. It probes whether task-aware RL training improves both graph-based reasoning and text retrieval performance across multiple question-answering benchmarks and model scales.

Datasets

• Natural Questions (NQ) — total 1000; splits: test (1000)
• PopQA — total 1000; splits: test (1000)
• HotpotQA — total 1000; splits: test (1000)
• 2WikiMultihopQA — total 1000; splits: test (1000)
• Musique — total 1000; splits: test (1000)

Metrics

• F1 score (primary) — range: percent
  • The harmonic mean of precision and recall computed over the final generated answers in the RAG pipeline. Calculated as 2 * (precision * recall) / (precision + recall).
• passage recall@5 — range: percent
  • The proportion of queries for which at least one gold supporting passage appears in the top-5 retrieved passages by the graph-based text retriever.

Input / output format

Input: A natural language query and a set of source documents (or a full corpus) from which the model must construct a knowledge graph.

Output: A set of extracted triples (subject, relation, object) forming the knowledge graph, followed by a final natural language answer generated by the downstream RAG pipeline.

Scoring recipe

def compute_f1(predictions, gold_answers):
    precisions = []
    recalls = []
    for pred, gold in zip(predictions, gold_answers):
        pred_tokens = set(normalize(pred))
        gold_tokens = set(normalize(gold))
        if not pred_tokens and not gold_tokens:
            precisions.append(1.0); recalls.append(1.0)
        elif not pred_tokens or not gold_tokens:
            precisions.append(0.0); recalls.append(0.0)
        else:
            tp = len(pred_tokens & gold_tokens)
            precisions.append(tp / len(pred_tokens))
            recalls.append(tp / len(gold_tokens))
    return 2 * (mean(precisions) * mean(recalls)) / (mean(precisions) + mean(recalls))

Common pitfalls

• The evaluation uses full text passages as evidence in the knowledge-index setting, which can mask retrieval improvements because rich context allows the generator to succeed even with imperfect retrieval.
• Llama models require a specific repetition penalty in the reward function during training to prevent repetitive triple generation, which is not applied to Qwen models.
• Hard negative mining is only applied to the text retrieval training scenario, not the graph knowledge retriever scenario.

Evidence (verbatim from paper)

For evaluation, a KG is first constructed over the entire document corpus for a each dataset. Then, depends on the type of retriever, the corresponding RAG is performed using this static graph. We report the final answer F1 score as the primary metric, consistent with prior work.

Citation

@misc{tsang2025autographr1,
  title={AutoGraph-R1: End-to-End Reinforcement Learning for Knowledge Graph Construction},
  author={Tsang et al. (2025)},
  year={2025},
  note={arXiv:2510.15339}
}

• arXiv: 2510.15339