Orgforge Eval

orgforge-eval

OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora — Flynt (2026) (arXiv:2603.14997, 2026)

What this evaluates

This benchmark evaluates RAG and retrieval-augmented agents on synthetic corporate corpora by testing their ability to retrieve artifacts, reason over causal and temporal chains, and detect knowledge gaps. It probes multi-hop reasoning, temporal knowledge-state tracking, and absence-of-evidence detection across structured enterprise artifacts like Slack, JIRA, and emails.

Datasets

• OrgForge Synthetic Corporate Corpus — total 1162; splits: test (83); repo https://github.com/aeriesec/orgforge

Metrics

• OrgForgeScorer (primary) — range: [0, 1]
  • A weighted composite score: min(1.0, 0.80 * s_primary + 0.20 * s_evidence). s_primary is a type-specific correctness score (0-1) based on exact matches, proximity bonuses, or Jaccard overlap. s_evidence is the Jaccard overlap between the ground-truth evidence chain and the artifact IDs retrieved by the agent.
• MRR@10 — range: [0, 1]
  • Mean Reciprocal Rank of the first ground-truth artifact ID found in the top-10 retrieved documents.
• Recall@10 — range: [0, 1]
  • Fraction of ground-truth evidence chain artifact IDs successfully retrieved within the top-10 results.

Input / output format

Input: A natural-language question (typed as RETRIEVAL, CAUSAL, TEMPORAL, GAP_DETECTION, ROUTING, PLAN, ESCALATION, or KNOWLEDGE_GAP) and a retrieval corpus of synthetic corporate artifacts (Slack messages, JIRA tickets, emails, logs, Confluence pages).

Output: A structured answer specific to the question type (e.g., artifact ID + timestamp, boolean, actor name, or ordered actor list) and an optional list of retrieved artifact IDs used as evidence.

Scoring recipe

def compute_score(q_type, pred_ans, pred_evid, gold_ans, gold_evid):
    if q_type == 'RETRIEVAL':
        s_p = 1.0 if pred_ans.id == gold_ans.id else 0.0
        if abs(pred_ans.ts - gold_ans.ts) <= 30: s_p += 0.1
    elif q_type == 'CAUSAL':
        s_p = 1.0 if (pred_ans.id == gold_ans.id and pred_ans.type == gold_ans.type) else 0.5 if pred_ans.id == gold_ans.id else 0.0
    elif q_type == 'TEMPORAL':
        s_p = 1.0 if pred_ans == gold_ans else 0.0
        if abs(pred_ans.day - gold_ans.day) <= 1: s_p += 0.1
    elif q_type == 'GAP_DETECTION':
        s_p = 0.6 * (1.0 if pred_ans == gold_ans else 0.0) + 0.4 * jaccard(pred_ans.downstream, gold_ans.downstream)
    elif q_type == 'KNOWLEDGE_GAP':
        s_p = jaccard(pred_ans.gap_areas, gold_ans.gap_areas)
    else:
        s_p = 1.0 if pred_ans == gold_ans else 0.0
    s_e = jaccard(pred_evid, gold_evid)
    return min(1.0, 0.80 * s_p + 0.20 * s_e)

Common pitfalls

• The evidence retrieval weight (0.20) is intentionally secondary; a lucky retrieval cannot mask an incorrect primary answer.
• TEMPORAL and GAP_DETECTION questions require boolean/absence reasoning rather than standard document retrieval, so baselines report only evidence recall, not answer correctness.
• PLAN and ESCALATION questions inherently score zero for keyword/semantic baselines due to templated prose and multi-hop actor-chain reasoning requirements.

Evidence (verbatim from paper)

OrgForgeScorer dispatches each question to a type-specific scorer. The final score is a weighted combination of primary answer correctness and evidence retrieval quality: score(q,a)=min(1.0, 0.80·s_primary(q,a)+0.20·s_evidence(q,a)) where s_primary∈[0,1] measures answer correctness and s_evidence is the Jaccard overlap between the ground-truth evidence chain and the artifact IDs the agent reports retrieving.

Citation

@misc{flynt2026orgforge,
  title={OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora},
  author={Flynt (2026)},
  year={2026},
  note={arXiv:2603.14997}
}

• arXiv: 2603.14997