qhjqhj00 co-sight-eval
Co-Sight: Enhancing LLM-Based Agents via Conflict-Aware Meta-Verification and Trustworthy Reasoning with Structured Facts — Hongwei Zhang et al. (2025) (arXiv:2510.21557, 2025)
What this evaluates
Evaluates long-horizon agentic reasoning, tool-augmented decision making, and factual grounding under conflict-aware verification. It probes how well systems can audit divergent reasoning steps, maintain structured knowledge, and produce accurate answers across multi-hop and interdisciplinary tasks.
Datasets
- GAIA — total 300; splits: test (300)
- HLE — total ?; splits: test (-1)
- Chinese-SimpleQA — total 3000; splits: test (3000)
Metrics
accuracy(primary) — range: percent- Percentage of questions where the model's final answer exactly matches the ground truth.
pass@N— range: percent- Reports success if any candidate among N generated trajectories matches the ground truth.
Input / output format
Input: Benchmark questions/tasks (text or multimodal) requiring tool-augmented reasoning and multi-step retrieval.
Output: Final answer or selection for each question.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
return (correct / len(gold)) * 100
def compute_pass_at_n(predictions_per_question, gold):
correct = 0
for preds, g in zip(predictions_per_question, gold):
if g in preds:
correct += 1
return (correct / len(gold)) * 100
Common pitfalls
- Confusing framework-level improvements with backbone model capabilities (e.g., HLE results must be compared against both the framework and the isolated Gemini 2.5 Pro baseline).
- Assuming full-reasoning-chain re-verification is used; the protocol explicitly audits only divergent/conflict nodes to save compute.
- Treating pass@N as equivalent to structured verification; pass@N simply rewards any correct sample in an ensemble, whereas CAMV uses a fixed audit budget across disagreements.
Evidence (verbatim from paper)
The oracle-style pass@N reports success if any candidate among N matches the ground truth. As shown in Table 1, the score of CAMV rises from 88.3% to 91.2% when N=1→2, surpassing pass@N under this small-ensemble setting. This improvement indicates that conflict-aware auditing effectively recovers and recombines partial micro-inferences that single trajectories alone fail to consolidate.
Citation
@misc{zhang2025cosight,
title={Co-Sight: Enhancing LLM-Based Agents via Conflict-Aware Meta-Verification and Trustworthy Reasoning with Structured Facts},
author={Hongwei Zhang et al. (2025)},
year={2025},
note={arXiv:2510.21557}
}
- arXiv: 2510.21557