Biosage Scientific Eval

biosage-scientific-eval

Cross-Disciplinary Knowledge Retrieval and Synthesis: A Compound AI Architecture for Scientific Discovery — Volkova et al. (2025) (arXiv:2511.18298, 2025)

What this evaluates

Evaluates a compound AI architecture's ability to retrieve, synthesize, and reason across cross-disciplinary scientific knowledge. It probes performance on established single-domain science benchmarks and a novel benchmark specifically designed for bio-AI cross-domain synthesis and reasoning.

Datasets

• LitQA2 — total ?; splits: test (-1)
• GPQA — total ?; splits: test (-1)
• WMDP — total ?; splits: test (-1)
• HLE-Bio — total ?; splits: test (-1)
• BioSage Cross-Disciplinary Benchmark — total 116; splits: test (116)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered multiple-choice questions out of the total number of questions in the benchmark.
• precision — range: [0, 1]
  • Proportion of relevant retrieved or synthesized items out of all items retrieved or generated by the system. Exact calculation thresholds are not specified in the text.

Input / output format

Input: Multiple-choice scientific questions requiring cross-disciplinary reasoning. Prompts utilize a zero-shot approach with hybrid chunking and step-by-step reasoning instructions.

Output: Model-generated answer selection and/or detailed reasoning components explaining the choice.

Scoring recipe

def compute_metrics(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    accuracy = correct / len(golds)
    # Precision calculation details are not provided in the text
    return {'accuracy': accuracy, 'precision': None}

Common pitfalls

• The paper does not specify exact prompt templates, temperature settings, or chunking parameters used during evaluation.
• Precision is reported as a metric but lacks a defined formula, relevance threshold, or calculation method.
• The novel benchmark was synthetically generated using GPT-o1, which may introduce distribution shifts or biases compared to human-annotated datasets.

Evidence (verbatim from paper)

Benchmarks include LitQA2, GPQA, WMDP, and HLE-Bio. Each configuration was evaluated on identical test sets to enable direct performance comparisons, with metrics including accuracy and precision.

Citation

@misc{volkova2025biosage,
  title={Cross-Disciplinary Knowledge Retrieval and Synthesis: A Compound AI Architecture for Scientific Discovery},
  author={Volkova et al. (2025)},
  year={2025},
  note={arXiv:2511.18298}
}

• arXiv: 2511.18298