Compare Deep Research

Comparing deep research providers

Send one prompt to several deep research providers, then measure the answers on axes a reader cannot eyeball: how much was written, how much was cited, whether the citations are real, and whether they bear on the question.

Three layers, in order. Never skip to layer 3 — the judgments depend on the identifiers the first two layers produce.

$SKILL_DIR below is this skill's directory. Run scripts with plain python3 — they are stdlib-only.

1. Pin the prompt and the panel

Write the prompt to a file (prompt.md) rather than passing it inline. Every arm must read byte-identical text; that is the whole basis of the comparison.

Check what the user can actually reach:

deep-research-client providers    # or: uvx --from deep-research-client deep-research-client providers

Providers without their API key are listed as unavailable. Report which arms are possible before running anything — do not silently drop an arm the user asked for. cyberian and claude_code need no key; mock needs ENABLE_MOCK_PROVIDER=true and is useful for a smoke test of this pipeline, never for a real comparison.

Two or more arms are needed. If only one provider is reachable, say so and stop: a one-arm comparison is a report, not a comparison.

2. Run the panel

python3 $SKILL_DIR/scripts/run_panel.py \
  --prompt-file prompt.md \
  --providers openai,perplexity,consensus \
  --out-dir runs/<YYYY-MM-DD>-<slug>

Per-arm models, provider params, or env vars need a panel file instead of --providers; copy references/panel.example.json and pass --panel.

Notes that matter:

• Cost. Real deep research calls are billed per run and can take many minutes. Confirm the panel with the user before the first paid run, and quote the arm count. Use --dry-run to show the exact commands first.
• Caching. The client caches by query; a repeated prompt returns instantly with cached: true. Pass --no-cache when timing is part of what is being compared, and check the cached column in the metrics output before reporting any duration.
• Variance. --repeat N runs each arm N times (<id>__rep1, <id>__rep2, …). Deep research providers are stochastic: without replicates you cannot tell a real gap from one provider's ordinary run-to-run swing. --repeat forces --no-cache — cached replicates would be byte-identical and report a variance of exactly zero, which looks like a finding but is an artifact. Two replicates give you a spread; three make it stable. Cost multiplies by N, so confirm with the user first.
• Arms fail independently. Failures land in run_manifest.json and the logs, and the run continues; check them, and report a failed arm as failed rather than as a low score.

Run directory afterwards:

runs/<run-id>/
  prompt.md              raw/<arm_id>.md            run_manifest.json
  logs/<arm_id>.log      raw/<arm_id>.citations.md

3. Countable metrics

python3 $SKILL_DIR/scripts/metrics.py --run-dir runs/<run-id>

Writes metrics.json: volume, structure, readability, citation profile (unique sources, duplicates, DOI/PMID counts, scholarly fraction, orphan citations never referenced in the prose), plus cross-arm text and citation overlap. citation_key matches the same paper across providers by DOI, then PMID, then normalized URL, so overlap is not fooled by different aggregator links.

With replicates, it also writes a replicates block: per-provider mean, SD, and coefficient of variation for each headline metric, plus self-overlap (how much a provider's own runs resemble each other). Overlap between different providers and overlap between replicates of one provider are kept strictly apart — pooling them would let a provider's instability read as agreement with a rival. Consensus citations count distinct providers, so one provider citing a paper on all three runs is one vote, not three.

4. Verify citations exist

python3 $SKILL_DIR/scripts/check_citations.py --run-dir runs/<run-id>

Resolves DOIs via Crossref, PMIDs via NCBI, everything else over HTTP, once per distinct source. Verdicts:

blocked and no_identifier are excluded from the rate on purpose. A publisher refusing a bot is not evidence a paper is fake, and a provider citing from an internal corpus is not fabricating — scoring either as dead would punish the provider citing more real paywalled literature. Even so, spot-check before ever calling a citation fabricated: dead can still mean a redirect or a URL the parser mangled.

--offline skips the network entirely.

4b. Ask the source authority (recommended)

python3 $SKILL_DIR/scripts/resolve_references.py --run-dir runs/<run-id>

Step 4 only proves a server answered. This asks PubMed (PMIDs) and Crossref/DataCite (DOIs) whether the record exists at all, via linkml-reference-validator, and caches the metadata. It splits the failure modes that matter:

The unresolved / hallucinated split is the point: a network error must never be reported as fabrication. Rates are computed only over resolvable citations.

Metadata caches to runs/.references_cache by default — shared across runs on purpose, so re-verifying the same literature costs nothing (18 references: ~2 minutes cold, 0.4s warm). Override with --cache-dir. --include-urls also pushes plain-URL citations through the validator, which fetches page text but cannot prove absence the way a registry can.

Requires uv; the validator runs in its own environment, so the harness itself stays stdlib-only. Before repeating any hallucinated or misattributed claim, open the entry in references.json — an identifier mangled upstream (a DOI with a /full suffix, a truncated URL) looks exactly like a fabrication.

4c. Check claims against the cited text (optional)

python3 $SKILL_DIR/scripts/validate_claims.py --run-dir runs/<run-id>

Step 4b proves a source exists. This asks whether it says what the report claims, using the validator's deterministic substring matcher on the cached text. It is deliberately narrow, and the narrowness is the point:

• Quoted spans — words the report puts in quotation marks beside a citation. Absent from the source, that is a misquotation and a real finding.
• Figures — doses, percentages, sample sizes stated beside a citation. Conclusive only against retrievable full text. Against an abstract, absence just means the number lives in the body of the paper, so it is recorded as not_checkable_in_abstract and excluded from every rate.
• Paraphrase is out of scope. Verified empirically: a faithful restatement of a cached abstract scores invalid under substring matching. Running whole sentences through this would flag accurate reporting as unsupported at scale. Paraphrase support stays with you and the rubric.

Coverage is therefore small and honest — on the xerophthalmia run, 2 of 145 citations had both retrievable text and a checkable span. Report it as a targeted check, never as a support rate for the bibliography.

5. Judge relevance and quality

Read references/judging-rubric.md and follow it. In short: derive the question's facets from the prompt before reading any report, score each arm in isolation, and write judgments/<arm_id>.json per the schema there, copying key values from citations_by_arm in metrics.json so the merge lines up.

Read the reports from raw/ yourself. Relevance is the metric the user cares most about and it is the one a script cannot compute — do not hand-wave it from domain names or let citation counts stand in for it.

With replicates, write one judgment per replicate file you judge (alpha__rep1, alpha__rep2) — render_report.py averages them into a composite with error bars. Judging every replicate is what makes the composite spread meaningful; if budget is tight, judge __rep1 of every arm first so the panel stays balanced, and say in the summary that the composite has no error bars. Never judge more replicates of one provider than another — that biases the mean.

Judging arms in parallel subagents (one per arm, rubric plus that arm's raw file and citation list) keeps each judgment independent and is worth it for panels of three or more — but only when the user has asked for or agreed to subagents.

6. Render the comparison

python3 $SKILL_DIR/scripts/render_report.py --run-dir runs/<run-id>

Writes comparison.md (headline table, volume, citation profile, verification, judged relevance and quality, per-arm notes, cross-provider agreement) and scorecard.json. Replicates are rolled up to one row per provider, reported as mean ± SD, and a run-to-run stability section compares the top-two gap against the noise floor — when the gap is smaller than either provider's own spread, the report says so outright. Honour that verdict when you write the summary: do not rank providers on a difference the harness has flagged as noise. The composite is coverage 30 / citation quality 30 / depth 20 / accuracy 20, where citation quality is judged relevance × verified-to-exist; missing components are dropped and the remainder re-normalized, with weight_covered recording how much of the rubric was actually scored. Adjust WEIGHTS in the script if the user wants a different emphasis, and say in the summary that you did.

It runs fine before any judging — you get the countable half of the report.

Reporting to the user

Lead with the ranking and the reason for it, then the caveats. Always state:

• how many arms ran, and which failed or were skipped for missing keys;
• whether any arm was served from cache (durations are meaningless if so);
• how many replicates each provider got, and any gap the stability section flagged as inside the noise floor — a single run per provider means every difference you report is unreplicated, and should be labelled that way;
• how many citations were judged versus sampled;
• that relevance and quality scores are model judgments, with the rubric's error bars — the counts and resolver verdicts are reproducible, the scores are not.

A single prompt ranks providers for that prompt. Three or four prompts across different question shapes (factual lookup, synthesis, recent-events, methods) is what supports a general claim; offer that as the next step rather than generalizing from one run.