# Compare Deep Research > Run one research prompt across multiple deep research providers (via the Deep Research Client) and compare the results side by side — length and structure, citation counts, whether cited sources actually resolve, and whether they are relevant to the question. Use when the user wants to benchmark, evaluate, bake off, or compare deep research tools/providers/agents, or asks which research provider is best for a question. - Skill: `monarch-initiative/compare-deep-research` (Agent Skill, multi-file: 12 files) - Install (CLI): `npx skillmds@latest add monarch-initiative/compare-deep-research` - Raw SKILL.md: https://api.skillmd.com/api/skills/monarch-initiative/compare-deep-research/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: monarch-initiative (https://skillmd.com/u/monarch-initiative) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/monarch-initiative/compare-deep-research --- # Comparing deep research providers Send one prompt to several deep research providers, then measure the answers on axes a reader cannot eyeball: how much was written, how much was cited, whether the citations are real, and whether they bear on the question. Three layers, in order. Never skip to layer 3 — the judgments depend on the identifiers the first two layers produce. | Layer | Produced by | Answers | |---|---|---| | Countable | `metrics.py` | how much content, how many citations, how much overlap | | Resolvable | `check_citations.py` | do the cited sources exist and match their titles | | Judged | you, against the rubric | are they relevant, and does the report answer the question | `$SKILL_DIR` below is this skill's directory. Run scripts with plain `python3` — they are stdlib-only. ## 1. Pin the prompt and the panel Write the prompt to a file (`prompt.md`) rather than passing it inline. Every arm must read byte-identical text; that is the whole basis of the comparison. Check what the user can actually reach: ```bash deep-research-client providers # or: uvx --from deep-research-client deep-research-client providers ``` Providers without their API key are listed as unavailable. Report which arms are possible before running anything — do not silently drop an arm the user asked for. `cyberian` and `claude_code` need no key; `mock` needs `ENABLE_MOCK_PROVIDER=true` and is useful for a smoke test of this pipeline, never for a real comparison. Two or more arms are needed. If only one provider is reachable, say so and stop: a one-arm comparison is a report, not a comparison. ## 2. Run the panel ```bash python3 $SKILL_DIR/scripts/run_panel.py \ --prompt-file prompt.md \ --providers openai,perplexity,consensus \ --out-dir runs/- ``` Per-arm models, provider params, or env vars need a panel file instead of `--providers`; copy `references/panel.example.json` and pass `--panel`. Notes that matter: - **Cost.** Real deep research calls are billed per run and can take many minutes. Confirm the panel with the user before the first paid run, and quote the arm count. Use `--dry-run` to show the exact commands first. - **Caching.** The client caches by query; a repeated prompt returns instantly with `cached: true`. Pass `--no-cache` when timing is part of what is being compared, and check the `cached` column in the metrics output before reporting any duration. - **Variance.** `--repeat N` runs each arm N times (`__rep1`, `__rep2`, …). Deep research providers are stochastic: without replicates you cannot tell a real gap from one provider's ordinary run-to-run swing. `--repeat` forces `--no-cache` — cached replicates would be byte-identical and report a variance of exactly zero, which looks like a finding but is an artifact. Two replicates give you a spread; three make it stable. Cost multiplies by N, so confirm with the user first. - Arms fail independently. Failures land in `run_manifest.json` and the logs, and the run continues; check them, and report a failed arm as failed rather than as a low score. Run directory afterwards: ``` runs// prompt.md raw/.md run_manifest.json logs/.log raw/.citations.md ``` ## 3. Countable metrics ```bash python3 $SKILL_DIR/scripts/metrics.py --run-dir runs/ ``` Writes `metrics.json`: volume, structure, readability, citation profile (unique sources, duplicates, DOI/PMID counts, scholarly fraction, orphan citations never referenced in the prose), plus cross-arm text and citation overlap. `citation_key` matches the same paper across providers by DOI, then PMID, then normalized URL, so overlap is not fooled by different aggregator links. With replicates, it also writes a `replicates` block: per-provider mean, SD, and coefficient of variation for each headline metric, plus *self-overlap* (how much a provider's own runs resemble each other). Overlap between different providers and overlap between replicates of one provider are kept strictly apart — pooling them would let a provider's instability read as agreement with a rival. Consensus citations count distinct providers, so one provider citing a paper on all three runs is one vote, not three. ## 4. Verify citations exist ```bash python3 $SKILL_DIR/scripts/check_citations.py --run-dir runs/ ``` Resolves DOIs via Crossref, PMIDs via NCBI, everything else over HTTP, once per distinct source. Verdicts: | Verdict | Meaning | Counts toward verified_rate | |---|---|---| | `ok` | resolves, metadata consistent | yes | | `mismatch` | resolves, but the real title is unrelated to the cited one | yes | | `dead` | 404/410 — may never have existed | yes | | `blocked` | 403/429/timeout — the host refused to answer | no | | `no_identifier` | cited by reference key, nothing to resolve | no | `blocked` and `no_identifier` are excluded from the rate on purpose. A publisher refusing a bot is not evidence a paper is fake, and a provider citing from an internal corpus is not fabricating — scoring either as `dead` would punish the provider citing more real paywalled literature. Even so, spot-check before ever calling a citation fabricated: `dead` can still mean a redirect or a URL the parser mangled. `--offline` skips the network entirely. ## 4b. Ask the source authority (recommended) ```bash python3 $SKILL_DIR/scripts/resolve_references.py --run-dir runs/ ``` Step 4 only proves a server answered. This asks PubMed (PMIDs) and Crossref/DataCite (DOIs) whether the **record exists at all**, via [linkml-reference-validator](https://github.com/linkml/linkml-reference-validator), and caches the metadata. It splits the failure modes that matter: | Status | Meaning | |---|---| | `verified` | record exists, cited title agrees, text retrievable | | `inaccessible` | record and metadata confirmed, no retrievable text (paywalled) | | `misattributed` | real record, but the cited title disagrees with the authoritative one | | `hallucinated` | the authority answered and has no such record | | `unresolved` | the lookup failed in transit — proves nothing either way | | `not_attempted` | no PMID or DOI to resolve | The `unresolved` / `hallucinated` split is the point: a network error must never be reported as fabrication. Rates are computed only over resolvable citations. Metadata caches to `runs/.references_cache` by default — **shared across runs on purpose**, so re-verifying the same literature costs nothing (18 references: ~2 minutes cold, 0.4s warm). Override with `--cache-dir`. `--include-urls` also pushes plain-URL citations through the validator, which fetches page text but cannot prove absence the way a registry can. Requires `uv`; the validator runs in its own environment, so the harness itself stays stdlib-only. Before repeating any `hallucinated` or `misattributed` claim, open the entry in `references.json` — an identifier mangled upstream (a DOI with a `/full` suffix, a truncated URL) looks exactly like a fabrication. ## 4c. Check claims against the cited text (optional) ```bash python3 $SKILL_DIR/scripts/validate_claims.py --run-dir runs/ ``` Step 4b proves a source exists. This asks whether it says what the report claims, using the validator's deterministic substring matcher on the cached text. It is deliberately narrow, and the narrowness is the point: - **Quoted spans** — words the report puts in quotation marks beside a citation. Absent from the source, that is a **misquotation and a real finding**. - **Figures** — doses, percentages, sample sizes stated beside a citation. Conclusive only against retrievable full text. Against an abstract, absence just means the number lives in the body of the paper, so it is recorded as `not_checkable_in_abstract` and excluded from every rate. - **Paraphrase is out of scope.** Verified empirically: a *faithful* restatement of a cached abstract scores invalid under substring matching. Running whole sentences through this would flag accurate reporting as unsupported at scale. Paraphrase support stays with you and the rubric. Coverage is therefore small and honest — on the xerophthalmia run, 2 of 145 citations had both retrievable text and a checkable span. Report it as a targeted check, never as a support rate for the bibliography. ## 5. Judge relevance and quality Read `references/judging-rubric.md` and follow it. In short: derive the question's facets from the prompt *before* reading any report, score each arm in isolation, and write `judgments/.json` per the schema there, copying `key` values from `citations_by_arm` in `metrics.json` so the merge lines up. Read the reports from `raw/` yourself. Relevance is the metric the user cares most about and it is the one a script cannot compute — do not hand-wave it from domain names or let citation counts stand in for it. With replicates, write one judgment per replicate file you judge (`alpha__rep1`, `alpha__rep2`) — `render_report.py` averages them into a composite with error bars. Judging every replicate is what makes the composite spread meaningful; if budget is tight, judge `__rep1` of every arm first so the panel stays balanced, and say in the summary that the composite has no error bars. Never judge more replicates of one provider than another — that biases the mean. Judging arms in parallel subagents (one per arm, rubric plus that arm's raw file and citation list) keeps each judgment independent and is worth it for panels of three or more — but only when the user has asked for or agreed to subagents. ## 6. Render the comparison ```bash python3 $SKILL_DIR/scripts/render_report.py --run-dir runs/ ``` Writes `comparison.md` (headline table, volume, citation profile, verification, judged relevance and quality, per-arm notes, cross-provider agreement) and `scorecard.json`. Replicates are rolled up to one row per provider, reported as `mean ± SD`, and a **run-to-run stability** section compares the top-two gap against the noise floor — when the gap is smaller than either provider's own spread, the report says so outright. Honour that verdict when you write the summary: do not rank providers on a difference the harness has flagged as noise. The composite is coverage 30 / citation quality 30 / depth 20 / accuracy 20, where citation quality is judged relevance × verified-to-exist; missing components are dropped and the remainder re-normalized, with `weight_covered` recording how much of the rubric was actually scored. Adjust `WEIGHTS` in the script if the user wants a different emphasis, and say in the summary that you did. It runs fine before any judging — you get the countable half of the report. ## Reporting to the user Lead with the ranking and the reason for it, then the caveats. Always state: - how many arms ran, and which failed or were skipped for missing keys; - whether any arm was served from cache (durations are meaningless if so); - how many replicates each provider got, and any gap the stability section flagged as inside the noise floor — a single run per provider means every difference you report is unreplicated, and should be labelled that way; - how many citations were judged versus sampled; - that relevance and quality scores are model judgments, with the rubric's error bars — the counts and resolver verdicts are reproducible, the scores are not. A single prompt ranks providers *for that prompt*. Three or four prompts across different question shapes (factual lookup, synthesis, recent-events, methods) is what supports a general claim; offer that as the next step rather than generalizing from one run.