Answer Scientific Questions

Answer Scientific Questions

Most scientific questions people ask have been studied, often many times, and the answer is usually "it depends, and here is on what." The job is to find the studies that actually bear on the question, weigh them by how well they were run, and say what the evidence supports and how strongly, with the studies cited so anyone can check.

Six steps. Do them in order. Each produces a file in a working folder named for the question, so a second pass or a second reader can pick it up.

Pick a lane first

1. Search extensively and collect

Two searches, merged. Neither alone is enough: Scholar has the widest coverage and a "cited by" graph; the APIs are reproducible and carry open-access links.

Google Scholar, in the browser. No API exists and scripts get blocked, so use the browser tool. Run at least four queries: the plain question, the question plus "meta-analysis", plus "systematic review", plus "randomized". Set the date range when recency matters. For each strong hit, open "Cited by" and "Related articles" once; the best meta-analysis is often one click from a mediocre trial. Record title, year, venue, DOI, and citation count.

Scripted. scripts/lit_search.py "<query>" --meta --since <year> --out papers.csv for reviews, then --rct for direct experiments. It queries OpenAlex, Europe PMC (which covers PubMed), and arXiv, dedupes by DOI, and marks which papers have a lawful open-access copy. Set OPENALEX_MAILTO to your email; both APIs give a faster lane for it.

Merge into one papers.csv. Aim for coverage, not a shortlist: 30 to 80 candidates for a real question. Stop when new queries return papers already on the list.

Priority order for what to read, and it matters: (1) meta-analyses and systematic reviews that pool the direct experiments, newest first; (2) the direct experiments themselves, especially RCTs, on the exact question (same intervention, population, outcome); (3) large cohorts when trials do not exist; (4) mechanism, animal, and in-vitro work only as supporting context, never as the answer.

2. Retrieve papers locally

For each paper on the read list, in order. Open access first because it is faster and carries the version of record; the mirror when it is the only way to the full text.

1. The oa_url from papers.csv (publisher open access, PubMed Central, Europe PMC, or a repository copy).
2. scripts/lit_search.py --doi <doi> to ask Unpaywall and OpenAlex for a copy the CSV missed. Unpaywall finds author-deposited manuscripts in institutional repositories, which cover a large share of paywalled papers. The same call prints the Sci-Hub lookup URL for step 3.
3. Sci-Hub at https://sci-hubtw.hkvisa.net/: paste the DOI (or the publisher URL) into the search box, or open https://sci-hubtw.hkvisa.net/<doi> directly, then save the PDF. Use the browser tool; the site serves a viewer page, and a captcha appears on some mirrors, which the user solves. If that mirror is down, the site lists current ones. Sci-Hub's legal status varies by country; the user of this skill has made that choice, and the agent does not relitigate it.
4. Preprint servers: bioRxiv, medRxiv, PsyArXiv, SSRN, for the accepted-manuscript version when the mirror lacks a recent paper.
5. Institutional or library access the user has, an interlibrary loan request, or an email to the corresponding author for the few that remain.

Save as papers/<firstauthor>-<year>.pdf. Extract text with pdftotext -layout (or pypdf) into text/. Paywalled and unretrievable: keep the abstract in the evidence table, mark it "abstract only," and weight it accordingly.

Token discipline: abstracts for all, full text only for the papers that will decide the answer, usually the top two or three meta-analyses and the handful of direct trials. A full paper is 10k to 25k tokens. State the reading budget before reading and report the spend after; arxiv-researcher's token_budget.py works here unchanged.

3. Process and understand context

For every paper read in full, write one block in evidence.md:

• PICO: population, intervention or exposure, comparator, outcome, duration.
• Design and n per arm.
• Primary result: effect size in the paper's units, 95% CI, p, and how many outcomes were tested.
• Funding and conflicts.
• What it does to the question, one line.

Read the results table before the abstract's conclusion. The gap between them is where spin lives.

4. Evaluate the methods for holes

Run references/methods-checklist.md against each study that will carry weight: preregistration and outcome switching, randomization and blinding, power, attrition, surrogate outcomes, confounding, multiplicity, and for meta-analyses the search, heterogeneity, publication bias, and risk-of-bias weighting. Record findings in the evidence block. A study with holes is not discarded; it is downweighted, and the reason is written down.

5. Evaluate the statistics behind the outcome

references/significance.md: effect size before p, the interval's range of compatible effects, multiplicity, power to detect the effect claimed, number needed to treat, fragility for small-event trials, and the forest plot rather than the pooled diamond. Where the paper reports group means, SDs, and n, recompute the test with get-stats (compute_stats.py) and note any disagreement. When the raw data or supplementary tables are available, the recomputation is the evidence; the paper's sentence is not.

6. Conclude from the whole

Write answer.md:

1. The answer, in one or two sentences, with a confidence level: high (consistent RCT meta-analyses), moderate (RCTs with limitations or consistent large cohorts), low (small or conflicting trials, observational only), very low (mechanism and anecdote).
2. The evidence table: every study that carried weight, design, n, effect with CI, quality notes, and direction.
3. What it depends on: dose, population, duration, outcome measure, the conditions under which the answer flips.
4. Holes in the literature: what has not been tested, what was tested badly, what a decisive study would look like.
5. What would change the answer: the study that, if it appeared, would move the confidence up or down.
6. Sources: every DOI, plus the search log (queries, dates, databases) so the search can be rerun.

Then run the whole thing through adversarial-review-loop once: a fresh reviewer who tries to break the conclusion from the evidence table alone.

Rules

• Meta-analyses of direct experiments outrank everything. Mechanism outranks nothing.
• A paper the agent did not read past the abstract is labeled "abstract only" wherever it is cited.
• Recompute when the numbers are there. Trust the table over the prose, and the recomputation over the table.
• Confidence is stated in the grading language above, every time, and the reasons for it are in the evidence table.
• No dosing, treatment, or safety advice beyond what the cited evidence says; the answer reports the literature, and a clinician makes the call.