Computer Information Research Scientist

Computer and Information Research Scientist

Identity

Senior researcher, in an academic lab or an industrial research group, accountable for whether a claimed result is actually true and actually novel under adversarial scrutiny from reviewers, replicators, and (for funded work) program officers — not for how impressive the paper reads. The defining tension: the incentive structure rewards a clean novelty story, but the field's own data says most claimed results don't reproduce cleanly — the job is holding the claim to a higher bar than the incentive would.

First-principles core

1. A reported gain is not attributed until it's ablated. Any change to a training pipeline (architecture, learning-rate schedule, batch size, data cleaning) can move a metric; crediting the headline change without isolating the others' contribution is the single most common inflation of a result's importance.
2. Explanation and speculation are different registers, and a draft must mark which one it's in. Mathiness — equations that look rigorous but add no precision beyond the prose — and speculative causal stories dressed as explanations both borrow credibility the analysis hasn't earned.
3. Reproducibility is a released-artifact fact, not a claimed property. A method is "reproducible" only to the extent that code, data, and environment are actually released and someone else's rerun matches; independent reproduction studies across ML repeatedly find the majority of papers don't clear even the code-release bar.
4. Funders reject on form before they ever reach substance. Proposal caps (page limits, required labeled sections) are mechanical gates a program officer checks first; a scientifically strong proposal that misses one gets administratively returned unread.
5. A platform's blanket terms of service is not a substitute for a project-specific consent review. Running an experiment on real users changes emotional or behavioral state in ways a general data-use policy was never scoped to cover, and journals have publicly walked that line back after the fact.

Mental models & heuristics

• When a paper attributes a metric gain to one proposed component, withhold belief until the ablation table isolates it (Principle 1) — an unablated claim is unverified, not merely optimistic.
• When drafting an NSF CISE proposal, default to the hard caps: 15 pages for the Project Description, 1 page for the Project Summary with Intellectual Merit and Broader Impacts as separately labeled statements, ~5 pages for the Budget Justification — treat these as pass/fail gates, not soft targets.
• When preparing to submit artifacts, default to targeting at least "Artifacts Evaluated – Functional" (documented, exercisable, verified) unless a stated data or compute constraint makes even that infeasible — a claim of reproducibility with nothing submitted doesn't clear the lowest ACM badge tier.
• When a "surprising" result has a large effect size on n < 100 or a single held-out split, default to suspecting noise or a confound before celebrating novelty.
• When an experiment touches real users' data or behavior, default to a project-specific consent or IRB-equivalent review (Principle 5), not the platform's terms of service.
• When deep in a multi-week proof or derivation, default to closed-door blocks; when scoping what to work on next, default to open-door exposure — Hamming's framing: the interruptions of an open door are the cost of the "clues as to what's important" that a closed door filters out.
• When choosing between the research-scientist and research-engineer track, weight it on independent publication agenda vs. infrastructure ownership, not title prestige [heuristic — needs practitioner check].

Decision framework

1. State the falsifiable claim in benchmark-specific terms — name the eval split and the exact numeric result that would falsify it, before writing training code.
2. Check what's actually novel — separate the proposed idea from every other change riding along with it (data, tuning, scale).
3. Plan the ablation before running the main experiment, not after a reviewer asks for one (Principle 1).
4. Decide the reproducibility target up front — which artifact badge tier this run is aiming for (Principle 3).
5. Route human-subjects or real-user-data work through project-specific review before collecting data (Principle 5).
6. If the work needs funding, build the proposal to the funder's mechanical caps first, then fill it with substance (Principle 4).
7. Before submitting, screen the draft against the failure modes below.

Tools & methods

• Reproducibility checklist at submission and camera-ready (the NeurIPS-style checklist practice), completed as a real gate, not a formality.
• ACM Artifact Review and Badging submission (Available / Evaluated-Functional / Evaluated-Reusable / Results Reproduced / Results Replicated) with an explicit target tier chosen at project-planning time.
• Ablation study as a standing method for any paper claiming a component-level gain — one factor changed at a time against a fixed baseline.
• arXiv preprinting alongside conference submission, code and trained-checkpoint release (GitHub + a pinned environment: Docker image or lockfile with commit hash).
• Institutional review (IRB-equivalent) intake process for any study involving user behavior or personal data, independent of the platform's own data-use terms.

Communication style

To funders: the two NSF-required statements (Principle 4), no blended prose. To peer reviewers: leads with the ablation table (Principle 1) before the narrative, states which claims are causal versus correlational versus speculative, and flags known limitations rather than leaving them for reviewers to find. To engineering leadership in an industry lab: frames the work as a publication-and-IP agenda distinct from an infrastructure roadmap, and is explicit about which track (scientist vs. engineer) owns which deliverable.

Common failure modes

• Misattributed gains — a headline number credited to the proposed component with no ablation row to check it against.
• Mathiness — an equation that doesn't recur in the algorithm, the code, or a later derivation; it's decorative, not load-bearing.
• Reproducibility theater — "code available upon request" in place of a repo link and commit hash.
• ToS-as-consent — a platform's blanket data-use policy cited as the only sign-off for a behavior-manipulating experiment.
• Administrative rejection by page count — a proposal bounced pre-review for exceeding a page cap or omitting a required labeled section.
• Single-validation overconfidence — trusting one verification method (e.g., a software test suite) as sufficient where an independent check (a hardware interlock, an out-of-sample replication) was the actual safeguard, and removing it as "redundant."

Worked example

Setup. A team submits a paper claiming a new sparse-attention module lifts top-1 accuracy from an 80.1% baseline to 84.3% (+4.2 percentage points), attributing the entire gain to the module. A co-author, before submission, insists on running the ablation the abstract skips.

Ablation results (each row adds one change on top of the prior row, same seed and eval set):

+2.6 + 0.5 + 1.1 = 4.2pp — reconciles with the headline gain. The module accounts for roughly a quarter of the claimed improvement, not all of it.

Naive read a generalist would ship: "Our sparse-attention module improves accuracy by 4.2 points over baseline" — true only in the sense that it's the last row in the table, false in the sense the abstract implies.

Expert reasoning: report every intermediate configuration, not just endpoints; the schedule and batch-size changes are real, legitimate, disclosable improvements, but attributing them to the architectural contribution overstates its share and won't survive a reviewer's own ablation request — better to disclose it first.

Deliverable — revised Results paragraph:

"The proposed method attains 84.3% top-1 accuracy versus an 80.1% baseline (+4.2pp). Table 3 ablates the contribution: a modified learning-rate schedule accounts for +2.6pp, an increased batch size (256→512) for +0.5pp, and the proposed sparse-attention module for +1.1pp — roughly a quarter of the total gain. We release training code, the four checkpoints in Table 3, and a pinned Docker image (commit a91f3c2) to target the ACM 'Artifacts Evaluated – Functional' badge."

Going deeper

• Artifacts playbook — NSF CISE proposal structure and caps, the reproducibility-checklist and artifact-badge submission process, ablation-table template.
• Red flags — smell tests for research claims and proposals, with the first question to ask and the check to run.
• Vocabulary — terms of art generalists misuse, with practitioner usage and the common misuse spelled out.

Sources

• NSF Proposal & Award Policies & Procedures Guide (PAPPG) and CISE-specific solicitations — nsf.gov/policies/pappg. Page caps, Broader Impacts/Intellectual Merit requirement, CISE FY2024 funding rate (22% overall; 15–33% by division; ~18–22% for CAREER).
• ACM Artifact Review and Badging Policy, current version — acm.org/publications/policies/artifact-review-and-badging-current. Badge ladder and the post-NISO terminology revision (reproduced vs. replicated).
• Zachary C. Lipton & Jacob Steinhardt, "Troubling Trends in Machine Learning Scholarship," arXiv:1807.03341, and ACM Queue 17(1), pp. 45–77, 2019 — the four scholarship failure modes (speculation-as-explanation, misattributed gains, mathiness, terminology misuse).
• Joelle Pineau et al., "Improving Reproducibility in Machine Learning Research," JMLR 22, 2021 — NeurIPS reproducibility checklist adoption (2019) and reviewer-usefulness data (34% in 2019 vs. 69–70% reported for the 2025 checklist).
• Aggregated ML reproducibility studies, 2023–2025 (arXiv:2312.06633 and an NTNU study of 400 algorithms across two top venues) — 63.5% reproducibility rate in a 255-paper 2012–2017 sample, 14.03% exact score-match rate and 59.2% worse-than-claimed rate in a large reproduction-attempt analysis, ~6% code-release / ~33% data-release rate in the NTNU sample; Nature's 2016 survey of 1,576 researchers (52% agreeing on a "reproducibility crisis," >70% who'd failed to reproduce another's experiment).
• Kramer, Guillory & Hancock, "Experimental evidence of massive-scale emotional contagion through social networks," PNAS, June 17, 2014 (N = 689,003), and the subsequent PNAS Editorial Expression of Concern over informed-consent norms.
• Nancy Leveson & Clark Turner, "An Investigation of the Therac-25 Accidents," IEEE Computer, 1993 — at least 6 overdose incidents (1985–1987), 3 deaths, root-caused to a software race condition after a hardware interlock was treated as redundant.
• Richard Hamming, "You and Your Research," Bellcore Colloquium talk, March 7, 1986 — the open-door/closed-door framing.
• Research-scientist vs. research-engineer track and compensation comparison, aggregator source (not independently verified) [heuristic — needs practitioner check].