Tufte Data Forensics

Data Forensics: Auditing an Analysis Before You Trust It

Overview

Seeing With Fresh Eyes ch. 4 (printed pp. 80–119) is a procedure, not a lament. Its organizing device is a proposed reference work — an imagined Data Measurement and Analysis: 1,000 Safeguards, Pitfalls, and Cheats in Statistical Practice, modeled on Khonsari and Sintek's cardiac-surgery encyclopedia of some 3,000 graded alerts and warnings (p. 82). The parallel is the whole argument: surgery already has an itemized, severity-graded, actionable literature of things that go wrong at the point of work, and statistical practice does not — even though a botched analysis reaches more lives than a botched operation.

This skill owns the step-by-step audit. Two siblings own adjacent material and are not repeated here:

What follows turns those named concepts into an executable audit: what to look at, in what order, what to demand, and what to write down.

The operating stance. Feynman's rule opens the chapter's facing page: the first principle is that you must not fool yourself, and you are the easiest person to fool (Feynman, quoted p. 80). Everything below is machinery for a person who has accepted that they are the primary threat. Tufte reinforces it with Darwin — that ignorance begets confidence more often than knowledge does (Darwin, quoted p. 86) — and with Van Wyck Brooks's principle that once you hold a point of view, all history will back you up (p. 81). The chapter's recurring villains are named as characters: Dr. Confirmation Bias and Dr. p-Hacker, who build narratives out of a fresh batch of random numbers (p. 86).

The selection-bias frame (p. 80). Tolstoy's remark that Napoleon "commanding" his armies compresses thousands of consecutive dependent commands, of which only the executed few are ever remembered, is the chapter's compact statement of causal selection bias: the record you audit has already been filtered by outcome before it reached you. Assume the survivors are in your dataset and the non-survivors are not.

§1. Why an Audit, Not a Vibe Check — Prevalence Replaces Anecdote

Tufte's methodological reform is to stop arguing from horror stories and start arguing from measured prevalence rates for specific, named error classes. An anecdote tells you a failure is possible; a prevalence rate tells you what to check first and how surprised to be when you find it.

Cross-reference: the same figures appear as an integrity table in tufte-visual-thinking §7 and as corruption evidence in tufte-evidence-corruption. The contribution here is the third column — each rate converted into a specific screening action.

Two prevalence claims to state carefully:

• Tufte reports the prevalence of regression toward the truth — early enthusiastic claims that later shrink or reverse — as perhaps 80% (p. 83). He hedges it; hedge it too.
• The 522 consulting biostatisticians surveyed (75% responded) reported inappropriate requests including: removing or altering records to better support the hypothesis; interpreting findings by expectation rather than by actual results; not reporting key missing data that would bias the result; and ignoring assumption violations that would reverse the conclusion (p. 92). This is a survey of requests made to analysts, which is why it belongs in an audit skill: it is a list of the pressures the analyst you are auditing was under.

Accuracy warning — the mock package insert (p. 98). Tufte prints a parody "BLACK BOX WARNING" for stepwise logistic multiple regression, listing adverse effects with percentages (cherry-picked models 80.1%, p-hacking >70%, multicollinearity 80.5%, and so on). It is a mock-up illustrating a proposal, not measured data. Never cite those percentages as prevalence rates. The transferable idea is the format: every statistical model should ship with a package insert naming its assumptions, breakdown conditions, and adverse reactions, the way lisinopril does.

§2. Step 1 — Observe the Data at the Moment of Measurement

The single highest-yield audit move, and the one almost never performed.

"You never learn more about a process than when you directly observe how data are actually measured." — Cuthbert Daniel, quoted in SWFE, p. 84

Go where the numbers are made. Walk around. Talk to the people holding the instrument. See how the numbers came to be, rather than how they arrived.

The standing questions at the point of measurement (p. 84):

Named failure mode — Sampling to Please (p. 84). A company polluted a river; agencies ordered daily water samples to monitor cleanup. Observe the actual collection and you see a small boat go out and the driver dip a container on a pole — after looking around for cleaner water. Nothing in the resulting time series records that scan. The general rule Tufte draws: people cannot keep their own score, and self-monitoring has early, structural limits. (Named in tufte-visual-thinking §7; the audit action is here.)

Do / don't:

Why the manuals matter (pp. 88–89). Tufte's positive example is the technical manual for a transesophageal echocardiography system, which enumerates dozens of named 2D and 3D imaging artifacts — acoustic saturation, aliasing, comet tail, mirroring, multi-path positioning, range ambiguity, reverberation, scattering, shadowing, side and grating lobes, speckle, spectral broadening, speed-of-sound error, and for 3D: acquisition, rendering, editing, color-gain, directional, motion, dropout, and pseudoclefting artifacts — each with its physical cause and its correction. His verdict is that this is what serious empirical analysis of measurement error looks like, and that it is enormously better than the notion of "error" in classical statistics (p. 88). Audit action: find the equivalent manual for whatever produced your data, and check which of its named artifacts your analysis assumes away.

Calibrate your expectations. Tufte's summary of what direct observation of medical measurement reveals: it is a two-digit science on a good day, and on other days getting the sign right is an achievement (p. 84). Import that expectation into whatever domain you audit before you accept a third significant figure.

§3. Step 2 — Track the Origin and Life History of the Measurement

A measurement is an object with a biography. Reconstruct it (p. 85).

• Why is this measurement made at all?
• Who looks at it — when, where, and for what purpose?
• What are its consequences, harms, and benefits?
• Where did the data go, and what is it doing now?
• Follow the money: who profits, and how much? For medical measurements Tufte states this as a standing instruction, not an accusation.
• Read the engineering and technical manuals, which describe measurement complexity and problems far more frankly than the research literature does.

Two provenance failures with names:

• Provenance not documented (p. 92, quoting Groskopf). Data are gathered by businesses, governments, nonprofits, sensors, satellites, and cranks. Survey data are not exhaustive; sensors vary in accuracy; governments are disinclined to hand over unbiased information; war-zone data are geographically biased by the danger of crossing battle lines; sources get daisy-chained together and policy analysts redistribute government data. Every stage in that chain is an opportunity for error.
• "Ground truth" is often just someone else's spreadsheet (p. 94). Database "ground truthers" never see their data at the moment of actual measurement. Attributing truth to a database, and then claiming proof of concept, is the fallacy of equivocation — a term used in two or more senses inside one argument so that a conclusion appears to follow when it does not. Tufte lists the puns that do this work: ground truth, error, power, optimal, explained/unexplained variance, intellectual property. His compressed audit slogans for the same failure: sketchy in, sketchy out; bias in, bias out; no causality in, no causality out; circularity in, circularity out (p. 94).

Named failure mode — Survival Bias (p. 94). Most medieval castles were built of wood; we believe most were stone because stone survived. Research databases are, definitionally, the databases that survived long enough to be selected for "ground truth" status. Survivor bias is subtle and inscrutable and demands both meta-cognition and detective work about database provenance.

§4. Step 3 — The Three Traceability Questions

The chapter's most directly reusable audit instrument, adapted from the September 2002 Lucent Technologies report on the investigation of possible scientific misconduct in the work of Hendrik Schön and coauthors, Appendix E (p. 85). Ask these three, in order, of any presented result.

Q1 — Can the presented data be traced back to primary data, free of processing or other manipulation?

Supporting logic worth carrying into policy: only credible primary data can corroborate published data; retention of primary data plus adequate record keeping is required for the ordinary conduct of work, not merely for investigating wrongdoing, because new questions arise that require returning to the primary data and revising the original analysis. Failure to keep primary data and records for a reasonable time is, by itself, a threat to the health of the enterprise — as true in the computer age as before.

Q2 — Is there clear evidence that the data do not come from the measurements described?

Three named evidence forms:

The Lucent committee's own caution is part of the tool: it set aside all but a few especially problematic Contradictory Physics cases, because the category is where genuine discovery also lives. Absent demonstrated special diligence, results contradicting known mechanism suggest simple error, self-deception, or misrepresentation — in that order.

Q3 — If the data are not valid, are there mitigating circumstances explaining the misrepresentation?

A clerical error — including the wrong data in a figure — reflects poor procedure but not misconduct. But innocent explanations require knowing the authors' state of mind at the time, which cannot be determined definitively. The credibility of a particular innocent explanation therefore depends on the overall credibility of the person in question, which in turn depends on whether there is an unreasonable number of problems or a pattern of questionable practices. Once the data problems are established, the live question is whether many improbable innocent explanations should be accepted.

Audit action: score Q3 across the whole body of work, not the single artifact. One improbable innocent explanation is plausible. Six is a pattern.

§5. Step 4 — Run the Forensic Audit Before Analysis, and Know It Is Not Data Cleaning

Heading, p. 91: conduct an independent forensic data audit of your data prior to analysis — don't just look around, see everything.

Why before. Audits get deflected because researchers and sponsors are anxious for an early peek at the findings, and early information exerts undue influence through anchoring bias (p. 91). Once anyone has seen a preliminary result, the audit is compromised — not by dishonesty but by anchoring. Sequence is a control.

The distinction that does the work (p. 91):

Named failure mode — Cleaning Mistaken for Auditing. A green cleaning report establishes that values are well-formed. It establishes nothing about whether they are true. (Named in tufte-visual-thinking §7; the operational split is here.)

The checklist substrate. Chris Groskopf's Guide to Bad Data enumerates 46 data-quality issues in spreadsheets, which Tufte calls a good start for forensic audits (p. 91). Grouped for use:

Named failure mode — the Chihuahua Syndrome (p. 92). A dog-licensing database was populated by a free-text breed field instead of a picklist, and ended up holding 250 spellings of "Chihuahua." Groskopf's rule: there is no worse way to wreck data than to let a single human type it in without validation, and even the best tools cannot save messy data. Audit action: for every categorical field, count distinct values and compare to the size of the legitimate vocabulary. A 250:1 ratio is a schema failure, not a data-entry failure.

Get it right at the start (p. 91). Early adjustments are cheap; later rescue revisions are costly. Jeff Atwood's contribution is that the worst technical debt he has encountered is bad names — for database columns, variables, functions — which must be fixed immediately before they metastasize across the codebase. Nate Silver's is the audit habit itself: when combining datasets or doing complex processing, be compulsive about missing data and sanity-check failures, because missing, miscoded, and outlier cases more often than not indicate a larger, more systematic problem with the code or the data. Errors compound; do the due diligence before moving on.

§6. Step 5 — Structural Checks the Eye Will Not Catch

These are the failures that survive a clean spreadsheet, a plausible chart, and a confident author.

Batch effects (p. 93). High-throughput measurement is affected by laboratory conditions, reagent lots, and personnel differences. The failure becomes serious when batch effects are correlated with the outcome of interest — then they do not add noise, they manufacture findings. Tufte's display is second-generation sequencing data from the 1000 Genomes Project: each row a HapMap sample processed in the same facility and platform, ordered by processing date, dark blue at 3 standard deviations below average and orange at 3 above, across a 3.5 Mb region of chromosome 16. Many batch effects are visible; the largest occupies days 243–251 as long orange horizontal streaks. Audit action: demand the batch metadata (date, site, instrument, operator, lot) as a first-class column, then plot the outcome against it before plotting anything else. If batch metadata was not recorded, the audit stops there.

Bang-bang duplicates / pseudo-replication (p. 86). An observer records a number and, seconds later, records another. If the underlying process changes daily, those two measurements are not independent: knowledge of the first gives near-full knowledge of the second, so the number of independent measurements is one, not two. Econometrics calls this autocorrelation or serial correlation; experimental design calls it bang-bang duplicates or pseudo-replications. Tufte's worked case: an experiment on 3 mice measuring the same variable 30 times per mouse does not have a sample size of 90. Audit action: for every reported n, ask what the independent unit actually is — subject, site, batch, session — and recompute.

Redefining groups breaks random assignment (p. 92). Researchers redefine the treatment group to include only patients treated rather than intended to treat. Darrel Francis's example: if treatment happens on the 9th floor of a building without elevators, less healthy patients never arrive — so even if the treatment has no effect at all, the treated group will appear to do better than placebo. Audit action: confirm the analysis is intention-to-treat, and reconstruct the attrition path between assignment and analysis.

Surrogates, proxies, and biomarkers (p. 94). Markers such as blood pressure, lipids, and patient hope or gratitude stand in for serious long-run outcomes such as strokes, heart attacks, quality of life, and all-cause mortality — and the link is often based on assumption, theory, folklore, marketing, or tradition rather than evidence. Tufte's diagnostic: changing hard outcomes to easier marker goals is a clear signal that the intervention doesn't work. The worked case is progression-free survival in oncology, where only about one-third of cancer drugs entering European and U.S. markets have evidence of overall survival or quality-of-life benefit, and post-approval incentive to evaluate real clinical benefit is weak. Audit action: for every metric, ask what decision it stands in for, and whether the substitution has ever been validated empirically.

Your database does not contain answers to all of your questions (p. 87). A database may simply lack the relevant explanatory variables — a devastating constraint. Many analyses are in fact model-specification searches ("let's try this, let's try that"), but this and that may not be in the database at all. Tufte's instruction is blunt: get better data. Audit action: before any modeling, list the variables the causal story requires and mark which are absent. An absent confounder is not a modeling problem.

Subgroup over-reach (p. 107). Subgroup analysis is essential for learning from data and cheap to abuse. Tufte's example via John Mandrola: a trial comparing angioplasty against standard medical therapy for angina found the primary endpoint of death and myocardial infarction in 6.3% of the angioplasty arm versus 3.3% of the medical arm. Rather than reporting that angioplasty was twice as bad on the primary endpoint, the abstract opened with a subgroup, a secondary endpoint within it, and a further sub-subgroup. Audit action: read the primary endpoint in the primary population first, and only then read the abstract.

Predictive models built on dirty data (p. 95). Predictive policing systems trained on data produced under flawed and sometimes unlawful practices cannot escape that legacy; the models document prevailing practice rather than the phenomenon. Tufte's twist is useful to an auditor: model outputs, in effect, measure their own biases — so prediction models rat themselves out. The supporting asymmetry: available studies estimate 49% of businesses and 25% of households have been victims of white-collar crime, against roughly 1.1% prevalence for violent crime and 7.4% for property crime — yet enforcement data reflect the opposite emphasis. Audit action: treat a model's output distribution as evidence about the data-generating institution, then check whether that institution is the thing you meant to measure.

§7. Step 6 — Audit the Model, Not Only the Data

Every model should ship a package insert (p. 98). Prescription drugs carry detailed inserts describing use, pitfalls, prevalence of adverse effects, and breakdowns. Tufte's proposal is the same document for statistical models, encountered everywhere the model is — textbooks, computer code, workaday practice, publications. His positive example (p. 99) is a real one: an engineering group's insert for Kaplan-Meier survival estimates, naming implicit factors that break the single-population assumption, lack of independence of censoring, lack of uniformity within a time interval, the effect of many censored values, patterns in plots of censored versus noncensored values, and special problems with small samples. Each use of a model should remind users of its constraints, assumptions, and breakdowns.

Model multiplicity is the default, not the exception.

• With enough parameters a model fits anything. Tufte quotes von Neumann: with four parameters he can fit an elephant, and with five he can make it wiggle its trunk (p. 106). Physics supplies the same point at scale — symmetry groups with plentiful parameters tuned to make protons live as long as one likes; one model might be correct, but no one would ever know.
• Feller's case (p. 106): a large literature established a transcendental "law of logistic growth" for human populations, bacterial colonies, and railroads, with lengthy tables and chi-square tests — but the normal, the Cauchy, and other distributions fit the same material equally well or better. The logistic played no distinguished role whatever. Goodness of fit does not select a mechanism.
• Curve-fitting choices carry messages independent of the data (p. 105, via xkcd): linear, quadratic, logarithmic, connecting lines, exponential, LOESS, ad-hoc filter, logistic, confidence interval, piecewise, house-of-cards. The same scatter supports all of them.

People cannot keep their own score (p. 106). If someone shows you simulations that only demonstrate the superiority of their own method, be suspicious; good simulations show where a method shines and where it breaks. The hidden mechanism named on the same page: authors tune their own method thoroughly and tune the competing methods insufficiently. Audit action: ask for the failure regime. A method with no published failure regime has not been characterized.

A good model explains data; it does not memorize it (p. 106). Overfitted models chasing data are brittle, break down, and regress toward the truth when confronted with new data. Computing millions of models is easy; explaining something well enough to yield replicated real-world explanations and successful interventions is very difficult. Tufte's emblem is Borges's map at 1:1 scale — perfect, useless, and eventually left to the inclemencies of sun and winter.

Fresh data remodels models (p. 107). The Phillips curve looked like a clean inflation–unemployment trade-off across U.S. data from 1961–1969; the subsequent 49 years, 1970–2018, scatter across the plane. Audit action: re-plot any foundational relationship with every year of data since the model was adopted, before using it.

Against the classical error model (p. 104). Tufte's claim is that empirical uncertainties and errors are not detected, measured, or modeled by standard statistical methods, and that estimating uncertainties under false assumptions — then announcing a hypothesis "true" or a difference "significant" — is falsification rather than inference. The assumptions are mentioned in textbooks and forgotten in workaday analysis and published reports. Linking math models to a noisy real world requires punning: calling two different things by the same name. Tukey's counterweight is the governing preference: far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise (Tukey, quoted p. 90).

But do not over-refine either (p. 90). Mosteller's essay, reproduced at length, records the statisticians' experience that when fairly crude measurements are refined, the change more often than not turns out to be small — policy decisions are frequently insensitive to the measure, and finer measures often act like weights that do not change the decision. This is not an argument against better information where it is needed; it is an argument for deciding where the next money for investigation should go. Aggregative statistics adequate for group policy may be inadequate for individual decisions, and that limit should be stated rather than hidden.

§8. Step 7 — Demand the Data Behind the Summary

Binned, smoothed, and summarized displays are where an audit dies. Tufte's heading (p. 101) is unusually direct: binning continuous data is medium-quality evidence of falsification.

The clutter defense, inverted (p. 101). The standard objection is that showing all the data creates clutter. Tufte's reply: clutter in a data graphic is evidence that your model does not fit the data — and that you know it. Summary graphics that cover contrary data and depict thresholds not present in the dataset are obvious cheats, easily detected, and damaging to credibility. His scale argument: a billion people a day read e-maps with data densities twenty times greater than a deceptive small chart, so there is no basis for assuming readers become stupid on contact with a research report.

Data-availability excuses are themselves data (p. 101). Tufte catalogs the refusals — trade secret, would violate patient privacy, hard drive crashed, intellectual property, in litigation, patent pending, IPO silent period. Treat the excuse as an audit finding. The positive contrast is the exomoon paper (p. 102): not paywalled, code and curve-fitting open-source and replicable, not claimed as intellectual property, not patented or monetized.

Cross-reference: graphic distortion itself (Lie Factor, dimension inflation, scale breaks) belongs to tufte-graphical-integrity; the culling and selection of evidence belongs to tufte-evidence-corruption. This section covers only the audit demand: get behind the summary to the measurements.

§9. Step 8 — Follow the Money

Tufte treats conflict of interest as a detectable, measurable property of a study — not an ad hominem.

"But financial conflicts are detectable definitively and represent a uniquely perverse influence on the search for scientific truth." — Colin Begg, quoted in SWFE, p. 81

The Memorial Sloan Kettering sequence (p. 116) is the chapter's worked case and the reason the audit item exists. Investigative reporting by ProPublica and The New York Times found that financial conflict-of-interest statements in published articles diverged from researchers' actual conflicts — service on drug-company boards of directors (a fiduciary, primary loyalty), private start-ups, large consulting payments. Deans and researchers departed; thousands of corrections were issued to previously published papers; the MSK physician-in-chief resigned three days after the report. The replacement physician-in-chief then defended financial conflicts from first-person clinical experience and argued for more faculty on corporate boards. The institution's chief of biostatistics and epidemiology took the opposite view: the substantive problem was not failure to disclose conflicts but the conflicts themselves. The reply he received is the line to remember, because it is the entire anti-audit stance compressed into four words:

"He works with data." — the MSK replacement physician-in-chief, on the chief of biostatistics, quoted in SWFE, p. 116

Tufte's answer, on the same page: working with data, epidemiologists have extended more lives than anyone — through vaccines, epidemic response, randomized trials, and the statistical analysis that proved smoking causes cancer, leading to cessation policies that prevented hundreds of millions of early deaths.

The two epigraph authorities (p. 114). Tufte frames professional self-interest as structural rather than personal:

"People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices." — Adam Smith, quoted in SWFE, p. 114

"It is difficult to get people to understand something, when their salary depends on their not understanding it." — Upton Sinclair, quoted in SWFE, p. 114

Alan Cassels, on the same page, supplies the operational form: across thousands of experts in psychiatric diagnosis, none ever proposed tightening the criteria for their favored diagnosis; all worried about missed cases, none about the harms and risks of mislabeling. Audit action: ask which direction of error the field is institutionally organized to fear. That direction is where the bias will be.

Money in guidelines (pp. 114–115). Guidelines are written exclusively by insiders, and joining a guideline panel advances an expert's specialty career. Some professional societies are large financial enterprises: one national cardiovascular association's fiscal 2016–2017 budget was $912 million with 20% from corporate support; 77% of a €60 million annual income at a European cardiology society came from industry. Ioannidis's proposed remodel (p. 115) is a straight outsider-comparison move: have guidelines written by methodologists and patients with content experts consulted, or recruit specialists unrelated to the subject matter, because what seems crucially important to a field expert may look like minutiae to a less personally involved outsider.

Gratitude is not an outcome measure (p. 115). A cardiology chair's defense — "I do angioplasty and I have grateful patients" — is self-scoring: post-op gratitude does not measure whether an intervention works, is free of harm, or is worth its cost against alternatives. Tufte's line: using patient gratitude to assess medical work is like winning a race that has only one runner. The empirical rebuttal on the same page: high patient satisfaction is associated mainly with hospitality, but also with more treatments, higher costs, and substantially higher mortality, even after adjusting for baseline health and comorbidities.

Regulatory capture as an audit finding (p. 112). Tufte's proposal is that credibility assessments of research should be as rigorous and frequent as restaurant sanitation inspections or pre-sale home inspections — his example being a 42-page pre-sale report with 55 photographs including infrared images of electrical switch boxes. He grants that performance audits are themselves distortable by financial interests and captured bureaucracies, but sets the stakes with the counterfactual: 270,000 U.S. deaths attributed to oxycodone, where every pill was approved by the FDA, made by licensed companies, prescribed by licensed doctors, sold by licensed pharmacists, and all 72 billion pills (about 500 per U.S. household) tracked to the exact place, time, and amount of sale by the DEA. The only thing worse than regulatory agencies is the theater of failed agencies captured by those they regulate.

Access and Ownership as an Audit Precondition (pp. 96–97)

Tufte breaks a two-page chapter-within-a-chapter into the middle of ch. 4, revisiting the electronic health record material from ch. 1. It belongs to the audit because you cannot audit data you are contractually forbidden to look at. Access is not a legal preliminary to the audit; it is the audit's first finding.

The design principle being violated (p. 96). Tufte invokes Berners-Lee's rule for data models: the point is to let a pool of information grow and evolve, which requires that the method of storage must not place its own restraints on the information. A record format that claims ownership of what it stores has inverted the rule — the container now governs the content.

The gag order, and why it is an evidence problem (pp. 96–97). Local EHR installations seize ownership of patient content by copyrighting it, and every login puts patients and staff through a click-through agreement. Tufte prints a parody terms-of-service that claims everything in the system — colors, words, photos, graphs, numbers, punctuation, artworks, logos, trademarks, and any compilation of them — and forbids reproduction "by any means," screenshots included; declining routes the user to a second box threatening to disable their medical record. He notes this is not a parody. The consequences that matter to an auditor:

…(truncated)