Data Bias And Quality Critical Reading

Data Bias & Quality Critical Reading

Purpose

Prevents the most common and costliest data error: accepting data as unquestioned truth. Data is always the result of someone's choices, measurements, and history — it is a representation of reality, not reality itself (see the pack's ../../CLAUDE.md). When an AI model is trained on biased data, or a decision is made on the basis of incomplete reporting, the error doesn't show up in the data itself — it only shows up in the outcome, often too late. This skill produces a systematic check before that moment.

Anchored in research

• The "Understanding data" area of data literacy (see ../data-literacy-competency-assessment/SKILL.md): critical thinking about a data source's origin, ownership, and representativeness is a foundation of data literacy, not a separate specialty.
• The established bias taxonomy from statistics and machine learning (a synthesis of multiple sources): selection bias, survivorship bias, historical/label bias, measurement bias, aggregation bias — commonly recognized, recurring bias types not tied to any single product or company.
• The "vanity metric vs. actionable metric" distinction (lean analytics tradition): a metric is useless for decision-making if it wouldn't change any decision regardless of which direction it moves.

Method

1. Ask the provenance questions first for every significant dataset: where does this data come from, who collected it and why, and who owns it today? If you can't answer all three, don't make a decision based on the data yet — establish provenance first.
2. Walk through the five bias types explicitly for every dataset driving a decision:
   • Selection bias: which cases/entities made it into the dataset and which were systematically left out? (e.g. only customers who contacted support, not all customers)
   • Survivorship bias: does the data only show the cases that "survived" a process (e.g. only approved applications, not rejected ones), leaving the reasons for failure invisible?
   • Historical/label bias: does the data reflect past inequity or one-sided decision-making (e.g. who was previously promoted or granted credit) in a way that a model would learn to repeat as a new "truth"?
   • Measurement bias: is the variable used a good proxy for what was actually meant to be measured, or is something being measured just because it's easy to measure (e.g. clicks instead of actual customer satisfaction)?
   • Aggregation bias: does meaningful variation disappear when data is combined into averages or totals (e.g. average utilization masking the fact that two customer groups behave completely differently)? For every bias found, note: which type, the likely direction of its effect, and whether it's correctable or only something to account for.
3. Explicitly ask what's missing from the data and whose perspective is missing. Missing data is not neutral — it's usually systematically missing for a particular group or situation. Name the missing group/situation, don't just note "data is missing."
4. Separate vanity metrics from decision-driving ones. Test every metric presented with the question: "If this number moved 20% in either direction, would it change any decision?" If the answer is no, the metric is "nice-to-know" and doesn't deserve a place at the core of decision-making — it can still be useful context, but shouldn't be presented as a key justification.
5. Assess the data's freshness and representativeness over time. Was the data collected under conditions matching the current situation (e.g. not collected during an exceptional period), and is it fresh enough for the decision it's being used to justify?
6. Produce a short "reliability note" for every key data finding before it moves forward: which biases were identified, how severe they are for the decision, and whether the finding can be used as-is or requires further validation.

What this skill does NOT do

• Doesn't fix the data technically (e.g. reweighting, imputation) — identifies and names the bias; correction is a separate technical task.
• Doesn't claim every bias must be removed before data can be used — many decisions can be made with a known, documented bias, as long as it's visible rather than hidden.
• Doesn't replace a statistical or machine-learning technical audit (e.g. a model's fairness metrics) — produces a business-level, critical first read before deeper technical analysis.
• Doesn't confirm figures or bias claims from memory — bases the assessment on data you provide, or marks an assumption clearly ([assumption — verify]).

Refinement notes

Areas to keep deepening with real practice:

• your own examples of biases you've found in client data and what they would have caused if they hadn't been noticed
• a concrete checklist template for walking through the five bias types (into ../../references/)
• rules of thumb for which industries/data types each bias type is most common in

Once this section is filled in and validated in practice, update the maturity field in skills_index.json to draft, validated, or canonical (see ../../../meta/maturity_levels.md). Don't add new fields to the frontmatter — name and description are the only ones allowed (see ../../../meta/frontmatter_schema.md).

Continue from here

• Before this (if the role is still unclear): ../data-role-diagnosis/SKILL.md
• Next in this pack: ../data-storytelling-and-business-translation/SKILL.md — use this skill's reliability notes when translating the finding into a story, and don't hide a known bias from the narrative.
• Related skill in another pack: ../../../business-case-and-analysis/skills/assumption-and-evidence-audit/SKILL.md — a broader check of assumptions and evidence at the whole business case level.
• Related skill in another pack: ../../../ai-strategy-and-governance/skills/responsible-ai-and-governance-check/SKILL.md — if the bias affects an AI model's training data, also check the responsibility/risk angle.
• A ready-made skill chain for this situation: see ../../../playbooks/
• This pack's shared guardrails: ../../CLAUDE.md

References

• ../../references/ — the pack's shared background material
• ../../CLAUDE.md — the pack's shared guardrails