Data Analysis Skill
Structured approach for analyzing metrics, attributing changes, and communicating findings.
When to Use
- Performance analysis from production metrics
- Attribution of improvements/regressions to code changes
- Creating executive summaries or stakeholder communications
- Any analysis requiring correlation of changes to measured outcomes
Quick Reference
Five Phases
Collection → Filtering → Curation → Questioning → Synthesis
| Phase |
Key Question |
Output |
| Collection |
What are we measuring? |
Baseline, scope, change list |
| Filtering |
What's signal vs. noise? |
Categorized changes with confidence |
| Curation |
What correlates with what? |
Attribution table |
| Questioning |
Do we KNOW or BELIEVE this? |
Validated claims with caveats |
| Synthesis |
Who needs to know what? |
Audience-appropriate artifacts |
Confidence Assignment
| Level |
Use When |
| High |
Clear mechanism + timing alignment + targets measured population |
| Medium |
Plausible mechanism but confounded by other changes |
| Low |
Speculative or enabling-only |
Attribution Table Template
| Change |
Evidence |
Release |
Metric |
Confidence |
Notes |
| [Description] |
[PR/commit] |
[version] |
[affected metric] |
High/Med/Low |
[mechanism or caveat] |
Process
1. Collection
**Metrics:** [What are you measuring?]
**Population:** [Who? All users, p75, specific cohort? A percentile is not a cohort: "p75 = power users" is an assumption until verified against one]
**Exposure:** [Fraction of the population running a build that contains the change, over the same window, as a number. If it is unknown, no outcome reading is valid]
**Period:** [Measurement window - release tags or dates]
**Source:** [APM, logs, synthetic benchmarks?]
**Baseline:** [Starting values with methodology]
Enumerate ALL changes in scope:
- Code changes (PRs, commits)
- Config changes
- External factors (traffic, user growth, infrastructure)
2. Filtering
Categorize each change:
- Direct: Clear causal path to measured metric
- Indirect: Enabling infrastructure (value materializes later)
- Unknown: In scope but mechanism unclear
- Noise: Unlikely to affect measured metrics
3. Curation
Build attribution table:
- Map changes to metric movements by release. Confirm each change's merge commit is contained in the release whose traffic you read, and segment by release so pre- and post-change clients are never pooled
- Note co-landed changes (shared attribution)
- Flag anomalies (improvement without cause, unexplained regression)
- Separate measured vs. post-cutoff work
4. Questioning
Challenge every attribution:
Document what's missing:
5. Synthesis
Create audience-appropriate artifacts:
| Artifact |
Audience |
Focus |
| Executive Summary |
Leadership |
Hard data, key wins, team recognition |
| Attribution Catalogue |
Engineering |
Detailed per-change analysis |
| Methodology Doc |
Future analysts |
Process, assumptions, data sources |
| Communication Post |
Stakeholders |
Exciting but honest, caveats visible |
Communication Template
**[Metric]: [Before] → [After] ([Change %])**
Population: [Who this measures]
Caveat: [Key limitation]
What's NOT included: [Equally interesting gaps]
Notable contributors:
- [Change 1] — [mechanism]
- [Change 2] — [mechanism]
Bottom line: [One sentence impact statement]
Anti-Patterns
| Don't |
Do Instead |
| Claim causation from correlation |
"Correlates with" or "plausible contributor" |
| Attribute release total to single change |
Note multiple changes, unknown isolated impact |
| Bury caveats in footnotes |
Caveats are part of the story |
| Use superlatives without data |
Let numbers speak |
| Hide uncertainty |
Use qualifiers: "likely," "plausible," "unknown" |
Checklist
Before finalizing:
1---2maturity: experimental3name: data-analysis4description: Structured approach for analyzing metrics, attributing changes, and communicating findings — five phases (collection → filtering → curation → questioning → synthesis), confidence assignment, audience-appropriate artifacts5metadata:6 source: https://skillmd.com/skills/jiayaoqijia/data-analysis7---89# Data Analysis Skill1011Structured approach for analyzing metrics, attributing changes, and communicating findings.1213---1415## When to Use1617- Performance analysis from production metrics18- Attribution of improvements/regressions to code changes19- Creating executive summaries or stakeholder communications20- Any analysis requiring correlation of changes to measured outcomes2122---2324## Quick Reference2526### Five Phases2728```29Collection → Filtering → Curation → Questioning → Synthesis30```3132| Phase | Key Question | Output |33| ----------- | --------------------------- | ----------------------------------- |34| Collection | What are we measuring? | Baseline, scope, change list |35| Filtering | What's signal vs. noise? | Categorized changes with confidence |36| Curation | What correlates with what? | Attribution table |37| Questioning | Do we KNOW or BELIEVE this? | Validated claims with caveats |38| Synthesis | Who needs to know what? | Audience-appropriate artifacts |3940### Confidence Assignment4142| Level | Use When |43| ---------- | ---------------------------------------------------------------- |44| **High** | Clear mechanism + timing alignment + targets measured population |45| **Medium** | Plausible mechanism but confounded by other changes |46| **Low** | Speculative or enabling-only |4748### Attribution Table Template4950| Change | Evidence | Release | Metric | Confidence | Notes |51| ------------- | ----------- | --------- | ----------------- | ------------ | --------------------- |52| [Description] | [PR/commit] | [version] | [affected metric] | High/Med/Low | [mechanism or caveat] |5354---5556## Process5758### 1. Collection5960```markdown61**Metrics:** [What are you measuring?]62**Population:** [Who? All users, p75, specific cohort? A percentile is not a cohort: "p75 = power users" is an assumption until verified against one]63**Exposure:** [Fraction of the population running a build that contains the change, over the same window, as a number. If it is unknown, no outcome reading is valid]64**Period:** [Measurement window - release tags or dates]65**Source:** [APM, logs, synthetic benchmarks?]66**Baseline:** [Starting values with methodology]67```6869Enumerate ALL changes in scope:7071- Code changes (PRs, commits)72- Config changes73- External factors (traffic, user growth, infrastructure)7475### 2. Filtering7677Categorize each change:7879- **Direct:** Clear causal path to measured metric80- **Indirect:** Enabling infrastructure (value materializes later)81- **Unknown:** In scope but mechanism unclear82- **Noise:** Unlikely to affect measured metrics8384### 3. Curation8586Build attribution table:87881. Map changes to metric movements by release. Confirm each change's merge commit is contained in the release whose traffic you read, and segment by release so pre- and post-change clients are never pooled892. Note co-landed changes (shared attribution)903. Flag anomalies (improvement without cause, unexplained regression)914. Separate measured vs. post-cutoff work9293### 4. Questioning9495Challenge every attribution:9697- [ ] "Do we KNOW this, or do we BELIEVE this?"98- [ ] "What would need to be true for this to be wrong?"99- [ ] "Are there alternative explanations?"100- [ ] "Could a change in the population's composition explain it?" Decompose within-unit change from composition before reporting an aggregate101102Document what's missing:103104- [ ] Unexplained improvements105- [ ] Unexplained regressions106- [ ] Work that SHOULD have helped but didn't107- [ ] Metrics you wish you had108109### 5. Synthesis110111Create audience-appropriate artifacts:112113| Artifact | Audience | Focus |114| --------------------- | --------------- | ------------------------------------- |115| Executive Summary | Leadership | Hard data, key wins, team recognition |116| Attribution Catalogue | Engineering | Detailed per-change analysis |117| Methodology Doc | Future analysts | Process, assumptions, data sources |118| Communication Post | Stakeholders | Exciting but honest, caveats visible |119120---121122## Communication Template123124```markdown125**[Metric]: [Before] → [After] ([Change %])**126127Population: [Who this measures]128Caveat: [Key limitation]129What's NOT included: [Equally interesting gaps]130131Notable contributors:132133- [Change 1] — [mechanism]134- [Change 2] — [mechanism]135136Bottom line: [One sentence impact statement]137```138139---140141## Anti-Patterns142143| Don't | Do Instead |144| ---------------------------------------- | ------------------------------------------------ |145| Claim causation from correlation | "Correlates with" or "plausible contributor" |146| Attribute release total to single change | Note multiple changes, unknown isolated impact |147| Bury caveats in footnotes | Caveats are part of the story |148| Use superlatives without data | Let numbers speak |149| Hide uncertainty | Use qualifiers: "likely," "plausible," "unknown" |150151---152153## Checklist154155Before finalizing:156157- [ ] Measurement methodology documented158- [ ] Baseline values recorded with source159- [ ] All changes in scope enumerated160- [ ] Confidence levels assigned with justification161- [ ] Unexplained anomalies noted162- [ ] Limitations explicitly stated163- [ ] What's NOT included documented164- [ ] Uncertainty reflected in language165- [ ] Links/references for all claims166- [ ] Multiple artifacts for different audiences167168---