# Duo Experimentation

> Duolingo-style experimentation as a decision protocol — hundreds of A/B tests a week, opinions resolved by data, ideas killed quickly. Covers hypothesis design, metric selection, guardrails, sample size, novelty effects, and explicit kill criteria. Use when the user wants to run an A/B test, decide between two designs without arguing, set up an experiment program, debug why a test isn't moving, or build a culture where "show don't tell" replaces seniority-based decisions. Source the Duolingo Handbook (Show Don't Tell) and engineering blog posts.

- Skill: `hktitan/duo-experimentation` (Agent Skill, multi-file: 11 files)
- Install (CLI): `npx skillmds@latest add hktitan/duo-experimentation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/hktitan/duo-experimentation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: HKTITAN (https://skillmd.com/u/hktitan)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/hktitan/duo-experimentation

---


# Duolingo Experimentation — Map of Content

Duolingo runs hundreds of experiments per week and treats the result as the decision, not as one input. The handbook calls this *Show Don't Tell*: when people disagree, the experiment is the answer, not the more senior person's intuition. This skill packages that protocol.

This skill is structured as a **graph**: scan the descriptions below, follow only the `[[wikilinks]]` you need.

## The protocol

- [[references/show-dont-tell]] — the principle: metrics decide, not titles.
- [[references/hypothesis-design]] — what makes a testable hypothesis vs. a wish.
- [[references/ab-test-structure]] — variant/control, randomization unit, exposure rules.

## Metrics

- [[references/metric-selection]] — picking a primary metric that actually maps to long-term retention, not the easiest-to-move one.
- [[references/guardrail-metrics]] — the metrics that *must not* go down, even if the primary does up.
- [[references/sample-size]] — how big, how long; the cost of stopping early.

## Pitfalls

- [[references/novelty-effects]] — the lift that decays after week three; how to detect and discount it.
- [[references/ship-and-iterate]] — when "good enough to ship" beats "perfect to test."

## Discipline

- [[references/kill-criteria]] — the threshold for stopping an experiment instead of milking it.
- [[references/experiment-cadence]] — weekly review rhythm that keeps "hundreds per week" survivable.

## Sibling skills

- [[../duo-retention/SKILL]] — every retention claim has an A/B test under it; cite both.
- [[../duo-product/SKILL]] — *Ship It* and *Show Don't Tell* are the same culture seen from two angles.
- [[../duo-culture/SKILL]] — experimentation only works if disagreement is safe; that's a culture problem too.

## Sources

- Duolingo Handbook (2025), Principle #4: *Show Don't Tell*
- blog.duolingo.com — engineering and experimentation posts