Product Analytics Experiment Review
Overview
Provide an evidence-first contract for experiment hypothesis, population, assignment, metrics, guardrails, instrumentation, segmentation, sample size, significance, novelty, ethics, and decisions.
When to use
Use only for the trigger conditions in the description and when the requested artifact needs explicit owners, evidence, limitations, and verification.
When NOT to use
Do not use only to define telemetry event fields or to launch an experiment.
Required inputs and context discovery
Collect exact project and build identity, requested scope, owners, dependencies, constraints, existing evidence, commands, artifact paths, risks, approvals, rollback or recovery, and unavailable information.
Safety and risk level
Risk level is read-only. Read-only review never authorizes mutation. Any load, rollout, publication, service, database, credential, or external-system action requires explicit human approval and bounded stop conditions.
Workflow
- State the product decision, falsifiable hypothesis, unit of randomization, and population.
Completion criterion: evidence is recorded and unresolved items are explicit.
- Define primary metric, guardrails, segments, attribution window, and instrumentation evidence.
Completion criterion: evidence is recorded and unresolved items are explicit.
- Review sample size, power, duration, peeking, novelty, interference, and data quality.
Completion criterion: evidence is recorded and unresolved items are explicit.
- Check player harm, fairness, privacy, rollback, and operational stop conditions.
Completion criterion: evidence is recorded and unresolved items are explicit.
- Return approve, revise, reject, or BLOCKED with pre-registered decision rules.
Completion criterion: evidence is recorded and unresolved items are explicit.
Evidence and output contract
Produce experiment-review.json with scope, snapshot, findings, owners, commands and exit codes, artifacts, acceptance criteria, Verified facts, Snapshot assumptions, Unverified hypotheses, BLOCKED items, and next actions.
Handoff contract
Record repository and path, goal, owned scope, do-not-touch scope, decisions, files changed, commands, exit codes, artifacts, restore information, blockers, next owner, and reactivation prompt.
Pitfalls and anti-rationalization
- Missing evidence is
BLOCKED, never PASS.
- A plan or review does not authorize production execution.
- Compile success, screenshots, or anecdotes do not prove broader coverage.
- Preserve unrelated work and generated-source ownership.
Verification checklist
References and scripts
No bundled runtime helper is required. Use project-owned plans, evidence, runbooks, schemas, and validation commands; repository governance tools remain full-clone-only.
1---2name: product-analytics-experiment-review3description: Use when a game product experiment or A/B test needs review across hypothesis, population, assignment, metrics, guardrails, instrumentation, segmentation, sample size, significance, novelty, ethics, and decision rules; not for telemetry schema alone.4license: MIT5---6# Product Analytics Experiment Review78## Overview9Provide an evidence-first contract for experiment hypothesis, population, assignment, metrics, guardrails, instrumentation, segmentation, sample size, significance, novelty, ethics, and decisions.1011## When to use12Use only for the trigger conditions in the description and when the requested artifact needs explicit owners, evidence, limitations, and verification.1314## When NOT to use15Do not use only to define telemetry event fields or to launch an experiment.1617## Required inputs and context discovery18Collect exact project and build identity, requested scope, owners, dependencies, constraints, existing evidence, commands, artifact paths, risks, approvals, rollback or recovery, and unavailable information.1920## Safety and risk level21Risk level is `read-only`. Read-only review never authorizes mutation. Any load, rollout, publication, service, database, credential, or external-system action requires explicit human approval and bounded stop conditions.2223## Workflow241. State the product decision, falsifiable hypothesis, unit of randomization, and population.25 Completion criterion: evidence is recorded and unresolved items are explicit.262. Define primary metric, guardrails, segments, attribution window, and instrumentation evidence.27 Completion criterion: evidence is recorded and unresolved items are explicit.283. Review sample size, power, duration, peeking, novelty, interference, and data quality.29 Completion criterion: evidence is recorded and unresolved items are explicit.304. Check player harm, fairness, privacy, rollback, and operational stop conditions.31 Completion criterion: evidence is recorded and unresolved items are explicit.325. Return approve, revise, reject, or BLOCKED with pre-registered decision rules.33 Completion criterion: evidence is recorded and unresolved items are explicit.3435## Evidence and output contract36Produce `experiment-review.json` with scope, snapshot, findings, owners, commands and exit codes, artifacts, acceptance criteria, Verified facts, Snapshot assumptions, Unverified hypotheses, BLOCKED items, and next actions.3738## Handoff contract39Record repository and path, goal, owned scope, do-not-touch scope, decisions, files changed, commands, exit codes, artifacts, restore information, blockers, next owner, and reactivation prompt.4041## Pitfalls and anti-rationalization42- Missing evidence is `BLOCKED`, never PASS.43- A plan or review does not authorize production execution.44- Compile success, screenshots, or anecdotes do not prove broader coverage.45- Preserve unrelated work and generated-source ownership.4647## Verification checklist48- [ ] Scope, identity, owners, and exclusions are explicit.49- [ ] Every verdict cites observed evidence or is labeled Unverified/BLOCKED.50- [ ] Risks have mitigations, triggers, owners, and dates.51- [ ] No unauthorized mutation, publication, deployment, or destructive action occurred.52- [ ] Handoff names the next bounded action.5354## References and scripts55No bundled runtime helper is required. Use project-owned plans, evidence, runbooks, schemas, and validation commands; repository governance tools remain full-clone-only.
hoatv2211