Experiment Verification Monitoring
Use this skill to prevent misconfigured or unhealthy experiments from producing
bad evidence. It focuses on verification before launch, canaries, A/A tests,
leakage and interference checks, active monitoring, and quality roadmap metrics.
Source Traceability
Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is
transformed and paraphrased from Chapter 5 on experiment effectiveness,
prelaunch verification, QA tooling, canaries, A/A tests, spillover effects, and
active monitoring.
Related skills:
ab-test-design-brief for planning an experiment before verification.
experimentation-throughput-strategy for monitoring overlap conflicts.
trustworthy-experiment-insights for statistical credibility after results.
Reference Routing
| Need |
Read |
| Verification concepts |
references/core/knowledge.md |
| QA, canary, A/A, and monitoring rules |
references/core/rules.md |
| Failure scenarios and examples |
references/core/examples.md |
| Step-by-step quality roadmap |
workflows/create-experiment-quality-roadmap.md |
Workflow
- Define the experiment quality risks the platform must catch.
- Add prelaunch verification for assignment, targeting, exposure, treatment,
metrics, and user experience.
- Add early launch canaries and active monitoring for misconfiguration.
- Run periodic A/A tests to validate infrastructure health.
- Track quality metrics and update the experimentation playbook.
- Define owners and escalation paths for active experiment issues.
Output Format
# Experiment Verification And Monitoring Plan
## Quality Risks
[What errors or trust failures this plan should prevent.]
## Prelaunch Checks
| Check | Method | Owner | Pass/Fail Criteria |
|-------|--------|-------|--------------------|
## Active Monitoring
- Canary:
- Dashboards:
- Alerts:
- Leakage/interference checks:
## Platform Health
- A/A test cadence:
- Quality metrics:
- Review process:
## Escalation Rules
- Pause if:
- Restart if:
- Investigate if:
Quality Bar
- Do not rely on manual QA alone when the platform has recurring setup errors.
- Do not launch experiments without verifying assignment, exposure, and metrics.
- Do not treat A/A tests as one-time setup checks; use them as periodic health
checks when platform trust matters.
- Do not monitor only final results; active tests need early health signals.
1---2name: experiment-verification-monitoring3description: Verify and monitor running experiments for operational quality. Use when designing prelaunch QA, spot-check tooling, experiment canaries, A/A tests, leakage checks, interference monitoring, active experiment dashboards, alerts, or an experimentation quality roadmap.4license: MIT5compatibility: Codex, Claude Code, and other Agent Skills-compatible clients.6metadata:7 source: https://skillmd.com/skills/hashgraph-online/experiment-verification-monitoring8 version: "0.1.0"9 displayName: Experiment Verification Monitoring10 category: Product Management11 tags: practical-ab-testing,next-level-ab-testing,ab-testing,experimentation,quality-assurance12---1314# Experiment Verification Monitoring1516Use this skill to prevent misconfigured or unhealthy experiments from producing17bad evidence. It focuses on verification before launch, canaries, A/A tests,18leakage and interference checks, active monitoring, and quality roadmap metrics.1920## Source Traceability2122Primary source: *Next-Level A/B Testing* by Leemay Nassery. Guidance is23transformed and paraphrased from Chapter 5 on experiment effectiveness,24prelaunch verification, QA tooling, canaries, A/A tests, spillover effects, and25active monitoring.2627Related skills:2829- `ab-test-design-brief` for planning an experiment before verification.30- `experimentation-throughput-strategy` for monitoring overlap conflicts.31- `trustworthy-experiment-insights` for statistical credibility after results.3233## Reference Routing3435| Need | Read |36|------|------|37| Verification concepts | `references/core/knowledge.md` |38| QA, canary, A/A, and monitoring rules | `references/core/rules.md` |39| Failure scenarios and examples | `references/core/examples.md` |40| Step-by-step quality roadmap | `workflows/create-experiment-quality-roadmap.md` |4142## Workflow43441. Define the experiment quality risks the platform must catch.452. Add prelaunch verification for assignment, targeting, exposure, treatment,46 metrics, and user experience.473. Add early launch canaries and active monitoring for misconfiguration.484. Run periodic A/A tests to validate infrastructure health.495. Track quality metrics and update the experimentation playbook.506. Define owners and escalation paths for active experiment issues.5152## Output Format5354```markdown55# Experiment Verification And Monitoring Plan5657## Quality Risks58[What errors or trust failures this plan should prevent.]5960## Prelaunch Checks61| Check | Method | Owner | Pass/Fail Criteria |62|-------|--------|-------|--------------------|6364## Active Monitoring65- Canary:66- Dashboards:67- Alerts:68- Leakage/interference checks:6970## Platform Health71- A/A test cadence:72- Quality metrics:73- Review process:7475## Escalation Rules76- Pause if:77- Restart if:78- Investigate if:79```8081## Quality Bar8283- Do not rely on manual QA alone when the platform has recurring setup errors.84- Do not launch experiments without verifying assignment, exposure, and metrics.85- Do not treat A/A tests as one-time setup checks; use them as periodic health86 checks when platform trust matters.87- Do not monitor only final results; active tests need early health signals.