# Simulation Based Validation

> Use when when you need to verify that a statistical correction (e.

- Skill: `holobiomicslab/simulation-based-validation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/simulation-based-validation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/simulation-based-validation/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/simulation-based-validation

---


# simulation-based-validation

## Summary

Generate synthetic methylation data using dataSim() to benchmark and validate statistical methods such as overdispersion-corrected differential methylation analysis. This skill enables controlled testing of analysis parameters and their effects on stringency and accuracy before applying methods to real data.

## When to use

When you need to verify that a statistical correction (e.g., overdispersion adjustment in calculateDiffMeth with overdispersion='MN') produces expected changes in test stringency, or when you want to validate that a new analysis workflow produces correct q-value distributions and variance adjustments under known ground-truth conditions.

## When NOT to use

- When you have sufficient real experimental replicates and real biological effect sizes — simulation is no substitute for validation on actual data with true biological variance.
- When your primary goal is differential methylation discovery on real samples — simulation-based validation is a method-vetting step, not a substitute for direct analysis.
- When computational resources are severely constrained and you must prioritize analysis speed over methodological validation.

## Inputs

- methylKit dataSim() parameters (number of replicates, number of methylation sites)
- methylBase object (simulated or real methylation data with coverage and methylation percentage columns)

## Outputs

- methylDiff object with q-values and test statistics from uncorrected run
- methylDiff object with q-values and test statistics from corrected run (overdispersion='MN')
- q-value distribution comparison (mean, median, and percentile values)

## How to apply

Use methylKit's dataSim() function to generate a synthetic methylBase object with controlled parameters (e.g., 6 replicates and 1000 methylation sites). Run calculateDiffMeth() on this object twice: once with your method of interest (e.g., overdispersion='MN' with test='Chisq') and once with a baseline uncorrected configuration (overdispersion=FALSE or default). Extract and compare the resulting q-value distributions between runs to verify that the corrected method produces higher average q-values (more stringent multiple-testing correction) and that the variance adjustment factor φ = X²/(N-P) is correctly applied, confirming the expected shift from Chi-square to F-test behavior.

## Related tools

- **methylKit** (Core R package providing dataSim() function for synthetic methylation data generation and calculateDiffMeth() for differential methylation analysis with overdispersion correction) — https://github.com/al2na/methylKit
- **R** (Runtime environment for executing methylKit functions and statistical comparisons)

## Examples

```
library(methylKit); sim <- dataSim(replicates=6, sites=1000); diff_corrected <- calculateDiffMeth(sim, overdispersion='MN', test='Chisq'); diff_uncorrected <- calculateDiffMeth(sim, overdispersion=FALSE, test='Chisq'); mean(diff_corrected@data$qvalue); mean(diff_uncorrected@data$qvalue)
```

## Evaluation signals

- Mean q-value from overdispersion-corrected run (overdispersion='MN') is higher than uncorrected baseline, confirming more stringent multiple-testing correction
- Statistical test automatically switches from Chi-square (uncorrected) to F-test (corrected with overdispersion='MN'), verifiable via test statistic type in methylDiff object
- Variance scaling factor φ = X²/(N-P) is applied only in the corrected run, evidenced by larger denominator in test statistic calculation
- Q-value distribution shift is consistent across the full range of methylation percentages (not an artifact of a few extreme sites)
- Simulated data with known effect sizes confirms that corrected method maintains or improves specificity while adjusting for overdispersion relative to binomial expectations

## Limitations

- Simulated data from dataSim() may not capture all real-world sources of variance and batch effects present in actual bisulfite-seq experiments.
- Validation on simulation alone does not guarantee equivalent performance on real data with complex sample heterogeneity, sequencing errors, or technical biases.
- The overdispersion='MN' correction assumes methylation-specific negative binomial distribution; if your data violates this assumption, validation results may not transfer to real analysis.

## Evidence

- [other] Generate a simulated methylBase object using methylKit's dataSim() function with parameters set to 6 replicates and 1000 methylation sites.: "Generate a simulated methylBase object using methylKit's dataSim() function with parameters set to 6 replicates and 1000 methylation sites."
- [other] Extract and compare the q-value distributions from the corrected run (overdispersion='MN') against a parallel uncorrected run (overdispersion=FALSE or default) to verify that the corrected method produces higher average q-values: "Extract and compare the q-value distributions from the corrected run (overdispersion='MN') against a parallel uncorrected run (overdispersion=FALSE or default) to verify that the corrected method"
- [other] The calculateDiffMeth() function with overdispersion='MN' parameter calculates a scaling parameter φ = X²/(N-P) to adjust variance as φ·n_i·π̂_i·(1-π̂_i), which makes statistical tests more stringent by correcting for variance in excess of binomial expectations and automatically switches from Chisq to F-test.: "The calculateDiffMeth() function with overdispersion='MN' parameter calculates a scaling parameter φ = X²/(N-P) to adjust variance as φ·n_i·π̂_i·(1-π̂_i), which makes statistical tests more stringent"
- [readme] methylKit is an R package for DNA methylation analysis and annotation from high-throughput bisulfite sequencing: "methylKit is an R package for DNA methylation analysis and annotation from high-throughput bisulfite sequencing."