# Batch Effect Preprocessing With Replicate Structure

> Use when you have metabolomics data from multiple experimental batches (≥2) with embedded replicate samples (identical samples run at different points within or across batches), log-transformed raw assays containing ≥50% missing values per batch, and a need to harmonise metabolite quantification.

- Skill: `holobiomicslab/batch-effect-preprocessing-with-replicate-structure` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/batch-effect-preprocessing-with-replicate-structure`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/batch-effect-preprocessing-with-replicate-structure/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/batch-effect-preprocessing-with-replicate-structure

---


# batch-effect-preprocessing-with-replicate-structure

## Summary

A hierarchical preprocessing workflow for multi-batch metabolomics data that leverages intra-batch and inter-batch replicate samples to remove unwanted variation while preserving biological signal. This skill combines missing-value filtering, feature intersection across batches, k-nearest neighbour imputation, loess-based signal drift correction, and RUV-III normalisation to produce batch-corrected assays suitable for downstream analysis.

## When to use

Apply this skill when you have metabolomics data from multiple experimental batches (≥2) with embedded replicate samples (identical samples run at different points within or across batches), log-transformed raw assays containing ≥50% missing values per batch, and a need to harmonise metabolite quantification across batches while controlling for intra-batch drift and inter-batch systematic bias. This is particularly relevant for large-scale studies (e.g. >200 samples, >5 batches) where batch effects dominate biological variance.

## When NOT to use

- Input contains no replicate samples embedded within or across batches—RUV-III estimation requires replicate information to decompose unwanted variation.
- Metabolomics data are already normalised, batch-corrected, or merged into a single assay—applying this workflow would destroy prior normalisation and introduce redundant imputation.
- Single-batch data or data with <50% missing values per batch—the threshold filtering and intersection logic are tuned for multi-batch studies with substantial sparsity.

## Inputs

- List of SummarizedExperiment objects (one per batch)
- logRaw assay (log2-transformed raw metabolite counts)
- Metadata table defining intra-batch and inter-batch replicate sample assignments

## Outputs

- rawImpute assay (filtered, intersected, and imputed metabolite abundances)
- Intra-batch normalised assay (loess-smoothed, RUV-III corrected)
- Final inter-batch normalised assay (batch-effect corrected, ready for analysis)

## How to apply

Load each batch as a SummarizedExperiment object with a log2-transformed raw assay (logRaw = log2(raw + 1)). First, apply hRUV::clean() with threshold=0.5, method='intersect', and assay='logRaw' to filter metabolites with >50% missing values per batch, retain only metabolites quantified across all batches via the intersect method, and impute remaining missing values using k-nearest neighbour to produce a rawImpute assay. Next, perform intra-batch normalisation by applying loess smoothing on samples within each batch and RUV-III with k=5 using intra-batch replicate samples to estimate and remove unwanted variation within runs. Finally, execute inter-batch normalisation by concatenating the hierarchically structured batches and using inter-batch replicate samples to estimate and correct systematic batch-level drift. Verify that the final assay contains no missing values and metabolite counts are consistent across all samples.

## Related tools

- **hRUV** (Core normalisation package implementing the hierarchical RUV strategy, clean() function for filtering/imputation, loess smoothing, and RUV-III batch effect removal) — https://github.com/SydneyBioX/hRUV
- **SummarizedExperiment** (Data container for organising metabolite abundance matrices and metadata across batches)
- **RUV-III** (Statistical method for estimating and removing unwanted variation using replicate samples)
- **DMwR2** (Provides k-nearest neighbour imputation functionality used by hRUV::clean())
- **dplyr** (Data manipulation and tabulation of batch and replicate metadata)
- **R** (Runtime environment for executing hRUV and related statistical workflows)

## Examples

```
dat_list = hRUV::clean(dat_list, threshold = 0.5, method = "intersect", assay = "logRaw", newAssay = "rawImpute")
```

## Evaluation signals

- rawImpute assay contains exactly the intersection of metabolites quantified across all five batches (e.g. 61 metabolites), with no missing values and no NaN or Inf entries.
- Metabolite counts match expected post-intersect totals; verify by comparing dim(assay(dat_list[[i]], 'rawImpute')) across all batches returns identical row counts.
- Replicate samples (intra- and inter-batch) cluster together in PCA or hierarchical clustering of the normalised assay, indicating unwanted variation has been removed.
- Batch-level bias (e.g. mean or median metabolite abundance per batch) is harmonised post-normalisation, assessed via boxplots or median absolute deviation (MAD) per batch before and after inter-batch normalisation.
- Absence of NaN, Inf, or extreme outlier values in the final assay; check via summary(assay(dat_clean, 'finalAssay')) and confirm all values are finite and within biological range.

## Limitations

- Requires well-designed replicate embedding (intra-batch and inter-batch) to reliably estimate unwanted variation; sparse or imbalanced replicate designs may lead to unstable RUV-III estimates.
- The intersect method for feature selection is conservative—retains only metabolites quantified in all batches, potentially discarding batch-specific metabolites of biological interest.
- k-nearest neighbour imputation assumes missing values are missing at random (MAR); values missing due to detection limits or batch-specific failures may introduce bias.
- Threshold=0.5 is dataset-specific; applying it unchanged to metabolomics data with different sparsity patterns (e.g. very low-abundance datasets) may over-filter or under-filter metabolites.
- RUV-III parameter k=5 is tuned for the BioHEART-CT study; different replicate structures or batch sizes may require re-optimisation of k.

## Evidence

- [readme] hRUV is a package for normalisation of multiple batches of metabolomics data in a hierarchical strategy: "`hRUV` is a package for normalisation of multiple batches of metabolomics data in a hierarchical strategy with use of samples replicates in large-scale studies."
- [intro] The clean function filters metabolites with >50% missing values per batch using the intersect method, then performs k-nearest neighbour imputation: "The `clean` function filters metabolites with >50% missing values per batch using the intersect method, then performs k-nearest neighbour imputation to produce a rawImpute assay containing 61"
- [intro] Log transformation of raw assay: "assay(dat, "logRaw", withDimnames = FALSE) = log2(assay(dat, "raw") + 1)"
- [intro] Intra-batch normalisation with loess smoothing and RUV-III with k=5: "For intra batch normalisation, we perform loess smoothing on samples and RUV-III using short replicates with parameter k set to 5"
- [intro] Inter-batch normalisation with concatenating hierarchical structure: "For inter batch normalisation, we perform concatenating hierarchical structure using batch replicate samples"
- [readme] The tool utilises 2 types of replicates to estimate unwanted variation within and between batches: "The tool utilises 2 types of replicates: intra-batch and inter-batch replicates to estimate the unwatned variation within and between batches with RUV-III."
- [readme] Novel hierarchical approach to removing unwanted variation by harnessing information from sample replicates embedded in experimental runs/batches: "Our novel tool is a novel hierarchical approach to removing unwanted variation by harnessing information from sample replicates embedded in the seequence of experimental runs/batches and applying"