Quality Assessment

Assess ENCODE Data Quality

When to Use

• User asks about data quality, QC metrics, or whether an experiment is reliable
• User wants to filter experiments by quality (FRiP, NSC, RSC, NRF, IDR, TSS enrichment)
• User asks "is this experiment good enough?" or "should I use this data?"
• User needs to interpret ENCODE audit flags (ERROR, NOT_COMPLIANT, WARNING)
• User wants to compare quality across multiple experiments
• User is selecting high-quality experiments for a meta-analysis or aggregation

Help the user evaluate whether ENCODE experiments meet quality standards for their analysis. Quality assessment is not a single-metric exercise — it requires integrating multiple orthogonal measures in the context of the specific assay, biological system, and analytical goals.

Literature Foundation

Step 1: Retrieve Experiment Details and Audit Status

Use encode_get_experiment with the accession to get full metadata including:

• Audit status (ERROR, NOT_COMPLIANT, WARNING, INTERNAL_ACTION)
• Replicate information (biological and technical replicates)
• Pipeline and analysis details (which ENCODE uniform pipeline was used)
• Quality metrics embedded in file objects

encode_get_experiment(accession="ENCSR...")

For batch assessment across multiple experiments:

encode_search_experiments(assay_title="...", organ="...", limit=50)
# Then iterate through results checking audit flags

Step 2: Interpret ENCODE Audit Flags

ENCODE audits are generated by automated validators during the ENCODE uniform processing pipeline (Hitz et al. 2023). They flag experiments by severity:

Common audit categories and what they mean:

Present every audit flag to the user and explain each one. A single ERROR audit does not automatically disqualify an experiment — context matters.

Step 3: Evaluate ChIP-seq Quality (Landt et al. 2012)

The ENCODE ChIP-seq guidelines (Landt et al. 2012) established the foundational metrics still used today. These were developed from analysis of hundreds of ChIP-seq experiments and reflect empirically-derived thresholds.

Core Metrics

Read Depth Requirements

IDR Analysis (Li et al. 2011)

The Irreproducible Discovery Rate provides principled assessment of replicate concordance:

Key insight: IDR thresholded peaks represent peaks passing replicate concordance analysis. Pseudoreplicated peaks = single-replicate fallback (lower confidence). Optimal IDR peaks from pooled data = most complete peak set.

Antibody Validation

ENCODE requires characterization for every antibody:

• Primary: IP followed by mass spectrometry or immunoprecipitation-western
• Secondary: At least one of: knockdown/knockout, motif enrichment, genomic annotation enrichment
• Check the antibody_lot_reviews field in experiment metadata

Step 4: Evaluate ATAC-seq Quality (Buenrostro et al. 2013; Ou et al. 2018)

ATAC-seq has a distinct quality profile driven by the transposase insertion mechanism.

Core Metrics

Read Depth Requirements

Fragment Size Interpretation (Buenrostro et al. 2013)

The fragment size distribution is the signature QC plot for ATAC-seq:

• Sub-nucleosomal (<150bp): Nucleosome-free regions — these are the "open chromatin" signal
• Mono-nucleosomal (~200bp): Single nucleosome wrapped
• Di-nucleosomal (~400bp): Two nucleosomes
• Tri-nucleosomal (~600bp): Three nucleosomes

A clean ATAC-seq library shows a clear nucleosomal ladder. Monotonic decay (no peaks) suggests either dead cells, over-transposition, or excessive DNA damage.

Step 5: Evaluate RNA-seq Quality (Conesa et al. 2016)

Core Metrics

Read Depth Requirements (Conesa et al. 2016)

Strand Specificity

ENCODE RNA-seq data may be stranded or unstranded:

• Stranded: Can distinguish sense vs antisense transcription. Required for accurate quantification of overlapping genes.
• Unstranded: Cannot resolve strand of origin. Check run_type and library_strand_specificity in metadata.

Step 6: Evaluate WGBS Quality (ENCODE data standards)

Core Metrics

Platform Considerations (Foox et al. 2021)

The SEQC2 EpiQC benchmark found significant platform effects:

• Different sequencing platforms (Illumina HiSeq, NovaSeq, MGI) can produce systematically different methylation calls
• Cross-platform comparisons require careful normalization
• RRBS (Reduced Representation Bisulfite Sequencing) covers only ~10% of CpGs — do NOT combine RRBS with WGBS directly

Coverage Requirements

Step 7: Evaluate Hi-C Quality (Yardimci et al. 2019)

Core Metrics

Read Depth and Resolution

Note: Hi-C resolution is not just about read depth — it also depends on restriction enzyme site density, ligation efficiency, and fragment size distribution. In situ Hi-C (Rao et al. 2014) generally produces cleaner data than dilution Hi-C.

Step 8: Evaluate CUT&RUN / CUT&Tag Quality (Skene & Henikoff 2017; Kaya-Okur et al. 2019)

These newer profiling methods have distinct quality profiles from ChIP-seq.

Key Differences from ChIP-seq

CUT&RUN Quality Metrics

CUT&RUN Suspect List (Nordin et al. 2023)

CUT&RUN and CUT&Tag generate artifacts at specific genomic regions (distinct from the ENCODE Blacklist). These are regions with apparent enrichment that is not target-specific:

• Use the CUT&RUN suspect list (Nordin et al. 2023) IN ADDITION to the ENCODE Blacklist
• Available at: https://github.com/Boyle-Lab/CUT-RUN_suspect_list
• Particularly important for H3K4me3 and H3K27me3 CUT&RUN data

Step 9: Evaluate Single-Cell Quality (scRNA-seq and scATAC-seq)

ENCODE includes scRNA-seq and scATAC-seq experiments (primarily 10X Chromium platform). Single-cell data has distinct quality metrics from bulk assays, focused on per-cell quality rather than per-experiment signal-to-noise.

scRNA-seq Quality Metrics

scATAC-seq Quality Metrics

Single-Cell-Specific Quality Pitfalls

• Ambient RNA contamination (scRNA-seq): Cell-free RNA from lysed cells during droplet capture inflates apparent expression of highly-expressed genes across all cells. Use CellBender (best-in-class), SoupX, or DecontX to estimate and remove ambient contamination BEFORE downstream analysis.
• Barcode multiplets (scATAC-seq): Multiple cells per droplet inflate fragment counts and blur cell-type signals. ArchR and SnapATAC2 include doublet detection modules.
• Cell-type composition bias: Quality metrics vary by cell type. A "low-quality" cell may be a small immune cell, not a damaged cell. Apply adaptive QC thresholds per cluster (e.g., miQC) rather than global cutoffs.
• Batch effects across donors: For ENCODE tissue scRNA-seq from multiple donors, batch correction (Harmony, scVI) is typically needed before integration. The single-cell-encode skill covers integration workflows.

Where to Find Single-Cell Quality in ENCODE

Single-cell experiments in ENCODE include cell-level quality summaries in their metadata. Check:

encode_get_experiment(accession="ENCSR...")

Look for audit flags — ENCODE applies automated QC checks including minimum cell counts, minimum genes per cell, and maximum doublet rates. Also check the replicates section for library preparation details.

Step 10: Assess Replication

ENCODE requires minimum 2 independent biological replicates for released data.

Replicate Types and Their Meaning

Concordance Assessment

Use encode_list_files to check for replicated peak files:

Cross-Replicate Correlation

For quantitative data (RNA-seq, signal tracks):

• Spearman correlation ≥0.9: Excellent concordance
• 0.8-0.9: Acceptable, check for outliers
• <0.8: Investigate — batch effect, sample mix-up, or biological variation

Step 11: Apply the ENCODE Blacklist (Amemiya et al. 2019)

Before interpreting any peak-based quality metric, confirm that the ENCODE Blacklist has been applied:

• Blacklist v2 regions: High-signal artifacts (satellite repeats, centromeric regions, high-copy sequences)
• Available at: https://github.com/Boyle-Lab/Blacklist/
• hg38: hg38-blacklist.v2.bed.gz (910 regions)
• mm10: mm10-blacklist.v2.bed.gz

Failure to remove blacklisted regions will inflate FRiP, create false peaks, and confound enrichment analyses. If analyzing CUT&RUN/CUT&Tag, also apply the CUT&RUN suspect list (Nordin et al. 2023).

Step 12: File Quality Tiers and Selection

When listing files with encode_list_files, use quality-informed selection:

# Get preferred default files (ENCODE's recommendation)
encode_list_files(experiment_accession="ENCSR...", preferred_default=True)

# Get IDR thresholded peaks (gold standard for ChIP-seq)
encode_list_files(experiment_accession="ENCSR...", output_type="IDR thresholded peaks", assembly="GRCh38")

# Get signal tracks for visualization
encode_list_files(experiment_accession="ENCSR...", output_type="fold change over control", assembly="GRCh38")

File Quality Hierarchy

Step 13: Summarize Quality Verdict

Provide a structured quality assessment:

Quality Tiers

Quality Summary Template

For each experiment assessed, provide:

1. Accession: ENCSR...
2. Assay: type and target
3. Audit status: list all flags with explanations
4. Key metrics: table with values and pass/fail
5. Replication: number and type of replicates, concordance
6. Pipeline: which ENCODE uniform pipeline version was used
7. Verdict: tier assignment with justification
8. Caveats: any specific limitations to note

Pitfalls and Common Mistakes

1. Single-metric decisions: No single metric captures quality. FRiP alone can be misleading — some TF ChIP-seq with biological signal has low FRiP due to focal binding patterns. Always evaluate collectively.

2. Comparing across assays: Do NOT compare ChIP-seq metrics to ATAC-seq metrics to CUT&RUN metrics. Each assay has its own quality profile and thresholds.

3. Ignoring batch effects: Experiments from different labs, dates, or platforms may have systematic quality differences. When combining data, check for batch-correlated quality variation.

4. Assembly mismatch: Quality metrics computed on different assemblies (hg19 vs GRCh38) may differ slightly. Always verify the assembly of quality metrics matches your analysis assembly.

5. Antibody lot variation: The same antibody target can show different enrichment across lots. Check antibody_lot_reviews in ENCODE metadata.

6. Read depth ≠ quality: A deeply sequenced bad library is still a bad library. Check NRF/PBC first — if complexity is exhausted, more sequencing wastes resources.

7. Control quality matters: An IP library is only as good as its control. Poor input/IgG control undermines all downstream peak-based metrics.

8. Newer assays, different rules: CUT&RUN and CUT&Tag have inherently different quality profiles from ChIP-seq. Applying ChIP-seq thresholds to CUT&RUN will incorrectly flag high-quality data.

Walkthrough: Quality Assessment of ENCODE ChIP-seq Before Analysis

Goal: Evaluate the quality of ENCODE ChIP-seq experiments against ENCODE consortium standards before including them in downstream analysis. Context: Not all ENCODE experiments meet the highest quality standards. Quality assessment prevents garbage-in-garbage-out in aggregation and integration analyses.

Step 1: Get experiment details and audit status

encode_get_experiment(accession="ENCSR000AKA")

Expected output:

{
  "accession": "ENCSR000AKA",
  "assay_title": "Histone ChIP-seq",
  "target": "H3K27ac",
  "biosample_summary": "GM12878",
  "replicates": 2,
  "status": "released",
  "audit": {"ERROR": 0, "NOT_COMPLIANT": 0, "WARNING": 1}
}

Interpretation: 0 ERRORs and 0 NOT_COMPLIANT = experiment meets ENCODE standards. 1 WARNING is acceptable.

Step 2: Check file-level quality

encode_list_files(accession="ENCSR000AKA", file_format="bed", output_type="IDR thresholded peaks", assembly="GRCh38")

Step 3: Review quality metrics

Key ChIP-seq quality thresholds (Landt et al. 2012):

Step 4: Track quality-verified experiments

encode_track_experiment(accession="ENCSR000AKA", notes="QC PASSED: FRiP=3.2%, NSC=1.12, RSC=0.95, 0 audit errors")

Integration with downstream skills

• Quality-filtered experiments feed into histone-aggregation and other aggregation skills
• QC metrics inform pipeline-chipseq parameter tuning
• Audit status guides search-encode experiment selection
• QC documentation supports data-provenance and scientific-writing

Code Examples

1. Check experiment audit status

encode_get_experiment(accession="ENCSR000AKA")

Expected output:

{
  "accession": "ENCSR000AKA",
  "audit": {"ERROR": 0, "NOT_COMPLIANT": 0, "WARNING": 1}
}

2. List files to check quality metrics

encode_list_files(accession="ENCSR000AKA", file_format="bed", assembly="GRCh38")

Expected output:

{
  "files": [
    {"accession": "ENCFF001ABC", "output_type": "IDR thresholded peaks", "file_size_mb": 1.1}
  ]
}

3. Get detailed file info for QC

encode_get_file_info(accession="ENCFF001ABC")

Expected output:

{
  "accession": "ENCFF001ABC",
  "file_format": "bed narrowPeak",
  "output_type": "IDR thresholded peaks",
  "assembly": "GRCh38",
  "quality_metrics": {"frip": 0.032, "nsc": 1.12, "rsc": 0.95}
}

Integration

Related Skills

• publication-trust: Assess scientific integrity of publications before relying on their methods or findings — complements experiment quality with publication quality
• histone-aggregation: Uses quality filtering before aggregating peaks
• accessibility-aggregation: ATAC-seq quality directly impacts aggregation
• data-provenance: Log quality decisions and thresholds used
• compare-biosamples: Quality must be comparable across samples being compared
• integrative-analysis: All data sources need quality assessment before integration
• single-cell-encode: Quality metrics for scRNA-seq and scATAC-seq (genes/cell, fragments/cell, TSS enrichment)
• epigenome-profiling: Quality assessment is a prerequisite for epigenomic profile assembly
• variant-annotation: Quality of ENCODE experiments determines reliability of variant annotation
• pipeline-guide: Pipeline version affects quality metric computation
• batch-analysis: Batch QC screening across multiple experiments for systematic quality filtering

Presenting Results

• Present QC metrics as a traffic-light table: metric | value | threshold | status (PASS/WARN/FAIL). Always include the ENCODE audit level. Suggest: "Would you like to filter to only experiments meeting all QC thresholds?"

For the request: "$ARGUMENTS"