Data Pipeline Manager

Data Pipeline Manager

Purpose

Guidance for designing, implementing, and troubleshooting data processing pipelines, with emphasis on quality validation, error handling, and reliability. Pipelines are a prevalent source of failure in automated systems: poor data quality, format mismatches, missing files, and transient errors cascade through stages and become hard to diagnose after the fact.

This skill covers the layer between the bioinformatician skill, which implements the specific analysis, and the systems-architect skill, which designs the overall system: pipeline orchestration, validation, and reliability.

When to Use

• Design and implementation: new pipelines from scratch; RNA-seq, ChIP-seq, or other genomics processing; ETL workflows for data integration; ML preprocessing; batch or streaming systems.
• Debugging: tracing errors through multi-stage workflows, diagnosing format mismatches between stages, finding bottlenecks, resolving dependency and ordering issues.
• Quality assurance: validation at each stage, format consistency between stages, integrity checks and checksums, quality metrics tracked over time.
• Error recovery: retry for transient failures, checkpoint and resume, partial failures in parallel processing, rollback, idempotent operations.
• Optimization: parallelizing independent stages, reducing inter-stage data transfer, caching expensive operations, scaling to larger volumes.

When a pipeline investigation spans several genuinely independent tracks — unrelated stages, separate sample batches — fanning the work out can pay off; see ../references/delegation-and-scope.md for when the overhead is worth it.

Pipeline Design

Start by pinning down the contract: input sources with their formats, schemas, and expected volume; each transformation stage's purpose and dependencies; output formats, destinations, and success criteria; resource estimates per stage; and the time, cost, and resource constraints you are working inside. The dependency map tells you which stages are sequential and which can run in parallel, which in turn selects the pattern below.

Build in observability from the start rather than retrofitting it: metrics per stage (records processed, time taken, errors), a logging strategy (what, where, at what level), alerting thresholds, and tracing for distributed components.

Pipeline Patterns

Sequential — each stage consumes the previous stage's output. RNA-seq (FASTQ → QC → alignment → count matrix → differential expression), classic ETL, image processing. Simple to reason about, with natural checkpointing at each stage; the slowest stage is the bottleneck and any failure stops the run. Give each stage a defined input/output contract, validate between stages, and keep intermediate outputs for debugging and resume.

Parallel — independent branches with no dependencies: many samples, several tools over the same input, partitioned data, fan-out of work to workers. High throughput and isolated failures, at the cost of resource management and result aggregation. Partition data deliberately, handle partial failures, and balance load to avoid stragglers.

Conditional — branch on data content or intermediate results: route by data type or quality, skip expensive steps that aren't needed. Efficient and adaptive, but every path needs testing, a consistent output format, and a log line recording which path each item took.

Hybrid — a mix, for multi-omics integration, complex bioinformatics workflows, or enterprise ETL across several sources. Most flexible and hardest to debug; document the DAG explicitly and modularize components.

For anything beyond a simple sequence, use a workflow manager (Snakemake, Nextflow, Airflow) rather than hand-rolled orchestration.

Input Validation

Validate inputs before processing rather than discovering problems mid-run. Structural checks: existence and accessibility, format against its specification (FASTQ, BAM, CSV, JSON), completeness (checksums, file size), schema (columns and data types), non-empty content, text encoding, read/write permissions on every path used.

Content checks go further: numeric values within expected ranges, required fields present and non-null, referential integrity across identifiers, internal consistency between related fields (start < end coordinates), domain quality metrics (quality scores, read lengths), and sample identifiers matching the expected manifest.

Fail fast on validation errors. Report the specific file, line, and field; say what is wrong and how to fix it; stop before consuming expensive compute; log the error so trends are visible across runs.

Concrete validators — FASTQ, CSV, BAM/SAM, DataFrame schema, gene-count matrix, genome coordinates, disk space, and cross-stage sample and dimension consistency — are in references/validation-patterns.md.

Transform

Each stage runs with a clear input/output contract and its own error handling. Practices that make transformations survivable:

• Idempotency: operations safe to retry without side effects.
• Atomic writes: write to a temp path, then rename.
• Batching and streaming: process in memory-sized batches; stream rather than loading whole datasets; chunk or window large files.
• Error context: capture the input data, parameters, and environment alongside the exception.
• Graceful degradation: keep processing remaining records when an individual record fails.
• Checkpointing: persist intermediate results so a failed run resumes rather than restarts.
• Progress and resource logging: log progress every N records; track memory, CPU, and disk to catch resource exhaustion before it becomes an opaque crash.

For large-scale data, add backpressure so a fast stage cannot overwhelm a slow downstream one, use efficient intermediate formats (Parquet, Arrow), and consider a distributed framework (Spark, Dask) when a single machine stops being enough.

Output Validation

Structure: correct format, expected schema, expected dimensions for tabular or array data, all expected files present, file sizes plausible (not empty, not suspiciously large).

Content: values within biologically or logically plausible ranges; no unexpected NaN, Inf, or null; distributions reasonable rather than all-zero or degenerate; logical relationships between output fields; domain-specific sanity checks (gene counts non-negative); key metrics compared against a baseline or previous run.

Report quality metrics with the outputs: record counts (input vs output, filtered, passed), success and error rates by stage, data quality scores and flags, processing time and resource usage, coverage and completeness, and domain metrics such as alignment and duplication rates.

Error Handling

Classify the failure first, since the class determines whether retrying can possibly help: transient (network timeouts, temporary unavailability, rate limits), permanent (format errors, missing required data, logic errors), configuration (wrong parameters, missing credentials, invalid paths), resource (out of memory, disk full, quota exceeded), and data (invalid values, corrupted files, schema mismatches).

Retry applies to transient failures: exponential backoff with jitter (1s, 2s, 4s, 8s), 3-5 retries as a typical ceiling, per-operation timeouts, a circuit breaker that stops retrying once the error rate crosses a threshold, and idempotent operations so a retry cannot double-process.

Logging for diagnosis: structured format (JSON) so logs can be aggregated; include input data, parameters, environment, and stack traces; use appropriate severity levels; carry correlation IDs across distributed components; keep credentials and PII out.

Checkpointing for resumability: save state after expensive operations, record which stages completed and what they produced along with what remains, resume from the last good checkpoint, clean up checkpoints on success, and handle concurrent checkpoint access in distributed runs.

Implementations — exponential backoff, conditional and async retry, file- and database-backed checkpoints, structured logging, circuit breaker, fallback and partial-success patterns, automatic recovery — are in references/error-handling-strategies.md.

Monitoring and Alerting

Track while a pipeline runs: progress through each stage, processing rate (records/bytes per second), input queue depth and backlog, error rate per stage and overall, CPU/memory/disk/network usage, and end-to-end plus per-stage latency.

Alert on failures immediately; on performance degrading below threshold; on quality metrics leaving expected ranges; on approaching resource limits; and on SLAs at risk. Aggregate and prioritize so alerts stay meaningful.

A useful dashboard shows overall health, stage-by-stage progress, error rates and types over time, resource utilization trends, data quality metrics, and comparison against historical performance.

Bioinformatics-Specific Considerations

FASTQ to Count Matrix Pipeline

Standard workflow:

1. Quality Control: FastQC on raw FASTQ files
2. Preprocessing: Trimming adapters and low-quality bases (Trimmomatic, cutadapt)
3. Alignment: Align to reference genome (STAR, HISAT2, BWA)
4. Quality Control: Check alignment metrics (samtools, picard)
5. Quantification: Count reads per gene (featureCounts, HTSeq, RSEM)
6. Count Matrix: Assemble counts into matrix for downstream analysis

Critical validation points:

• Raw read quality scores (median Q30+)
• Adapter contamination levels
• Alignment rate (>70% for RNA-seq)
• Duplicate rate (<30%)
• Gene detection rate (>10,000 genes for human)
• Count distribution (not zero-inflated; mean count >100, coefficient of variation <10)

Genome Build Consistency

Mixing genome builds (hg19 vs hg38, mm9 vs mm10) causes misalignments and silently incorrect results. Document the build in metadata and file headers, validate build consistency across all samples, check the reference FASTA matches the annotation GTF, verify chromosome naming conventions (chr1 vs 1), pass the build as an explicit pipeline parameter, and include it in output file names.

Sample Metadata Tracking

Essential fields: sample identifier (unique across the study), biological condition/phenotype, technical batch, sequencing run, library preparation protocol, genome build and annotation version, processing date and pipeline version.

Validate that sample IDs match between FASTQ files and metadata, that no sample ID is duplicated, that required fields are present, that controlled-vocabulary terms are valid, and that metadata agrees with the data (species matches the reference).

Reference Data Management

Version-control reference genomes and annotations, keep naming consistent, store checksums, document source and version, maintain separate references per genome build, and check reference integrity before a run.

Integration with Other Skills

bioinformatician implements the analysis inside the framework this skill provides: it selects tools and debugs analysis-specific issues and algorithm performance, while this skill supplies the pipeline architecture, validation, error handling, and orchestration around it.

systems-architect designs the surrounding system and infrastructure — scaling, infrastructure reliability, monitoring platforms — while this skill designs the pipeline components, parallelization, reliability, and pipeline-specific monitoring that sit on top.

References

• references/validation-patterns.md — validators for filesystem, formats, schemas, data quality, bioinformatics-specific and cross-stage checks
• references/error-handling-strategies.md — retry, checkpointing, logging, circuit breakers, graceful degradation, recovery
• ../references/delegation-and-scope.md — when to fan work out to subagents

Examples

• examples/rnaseq-pipeline-example.md — full FASTQ → counts pipeline with validation
• examples/pipeline-debugging-example.md — tracing a failed multi-stage run