ETL & Pipelines

Runs NGS read QC, alignment, and BAM processing, wrapping FastQC, BWA/Bowtie2/Minimap2, SAMtools, and MultiQC for automated read-to-BAM workflows.

Downloads genomes, genes, virus sequences, and taxonomy data from NCBI using the datasets and dataformat CLI tools.

Builds a configurable scraping agent that collects data from APIs, HTML, or RSS, enriches it with Gemini AI scoring, and stores results in Notion, Google Sheets, Supabase, or local files.

Provides dbt patterns for data transformation and analytics engineering, including model structures, incremental models, and testing.

Transforms raw data into analytics-ready models using dbt, covering models, tests, macros, sources, snapshots, documentation, and packages.

Query, filter, transform, and aggregate JSON data using jq, with practical patterns for shell pipelines and CLI integration.

Process in-memory datasets with Polars' expression API, lazy evaluation, and parallel execution, including pandas migration patterns and I/O for CSV, Parquet, and JSON.

Process large ML datasets in parallel across CPU or GPU clusters, with streaming execution, multi-format I/O, and integration with Ray Train, PyTorch, and TensorFlow for batch inference and preprocessing pipelines.

Guides database design, implementation, query optimization, migrations, data pipeline development, and operations across SQL and NoSQL platforms.