How do I install the Etl Patterns skill?

Run npx skillmds@latest add majiayu000/etl-patterns in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Etl Patterns safe to use?

This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. Capability flags: docs only. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Etl Patterns?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Etl Patterns free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Etl Patterns?

majiayu000 (@majiayu000) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Etl Patterns by majiayu000 — Agent Skill

Name: Etl Patterns
Rating: 4.666666666666667 (3 reviews)
Author: majiayu000

File contents

ETL Patterns

Orchestrator for production-grade Extract-Transform-Load patterns.

Skill Routing

Need	Skill	Content
Reliability patterns	`etl-core-patterns`	Idempotency, checkpointing, error handling, chunking, retry, logging
Load strategies	`etl-incremental-patterns`	Backfill, timestamp-based, CDC, pipeline orchestration

Pattern Selection Guide

By Reliability Need

Need	Pattern	Skill
Repeatable runs	Idempotency	`etl-core-patterns`
Resume after failure	Checkpointing	`etl-core-patterns`
Handle bad records	Error handling + DLQ	`etl-core-patterns`
Memory management	Chunked processing	`etl-core-patterns`
Network resilience	Retry with backoff	`etl-core-patterns`
Observability	Structured logging	`etl-core-patterns`

By Load Strategy

Scenario	Pattern	Skill
Small tables (<100K)	Full refresh	`etl-incremental-patterns`
Large tables	Timestamp incremental	`etl-incremental-patterns`
Real-time sync	CDC events	`etl-incremental-patterns`
Historical migration	Parallel backfill	`etl-incremental-patterns`
Zero-downtime refresh	Swap pattern	`etl-incremental-patterns`
Multi-step pipelines	Pipeline orchestration	`etl-incremental-patterns`

Quick Reference

Idempotency Options

# Small datasets: Delete-then-insert
# Large datasets: UPSERT on conflict
# Change detection: Row hash comparison

Load Strategy Decision

Is table < 100K rows?
  → Full refresh

Has reliable timestamp column?
  → Timestamp incremental

Source supports CDC?
  → CDC event processing

Need zero downtime?
  → Swap pattern (temp table → rename)

One-time historical load?
  → Parallel backfill with date ranges

Common Pipeline Structure

# 1. Setup
checkpoint = Checkpoint('.etl_checkpoint.json')
processor = ETLProcessor()

# 2. Extract (with incremental)
df = incremental_by_timestamp(source_table, 'updated_at')

# 3. Transform (with error handling)
transformed = processor.process_batch(df.to_dict('records'))

# 4. Load (with idempotency)
upsert_records(pd.DataFrame(transformed))

# 5. Checkpoint
checkpoint.set_last_processed('sync', df['updated_at'].max())

# 6. Handle failures
processor.save_failures('failures/')

Related Skills

data-validation - Validate data quality during ETL
data-quality - Monitor data quality metrics
pandas-coder - DataFrame transformations

---name: etl-patternsdescription: Production ETL patterns orchestrator. Routes to core reliability patterns and incremental load strategies.--- # ETL Patterns Orchestrator for production-grade Extract-Transform-Load patterns. ## Skill Routing | Need | Skill | Content | |------|-------|---------| | Reliability patterns | `etl-core-patterns` | Idempotency, checkpointing, error handling, chunking, retry, logging | | Load strategies | `etl-incremental-patterns` | Backfill, timestamp-based, CDC, pipeline orchestration | ## Pattern Selection Guide ### By Reliability Need | Need | Pattern | Skill | |------|---------|-------| | Repeatable runs | Idempotency | `etl-core-patterns` | | Resume after failure | Checkpointing | `etl-core-patterns` | | Handle bad records | Error handling + DLQ | `etl-core-patterns` | | Memory management | Chunked processing | `etl-core-patterns` | | Network resilience | Retry with backoff | `etl-core-patterns` | | Observability | Structured logging | `etl-core-patterns` | ### By Load Strategy | Scenario | Pattern | Skill | |----------|---------|-------| | Small tables (<100K) | Full refresh | `etl-incremental-patterns` | | Large tables | Timestamp incremental | `etl-incremental-patterns` | | Real-time sync | CDC events | `etl-incremental-patterns` | | Historical migration | Parallel backfill | `etl-incremental-patterns` | | Zero-downtime refresh | Swap pattern | `etl-incremental-patterns` | | Multi-step pipelines | Pipeline orchestration | `etl-incremental-patterns` | ## Quick Reference ### Idempotency Options ```python # Small datasets: Delete-then-insert # Large datasets: UPSERT on conflict # Change detection: Row hash comparison ``` ### Load Strategy Decision ``` Is table < 100K rows? → Full refresh Has reliable timestamp column? → Timestamp incremental Source supports CDC? → CDC event processing Need zero downtime? → Swap pattern (temp table → rename) One-time historical load? → Parallel backfill with date ranges ``` ## Common Pipeline Structure ```python # 1. Setup checkpoint = Checkpoint('.etl_checkpoint.json') processor = ETLProcessor() # 2. Extract (with incremental) df = incremental_by_timestamp(source_table, 'updated_at') # 3. Transform (with error handling) transformed = processor.process_batch(df.to_dict('records')) # 4. Load (with idempotency) upsert_records(pd.DataFrame(transformed)) # 5. Checkpoint checkpoint.set_last_processed('sync', df['updated_at'].max()) # 6. Handle failures processor.save_failures('failures/') ``` ## Related Skills - `data-validation` - Validate data quality during ETL - `data-quality` - Monitor data quality metrics - `pandas-coder` - DataFrame transformations

Frequently asked questions

Etl Patterns