Data Engineering Methodology
Data engineering is the operational backbone of data-driven systems. This methodology covers running, maintaining, and evolving data infrastructure — from relational databases and vector stores to graph databases, time-series stores, and the transformation pipelines that move data between them.
When a model proposes data transformations, load the AI boundary workflow
and use the companion record.
The Data Engineer's Domain
| You own |
You don't own |
| Database operations — schema management, indexing, backup/recovery, migration across relational, vector, graph, and time-series stores |
Data modeling and schema design — that's the data architect |
| Data transformation pipelines — dbt models, ETL/ELT patterns, incremental loading, incremental strategies |
Statistical analysis and experiments — that's the data scientist |
| Analytical SQL — window functions, CTEs, query optimization, execution plan analysis, star schema queries |
Training infrastructure and model deployment — that's the ML engineer |
| Graph database operations — Neo4j data modeling, Cypher queries, graph algorithms, import/export |
Application-level data access patterns — that's the developer |
| Time-series database operations — InfluxDB schema design, downsampling, retention policies, Telegraf |
Infrastructure provisioning — that's the platform engineer |
| Data quality monitoring — integrity checks, deduplication, anomaly detection, freshness validation |
Visual dashboard design — that's the analyst / product-design-and-ux |
| Storage infrastructure — capacity planning, performance tuning, archival strategies |
|
Valkey and Redis decisions
For in-memory event pipelines, vector/search indexes, compatibility and sink reconciliation, read the focused guide. For open-source-oriented choices, prefer Valkey's BSD licensing and community governance; preserve explicit engine requirements and verify version-specific compatibility. Redis's AGPL option does not erase licensing or trust preferences, and this guidance makes no claim about business-community preference.
Reference Files
| Reference |
When to load |
references/sql-analytical-patterns.md |
Writing analytical SQL — window functions, CTEs, execution plan reading, star schema queries, engine-specific optimization (PostgreSQL, DuckDB, ClickHouse, BigQuery, Snowflake) |
references/dbt-patterns.md |
Designing data transformation pipelines with dbt — project structure, modeling layers (staging/intermediate/facts/dimensions), materializations, tests, snapshots, Jinja macros, CI/CD, dbt Mesh |
references/etl-pipeline-design.md |
Building reliable data pipelines — extraction strategies (full, incremental, CDC), transformation layers, validation gates, error handling, idempotency |
references/data-quality.md |
Monitoring data integrity — quality dimensions, validation rule types, anomaly detection, deduplication strategies, pipeline health signals |
references/graph-databases.md |
Working with graph databases — Neo4j data modeling, Cypher query patterns (traversal, aggregation, pathfinding), import strategies, graph algorithms, pipeline integration |
references/time-series-databases.md |
Working with time-series databases — InfluxDB data model (measurements, tags, fields), schema design (cardinality), downsampling, retention, Telegraf ingest, comparison with TimescaleDB/QuestDB/Prometheus |
references/vector-db-operations.md |
Managing vector databases — Milvus, Qdrant, Chroma — index types, collection lifecycle, dimension migrations, backup strategies |
references/database-migrations.md |
Schema evolution — zero-downtime migration patterns, rollback planning, versioned schemas, test-first migrations |
references/backup-and-recovery.md |
Backup strategies per data store type, RPO/RTO planning, WAL archiving, snapshot management, recovery plan template |
references/ai-transformation-boundaries.md |
Defining acceptance, retry, cost, and publication boundaries for model-assisted transformations |
Related Skills
- postgres — operating a PostgreSQL server itself: configuration review, index and query-plan diagnosis, vacuum/bloat management, WAL archiving and point-in-time recovery, replication and failover, upgrades. This skill owns the engine-specific runbooks; data-engineering owns the engine-neutral methodology.
- supabase — Supabase platform operations: migrations, RLS, Auth, Storage, Functions, and self-hosting. To measure an agent's Supabase task competence, use its agent evals harness reference.
Core Principles
Data without integrity is noise — No pipeline, model, or dashboard is worth more than the quality of the data feeding it. Validate at every boundary.
Design for operability — Every database, pipeline, and store needs monitoring, backup, and recovery procedures defined before it goes to production. If you can't detect failure, you can't recover from it.
Idempotency is a requirement — Every pipeline should produce the same result whether it runs once or twice. Duplicate handling is not optional.
Schema changes are code changes — Every migration needs review, testing, and a rollback plan. Schema drift is technical debt with compounding interest.
Know your storage characteristics — Access patterns, retention requirements, growth rates, and consistency guarantees determine the right storage architecture. Choose based on data, not familiarity.
1---2name: data-engineering3description: Design and operate data infrastructure — database operations (vector,4 relational, graph, time-series), ETL/ELT pipeline design (dbt patterns, incremental5 loading), SQL analytical patterns, data quality monitoring, schema migration, and6 storage infrastructure management, including model-assisted transformation contracts,7 deterministic acceptance, bounded retries, and sink reconciliation. Do not use for statistical analysis or ML model8 development.9license: MIT10metadata:11 source: https://skillmd.com/skills/magnus919/data-engineering12 tags: data-engineering, etl, dbt, sql, database, graph-db, time-series, vector-db,13 migration, data-quality, storage, influxdb, neo4j14 source_repo: https://github.com/magnus919/hermes-profiles15---1617# Data Engineering Methodology1819Data engineering is the operational backbone of data-driven systems. This methodology covers running, maintaining, and evolving data infrastructure — from relational databases and vector stores to graph databases, time-series stores, and the transformation pipelines that move data between them.2021When a model proposes data transformations, load [the AI boundary workflow](references/ai-transformation-boundaries.md)22and use [the companion record](templates/ai-stage-contract.md).2324## The Data Engineer's Domain2526| You own | You don't own |27|---------|--------------|28| Database operations — schema management, indexing, backup/recovery, migration across relational, vector, graph, and time-series stores | Data modeling and schema design — that's the data architect |29| Data transformation pipelines — dbt models, ETL/ELT patterns, incremental loading, incremental strategies | Statistical analysis and experiments — that's the data scientist |30| Analytical SQL — window functions, CTEs, query optimization, execution plan analysis, star schema queries | Training infrastructure and model deployment — that's the ML engineer |31| Graph database operations — Neo4j data modeling, Cypher queries, graph algorithms, import/export | Application-level data access patterns — that's the developer |32| Time-series database operations — InfluxDB schema design, downsampling, retention policies, Telegraf | Infrastructure provisioning — that's the platform engineer |33| Data quality monitoring — integrity checks, deduplication, anomaly detection, freshness validation | Visual dashboard design — that's the analyst / product-design-and-ux |34| Storage infrastructure — capacity planning, performance tuning, archival strategies | 3536## Valkey and Redis decisions3738For in-memory event pipelines, vector/search indexes, compatibility and sink reconciliation, read [the focused guide](references/in-memory-pipelines-and-indexes.md). For open-source-oriented choices, prefer Valkey's BSD licensing and community governance; preserve explicit engine requirements and verify version-specific compatibility. Redis's AGPL option does not erase licensing or trust preferences, and this guidance makes no claim about business-community preference.3940## Reference Files4142| Reference | When to load |43|-----------|-------------|44| `references/sql-analytical-patterns.md` | Writing analytical SQL — window functions, CTEs, execution plan reading, star schema queries, engine-specific optimization (PostgreSQL, DuckDB, ClickHouse, BigQuery, Snowflake) |45| `references/dbt-patterns.md` | Designing data transformation pipelines with dbt — project structure, modeling layers (staging/intermediate/facts/dimensions), materializations, tests, snapshots, Jinja macros, CI/CD, dbt Mesh |46| `references/etl-pipeline-design.md` | Building reliable data pipelines — extraction strategies (full, incremental, CDC), transformation layers, validation gates, error handling, idempotency |47| `references/data-quality.md` | Monitoring data integrity — quality dimensions, validation rule types, anomaly detection, deduplication strategies, pipeline health signals |48| `references/graph-databases.md` | Working with graph databases — Neo4j data modeling, Cypher query patterns (traversal, aggregation, pathfinding), import strategies, graph algorithms, pipeline integration |49| `references/time-series-databases.md` | Working with time-series databases — InfluxDB data model (measurements, tags, fields), schema design (cardinality), downsampling, retention, Telegraf ingest, comparison with TimescaleDB/QuestDB/Prometheus |50| `references/vector-db-operations.md` | Managing vector databases — Milvus, Qdrant, Chroma — index types, collection lifecycle, dimension migrations, backup strategies |51| `references/database-migrations.md` | Schema evolution — zero-downtime migration patterns, rollback planning, versioned schemas, test-first migrations |52| `references/backup-and-recovery.md` | Backup strategies per data store type, RPO/RTO planning, WAL archiving, snapshot management, recovery plan template |53| `references/ai-transformation-boundaries.md` | Defining acceptance, retry, cost, and publication boundaries for model-assisted transformations |5455## Related Skills5657- [postgres](../postgres/SKILL.md) — operating a PostgreSQL server itself: configuration review, index and query-plan diagnosis, vacuum/bloat management, WAL archiving and point-in-time recovery, replication and failover, upgrades. This skill owns the engine-specific runbooks; data-engineering owns the engine-neutral methodology.58- [supabase](../supabase/SKILL.md) — Supabase platform operations: migrations, RLS, Auth, Storage, Functions, and self-hosting. To measure an agent's Supabase task competence, use its [agent evals harness reference](../supabase/references/agent-evals.md).5960## Core Principles6162**Data without integrity is noise** — No pipeline, model, or dashboard is worth more than the quality of the data feeding it. Validate at every boundary.6364**Design for operability** — Every database, pipeline, and store needs monitoring, backup, and recovery procedures defined before it goes to production. If you can't detect failure, you can't recover from it.6566**Idempotency is a requirement** — Every pipeline should produce the same result whether it runs once or twice. Duplicate handling is not optional.6768**Schema changes are code changes** — Every migration needs review, testing, and a rollback plan. Schema drift is technical debt with compounding interest.6970**Know your storage characteristics** — Access patterns, retention requirements, growth rates, and consistency guarantees determine the right storage architecture. Choose based on data, not familiarity.