How do I install the Data Intensive skill?

Run npx skillmds@latest add claude-dev-suite/data-intensive in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Data Intensive safe to use?

This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. Capability flags: docs only. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Data Intensive?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Data Intensive free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Data Intensive?

claude-dev-suite (@claude-dev-suite) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Data Intensive by claude-dev-suite — Agent Skill

Name: Data Intensive
Rating: 4.2 (5 reviews)
Author: claude-dev-suite

Data Intensive

Data-intensive / data-platform architecture: warehouse vs lakehouse, OLTP vs OLAP, batch vs streaming (lambda/kappa), change-data-capture, and data mesh. Architect-level platform-shape decisions, not SQL or ETL code. USE WHEN: designing a data platform/pipeline architecture, "data warehouse", "lakehouse", "OLAP vs OLTP", "lambda/kappa", "streaming vs batch", "CDC", "data mesh", "medallion", analytics platform, columnar store, ingestion topology. DO NOT USE FOR: writing SQL/ORM (use database skills); a specific ETL tool (use data/data-processing skills); storage-engine internals (use `storage-engines`).

claude-dev-suite Updated Sep 8, 2026 28 repo stars

File contents

Data-Intensive Architecture

OLTP vs OLAP — separate the workloads

OLTP: many small row-oriented transactions, low latency, normalized.
OLAP: few large column-oriented scans/aggregations, throughput-oriented. Don't run heavy analytics on the OLTP store — replicate/ETL/CDC into an analytics store. Columnar formats (Parquet/ORC) + columnar engines win OLAP.

Storage architecture choice

Option	Idea	Fits
Warehouse (Snowflake, BigQuery, Redshift)	Managed columnar SQL store	Structured BI, governance, SQL-first
Data lake (object storage + files)	Cheap, schema-on-read, any format	Raw/varied data, ML feature sources
Lakehouse (Delta/Iceberg/Hudi on object storage)	Lake economics + ACID tables + time travel	Unified BI + ML, open formats, avoids lock-in

Lakehouse (open table formats) is the common 2026 default when you want one copy of data serving both BI and ML without warehouse lock-in.

Batch vs streaming topology

Batch: periodic, simple, high-latency. Streaming (Kafka/Flink): continuous, low-latency, more complex (exactly-once, watermarks, state).
Lambda: batch layer (accurate) + speed layer (fresh) merged — two codebases, complexity. Kappa: stream-only, reprocess from the log — simpler; prefer it unless you truly need a separate batch layer.
CDC (Debezium): stream row changes out of OLTP without dual-writes — the clean way to feed lake/warehouse/search in near-real-time.

Modeling & ownership

Medallion (bronze/silver/gold) layering for lake/lakehouse refinement.
Data mesh: domain-owned data products + federated governance — organizational, for large orgs; overkill for small teams (start centralized).

When to recommend what

BI on structured data, SQL team → warehouse (or lakehouse w/ SQL engine).
Mixed BI + ML, open formats, scale → lakehouse (Iceberg/Delta) + Kappa + CDC.
Real-time decisions → streaming (Kafka + Flink), exactly-once where it matters.
Don't reach for data mesh until org scale forces decentralization.

---name: data-intensivedescription: Data-intensive / data-platform architecture: warehouse vs lakehouse, OLTP vs OLAP, batch vs streaming (lambda/kappa), change-data-capture, and data mesh. Architect-level platform-shape decisions, not SQL or ETL code. USE WHEN: designing a data platform/pipeline architecture, "data warehouse", "lakehouse", "OLAP vs OLTP", "lambda/kappa", "streaming vs batch", "CDC", "data mesh", "medallion", analytics platform, columnar store, ingestion topology. DO NOT USE FOR: writing SQL/ORM (use database skills); a specific ETL tool (use data/data-processing skills); storage-engine internals (use `storage-engines`).---# Data-Intensive Architecture## OLTP vs OLAP — separate the workloads- **OLTP**: many small row-oriented transactions, low latency, normalized.- **OLAP**: few large column-oriented scans/aggregations, throughput-oriented.Don't run heavy analytics on the OLTP store — replicate/ETL/CDC into ananalytics store. Columnar formats (Parquet/ORC) + columnar engines win OLAP.## Storage architecture choice| Option | Idea | Fits ||---|---|---|| **Warehouse** (Snowflake, BigQuery, Redshift) | Managed columnar SQL store | Structured BI, governance, SQL-first || **Data lake** (object storage + files) | Cheap, schema-on-read, any format | Raw/varied data, ML feature sources || **Lakehouse** (Delta/Iceberg/Hudi on object storage) | Lake economics + ACID tables + time travel | Unified BI + ML, open formats, avoids lock-in |Lakehouse (open table formats) is the common 2026 default when you want onecopy of data serving both BI and ML without warehouse lock-in.## Batch vs streaming topology- **Batch**: periodic, simple, high-latency. **Streaming** (Kafka/Flink): continuous, low-latency, more complex (exactly-once, watermarks, state).- **Lambda**: batch layer (accurate) + speed layer (fresh) merged — two codebases, complexity. **Kappa**: stream-only, reprocess from the log — simpler; prefer it unless you truly need a separate batch layer.- **CDC** (Debezium): stream row changes out of OLTP without dual-writes — the clean way to feed lake/warehouse/search in near-real-time.## Modeling & ownership- **Medallion** (bronze/silver/gold) layering for lake/lakehouse refinement.- **Data mesh**: domain-owned data products + federated governance — organizational, for large orgs; overkill for small teams (start centralized).## When to recommend what- BI on structured data, SQL team → warehouse (or lakehouse w/ SQL engine).- Mixed BI + ML, open formats, scale → lakehouse (Iceberg/Delta) + Kappa + CDC.- Real-time decisions → streaming (Kafka + Flink), exactly-once where it matters.- Don't reach for data mesh until org scale forces decentralization.

Frequently asked questions