DDIA — Agent Skill (Designing Data-Intensive Applications)
Rules from Designing Data-Intensive Applications (Martin Kleppmann, O'Reilly 2017). Data is usually the hard part of backend web apps — choose tools and trade-offs explicitly.
Pair with web-api-design at the boundary; domain-driven-design for domain models; software-architecture for style selection.
When to apply
- Choosing database, cache, queue, search, or stream tech
- Data modeling (relational vs document vs graph)
- Replication, sharding, consistency, transactions
- Batch vs stream processing pipelines
- User says: DDIA, data-intensive, replication, partition, consistency, event log
Core laws
- Reliability, scalability, maintainability — every data decision serves these three.
- There is no free lunch — latency, consistency, availability, cost — pick trade-offs.
- Know your access patterns — schema and storage follow queries, not the reverse.
- Assume faults — hardware, software, humans, networks fail.
- Simplest thing that works — one database until proven insufficient.
- Evolve schemas and formats — forward/backward compatibility is architecture.
Agent workflow
1. LOAD — read/write ratio, volume, latency, growth
2. MODEL — relational / document / graph for the domain
3. STORE — OLTP engine, indexes, analytics path if needed
4. DISTRIBUTE — replicate? partition? leader/follower?
5. CONSISTENCY — transactions vs eventual; isolation level
6. DERIVE — caches, search indexes, streams off source of truth
7. OPERATE — backups, monitoring, human error guards
Part I — Foundations
Reliable, scalable, maintainable (Ch 1)
Reliability: faults happen — design to survive (redundancy, retries, graceful degradation).
Scalability: define load (QPS, users, data size) and latency targets; measure percentiles (p99), not averages.
Maintainability:
- Operability — easy for ops to run
- Simplicity — manage complexity (small number of composable ideas)
- Evolvability — easy to change as requirements shift
Data models (Ch 2)
| Model |
Good for |
| Relational |
Joins, constraints, many-to-many, reporting |
| Document |
Aggregate-shaped data, schema flexibility, one-to-few |
| Graph |
Many-to-many relationships, connected data, traversals |
Mismatch smell: ORM fighting schema → wrong model or wrong boundaries.
Query languages: declarative (SQL) vs imperative; MapReduce for large batch.
Storage engines (Ch 3)
- OLTP — row-oriented, B-trees or LSM-trees, indexes
- OLAP — column storage, warehouses, data cubes
- Don't mix transactional and heavy analytics on same DB without plan
Encoding & evolution (Ch 4)
- JSON, Avro, Protobuf — schema evolution matters for services and streams
- Backward/forward compatibility when rolling deploys
Part II — Distributed data
Replication (Ch 5)
- Leader/follower — common default; sync vs async replication
- Replication lag problems: read-your-writes, monotonic reads, consistent prefix
- Multi-leader — multi-datacenter; write conflicts need resolution
- Leaderless — quorums (Dynamo-style); sloppy quorum, hinted handoff
Partitioning (Ch 6)
- By key range vs hash of key — skew/hot spots
- Secondary indexes — global vs document-local
- Rebalancing when cluster grows
Transactions (Ch 7)
ACID — understand what your DB actually guarantees.
Isolation levels: Read Committed → Snapshot Isolation → Serializable (2PL, SSI).
Distributed transactions — 2PC is fragile; prefer sagas / idempotent ops for cross-service (see domain-driven-design).
Distributed systems reality (Ch 8–9)
Fallacies: network not reliable, latency not zero, clocks lie, partial failures normal.
Consensus — for leader election, atomic commit when truly needed.
Don't: assume synchronous cross-service transactions fix everything.
Part III — Derived data
Batch (Ch 10) vs Stream (Ch 11)
|
Batch |
Stream |
| Latency |
Hours/minutes |
Seconds/ms |
| Tooling |
MapReduce, Spark |
Kafka, Flink |
| Use |
Reports, rebuild indexes |
Notifications, sync, CDC |
Change data capture (CDC) — log as source of truth; derive views.
Future of data systems (Ch 12)
- Unbundled databases — specialized tools + derived data keeps sync
- Eventual consistency with end-to-end correctness (constraints, verification)
- Design apps around dataflow, not only request/response
Decision heuristics
| Question |
Heuristic |
| One service, moderate load |
PostgreSQL (or similar) until proven otherwise |
| Need flexible nested docs |
Document DB or JSON column + careful indexing |
| Heavy graph queries |
Graph DB or adjacency in SQL with limits |
| Read-heavy, tolerate stale |
Cache (Redis) + TTL; know invalidation |
| Full-text search |
Dedicated index (Elasticsearch, etc.) — derived from source |
| Cross-service updates |
Saga + idempotency, not 2PC across microservices |
| Analytics |
Separate OLAP path; don't crush OLTP |
| Real-time fan-out |
Stream/log + consumers |
Smells to flag
| Smell |
Fix |
| One DB for everything including BI |
Split OLTP/OLAP |
| No replication on prod data |
Replicas + backup tested |
| Ignoring replication lag in UX |
Read-your-writes strategy |
| Hot partition key |
Reshard, salting |
| Serializable everywhere "just in case" |
Match isolation to need |
| Cache without invalidation plan |
TTL, event-driven invalidation |
| Dual writes to DB and search |
CDC / outbox / single writer |
| Distributed monolith + shared DB |
Partition data with services |
Review output format
## Data profile
[Load, size, read/write, latency needs]
## Model & storage
[Choice + why]
## Distribution
[Replication, partitioning, consistency]
## Failure modes
[What breaks; mitigations]
## Derived systems
[Cache, search, streams]
## Trade-offs accepted
[What we give up and why]
Source
Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017).
1---2name: ddia3description: Apply Martin Kleppmann's Designing Data-Intensive Applications principles when choosing databases, designing data models, replication, partitioning, transactions, streams, or backend scalability. Use for data architecture, consistency, caching, queues, batch/stream processing, or when the user mentions DDIA, data-intensive, replication, or CAP.4---56# DDIA — Agent Skill (Designing Data-Intensive Applications)78Rules from *Designing Data-Intensive Applications* (Martin Kleppmann, O'Reilly 2017). **Data is usually the hard part of backend web apps — choose tools and trade-offs explicitly.**910Pair with **web-api-design** at the boundary; **domain-driven-design** for domain models; **software-architecture** for style selection.1112## When to apply1314- Choosing database, cache, queue, search, or stream tech15- Data modeling (relational vs document vs graph)16- Replication, sharding, consistency, transactions17- Batch vs stream processing pipelines18- User says: DDIA, data-intensive, replication, partition, consistency, event log1920---2122## Core laws23241. **Reliability, scalability, maintainability** — every data decision serves these three.252. **There is no free lunch** — latency, consistency, availability, cost — pick trade-offs.263. **Know your access patterns** — schema and storage follow queries, not the reverse.274. **Assume faults** — hardware, software, humans, networks fail.285. **Simplest thing that works** — one database until proven insufficient.296. **Evolve schemas and formats** — forward/backward compatibility is architecture.3031---3233## Agent workflow3435```361. LOAD — read/write ratio, volume, latency, growth372. MODEL — relational / document / graph for the domain383. STORE — OLTP engine, indexes, analytics path if needed394. DISTRIBUTE — replicate? partition? leader/follower?405. CONSISTENCY — transactions vs eventual; isolation level416. DERIVE — caches, search indexes, streams off source of truth427. OPERATE — backups, monitoring, human error guards43```4445---4647## Part I — Foundations4849### Reliable, scalable, maintainable (Ch 1)5051**Reliability:** faults happen — design to survive (redundancy, retries, graceful degradation).5253**Scalability:** define load (QPS, users, data size) and latency targets; measure percentiles (p99), not averages.5455**Maintainability:**56- **Operability** — easy for ops to run57- **Simplicity** — manage complexity (small number of composable ideas)58- **Evolvability** — easy to change as requirements shift5960### Data models (Ch 2)6162| Model | Good for |63|-------|----------|64| **Relational** | Joins, constraints, many-to-many, reporting |65| **Document** | Aggregate-shaped data, schema flexibility, one-to-few |66| **Graph** | Many-to-many relationships, connected data, traversals |6768**Mismatch smell:** ORM fighting schema → wrong model or wrong boundaries.6970**Query languages:** declarative (SQL) vs imperative; MapReduce for large batch.7172### Storage engines (Ch 3)7374- **OLTP** — row-oriented, B-trees or LSM-trees, indexes75- **OLAP** — column storage, warehouses, data cubes76- **Don't mix** transactional and heavy analytics on same DB without plan7778### Encoding & evolution (Ch 4)7980- JSON, Avro, Protobuf — schema evolution matters for services and streams81- **Backward/forward compatibility** when rolling deploys8283---8485## Part II — Distributed data8687### Replication (Ch 5)8889- **Leader/follower** — common default; sync vs async replication90- **Replication lag** problems: read-your-writes, monotonic reads, consistent prefix91- **Multi-leader** — multi-datacenter; write conflicts need resolution92- **Leaderless** — quorums (Dynamo-style); sloppy quorum, hinted handoff9394### Partitioning (Ch 6)9596- **By key range** vs **hash of key** — skew/hot spots97- **Secondary indexes** — global vs document-local98- **Rebalancing** when cluster grows99100### Transactions (Ch 7)101102**ACID** — understand what your DB actually guarantees.103104**Isolation levels:** Read Committed → Snapshot Isolation → Serializable (2PL, SSI).105106**Distributed transactions** — 2PC is fragile; prefer **sagas** / idempotent ops for cross-service (see domain-driven-design).107108### Distributed systems reality (Ch 8–9)109110**Fallacies:** network not reliable, latency not zero, clocks lie, partial failures normal.111112**Consensus** — for leader election, atomic commit when truly needed.113114**Don't:** assume synchronous cross-service transactions fix everything.115116---117118## Part III — Derived data119120### Batch (Ch 10) vs Stream (Ch 11)121122| | Batch | Stream |123|---|-------|--------|124| Latency | Hours/minutes | Seconds/ms |125| Tooling | MapReduce, Spark | Kafka, Flink |126| Use | Reports, rebuild indexes | Notifications, sync, CDC |127128**Change data capture (CDC)** — log as source of truth; derive views.129130### Future of data systems (Ch 12)131132- **Unbundled databases** — specialized tools + derived data keeps sync133- **Eventual consistency** with **end-to-end correctness** (constraints, verification)134- Design apps around **dataflow**, not only request/response135136---137138## Decision heuristics139140| Question | Heuristic |141|----------|-----------|142| One service, moderate load | PostgreSQL (or similar) until proven otherwise |143| Need flexible nested docs | Document DB or JSON column + careful indexing |144| Heavy graph queries | Graph DB or adjacency in SQL with limits |145| Read-heavy, tolerate stale | Cache (Redis) + TTL; know invalidation |146| Full-text search | Dedicated index (Elasticsearch, etc.) — derived from source |147| Cross-service updates | Saga + idempotency, not 2PC across microservices |148| Analytics | Separate OLAP path; don't crush OLTP |149| Real-time fan-out | Stream/log + consumers |150151---152153## Smells to flag154155| Smell | Fix |156|-------|-----|157| One DB for everything including BI | Split OLTP/OLAP |158| No replication on prod data | Replicas + backup tested |159| Ignoring replication lag in UX | Read-your-writes strategy |160| Hot partition key | Reshard, salting |161| Serializable everywhere "just in case" | Match isolation to need |162| Cache without invalidation plan | TTL, event-driven invalidation |163| Dual writes to DB and search | CDC / outbox / single writer |164| Distributed monolith + shared DB | Partition data with services |165166---167168## Review output format169170```markdown171## Data profile172[Load, size, read/write, latency needs]173174## Model & storage175[Choice + why]176177## Distribution178[Replication, partitioning, consistency]179180## Failure modes181[What breaks; mitigations]182183## Derived systems184[Cache, search, streams]185186## Trade-offs accepted187[What we give up and why]188```189190---191192## Source193194Martin Kleppmann, *Designing Data-Intensive Applications* (O'Reilly, 2017).
poudatmorteza