# Data Lake Architecture

> Data Lake & Big Data Architecture

- Skill: `j4flmao/data-lake-architecture` (Agent Skill)
- Install (CLI): `npx skillmds@latest add j4flmao/data-lake-architecture`
- Raw SKILL.md: https://api.skillmd.com/api/skills/j4flmao/data-lake-architecture/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: j4flmao (https://skillmd.com/u/j4flmao)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/j4flmao/data-lake-architecture

---

# Data Lake & Big Data Architecture

## Core Concepts

Modern data architectures separate compute from storage to process Petabytes of data affordably.

### 1. Data Lake vs Data Warehouse
- **Data Warehouse:** Structured data, Schema-on-Write (ETL). Expensive, fast queries (Snowflake, BigQuery).
- **Data Lake:** Raw, unstructured data, Schema-on-Read (ELT). Cheap storage in AWS S3/GCS using optimized columnar formats like Apache Parquet.

### 2. Lakehouse Architecture (Delta Lake / Apache Iceberg)
Combines the cheap storage of Data Lakes with the ACID transactions of Data Warehouses. It allows you to run `UPDATE` and `DELETE` commands directly on Parquet files in S3 by maintaining a transaction log.

### Data Pipeline Map

```mermaid
%%{init: {"theme": "default", "flowchart": {"useMaxWidth": true}}}%%
flowchart TD
    subgraph Ingestion ["Data Sources"]
        A["Postgres (CDC via Debezium)"]
        B["App Logs (Kafka)"]
    end
    
    subgraph Storage ["Data Lake (S3/GCS)"]
        C["Bronze (Raw JSON/CSV)"]
        D["Silver (Cleaned Parquet)"]
        E["Gold (Aggregated Delta Lake)"]
    end
    
    subgraph Compute ["Spark / Databricks"]
        F["Batch Processing Job"]
        G["Streaming Processing Job"]
    end
    
    A --> C
    B --> C
    C -->|"Extract & Clean"| F
    F -->|"Load"| D
    D -->|"Aggregate"| G
    G -->|"Load"| E
```