# Tech Stack

> Describes the primary technologies, frameworks, libraries, and language conventions used in this codebase.

- Skill: `majiayu000/tech-stack` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add majiayu000/tech-stack`
- Raw SKILL.md: https://api.skillmd.com/api/skills/majiayu000/tech-stack/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: majiayu000 (https://skillmd.com/u/majiayu000)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/majiayu000/tech-stack

---


# Tech Stack Overview

## Language-Specific Conventions

- **Python**: See [PYTHON.md](./PYTHON.md) for detailed conventions and examples

## Primary Languages

| Language | Usage | Version |
|----------|-------|---------|
| Python | datagen, notebooks | 3.10+ |
| PySpark | Fabric notebooks | Spark 3.x |
| KQL | Eventhouse queries | N/A |
| JSON/YAML | Fabric item definitions | N/A |

## Frameworks & Libraries

### Data Generation (datagen)
- **DuckDB**: Local analytical database for historical data
- **Faker**: Synthetic data generation
- **Pydantic**: Data validation and event schemas
- **azure-eventhub**: Event streaming to Azure

### Lakehouse
- **Delta Lake**: ACID transactions, schema enforcement
- **PySpark**: Distributed data processing

### Real-Time Analytics
- **Microsoft Fabric Eventhouse**: KQL-based analytics
- **Eventstream**: Event routing and transformation

## Project Architecture

```
Event Flow:
  datagen (Python)
    → Azure Event Hubs
    → Eventstream
    → KQL Tables + Lakehouse Bronze

Data Layers:
  Bronze (raw JSON)
    → Silver (typed Delta)
    → Gold (aggregated Delta)
    → Semantic Model (Power BI)
```

## Key Dependencies

See `datagen/pyproject.toml` for Python dependencies.

Core packages:
- `pydantic` - Schema validation
- `duckdb` - Local analytics
- `faker` - Data generation
- `azure-eventhub` - Event streaming