# Iceberg Advanced

> Model, implement, and orchestrate the Write-Audit-Publish (WAP) pattern natively using Apache Iceberg branches.

- Skill: `2dmurali/iceberg-advanced` (Agent Skill)
- Install (CLI): `npx skillmds@latest add 2dmurali/iceberg-advanced`
- Raw SKILL.md: https://api.skillmd.com/api/skills/2dmurali/iceberg-advanced/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: 2dmurali (https://skillmd.com/u/2dmurali)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/2dmurali/iceberg-advanced

---


# Skill: Advanced Iceberg & WAP (Write-Audit-Publish)

## Description
Enables the agent to model, implement, and orchestrate the Write-Audit-Publish (WAP) pattern natively using Apache Iceberg branches. 

## Context
In an Enterprise data lakehouse, writing directly to production tables (`main` branch) during ETL processes poses a massive risk of data corruption. Apache Iceberg solves this natively with Git-like branching. The agent must implement WAP patterns over traditional staging tables.

## Instructions

### 1. WAP Pattern Flow
Instead of creating `_stg` tables, instruct the user to use Iceberg branches.

**Step 1: Create a Branch**
Create a branch pointing to the current state of `main`.
```sql
ALTER TABLE prod.sales.transactions CREATE BRANCH etl_job_123;
```

**Step 2: Configure Spark to Write to Branch**
Spark must be told to write to the specific branch.
```python
# PySpark Example
spark.conf.set("spark.wap.branch", "etl_job_123")

# Now any INSERT or MERGE INTO statement goes to the branch
df.writeTo("prod.sales.transactions").append()
```

**Step 3: Audit / Data Quality Check**
Read from the branch to verify row counts, nulls, and constraints.
```sql
SELECT count(*) FROM prod.sales.transactions VERSION AS OF 'etl_job_123';
-- Trigger Soda or Great Expectations on this branch
```

**Step 4: Publish (Fast-Forward)**
Once the audit passes, publish the branch back to main.
```sql
CALL catalog.system.fast_forward('prod.sales.transactions', 'main', 'etl_job_123');
```

**Step 5: Cleanup**
```sql
ALTER TABLE prod.sales.transactions DROP BRANCH etl_job_123;
```

### 2. Hidden Partitioning
Iceberg uses Hidden Partitioning. If a user tries to partition by `year` or `month` by extracting it manually:
- ❌ **Bad:** `df.withColumn("year", year("ts")).write.partitionBy("year")`
- ✅ **Good:** `CREATE TABLE ... PARTITIONED BY (years(ts))`

Explain to the user that Iceberg handles the partition extraction automatically without needing explicit columns.

## Output Format: Iceberg WAP Implementation Guide
```markdown
### 🛡️ Write-Audit-Publish (WAP) Plan: `[Table Name]`

#### 1. Branch Creation
Run: `ALTER TABLE ... CREATE BRANCH ...`

#### 2. Spark Write Configuration
Ensure your SparkSession has: `spark.wap.branch=[branch_name]`

#### 3. Audit Queries
- Run: `SELECT ... VERSION AS OF '[branch_name]'`
- Expected: 0 Nulls.

#### 4. Publish Command
Run: `CALL catalog.system.fast_forward(...)`
```