# Finops

> Analyze, attribute, and reduce cloud costs associated with Lakehouse architecture (Storage, Compute, API operations).

- Skill: `2dmurali/finops` (Agent Skill)
- Install (CLI): `npx skillmds@latest add 2dmurali/finops`
- Raw SKILL.md: https://api.skillmd.com/api/skills/2dmurali/finops/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: 2dmurali (https://skillmd.com/u/2dmurali)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/2dmurali/finops

---


# Skill: Enterprise Lakehouse FinOps & Cost Management

## Description
Enables the AI agent to analyze, attribute, and reduce cloud costs associated with Lakehouse architecture (Storage, Compute, and API operations).

## Context
Lakehouses can quickly become expensive due to stale snapshots, orphaned files, excessive API calls (e.g., S3 PUT/GET), and unoptimized clusters. This skill teaches the agent how to implement strict FinOps practices across Databricks, AWS EMR, and Snowflake.

## Instructions

### 1. Storage Optimization (Snapshot & Orphan Cleanup)
Data table versions and snapshots accrue over time. Recommend cleanup thresholds based on environment.
- **Production:** Retain 7-14 days for Time Travel.
- **Dev/Test:** Retain 1-3 days.

**Delta Lake Cleanup:**
```sql
-- Delta Lake (Databricks)
VACUUM catalog.schema.table_name RETAIN 168 HOURS;
```

**Apache Iceberg Cleanup:**
```sql
-- Apache Iceberg (Spark)
CALL catalog.system.expire_snapshots(
  table => 'catalog.schema.table_name',
  older_than => TIMESTAMP '2023-10-01 00:00:00.000',
  retain_last => 5
);

-- Clean up orphaned files not referenced by any snapshot
CALL catalog.system.remove_orphan_files(
  table => 'catalog.schema.table_name'
);
```

### 2. Compute FinOps (Databricks / Spark)
If analyzing PySpark job configurations, look for these anti-patterns:
- **Over-provisioning:** Using massive instance types (e.g., `r6g.16xlarge`) when cluster utilization metrics show < 30% usage.
- **Auto-scaling limits:** Recommend setting strict `max_workers` to prevent runaway costs during data skew.
- **Spot Instances:** For ETL batch jobs, mandate the use of Spot/Preemptible instances for worker nodes.

**Example Agent Recommendation for Terraform:**
```hcl
resource "databricks_cluster" "etl_cluster" {
  # ...
  autoscale {
    min_workers = 2
    max_workers = 10 # 🛡️ FinOps Guardrail
  }
  aws_attributes {
    spot_bid_price_percent = 100
    first_on_demand        = 1
    availability           = "SPOT_WITH_FALLBACK" # 🛡️ Cost Saver
  }
}
```

### 3. S3/Cloud Storage API Costs
Iceberg and Delta require frequent `LIST` and `PUT` requests.
- **Agent Check:** If a streaming job commits every 10 seconds to S3, calculate the API cost (`(60*60*24*30 / 10) * S3_PUT_Cost`).
- **Recommendation:** Suggest increasing the trigger interval (e.g., 1-5 minutes) for cost-efficiency unless real-time latency is a strict SLA requirement.

### 4. Tagging & Attribution
Ensure all cloud resources and tables are tagged for FinOps attribution.
- Require tags: `CostCenter`, `Environment`, `DataDomain`.
- For Unity Catalog: `ALTER TABLE main.sales SET TAGS ('CostCenter' = 'Finance');`

## Output Format: FinOps Report
```markdown
### 💰 FinOps Audit Report: `[Pipeline/Table Name]`

#### 🔍 Compute Analysis
- **Finding:** The Spark job `Daily_Agg_Job` is using 20 on-demand `i3.4xlarge` nodes.
- **Recommendation:** Switch worker nodes to `SPOT_WITH_FALLBACK`. Estimated monthly savings: **~60%**.

#### 🗄️ Storage Analysis
- **Finding:** Table `clickstream_raw` has 12,000 retained snapshots dating back 2 years.
- **Recommendation:** Implement a daily `VACUUM` / `expire_snapshots` job retaining only the last 7 days.

#### 🏷️ Tagging
- ❌ Missing `CostCenter` tag on the S3 destination bucket.
```