Finops

Skill: Enterprise Lakehouse FinOps & Cost Management

Description

Enables the AI agent to analyze, attribute, and reduce cloud costs associated with Lakehouse architecture (Storage, Compute, and API operations).

Context

Lakehouses can quickly become expensive due to stale snapshots, orphaned files, excessive API calls (e.g., S3 PUT/GET), and unoptimized clusters. This skill teaches the agent how to implement strict FinOps practices across Databricks, AWS EMR, and Snowflake.

Instructions

1. Storage Optimization (Snapshot & Orphan Cleanup)

Data table versions and snapshots accrue over time. Recommend cleanup thresholds based on environment.

• Production: Retain 7-14 days for Time Travel.
• Dev/Test: Retain 1-3 days.

Delta Lake Cleanup:

-- Delta Lake (Databricks)
VACUUM catalog.schema.table_name RETAIN 168 HOURS;

Apache Iceberg Cleanup:

-- Apache Iceberg (Spark)
CALL catalog.system.expire_snapshots(
  table => 'catalog.schema.table_name',
  older_than => TIMESTAMP '2023-10-01 00:00:00.000',
  retain_last => 5
);

-- Clean up orphaned files not referenced by any snapshot
CALL catalog.system.remove_orphan_files(
  table => 'catalog.schema.table_name'
);

2. Compute FinOps (Databricks / Spark)

If analyzing PySpark job configurations, look for these anti-patterns:

• Over-provisioning: Using massive instance types (e.g., r6g.16xlarge) when cluster utilization metrics show < 30% usage.
• Auto-scaling limits: Recommend setting strict max_workers to prevent runaway costs during data skew.
• Spot Instances: For ETL batch jobs, mandate the use of Spot/Preemptible instances for worker nodes.

Example Agent Recommendation for Terraform:

resource "databricks_cluster" "etl_cluster" {
  # ...
  autoscale {
    min_workers = 2
    max_workers = 10 # 🛡️ FinOps Guardrail
  }
  aws_attributes {
    spot_bid_price_percent = 100
    first_on_demand        = 1
    availability           = "SPOT_WITH_FALLBACK" # 🛡️ Cost Saver
  }
}

3. S3/Cloud Storage API Costs

Iceberg and Delta require frequent LIST and PUT requests.

• Agent Check: If a streaming job commits every 10 seconds to S3, calculate the API cost ((60*60*24*30 / 10) * S3_PUT_Cost).
• Recommendation: Suggest increasing the trigger interval (e.g., 1-5 minutes) for cost-efficiency unless real-time latency is a strict SLA requirement.

4. Tagging & Attribution

Ensure all cloud resources and tables are tagged for FinOps attribution.

• Require tags: CostCenter, Environment, DataDomain.
• For Unity Catalog: ALTER TABLE main.sales SET TAGS ('CostCenter' = 'Finance');

Output Format: FinOps Report

### 💰 FinOps Audit Report: `[Pipeline/Table Name]`

#### 🔍 Compute Analysis
- **Finding:** The Spark job `Daily_Agg_Job` is using 20 on-demand `i3.4xlarge` nodes.
- **Recommendation:** Switch worker nodes to `SPOT_WITH_FALLBACK`. Estimated monthly savings: **~60%**.

#### 🗄️ Storage Analysis
- **Finding:** Table `clickstream_raw` has 12,000 retained snapshots dating back 2 years.
- **Recommendation:** Implement a daily `VACUUM` / `expire_snapshots` job retaining only the last 7 days.

#### 🏷️ Tagging
- ❌ Missing `CostCenter` tag on the S3 destination bucket.