Governance

Skill: Enterprise Lakehouse Governance & Security

Description

Guides the agent in identifying sensitive data, extracting ETL lineage, and generating least-privilege IAM and RBAC policies for cloud storage and metadata catalogs (Unity Catalog, AWS Lake Formation).

Context

Lakehouses centralize an organization's data, making security and governance paramount. It is crucial to identify Personally Identifiable Information (PII), trace where data comes from (lineage), and ensure that roles accessing the lake have tightly scoped permissions.

Instructions

1. PII & Sensitive Column Detection

Analyze table schemas for column names that imply sensitive information.

• Target Keywords: email, ssn, phone, address, dob, credit_card, password, token, ip_address, health_record.
• Action: If sensitive columns are found, recommend applying column-level masking, Dynamic Data Masking (if supported), or replacing raw values with hashes.

Databricks Unity Catalog Masking Example:

CREATE MASKING FUNCTION email_mask(email STRING)
  RETURN CASE WHEN is_member('hr_role') THEN email ELSE '***@***.com' END;

ALTER TABLE users ALTER COLUMN email SET MASK email_mask;

2. Lineage Extraction

If analyzing ETL code (Spark, dbt, SQL):

• Identify Sources (Reads): spark.read, FROM, ref(), source().
• Identify Targets (Writes): df.write, INSERT INTO, CREATE TABLE AS, models.
• Build a text-based lineage graph mapping inputs to outputs.

3. Least-Privilege IAM & Security Guidance

When a user asks for permissions to access the lakehouse, generate strict, least-privilege policies. Avoid wildcard * permissions on buckets.

Example Guidance for AWS S3 & Glue: To grant an ETL job read/write access to an Iceberg table:

1. S3 Bucket permissions must be scoped to the exact prefix.
2. Glue Catalog permissions must be scoped to the specific database and table.

Example IAM Policy JSON Generation:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::my-lake-bucket/path/to/table/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetTable",
        "glue:UpdateTable"
      ],
      "Resource": [
        "arn:aws:glue:us-east-1:123456789012:catalog",
        "arn:aws:glue:us-east-1:123456789012:database/my_db",
        "arn:aws:glue:us-east-1:123456789012:table/my_db/my_table"
      ]
    }
  ]
}

4. Row and Column Level Security (RLS/CLS)

When users request specific access restrictions based on attributes:

• Recommend Unity Catalog Row Filters or Snowflake Row Access Policies.
• Do NOT suggest creating multiple duplicate tables or views unless performance dictates it.

Output Format: Governance & Security Audit

### 🔒 Governance & Security Audit: `[Pipeline/Table Name]`

#### 🕵️ PII Detection
- 🚨 **High Risk:** Column `customer_email` detected in `raw_users` table. 
  - *Recommendation:* Apply a Unity Catalog Masking Function or Hash before writing to the gold zone.

#### 🔗 Lineage
- **Sources:** `s3://landing/users/`, `s3://landing/transactions/`
- **Target:** `glue_catalog.silver.user_transactions`

#### 🔑 IAM Posture
- The current Terraform policy uses `s3:*` on the whole bucket.
- *Recommendation:* Restrict to `s3:PutObject` on the `silver/user_transactions/` prefix.