Table Inspection

Skill: Enterprise Lakehouse Table Inspection & Discovery

Description

Provides the agent with methodologies to discover data lake assets, connect to Enterprise catalogs (Unity Catalog, AWS Glue, Polaris), and inspect Delta Lake, Apache Iceberg, and Apache Hudi tables.

Context

Engineers need deep metadata without heavy compute. The agent must seamlessly connect to the configured metadata catalog and use lightweight Python libraries to retrieve schemas, partition layouts, and snapshot histories.

Instructions

1. Catalog Connection

Always ask the user for their preferred catalog or check their environment variables (AWS_PROFILE, DATABRICKS_HOST).

Iceberg via REST/Polaris:

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "default",
    **{
        "type": "rest",
        "uri": "https://api.polaris.com/v1",
        "credential": "oauth2_token..."
    }
)
table = catalog.load_table("namespace.table")

Delta via Unity Catalog (Databricks SDK):

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
table = w.tables.get("catalog.schema.table_name")
print(table.columns)

2. Time Travel & Table Recovery

If a user mentions corrupted data or dropped tables:

• Suggest querying a specific snapshot/version.
• Delta: SELECT * FROM my_table VERSION AS OF 12
• Iceberg: SELECT * FROM my_table FOR SYSTEM_VERSION AS OF 12345

To restore a table:

-- Delta Lake
RESTORE TABLE my_table TO VERSION AS OF 12;

-- Apache Iceberg
CALL catalog.system.rollback_to_snapshot('my_table', 12345);

3. RAG-Ready Metadata Summary

Always output findings in structured Markdown so other agents or humans can easily read it.

Example Format:

### 🗂️ Enterprise Table Profile: `[Table Name]`

- **Catalog:** [Glue / Unity / Polaris]
- **Format:** [Delta/Iceberg/Hudi]
- **Location:** `[s3/adls/gcs path]`
- **Latest Version/Snapshot:** [Version ID]
- **Time Travel Retained:** [X days]

#### Schema
| Column | Type | Nullable | Masking/Tag |
|--------|------|----------|-------------|
| id     | long | False    | -           |
| email  | str  | True     | `[MASKED]`  |