Data Catalog

data-catalog

Purpose

This skill manages metadata for data assets, enabling discovery, governance, and lineage tracking in data engineering workflows. It catalogs datasets, schemas, and dependencies to support data-driven projects.

When to Use

Use this skill when you need to track data assets in a project, such as during ETL processes, data governance audits, or when building data pipelines. Apply it in scenarios involving large-scale data repositories, compliance requirements, or collaborative data teams.

Key Capabilities

• Register and update metadata for datasets using JSON structures, e.g., {"name": "sales_data", "schema": {"columns": ["id", "date"]}}.
• Search and query assets via full-text or tag-based filters, supporting lineage queries like tracing data origins.
• Enforce governance policies, such as access controls, by associating tags like "sensitive" to assets.
• Generate lineage graphs in JSON format, e.g., {"source": "raw_logs", "target": "processed_reports"}.
• Integrate with storage systems like S3 or databases, using connectors that require API keys via $DATA_CATALOG_API_KEY.

Usage Patterns

To use this skill, first authenticate with an environment variable like export DATA_CATALOG_API_KEY=your_key. Then, follow a pattern: initialize the catalog, register assets, query as needed, and handle updates. For pipelines, embed it in scripts to auto-register outputs. Always validate metadata before operations to avoid conflicts.

Common Commands/API

Use the dcatalog CLI or REST API for interactions. Authentication requires $DATA_CATALOG_API_KEY in requests.

• CLI Commands:

  • Register an asset: dcatalog register --asset-name sales_data --type dataset --metadata '{"schema": ["id", "amount"]}' --api-key $DATA_CATALOG_API_KEY
  • Query assets: dcatalog search --query "sales" --tags metadata --limit 10
  • Update lineage: dcatalog update-lineage --source raw_data --target processed_data --relation depends_on

• API Endpoints:

  • POST /api/v1/assets: Create a new asset. Example curl: curl -H "Authorization: Bearer $DATA_CATALOG_API_KEY" -d '{"name": "sales_data", "tags": ["metadata"]}' -X POST https://api.opencclaw.com/api/v1/assets
  • GET /api/v1/assets/search?query=sales: Search assets. Example: curl -H "Authorization: Bearer $DATA_CATALOG_API_KEY" https://api.opencclaw.com/api/v1/assets/search?query=sales
  • PUT /api/v1/lineage: Update lineage. Code snippet:

    import requests
    headers = {"Authorization": f"Bearer {os.environ['DATA_CATALOG_API_KEY']}"}
    response = requests.put('https://api.opencclaw.com/api/v1/lineage', headers=headers, json={"source": "raw_data", "target": "report"})
    

Config formats are JSON-based, e.g., for CLI config file (~/.dcatalog/config.json):
{"default_tags": ["data-governance"], "api_endpoint": "https://api.opencclaw.com"}

Integration Notes

Integrate this skill with data tools like Apache Airflow or AWS Glue by wrapping API calls in custom operators. For example, in a Python script, import the API client and pass $DATA_CATALOG_API_KEY. Ensure compatibility by matching schema versions; use JSON configs for mappings, e.g., link to S3 buckets via {"bucket": "my-bucket", "prefix": "data/"}. Test integrations in a sandbox environment before production.

Error Handling

Handle errors by checking HTTP status codes in API responses; for example, if status is 401, prompt for $DATA_CATALOG_API_KEY revalidation. For CLI, use try-catch in scripts:

try:
    subprocess.run(["dcatalog", "register", "--asset-name", "test"], check=True)
except subprocess.CalledProcessError as e:
    print(f"Error: {e.returncode} - {e.output}")

Common issues include invalid JSON metadata (fix by validating with json.loads() before sending) or authentication failures (retry with refreshed keys). Log errors with timestamps for debugging.

Graph Relationships

• Related Cluster: data-engineering
• Connected Tags: metadata, data-governance, data-discovery
• Dependencies: Often links to skills in storage or processing clusters, e.g., for data ingestion or transformation.