Data Data Catalog

Data Data Catalog

Purpose

Design and deploy a data catalog platform with metadata ingestion, column-level lineage, business glossary, data discovery, ownership model, and usage analytics.

Agent Protocol

Trigger

Exact user phrases: "data catalog", "DataHub", "Amundsen", "Apache Atlas", "OpenMetadata", "metadata management", "data discovery", "business glossary", "data lineage", "data ownership", "column-level lineage", "metadata ingestion".

Input Context

• Existing data stack (warehouse, lake, BI tools)
• Data catalog tool in consideration (if any)
• Data governance maturity level
• Number of data assets and teams
• Compliance requirements (GDPR, SOX, HIPAA)
• Current metadata sources (alerts, logs, metastore)

Output Artifact

Data catalog architecture with platform selection, metadata ingestion pipeline design, governance model with glossary and ownership, and usage analytics framework.

Response Format

# Catalog platform selection matrix
# Metadata ingestion pipeline
# Business glossary structure
# Ownership & stewardship model
# Usage analytics schema

No preamble. No postamble. No explanations. No filler/hedging/transitions. Compress output — why use many token when few do trick.

Completion Criteria

• Catalog platform selected with rationale
• Metadata ingestion pipelines configured for all sources
• Column-level lineage established for critical datasets
• Business glossary defined with domains and terms
• Data ownership assigned with stewardship workflow
• Search and discovery configured with facets
• Usage analytics tracking enabled

Max Response Length

350 lines of configuration.

Workflow

Step 1: Select Catalog Platform

Default: DataHub for lineage-heavy, OpenMetadata for governance-heavy. Amundsen only if search is primary need and lineage secondary.

Decision tree:

• Need real-time column-level lineage? → DataHub
• Need built-in governance workflows? → OpenMetadata
• Existing Hadoop/Hive ecosystem? → Apache Atlas
• Simple search, small team? → Amundsen

Step 2: Metadata Ingestion

DataHub Ingestion YAML

source:
  type: snowflake
  config:
    account_id: xy12345.us-east-1
    warehouse: COMPUTE_WH
    role: DATAHUB_ROLE
    include_views: true
    include_tables: true
    profiling:
      enabled: true
      profile_table_level_only: false
    capture_column_lineage:
      enabled: true
sink:
  type: datahub-rest
  config:
    server: "http://datahub-gms:8080"

OpenMetadata Connector Config

source:
  type: snowflake
  serviceName: snowflake_prod
  serviceConnectionType: Snowflake
  sourceConfig:
    config:
      type: DatabaseMetadata
      markDeletedTables: true
      includeTables: true
      includeViews: true
sink:
  type: metadata-rest
  config:
    apiEndpoint: http://openmetadata-server:8585/api

Step 3: Column-Level Lineage

Configure via OpenLineage for Spark/Airflow. Use dbt catalog.json + manifest.json for dbt models. For SQL-based ETL, use DataHub SQL parser to extract column-level lineage from query logs.

# DataHub lineage extraction from SQL
source:
  type: glue
  config:
    aws_region: us-east-1
    extract_transforms: true
    sql_parser:
      enabled: true
      lineage_type: COLUMN

Column-level lineage enables impact analysis: "which dashboards break if I change this column?" and root cause analysis: "which source column caused this data quality issue?"

Step 4: Business Glossary

Glossary terms linked to tables and columns via schemaField tags. Terms inherit from parent to child datasets.

Step 5: Ownership & Stewardship

Ownership stored as metadata on each dataset entity. Stewards review metadata quarterly.

Step 6: Search & Discovery

Search rankings: popularity > freshness > completeness > documentation score.

Step 7: Usage Analytics

Track: dataset_queries, column_access_frequency, top_users, search_queries. Publish dashboard: most queried datasets, orphan datasets (no reads in 90 days), top searches without results.

Step 8: DataHub API Integration

from datahub.ingestion.graph.client import DataHubGraph

graph = DataHubGraph(
    server="http://datahub-gms:8080",
    token="<personal-access-token>",
)

# Search datasets
results = graph.search_entities(
    type="dataset",
    query="orders",
    facets=["platform", "domain", "owner"],
    start=0, count=20,
)
for dataset in results.entities:
    print(dataset.urn, dataset.properties.name)

# Get lineage
lineage = graph.get_lineage(
    urn="urn:li:dataset:(urn:li:dataPlatform:snowflake,PROD_DB.COMMERCE.FCT_ORDERS,O)",
    depth=2, direction="UPSTREAM",
)

Step 9: Catalog Governance Workflows

Implement data access request workflows through the catalog. When a consumer finds a dataset, they click "Request Access" which creates a ticket for the data owner. The owner approves or denies. On approval, permissions are provisioned automatically. Catalog tracks access grants and flags stale permissions for quarterly review.

Step 10: Automated Metadata Quality Scoring

Score each dataset on metadata completeness: description (exists, > 50 chars), owner (assigned), glossary terms (linked), lineage (column-level), documentation score (fields documented). Publish scores on catalog dashboard. Domain stewards responsible for improving low-scoring datasets.

Architecture / Decision Trees

Catalog Platform Selection

Need data catalog?
  ├── Real-time column lineage required? → DataHub
  ├── Built-in governance + glossary workflows? → OpenMetadata
  ├── Existing Hadoop/Hive infrastructure? → Apache Atlas
  ├── Simple search, limited resources? → Amundsen
  └── Need managed service? → DataHub Cloud / Atlan / Alation

Metadata Ingestion Priority

Which sources first?
  ├── Tier 1: Critical BI datasets (Snowflake, BigQuery, Redshift)
  ├── Tier 2: Transformation layer (dbt, Spark, Airflow)
  ├── Tier 3: Streaming (Kafka, Schema Registry)
  └── Tier 4: BI tools (Tableau, PowerBI, Looker)

Common Pitfalls

1. Metadata staleness: ingestion pipelines fail silently and metadata becomes stale. Monitor ingestion health and alert on failures.
2. No ownership assigned: datasets without owners become orphaned. Enforce ownership requirement in ingestion.
3. Glossary disconnected from technical metadata: business terms not linked to columns. Ensure lineage connection.
4. Over-instrumentation: ingesting every column from every table creates noise. Focus on critical data assets first.
5. Catalog as shelf-ware: catalog deployed but no one uses it. Invest in search quality and usage analytics.
6. Permission model not defined: catalog must respect data access policies or risk exposing sensitive metadata.
7. Lineage depth too shallow: only table-level lineage without column-level for critical datasets.
8. No enforcement of metadata requirements: datasets in catalog without required fields degrade trust.
9. Ingestion pipeline not monitoring source changes: schema changes in source break ingestion silently.
10. Search not optimized: poor search rankings make catalog useless. Tune based on user behavior.

Best Practices

• Start with 2-3 critical data sources and expand incrementally.
• Enforce ownership for every production dataset during ingestion.
• Conduct quarterly metadata health reviews with domain stewards.
• Monitor ingestion pipeline health and alert on failures within 24 hours.
• Link business glossary terms to technical columns for complete discoverability.
• Use catalog as the access request gateway for all data assets.
• Tier data assets: Critical (C-level reports), Important (team dashboards), Operational (internal tools).
• Purge metadata for deleted datasets within 30 days.
• Catalog should support both push (real-time API) and pull (scheduled ingestion) metadata collection.
• Implement metadata quality scoring to drive improvement.
• Use catalog API for automated metadata operations in CI/CD.
• Set up catalog access for all data consumers (analysts, engineers, scientists).
• Publish catalog usage dashboards to demonstrate ROI.

Compared With

Data catalog vs schema registry: catalog manages business metadata, ownership, and discovery. Schema registry manages technical schema, compatibility, and serialization. They integrate: schema registry feeds schemas to the catalog as technical metadata.

Data catalog vs data quality tools: catalog is about discovery and governance. Quality tools (Great Expectations, Soda) validate data against expectations. They integrate: quality results feed into catalog as dataset health scores.

Performance

• Metadata ingestion: DataHub push API handles 1000+ events/second per instance.
• Search latency: < 200ms for 10M+ indexed entities with Elasticsearch backend.
• Lineage graph queries: < 1s for full lineage of highly-connected datasets.
• Profiling: enable on critical tables only (expensive on large tables).
• Ingestion pipeline: schedule during off-peak hours to minimize load on source systems.
• Catalog storage: multiply number of datasets by 10KB for metadata storage estimate.
• Batch ingestion: 10K-50K entities/hour per ingestion worker.
• Real-time ingestion (push): < 1s end-to-end from emit to searchable.
• API rate limits: 100 req/s for GraphQL, 500 req/s for REST (configurable).
• Elasticsearch cluster sizing: 3 nodes minimum for HA, index per entity type.

Scalability: DataHub scales horizontally by adding GMS instances. Search scales with Elasticsearch cluster. For 100K+ datasets, partition by domain and use tiered storage for older metadata.

Tooling

Catalog Implementation Patterns

DataHub Ingestion Pipeline

# DataHub ingestion recipe: snowflake + dbt + tableau
datahub_ingestion:
  source:
    type: snowflake
    config:
      account_id: "xyz12345"
      warehouse: "transforming"
      include_views: true
      include_tables: true
      profiling:
        enabled: true
        profile_table_level_only: false
        profile_pattern:
          allow: ["analytics.*"]
  
  source:
    type: dbt
    config:
      manifest_path: "./target/manifest.json"
      catalog_path: "./target/catalog.json"
      sources_path: "./target/sources.json"
      enable_meta_mapping: true
      node_type_pattern:
        allow: ["model", "source", "test"]
  
  source:
    type: tableau
    config:
      connect_uri: "https://10ay.online.tableau.com"
      site: "my-site"
      projects: ["default", "executive"]
      extract_lineage: true
      ingest_dashboards: true
  
  sink:
    type: datahub-rest
    config:
      server: "http://datahub-gms:8080"

Metadata Model

metadata_model:
  dataset:
    required_fields:
      - id: "fully qualified name"
      - name: "human-readable name"
      - schema: { columns: [{ name, type, description, nullable }] }
      - owner: "team or individual"
      - tier: [1, 2, 3]
      - domain: "business domain tag"
    
    optional_fields:
      - glossary_terms: ["term1", "term2"]
      - tags: ["PII", "FINANCIAL", "EXPERIMENTAL"]
      - lineage: { upstream: [], downstream: [] }
      - quality: { score: 0.95, last_checked: "2026-05-01" }
      - usage: { weekly_queries: 500, top_users: [] }
      - profile: { row_count: 5000000, size_bytes: 1073741824 }
  
  glossary_term:
    required_fields:
      - id: "term_id"
      - name: "term_name"
      - description: "term definition"
      - domain: "responsible domain"
    
    optional_fields:
      - synonyms: ["also_known_as"]
      - related_terms: ["parent", "child"]
      - stewardship: { owner: "data_governance@org.com" }

  tag:
    required_fields:
      - name: "tag_name"
      - description: "tag purpose"
    optional_fields:
      - color: "#hex"
      - inherit_from_dataset: true

Catalog API Usage

# DataHub GraphQL queries for catalog integration
import requests

def search_datasets(keyword, domain=None, tier=None):
    query = """
    query searchDatasets($query: String!, $filters: [FacetFilterInput!]) {
        search(input: { type: DATASET, query: $query, filters: $filters }) {
            total results { entity { urn type ...on Dataset { name platform { name } } } }
        }
    }
    """
    variables = {"query": keyword, "filters": []}
    if domain: variables["filters"].append({"field": "domains", "value": domain})
    if tier: variables["filters"].append({"field": "tier", "value": str(tier)})
    
    response = requests.post(
        "http://datahub-gms:8080/api/graphql",
        json={"query": query, "variables": variables},
        headers={"Authorization": "Bearer <token>"}
    )
    return response.json()

def update_metadata(dataset_urn, metadata):
    """Update dataset description and tags via DataHub API"""
    mutation = """
    mutation updateDataset($urn: String!, $input: DatasetUpdateInput!) {
        updateDataset(urn: $urn, input: $input) { urn }
    }
    """
    variables = {
        "urn": dataset_urn,
        "input": {
            "editableProperties": {
                "description": metadata.get("description", ""),
                "tags": metadata.get("tags", [])
            }
        }
    }
    return requests.post("http://datahub-gms:8080/api/graphql",
        json={"query": mutation, "variables": variables})

Decision Trees

Catalog Implementation

Team size and metadata requirements?
├── Small team (< 10), need quick data discovery
│   └── Amundsen (simple, search-focused)
├── Mid-size team, need lineage + governance
│   ├── Python/Spark/Airflow stack → DataHub (best lineage support)
│   └── dbt-native stack → OpenMetadata (dbt integration)
├── Large enterprise, compliance-driven
│   └── DataHub or Atlan (strong governance, RBAC, audit)
└── Hadoop/Hive-centric ecosystem
    └── Apache Atlas (native Hive/HBase/Spark integration)

Rules

• Every production dataset documented in catalog with owner
• Column-level lineage tracked for all critical data assets
• Business glossary terms reviewed quarterly by domain stewards
• Metadata freshness monitored — stale entries flagged within 7 days
• Access requests routed through catalog (not ad-hoc)
• Deprecated datasets marked with expiration date and replacement
• No undocumented dataset promoted to production
• Ingestion pipeline health monitored and alerted
• Catalog search optimized with usage analytics feedback
• Metadata quality scored and published for accountability
• Glossary terms linked to columns, not just tables
• Catalog used as single entry point for data discovery
• Automate metadata ingestion from data sources, dbt, and BI tools
• Link glossary terms, tags, and ownership for searchable metadata
• Expose catalog via API for integration with other tools

References

• references/catalog-api-examples.md — Catalog API Examples
• references/catalog-data-discovery.md — Catalog Data Discovery
• references/catalog-governance.md — Catalog Governance
• references/catalog-metadata-automation.md — Catalog Metadata Automation
• references/catalog-platforms.md — Catalog Platforms Comparison
• references/catalog-search-discovery.md — Catalog Search and Discovery
• references/metadata-ingestion-patterns.md — Metadata Ingestion Patterns
• references/metadata-management.md — Metadata Management
• references/data-catalog-metadata-management.md — Metadata Management Deep Dive
• references/data-catalog-search-discovery.md — Search Discovery Reference

Architecture Decision Trees

Catalog Platform Selection
├── Cloud-native SaaS desired?
│   ├── Yes → Atlan / Datahub Cloud / Alation
│   └── No → Open-source self-hosted (Datahub, Amundsen)
├── dbt integration required?
│   ├── Yes → Datahub (dbt ingestion native)
│   └── No → Evaluate feature completeness
├── Real-time metadata sync?
│   ├── Yes → Datahub with Kafka-based ingestion
│   └── No → Batch ingestion (daily cron)
└── Schema drift detection needed?
    ├── Yes → Datahub with column-level lineage
    └── No → Basic table/column catalog (Amundsen)

Decision criteria: Evaluate metadata freshness, integration depth, operating cost, and team size for administration.

Implementation Patterns

Metadata Ingestion Pattern

# data_catalog/ingestion.py
from dataclasses import dataclass
from datetime import datetime

@dataclass
class TableMetadata:
    database: str
    schema: str
    table: str
    columns: list[dict]
    row_count: int
    last_updated: datetime
    description: str

class MetadataIngestor:
    def __init__(self, catalog_api: str, token: str):
        self.api = catalog_api
        self.headers = {"Authorization": f"Bearer {token}"}

    async def ingest_table(self, metadata: TableMetadata):
        payload = {
            "urn": f"urn:li:dataset:{metadata.database}.{metadata.schema}.{metadata.table}",
            "schema": metadata.columns,
            "last_updated": metadata.last_updated.isoformat(),
            "description": metadata.description,
        }
        async with httpx.AsyncClient() as client:
            resp = await client.post(f"{self.api}/entities", json=payload, headers=self.headers)
            resp.raise_for_status()

Search Index Pattern

# data_catalog/search_config.yml
index:
  name: data_catalog
  fields:
    - name: table_name
      type: text
      boost: 3.0
    - name: description
      type: text
      boost: 1.5
    - name: tags
      type: keyword
      boost: 2.0
  synonyms:
    - ["user", "customer", "account"]
    - ["revenue", "sales", "income"]

Production Considerations

• Ingestion reliability: Use dead-letter queues for failed metadata syncs; retry with exponential backoff.
• Schema versioning: Store schema history to track column additions, removals, and type changes.
• Access control: Restrict sensitive table discovery by role (PII tables hidden from non-privileged users).
• Crawl scheduling: Set full crawl daily, incremental crawl every 15 min for high-priority sources.
• Search optimization: Index descriptions, column names, and tags; boost popular tables in results.
• Ownership tracking: Require each dataset to have a steward; alert on unowned datasets > 7 days.

Anti-Patterns

Performance Optimization

• Search indexing: Use Elasticsearch with field boosting; reindex on schema change only.
• API caching: Cache catalog API responses at reverse proxy (Varnish/nginx, TTL 60s).
• Batch ingestion: Bulk ingest 500 tables per API call instead of individual requests.
• Query optimization: Index the catalog database on (database, schema, table) and last_updated.
• Thumbnail generation: Pre-generate data preview thumbnails during ingestion, not on request.

Security Considerations

• PII classification: Auto-tag columns with PII/PCI using regex patterns; restrict discovery by role.
• Audit trail: Log all catalog searches, views, and ownership changes for compliance.
• API security: Require service tokens with least privilege; rotate tokens every 90 days.
• Data masking: Preview queries run on sampled masked data; never expose raw sensitive values.
• RBAC integration: Sync catalog roles from enterprise IdP (Okta, Azure AD) via SCIM.

Handoff

data-data-platform for platform infrastructure. data-data-quality for linking quality metadata to catalog. data-data-observability for freshness monitoring. data-data-contracts for contract metadata in catalog.