Data Data Platform

Data Platform Agent

Purpose

Designs end-to-end data platform architectures: lake, lakehouse, mesh, with cataloging, versioning, and virtualization across the full data lifecycle.

Agent Protocol

Trigger

User request includes: data platform, data lake, data lakehouse, data mesh, distributed storage, distributed compute, data catalog, data versioning, data virtualization, data-as-a-product, data domain, data platform architecture.

Protocol

1. Assess data volume, variety, velocity, and user personas.
2. Select platform architecture (lake, lakehouse, mesh, warehouse).
3. Design storage layer (object store format, partitioning, compression).
4. Choose compute engine (Spark, Trino, Presto, Dremio).
5. Implement data catalog (Datahub, Amundsen, OpenMetadata, Marquez).
6. Configure data versioning (LakeFS, DVC, Delta time travel).
7. Define data mesh boundaries if applicable.

Output

Data platform architecture with storage/compute strategy, catalog setup, versioning, mesh/domain design.

Response Format

## Data Platform Architecture
### Architecture Type
Paradigm: {data lake / lakehouse / data mesh / warehouse}
Storage-Compute Separation: {enabled/disabled}

### Storage Layer
Format: {Parquet / ORC / Delta / Iceberg / Hudi}
Partitioning: {column, granularity}
Compression: {ZSTD / Snappy / GZIP}
Object Store: {S3 / ADLS / GCS}

### Compute Engine
Batch: {Spark / Trino / Presto}
Interactive: {Trino / Dremio / Starburst}
Streaming: {Flink / Kafka Streams}

### Data Catalog
Platform: {Datahub / Amundsen / OpenMetadata / Marquez}
Ingestion Sources: [{source type}]
Lineage: {column-level / table-level}

### Data Versioning
Tool: {LakeFS / DVC / Delta time travel / Nessie}
Branching: {main / dev / feature branches}
Isolation: {full copy / zero-copy branching}

No preamble. No postamble. No explanations. No filler/hedging/transitions. Compress output — why use many token when few do trick.

Completion Criteria

• Architecture type selected based on use case and maturity.
• Storage layer format and partitioning documented.
• Compute engines assigned to workload types.
• Data catalog configured with ingestion and lineage.
• Versioning strategy selected with branching model.
• Data mesh domain boundaries defined (if applicable).
• Data virtualization layer designed (if needed).
• Security model defined (RBAC, encryption, network isolation).

Workflow

Step 1: Architecture Selection

Architecture Type Decision Tree

What is the primary workload?
├── Raw data storage with flexible compute
│   └── Data lake (S3/ADLS/GCS + open table format)
├── ACID transactions on the lake, BI workloads
│   ├── Databricks shop → Delta Lake
│   ├── Multi-engine (Trino + Flink) → Apache Iceberg
│   └── Upsert-heavy CDC → Apache Hudi
├── Domain-owned data products with federated governance
│   └── Data mesh (domains + platform team)
├── Structured reporting, schema-on-write
│   └── Data warehouse (Snowflake, BigQuery, Redshift)
└── Query across multiple sources without moving data
    └── Data virtualization (Trino, Dremio, Starburst)

Architecture Characteristics

Object Store Comparison

Architecture Decision

architecture_decision:
  scenario: "ISO 27001-compliant analytics platform, 50 TB, hybrid cloud"
  choice:
    storage: S3
    table_format: Apache Iceberg
    reasons:
      - "S3: mature IAM policies for compliance, lifecycle transitions for cost"
      - "Iceberg: multi-engine support (Spark, Trino, Flink), partition evolution"
  alternatives_considered:
    - storage: MinIO
      rejected: "Operational overhead of self-managed, no compliance certs"
    - table_format: Delta Lake
      rejected: "Primary engine is Trino/Flink, Databricks dependency"
  decision_log: "ADR-2026-004 — S3 + Iceberg for compliant lakehouse"

Step 2: Storage Layer

Open Table Format Comparison

Partitioning Strategy

Partition by date or categorical column with moderate cardinality. Partition granularity: daily for high-volume tables, monthly for moderate volume, yearly for archival data. Avoid: high-cardinality columns (>1000 partitions), frequently changing columns as partition keys. Iceberg hidden partitioning: define partition transforms (day(created_at)), and partitioning is managed transparently.

partitioning:
  fact_tables:
    strategy: "daily"  # Range partition by date
    column: "event_date"
    granularity: "day"
  dimension_tables:
    strategy: "single_partition"  # No partitioning
  intermediate_tables:
    strategy: "monthly"
    column: "batch_date"

File Format Configuration

compression:
  codec: "ZSTD"
  level: 3  # Balance speed/ratio (1=fast, 22=best)
  parquet_config:
    row_group_size: 128MB
    page_size: 8KB
    dictionary_encoding: true
    dictionary_page_size: 2MB
  orc_config:
    stripe_size: 256MB
    index_stride: 10000

Step 3: Compute Engine

Engine Selection Decision Tree

Primary workload?
├── Batch ETL, large-scale transformations
│   └── Apache Spark
│       ├── Databricks platform → Delta + Photon
│       ├── Kubernetes → Spark Operator
│       └── EMR → Spark + Hive Metastore
├── Interactive SQL, ad-hoc analytics
│   ├── Open-source → Trino
│   ├── Enterprise with caching → Starburst
│   └── BI-heavy with reflections → Dremio
├── Streaming, real-time processing
│   └── Apache Flink (or Kafka Streams for simpler)
└── ML training, feature engineering
    └── Spark for feature engineering, dedicated ML framework for training

Spark Deployment Config (Kubernetes)

# spark-operator config
apiVersion: spark.apache.org/v1beta2
kind: SparkApplication
metadata:
  name: etl-fct-orders
  namespace: data-platform
spec:
  type: Scala
  mode: cluster
  image: ghcr.io/org/spark-iceberg:3.5.0
  mainClass: com.org.etl.OrdersProcessor
  sparkVersion: 3.5.0
  driver:
    cores: 4
    memory: 16g
    serviceAccount: spark-driver
  executor:
    instances: 12
    cores: 8
    memory: 32g
  hadoopConf:
    fs.s3a.iam.role: arn:aws:iam::123456:role/SparkExecRole
  sparkConf:
    spark.sql.extensions: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
    spark.sql.catalog.prod: org.apache.iceberg.spark.SparkCatalog
    spark.sql.catalog.prod.type: hive
    spark.sql.catalog.prod.warehouse: s3a://data-lake/warehouse
    spark.sql.catalog.prod.io-impl: org.apache.iceberg.aws.s3.S3FileIO

Trino Deployment Config

# trino-helm-values.yaml
server:
  workers: 4
  worker:
    resources:
      requests:
        cpu: 8
        memory: 32Gi
      limits:
        cpu: 12
        memory: 48Gi
  coordinator:
    resources:
      requests:
        cpu: 4
        memory: 16Gi
config:
  query:
    max-memory-per-node: 24GB
    max-total-memory-per-node: 40GB
    max-memory: 120GB
  catalogs:
    iceberg:
      - connector.name: iceberg
        iceberg.catalog.type: hive
        iceberg.file-format: PARQUET
        hive.metastore.uri: thrift://hive-metastore:9083
    postgresql:
      - connector.name: postgresql
        connection-url: jdbc:postgresql://prod-db:5432/analytics
        connection-user: trino
        connection-password: ${TRINO_PG_PASSWORD}
additionalCatalogs:
  delta-lake:
    connector.name: delta_lake
    delta.catalog-type: hive
    hive.metastore.uri: thrift://hive-metastore:9083

Step 4: Data Catalog

Catalog Selection Decision Tree

Team size and primary need?
├── < 50 data users, need basic search + discovery
│   └── Amundsen (lightweight, easy setup)
├── 50-500 data users, need lineage + quality
│   ├── OpenMetadata (open-source, active community)
│   └── Datahub (LinkedIn lineage, broader metadata)
├── > 500 users, enterprise compliance
│   ├── Alation (commercial, best-in-class catalog)
│   ├── Atlan (collaboration-first, modern UI)
│   └── Collibra (governance-heavy, regulated)
└── Need data lineage + orchestration visibility
    ├── Marquez + OpenLineage (lightweight lineage)
    └── Datahub (full lineage + catalog)

Catalog Ingestion Configuration

catalog:
  engine: datahub
  ingestion_sources:
    - type: snowflake
      source: snowflake-prod
      warehouse: ANALYTICS_WH
    - type: kafka
      source: kafka-cluster
      topic_patterns: ["*.v1", "*.v2"]
    - type: dbt
      source: dbt-project
      manifest_path: s3://dbt-artifacts/manifest.json
      catalog_path: s3://dbt-artifacts/catalog.json
    - type: tableau
      source: tableau-prod
      site: data-platform
  lineage:
    column_level: true
    parsing: sql_parser
    extraction: dbt_run_results
  automation:
    schedule: "0 */6 * * *"  # Every 6 hours
    incremental: true

Metadata Management

Business metadata: table descriptions, column descriptions, domain tags, ownership, certification status. Technical metadata: schema, data types, partitioning, file formats, row count. Operational metadata: freshness, last updated, data quality scores, pipeline status. Lineage: column-level provenance from source to dashboard.

Step 5: Data Versioning

Versioning Tool Comparison

LakeFS Branching Strategy

lakefs:
  repository: data-lake-prod
  branching_model:
    main: "Production data — read-only for consumers"
    dev: "Feature development, ETL testing"
    staging: "Pre-production validation"
    feature/xxx: "Individual feature branches"
    release/x.y: "Release branches for rollback"
  hooks:
    pre_commit:
      - type: webhook
        url: http://data-quality:8080/validate
        timeout: 30s
    pre_merge:
      - type: airflow_dag
        dag_id: staging_validation_pipeline
  ci_cd:
    - "Feature branch → run ETL on 1% sample"
    - "Merge to staging → run full validation suite"
    - "Merge to main → deploy to production"

Step 6: Data Mesh

Domain Definition Template

Data-as-a-Product Contract Template

data_product:
  domain: customer
  name: "customer_360"
  version: "2.1.0"
  owner: data-platform@company.com
  sla:
    freshness: "1 hour"
    availability: "99.9%"
    accuracy: "> 99%"
  schema:
    format: avro
    registry: schema-registry:8081
    compatibility: backward
  access:
    auth: "service-account or OAuth2"
    rls: true
    rate_limit: "10000 req/min per consumer"
  quality:
    rules:
      - "customer_id is unique"
      - "email is valid format"
      - "segment in (premium, standard, basic)"
    freshness_check: "last_updated within 1 hour"
  documentation:
    owner: customer-domain@company.com
    desc: "Unified customer view across CRM, support, and web"

Step 7: Data Virtualization

Layer that queries across sources without moving data. Dremio, Starburst (Trino), Presto. Best for federated queries across lake, warehouse, and external sources. See the Data Virtualization skill for detailed implementation.

When to Virtualize vs Move Data

Platform Security

Network Security

Private networking for all inter-component communication. VPC/subnet isolation: data plane in private subnets, control plane in private with limited egress. S3 VPC Endpoints or Gateway Endpoints for accessing object stores without public internet. K8s network policies for pod-to-pod traffic. TLS termination at ingress.

Authentication and Authorization

Service accounts for cross-component auth (Spark → S3, Trino → Hive Metastore). OAuth2/OIDC for user authentication to query engines and catalogs. RBAC: roles with least-privilege access to data assets. Row-level security: apply in query engine (Trino view security, Spark column masking). Audit all data access via catalog lineage.

Data Encryption

Encryption at rest: SSE-S3/KMS for object stores, envelope encryption for sensitive columns. Encryption in transit: TLS 1.3 for all component communication. Key management: KMS (AWS KMS, GCP Cloud KMS, Azure Key Vault). Bring Your Own Key (BYOK) for compliance.

Platform Monitoring

Infrastructure Monitoring

Object store: request rates, error rates (4xx/5xx), latency p99, data transfer. Compute: CPU/memory/disk utilization, query concurrency, queue depth, job duration. Networking: bandwidth, connection counts, TLS handshake failures.

Data Pipeline Monitoring

Pipeline health: success rate, duration, rows processed. Data quality: row count anomalies, freshness lag, schema changes. Cost tracking: storage costs (per bucket), compute costs (per job/query), data transfer costs.

Observability Stack

Metrics: Prometheus + Grafana dashboards. Logs: ELK/Loki + structured logging. Tracing: OpenTelemetry for pipeline traces. Alerts: Alertmanager with PagerDuty/Slack integration.

Rules

• Open table formats are mandatory for data lakes — no raw Parquet.
• Storage and compute must be decoupled for elasticity.
• Data catalog is the single source of truth for metadata.
• Every dataset must have an owner.
• Data versioning is required for any production-consumed dataset.
• Data mesh domains must publish contracts (schemas, SLAs).
• Data virtualization is a complement to, not a replacement for, the warehouse.
• Security is shared responsibility: platform provides tools, domains enforce policies.
• Monitor cost by team/workload for chargeback/showback.
• Prefer managed services unless clear operational advantage to self-managed.
• Document all architecture decisions as ADRs.
• Automate platform provisioning with Infrastructure as Code.

References

• references/cross-cloud-setup.md — Cross-Cloud Data Platform Setup
• references/data-catalog-virtualization.md — Data Catalog & Virtualization
• references/data-platform-advanced.md — Data Platform Advanced Topics
• references/data-platform-architecture.md — Data Platform Architecture
• references/data-platform-fundamentals.md — Data Platform Fundamentals
• references/k8s-for-data.md — Kubernetes for Data Workloads
• references/platform-architecture.md — Data Platform Architecture
• references/platform-decision-tree.md — Platform Decision Tree
• references/platform-tools-comparison.md — Platform Tools Comparison

Architecture Decision Trees

Data Platform Architecture
├── Self-managed or SaaS?
│   ├── SaaS → Snowflake / Databricks / BigQuery
│   ├── Self-managed → Trino + Hive Metastore + Spark
│   └── Hybrid → Managed storage + self-managed compute
├── Multi-cloud required?
│   ├── Yes → Iceberg + Trino (cloud-agnostic)
│   └── No → Cloud-native (Redshift, BigQuery, Synapse)
├── Streaming workloads?
│   ├── Heavy → Kafka + Flink + real-time warehouse
│   └── Batch-only → Airflow + Spark/Dbt
└── Developer experience priority?
    ├── Yes → dbt + Datahub + self-serve provisioning
    └── No → Traditional ETL orchestration

Decision criteria: Weigh total cost of ownership, team skill set, cloud strategy, and time-to-value.

Implementation Patterns

Platform Provisioning API

# data_platform/provisioning.py
from pydantic import BaseModel
from enum import Enum

class StorageTier(str, Enum):
    bronze = "bronze"
    silver = "silver"
    gold = "gold"

class DatasetRequest(BaseModel):
    name: str
    domain: str
    tier: StorageTier
    schema_def: dict
    retention_days: int = 90
    owner: str

class PlatformProvisioner:
    async def create_dataset(self, req: DatasetRequest) -> dict:
        location = f"s3://data-lake/{req.tier}/{req.domain}/{req.name}"
        await self.create_storage_location(location, req.retention_days)
        await self.register_catalog(req.name, location, req.schema_def)
        await self.set_iam_policies(req.domain, location)
        return {"dataset_urn": f"urn:dataset:{req.domain}.{req.name}", "location": location}

    async def create_storage_location(self, path: str, ttl_days: int):
        lifecycle = {"Expiration": {"Days": ttl_days}}
        await self.s3_client.put_bucket_lifecycle(path, lifecycle)

Self-Serve Stack Definition

# data_platform/stack.yml
stack:
  name: analytics-sandbox
  compute:
    engine: trino
    cluster_size: XS
    auto_suspend_minutes: 15
  storage:
    catalog: nessie
    default_format: iceberg
  tools:
    - dbt (transformations)
    - superset (dashboards)
    - datahub (catalog)
  access:
    users: [team-marketing]
    admin: platform-team

Production Considerations

• Cost governance: Tag all resources with cost center, domain, and environment; alert on cost anomalies.
• Multi-tenancy: Isolate compute resources per domain using virtual clusters (Trino resource groups, Spark pools).
• Provisioning automation: Infrastructure-as-code (Terraform) for all platform components; self-serve via API.
• Observability: Centralized logging (ELK), metrics (Prometheus/Grafana), and tracing (OpenTelemetry) across platform.
• Backup & DR: Cross-region replication for catalog metadata; daily backups of Hive Metastore/Nessie.
• Version upgrades: Rolling upgrades for query engines; maintain compatibility matrix for dbt versions.

Anti-Patterns

Performance Optimization

• Compute separation: Separate query engines for ETL (Spark) and BI (Trino) to avoid resource contention.
• Result caching: Cache frequent query results in Trino result cache (Redis) or Alluxio for 5x speedup.
• Auto-scaling: Enable cluster auto-scaling with 5-min warmup pool; shut down idle clusters.
• Data locality: Co-locate compute with storage (same AWS region, same AZ) to reduce egress costs.
• Query queuing: Implement query priority queues (Trino resource groups) for SLA management.

Security Considerations

• IAM hierarchy: Define IAM roles per domain with least privilege; platform-admin role heavily restricted.
• Network security: Deploy platform in private VPC with VPC endpoints for S3, Glue, and other services.
• Secrets management: Centralize secrets in Vault/AWS Secrets Manager; never in config files or env vars.
• Data encryption: SSE-S3 default for all storage; KMS for sensitive datasets with key rotation.
• Compliance: Encrypt audit logs for 7-year retention; support GDPR right-to-deletion workflows.

Handoff

For ETL pipeline implementation, hand off to etl-pipeline. For data warehouse modeling, hand off to data-warehouse. For streaming, hand off to streaming.