MCP Data Observability Integration
Overview
Agents that cannot see production signals guess at fixes — scaling clusters blindly, rerunning jobs that OOM for structural reasons, or changing Kafka consumers without checking lag. This skill pairs MCP server templates with safe read-only observability workflows so diagnosis precedes mutation.
When to Use
- diagnosing Spark OOM, shuffle skew, or stage failures
- inspecting Kafka consumer group lag before replay or consumer changes
- reviewing Airflow DAG run state, task duration drift, or failed retries
- validating warehouse publish state before reopening downstream consumers
- setting up IDE-integrated observability for data engineering agents
Do not use MCP write capabilities for destructive fixes during incident triage unless explicitly approved and scoped.
Workflow
Choose the smallest MCP surface for the question.
- Spark plans and stage metrics:
mcp/spark.mcp.json or mcp/databricks.mcp.json
- Kafka lag and topic metadata:
mcp/kafka.mcp.json
- Orchestration run state:
mcp/airflow.mcp.json
- Warehouse inspection:
mcp/snowflake.mcp.json, mcp/bigquery.mcp.json, or mcp/postgres.mcp.json
- Release and job metadata:
mcp/dbt-cloud.mcp.json, mcp/github.mcp.json
Configure read-only first.
From mcp/README.md:
- replace placeholder
command with the MCP server binary in use
- scope credentials to read-only inspection roles
- set allowlists (
topic, catalog, schema) to minimum necessary scope
- validate connectivity outside the agent session before relying on tool output
Establish the diagnostic sequence.
Typical order:
- confirm symptom (lag, failure rate, freshness breach)
- pull live metadata (consumer lag, last successful run, stage skew)
- compare against baseline or SLA from
data-observability-and-sla-management
- form hypothesis before proposing code or infra changes
- record findings in incident or backfill evidence templates when action follows
Apply platform-specific investigation patterns.
- Spark OOM or timeout: inspect stage breakdown, shuffle read/write bytes, spill metrics, and partition counts; prefer physical plan fixes over blind scale-up
- Kafka lag: inspect group lag per partition, DLQ rate, and retention headroom; pause replay until lag root cause is classified
- Orchestration failures: inspect task logs, retry history, and upstream sensors; do not trigger wide backfills from a single red task without plan gates
Load references/mcp-data-observability-patterns.md for tool-to-symptom mapping.
Pair observability with safety skills.
- replay or repair:
safe-backfill-and-replay-orchestration
- streaming hardening:
kafka-resilience-and-schema-evolution
- serverless Spark:
spark-serverless-reliability-and-state-management
- incident response:
incident-triage-and-pipeline-recovery
Close the loop after changes.
Re-query MCP signals to confirm lag recovery, run success, or freshness restoration before publish reopen.
Common Rationalizations
| Rationalization |
Reality |
| "We can infer the problem from logs alone." |
Logs without lag, plan, and run metadata miss systemic causes like skew and consumer stall. |
| "MCP access should use admin credentials for speed." |
Over-scoped tokens increase blast radius when agents suggest broad changes. |
| "Observability is only for on-call humans." |
Agents without live signals replicate dangerous guesswork at machine speed. |
| "One MCP server is enough for every stack." |
Multi-platform estates need focused templates per system, not one mega-connection. |
Red Flags
- agents propose infra or code changes without querying live signals
- MCP templates use write-capable credentials by default
- no allowlists on Kafka topics or warehouse schemas
- lag or run failure is acknowledged but not tied to an owner or SLA
- observability findings are not linked to backfill or incident evidence
Verification
1---2name: mcp-data-observability-integration3description: Guides agents to wire Model Context Protocol servers for live data platform observability including Spark execution plans, OOM diagnosis, Kafka consumer lag, and orchestration run state. Use when agents need cluster metrics, streaming lag, or job diagnostics instead of blind code changes.4---56# MCP Data Observability Integration78## Overview910Agents that cannot see production signals guess at fixes — scaling clusters blindly, rerunning jobs that OOM for structural reasons, or changing Kafka consumers without checking lag. This skill pairs MCP server templates with safe read-only observability workflows so diagnosis precedes mutation.1112## When to Use1314- diagnosing Spark OOM, shuffle skew, or stage failures15- inspecting Kafka consumer group lag before replay or consumer changes16- reviewing Airflow DAG run state, task duration drift, or failed retries17- validating warehouse publish state before reopening downstream consumers18- setting up IDE-integrated observability for data engineering agents1920Do not use MCP write capabilities for destructive fixes during incident triage unless explicitly approved and scoped.2122## Workflow23241. Choose the smallest MCP surface for the question.25 - Spark plans and stage metrics: `mcp/spark.mcp.json` or `mcp/databricks.mcp.json`26 - Kafka lag and topic metadata: `mcp/kafka.mcp.json`27 - Orchestration run state: `mcp/airflow.mcp.json`28 - Warehouse inspection: `mcp/snowflake.mcp.json`, `mcp/bigquery.mcp.json`, or `mcp/postgres.mcp.json`29 - Release and job metadata: `mcp/dbt-cloud.mcp.json`, `mcp/github.mcp.json`30312. Configure read-only first.32 From `mcp/README.md`:33 - replace placeholder `command` with the MCP server binary in use34 - scope credentials to read-only inspection roles35 - set allowlists (`topic`, catalog, schema) to minimum necessary scope36 - validate connectivity outside the agent session before relying on tool output37383. Establish the diagnostic sequence.39 Typical order:40 - confirm symptom (lag, failure rate, freshness breach)41 - pull live metadata (consumer lag, last successful run, stage skew)42 - compare against baseline or SLA from `data-observability-and-sla-management`43 - form hypothesis before proposing code or infra changes44 - record findings in incident or backfill evidence templates when action follows45464. Apply platform-specific investigation patterns.47 - **Spark OOM or timeout**: inspect stage breakdown, shuffle read/write bytes, spill metrics, and partition counts; prefer physical plan fixes over blind scale-up48 - **Kafka lag**: inspect group lag per partition, DLQ rate, and retention headroom; pause replay until lag root cause is classified49 - **Orchestration failures**: inspect task logs, retry history, and upstream sensors; do not trigger wide backfills from a single red task without plan gates5051 Load `references/mcp-data-observability-patterns.md` for tool-to-symptom mapping.52535. Pair observability with safety skills.54 - replay or repair: `safe-backfill-and-replay-orchestration`55 - streaming hardening: `kafka-resilience-and-schema-evolution`56 - serverless Spark: `spark-serverless-reliability-and-state-management`57 - incident response: `incident-triage-and-pipeline-recovery`58596. Close the loop after changes.60 Re-query MCP signals to confirm lag recovery, run success, or freshness restoration before publish reopen.6162## Common Rationalizations6364| Rationalization | Reality |65| --- | --- |66| "We can infer the problem from logs alone." | Logs without lag, plan, and run metadata miss systemic causes like skew and consumer stall. |67| "MCP access should use admin credentials for speed." | Over-scoped tokens increase blast radius when agents suggest broad changes. |68| "Observability is only for on-call humans." | Agents without live signals replicate dangerous guesswork at machine speed. |69| "One MCP server is enough for every stack." | Multi-platform estates need focused templates per system, not one mega-connection. |7071## Red Flags7273- agents propose infra or code changes without querying live signals74- MCP templates use write-capable credentials by default75- no allowlists on Kafka topics or warehouse schemas76- lag or run failure is acknowledged but not tied to an owner or SLA77- observability findings are not linked to backfill or incident evidence7879## Verification8081- [ ] The correct MCP template is selected and validated read-only82- [ ] Live signals were queried before structural changes were proposed83- [ ] Findings map to a named safety skill when replay or publish is involved84- [ ] Post-change signals confirm recovery before publish reopen
vaquarkhan