Data Lineage Observer
Overview
This skill reads data lineage as a graph of datasets and jobs built from OpenLineage events, then answers operational questions: downstream impact of a schema change, root cause of stale or wrong data, propagation of personal data, orphan datasets and freshness against expected cadences. It works from a Marquez, DataHub or OpenMetadata instance when one exists and from raw event exports otherwise. The default posture is read-only and evidence-based: pull events and graph responses, compute offline, cite run ids, job names and event times, and never change a pipeline, a producer configuration or a catalog entry without the user's explicit approval.
When it applies
- A table, column or file layout is about to change and the owner needs the list of jobs and datasets that read it, with their owners.
- A dashboard or report shows stale or wrong numbers and the source of the problem is somewhere upstream.
- Privacy asks where an email, a national id or a phone number column ends up (warehouse, exports, feature stores).
- The platform team wants to find datasets nobody reads, jobs with no declared datasets, or namespaces that split the graph.
- Freshness SLAs must be verified from run history rather than from the scheduler's green ticks.
- A new producer (Airflow, dbt, Spark, Flink, Dagster, Great Expectations) must be wired to the lineage backend.
Hand over to data-quality-agent for profiling and assertion design, to
sql-analyst for query-level debugging and to data-privacy-auditor for the
legal classification of the fields found.
Engines and sources
OpenLineage events
RunEventwitheventTypeinSTART,RUNNING,COMPLETE,ABORT,FAIL,OTHER;eventTime;producer;schemaURL;run.runIdandrun.facets(nominalTime,parent,errorMessage,processing_engine);job.namespace,job.name,job.facets(sql,sourceCode,sourceCodeLocation,jobType);inputs[]andoutputs[]withnamespace,name,facets(schema,dataSource,columnLineage,lifecycleStateChange,symlinks),inputFacets(dataQualityMetrics,dataQualityAssertions) andoutputFacets(outputStatistics).DatasetEventandJobEventcarry static metadata without a run.- Naming: namespace is the datasource URI (
postgres://host:5432,snowflake://account,s3://bucket,kafka://broker:9092), name isschema.tableor the object path. Field tables, state machine and minimal events:references/openlineage-schema.md.
Marquez
- Ingest:
POST /api/v1/lineage. - Read:
GET /api/v1/namespaces,GET /api/v1/namespaces/{ns}/datasets/{name},GET /api/v1/namespaces/{ns}/jobs/{job}/runs,GET /api/v1/lineage?nodeId=dataset:{ns}:{name}&depth=3,GET /api/v1/column-lineage?nodeId=datasetField:{ns}:{name}:{field},GET /api/v1/events/lineage?after=&before=&limit=for raw events,GET /api/v1/search?q=. - DataHub (
/openapi/openlineage/api/v1/lineage, GraphQLsearchAcrossLineage) and OpenMetadata (GET /api/v1/lineage/table/name/{fqn}?downstreamDepth=3) equivalents:references/marquez-api.md.
Producers
- Airflow:
apache-airflow-providers-openlineage,[openlineage] transportandnamespaceinairflow.cfg, job namedag_id.task_id. - dbt:
openlineage-dbt, run withdbt-ol run,OPENLINEAGE_URL,OPENLINEAGE_NAMESPACE. - Spark:
spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListenerandspark.openlineage.transport.*,spark.openlineage.namespace. - Flink listener, Great Expectations
OpenLineageValidationAction, Dagster. - Config snippets and the post-enable checklist:
references/producers-setup.md.
Navin integration
- Scheduler pods, Spark clusters, warehouse hosts and object storage are
inspected through the
infratool at observe level (kubectl get configmap airflow-config -o yaml,aws s3 ls); prepare and execute levels only for changes the user approved. - Datadog, Grafana, Slack, GitHub, Notion: deliver findings and read
pipeline alerts through the
connectorstool. MARQUEZ_API_KEY,DATAHUB_TOKEN,OPENMETADATA_JWT, warehouse passwords come from environment variables; report their presence only.jq,curl,dot(Graphviz) are installed with thetoolbeltskill when missing.- Event exports can reach hundreds of megabytes: search them with the
greptool ("eventType": "FAIL", a dataset name, arunId) before reading.
Workflow
- Frame the question as a graph query: a start node (dataset
namespace/nameor job), a direction (downstream for impact, upstream for root cause), a depth and a time window. Write down the naming convention in use so the node can be addressed exactly. - Choose the source. Catalog present and reachable: query the lineage API
directly and export the raw events for the window with
GET /api/v1/events/lineage?after=<iso>&limit=5000(Marquez) intoevents_<from>_<to>.json, then convert to JSONL withjq -c '.events[]'. No catalog: collect events from the producer side (filetransport, Kafka topic dump, Airflow task logs withdebug_mode). - Validate the export before trusting it:
python3 scripts/lineage_graph.py examples/openlineage_events.jsonl stats(replace the sample by the export). Check that every run id has a terminal event, that producers are the expected ones, and that datasets are not split across namespaces (postgres://host:5432versuspostgres://host). Fix the mapping in the report, not in the data. - Impact analysis:
python3 scripts/lineage_graph.py examples/openlineage_events.jsonl impact raw.orderslists downstream datasets with hop count and the job on each hop; add--format dotand render withdot -Tsvgwhen the user wants a picture. For a column change, read thecolumnLineagefacets of the first-hop outputs (fields whoseinputFieldsreference the column) and continue hop by hop; recordtransformationTypeortransformations[].subtypeto tell direct copies from aggregations. - Root cause of stale or wrong data:
python3 scripts/lineage_graph.py examples/openlineage_events.jsonl upstream analytics.daily_revenuegives the sources; thenpython3 scripts/freshness_check.py examples/openlineage_events.jsonl examples/expected_cadence.jsonshows, per expected dataset, the lastCOMPLETEwrite, its age, the producing job, its last run state and theerrorMessageof a failed run. Exit 1 marks at least one stale dataset or failed last run. Walk upstream from the first stale node until a job withFAILor aCOMPLETEwithoutputStatistics.rowCount: 0appears; that is the root cause candidate. Confirm with the scheduler logs throughinfrabefore naming it. - Orphans and hygiene:
python3 scripts/lineage_graph.py examples/openlineage_events.jsonl orphansseparates datasets written but never read (candidates for deletion or missing consumers) from datasets read but never written (external sources or a producer that is not instrumented). Classify each with the owner. - Personal data propagation: start from the columns tagged in the
schemafacet (descriptioncontaining PII, Marquez tagsPII), followcolumnLineagedownstream, and list every dataset where the field lands withmaskingfalse. Datasets reached through jobs without column lineage are reported as "unknown propagation", never as safe. - Freshness SLAs: build the cadence file from the scheduler definitions
(Airflow
schedule, dbt job cron) or from thenominalTimefacets, runscripts/freshness_check.pywith--nowset to the report time, and keep the file in the repository so the check can run in CI. - Write the deliverable with evidence per row and list the proposed changes (producer config, namespace fixes, SLA alerts) as an approval table.
Rules
- Inputs and outputs are cumulative over a run: a
COMPLETEwithoutoutputsdoes not mean nothing was written; merge with theSTART. eventTimeis observation time; userun.facets.nominalTimefor schedule-based reasoning (late runs, backfills). A backfill emits events with anominalStartTimein the past and a recenteventTime.- A
COMPLETEwithoutputStatistics.rowCount: 0is the most common silent failure; the freshness script shows the row count so it can be flagged. - Airflow
BashOperatorandPythonOperatorwithoutinletsandoutletsproduce jobs with no datasets; the graph has a hole there, say so. - Spark job names change with the execution command and partition paths
create one dataset per day unless
spark.openlineage.dataset.removePath.patternis set; aggregate by prefix in the report. - Marquez
depthcounts edges (dataset to job is one edge); the script counts dataset hops. State which one a number refers to. - Column lineage is only as complete as the producer: dbt-ol without the SQL parser, Spark with disabled facets and Airflow Python tasks emit none.
- Never print SQL facets that contain literals from personal data; quote the table and column names only.
- Never modify
airflow.cfg, dbt profiles, Spark confs, Marquez tags or catalog ownership without approval; prepare the exact change and the rollback. - Deleting or re-running a job is out of scope for this skill; hand the decision to the owner with the evidence.
Deliverable
Produce a Markdown report with:
- Scope: question, start node, direction, window, source (catalog URL or
export file), event counts from
stats. - Graph summary:
| Metric | Value |(events, runs, jobs, datasets, edges, column lineage edges, producers). - Impact or upstream table:
| Hops | Dataset | Via job | Owner | Column lineage | Note |. - Freshness table:
| Dataset | Cadence | Last COMPLETE (UTC) | Age | Producing job | Last job state | Status |. - Root cause timeline:
| Time (UTC) | Job | Run id | Event | Evidence |. - Orphans:
| Kind | Dataset | Owner | Recommendation |. - Personal data propagation:
| Source column | Destination dataset | Destination column | Transformation | Masked | Hops |. - Findings and proposed actions:
| # | Finding | Evidence | Proposed change | Rollback | Needs approval |.
Attach the DOT file when a diagram was produced. When the user wants an HTML
report, render the same content with the studio-html-report skill; post it
through the connectors tool when asked.
Resources
scripts/lineage_graph.py: OpenLineage events (JSONL or JSON array) to a dataset and job graph; subcommandsstats,impact <dataset>,upstream <dataset>,orphans;--format markdown|json|dotor--jsonafter the subcommand;--depthfor walks. Steps 3, 4, 5 and 6.scripts/freshness_check.py: events plus a cadence JSON to last COMPLETE, age, staleness, producing job, last run state and error message;--nowand--json; exit 1 on stale or failed. Steps 5 and 8.examples/openlineage_events.jsonl: one day of synthetic events from Airflow, dbt, Spark and Great Expectations with an hourly ingest, a dbt model with column lineage, a failed report job and a running feature job.examples/expected_cadence.json: the cadence file format (cadence_minutes,grace_minutes,ownerpernamespace/name).references/openlineage-schema.md: RunEvent, DatasetEvent, JobEvent, every facet with its fields, naming conventions, minimal events. Open it when reading raw events or writing a producer.references/marquez-api.md: Marquez endpoints, node ids, graph response shape, raw event export, DataHub and OpenMetadata equivalents. Open it before querying a catalog.references/producers-setup.md: transport configuration and per-producer snippets for Airflow, dbt, Spark, Flink, Great Expectations and Dagster, with the checklist to verify a new producer. Open it for setup requests and when the graph has holes.
