# Lineage Extract Static > Use when extracting data/process lineage from a codebase or file tree — Python/Java/SQL/DSX/Control-M/dbt/COBOL/Pick-MultiValue-BASIC/scheduler configs/log files into spec-correct OpenLineage 2.0.2 JobEvent + DatasetEvent ndjson plus dual visual report (HTML Cytoscape DAG + Mermaid markdown + CSV). Skill is a framework — model-neutral prompts the in-session AI CLI (Claude Code, Codex CLI, Antigravity CLI (agy), Copilot CLI) uses to analyse chunks via its own LLM context, NOT per-format parser plug-ins. Static analysis only (v1); runtime producers in v2. Also trigger on "lineage report", "data flow extraction", "openlineage", "trace data through the codebase", "where does this dataset come from", "Java lineage", "JDBC/JPA lineage", "Pick/MultiValue lineage". - Skill: `joogy06/lineage-extract-static` (Agent Skill, multi-file: 34 files) - Install (CLI): `npx skillmds@latest add joogy06/lineage-extract-static` - Raw SKILL.md: https://api.skillmd.com/api/skills/joogy06/lineage-extract-static/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Web & Frontend - Author: joogy06 (https://skillmd.com/u/joogy06) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/joogy06/lineage-extract-static --- # lineage-extract-static — Cross-System Data/Process Lineage Skill S033 design at `docs/plans/2026-05-14-lineage-extract-static-skill-design.md`. ## What it does Reads any file tree and emits: - **`openlineage.ndjson`** — canonical stream of `JobEvent` + `DatasetEvent` (one event per line). Spec-correct static lineage per OpenLineage 2.0.2 — NO synthesised `Run` by default. HARD-RULE 1. - **`openlineage.json`** — derived `{"events": [...]}` bundle for tooling that prefers single-document input. - **`lineage_edges.csv`** — single denormalized CSV: `src_dataset_namespace, src_dataset_name, src_kind, target_job_namespace, target_job_name, target_job_kind, edge_kind, confidence, evidence_file, evidence_line, extractor_id`. Opt-in `--output-format=ol-relational` splits into 4 CSVs. - **`report.html`** — self-contained Cytoscape DAG with sortable tables and download links. Air-gap safe (vendored cytoscape.min.js OR CDN fallback OR `--no-vendor` Mermaid-only). - **`report.md`** — GitHub-renderable Mermaid summary (flowchart capped at 50 nodes + truncation pointer to `report.html` for over-cap). - **`manifest.json`** — per-run metadata (run_id, costs, counters, redaction_count, errors). Validated against `schemas/lineage-manifest.v1.json`. - **`errors.jsonl`** — per-file extraction failures. ## When to use it Trigger on these user phrasings: - "build me a lineage report on /path/to/project" - "trace data flow through this codebase" - "where does this dataset come from" - "extract lineage to OpenLineage format" - "generate a data flow visualisation" - "what jobs read/write to this table" Skip when: - The user wants RUNTIME lineage (job execution traces) — that's openlineage-spark / openlineage-airflow producers, v2. - The user wants COLUMN-level lineage — defer to v1.1 (needs schema metadata not available statically). - The user wants CROSS-REPO lineage — single project root only in v1. - The user wants BUSINESS / FUNCTIONAL lineage (semantic annotations on jobs) — post-v1 enrichment composing `intent-extract`. ## Architecture (the LLM-driven framework) The skill is the **framework**, the in-session AI CLI is the **parser**. There are NO per-format AST parsers (`sqlglot` / `tree-sitter` / `xml.sax`) in `scripts/`. The agent's LLM handles arbitrary input formats — Python, Java, SQL, DSX, Control-M, dbt, COBOL, Pick/MultiValue BASIC, log files — anything in its training coverage. The skill provides: - **5 model-neutral prompts** in `prompts/*.md` — instructions the agent uses for per-chunk analysis, chunk merging, identity resolution, and redaction. - **8 deterministic Python scripts** in `scripts/*.py` — chunk_file, accumulate, merge_into_ol, validate_ol, redact, project_views, render_report, install_vendor. - **6 reference docs** in `references/*.md` — OL spec, confidence classifier, dataset identity, output formats, chunking strategy, anti-patterns. - **3 JSON schemas** in `schemas/` — lineage-finding.v1, lineage-manifest.v1, openlineage-2.0.2-vendored. Scripts have NO LLM calls. Prompts have NO Python code. The two halves are interoperable across Claude Code / Codex CLI / Antigravity CLI (agy) / Copilot CLI because the prompts are model-neutral (HARD-RULE 8). ## Invocation flow When a user issues `lineage-extract-static ` (or equivalent NL: *"build me a lineage report on /path/to/project"*): ``` 1. Agent reads this SKILL.md (the orchestration playbook). 2. Agent runs `scripts/chunk_file.py` over the project tree. Output: per-file `manifest.json` at ~/.cache/lineage-extract-static/runs//files// carrying path, sha256, size, line_count, chunked: bool, chunk_count. 3. For each file: 3a. Small file (<= 5 MB / <= 20k lines) → agent reads the whole file, applies `prompts/analyze-file.md`, emits ONE `lineage-finding.v1` JSON object to `chunk_0001.jsonl` (single chunk). 3b. Large file → agent reads chunk-by-chunk (default 2000 lines/chunk with 50-line overlap), applies `prompts/analyze-file.md` per chunk, writes per-chunk JSONL. Then applies `prompts/merge-chunks-within-file.md` to reconcile boundary_status markers; agent calls `scripts/accumulate.py` to perform the deterministic boundary-pairing predicate (NOT an LLM judgment) and emit the file-level rollup at `summary.json`. 4. After all files done, agent applies `prompts/merge-across-files.md` to combine per-file rollups + runs `prompts/resolve-identity.md` to canonicalize dataset URIs (3-step waterfall: SQL FQN → repo-rel path → alias map). 5. Agent runs `scripts/redact.py` on the combined rollup (fail-closed). 6. Agent runs `scripts/merge_into_ol.py` → emits openlineage.ndjson + openlineage.json + lineage_edges.csv to /tmp/lineage-extract-static-/. JobEvents now carry a `sourceCodeLocation.contentSha256` JOB facet (sha256 of the RAW on-disk source bytes — the cross-engine v1.1 join key; see below). DatasetEvents carry a `schema` facet (SchemaDatasetFacet, 2026-07-01 uplift) when the rollup's `dataset_schemas` names the dataset's columns — named-in-source only (dbt schema.yml, SQL DDL, CSV headers, COBOL copybooks, staged SELECT lists), `type` only when stated, never invented (see references/output-formats.md). OUTPUT DatasetEvents additionally carry a `columnLineage` 1-2-0 facet (2026-07-02 uplift) when the rollup's `column_lineage` names per-column derivations (SQL rename lists, explicit passthroughs, expressions naming their sources; `select * from ` expands 1:1 via the passthrough marker — multi-input or unknown-parent SELECT-* yields NOTHING), and a `documentation` facet when `dataset_descriptions` states one. Schema facet fields carry `description` when stated in source. No confidence key ever rides on facets — confidence stays in the edge/report layer. §9 corollary (a) (user-approved 2026-07-02): a `select * from ` model with no own column set inherits the parent's NAMED set as its own schema facet (name + parent-stated type only) — deterministic SQL semantics, same single-input/known-parent constraints, never invented. 6b. Agent runs `scripts/project_views.py` → consumes the just-emitted openlineage.ndjson + lineage_edges.csv and emits `views.json` (the L1/L2 render payload). DETERMINISTIC, stdlib, NO LLM — pure graph algebra. Skipped only if `--views=` is passed empty. 7. Agent runs `scripts/render_report.py` (ONCE, passing `views.json`) → produces the single self-contained `report.html` (a 3-tab L1/L2/L3-hidden Cytoscape switcher) + report.md + datasets.csv + jobs.csv + edges.csv (the OL-relational opt-in) to same dir. 8. Agent prints the location of the report directory to the user. ``` ## CLI flags ``` lineage-extract-static analyze --output-dir (default: /tmp/lineage-extract-static-/) --include-low-precision (opt-in for plug-ins with precision_floor < 0.85) --output-format=ol-relational (opt-in for 4-CSV split; default = single denormalized) --views=l1,l2 (multi-view projection; DEFAULT ON. L1 = file interaction (job-retained); L2 = table/data + column. Pass --views= empty to disable. Folded into step 6b so render is called ONCE with the views.json payload.) --with-static-run (opt-in for RunEvent-wrapped compatibility export) --merge-by-basename (advisory speculative-confidence basename matching) --no-vendor (Mermaid-only output if Cytoscape vendor absent) --since (incremental scan) --parallel N (per-file ProcessPoolExecutor workers; default = CPU) --aliases (alias map; default .lineage/aliases.yaml) lineage-extract-static diff exit 0 = no breaking changes / exit 1 = datasets removed / exit 2 = jobs removed ``` ## HARD-RULEs 1. **Spec-correct OL output, NOT synthesised runs.** Canonical emission is `JobEvent` + `DatasetEvent` with NO `Run` wrapper. `RunEvent` wrapping is opt-in via `--with-static-run` for consumer compatibility only. Phantom runs pollute downstream catalogs (Marquez, Atlan, DataHub). 2. **Bright-line confidence classifier.** Any extraction whose `evidence_snippet` contains f-string syntax, `.format()`, `%`-format, env-var template, OR any symbol the LLM cannot resolve to a literal within the chunk → forced `speculative`. NEVER `grounded` when interpolation/dynamic-resolution is present. Enforced by the prompt template + post-emission validator in `scripts/validate_ol.py`. 3. **Identity waterfall pre-computed, never inferred by LLM.** The 3-step resolution (SQL FQN → repo-root-relative path → alias map) runs in `prompts/resolve-identity.md` BEFORE OL emission. Basename-only merge OFF by default; advisory speculative only when `--merge-by-basename` is explicitly enabled. 4. **Secret redaction is fail-closed.** `scripts/redact.py` runs on every aggregated rollup BEFORE OL emission. Any redaction error = abort, NEVER emit partial. Two-layer redaction: prompt-level (LLM instructed not to emit credentials) + post-LLM (regex-based scrubbing). 5. **Sandbox + cache discipline.** Per-run cache at `~/.cache/lineage-extract-static/runs//` mode `0700` (NEVER `/tmp`). Atomic writes via `.tmp.` + `os.replace()`. TTL 30 days + LRU eviction at `LINEAGE_CACHE_MAX_GB=10`. 6. **DoS hard caps.** `LINEAGE_HARD_FILE_LIMIT_MB=50` skip-with-warn, `LINEAGE_MAX_DURATION_S=3600` global wall-clock cap, `LINEAGE_CACHE_MAX_GB=10` cache cap. Exit `PARTIAL` (not failure) when caps hit; surface in `manifest.json`. 7. **XSS-safe rendering.** Every user-controlled string (file paths, dataset names, evidence snippets) interpolated into HTML via `html.escape()` server-side + `textContent =` / `cy.json({elements})` (never any HTML-string sink — `innerHTML` / `outerHTML` / `insertAdjacentHTML` / `document.write`) client-side, on EVERY view tab. Required test `test_html_escape_hostile_filenames_inert_across_tabs` enforces this across the L1 and L2 element sets. Hostile filenames must NOT execute. 8. **Cross-tool portability.** Prompts in `prompts/*.md` are model-neutral — no Anthropic-specific anchors, no Claude-only facets, no `<>` tags. Tested against Claude Code + Codex CLI + Gemini CLI (pre-retirement) + Copilot CLI; the gemini CLI retired 2026-06-18 — use Antigravity CLI (agy) instead. ## Confidence taxonomy (bright-line classifier) See `references/confidence-classifier.md` for worked examples. Quick reference: | Tier | Rule | OL emission | |---|---|---| | `grounded` | Literal string token (path/table-name); all symbols resolve in the local context (function args, top-level constants) | Solid edge; feeds gate decisions | | `inferred` | Name-resolution heuristic — env-var resolved against in-repo `.env` / `config.yaml`; relative path resolved against `__file__`; SQL `FROM ` resolved against same-file CTE | Dashed edge; advisory only | | `speculative` | String interpolation (f-string / `.format()` / `%`-format / template literal) within ±20 lines; env-var without in-repo resolution; unresolved symbol; `SELECT *` without schema metadata; basename-only match | Dotted edge; collapsed by default in HTML; never feeds gates | ## Identity resolution (3-step waterfall) See `references/dataset-identity.md` for full rules. Quick reference: 1. **SQL FQN** (highest precedence). `namespace = ://:/`, `name = .`. Default schemas: PostgreSQL `public`, Oracle ``, Snowflake `.`. 2. **Repo-root-relative absolute path** (filesystem datasets). `realpath` to dereference symlinks. `namespace = file://`, `name = `. 3. **Configurable alias map** (`.lineage/aliases.yaml`, optional). Resolves DSN strings, NFS mount mirrors, etc., to canonical URIs. Basename-only merge is OFF by default. Behind `--merge-by-basename` flag (advisory), basename matches emit edges with `confidence: speculative` + `possible_alias` facet linking candidates — NEVER silently merged. ## Output layout ``` /tmp/lineage-extract-static-/ ├── openlineage.ndjson ← canonical OL stream (JobEvent + DatasetEvent per line) ├── openlineage.json ← derived bundle {"events": [...]} ├── lineage_edges.csv ← single denormalized CSV (default) ├── datasets.csv ← OL-relational opt-in ├── jobs.csv ← OL-relational opt-in ├── edges.csv ← OL-relational opt-in ├── runs.csv ← OL-relational opt-in (only when --with-static-run) ├── manifest.json ← per-run metadata (validated against lineage-manifest.v1) ├── errors.jsonl ← per-file extraction failures ├── views.json ← L1/L2 render payload (project_views.py; default on) ├── report.md ← Mermaid summary (air-gap L1 fallback) └── report.html ← single self-contained 3-tab (L1/L2/L3-hidden) Cytoscape switcher + sortable tables + downloads ``` ## Multi-view projection (L1 / L2) + the contentSha256 join key `scripts/project_views.py` is a DETERMINISTIC, stdlib, NO-LLM post-pass that consumes the already-emitted `openlineage.ndjson` + `lineage_edges.csv` (the CSV carries the per-edge confidence/evidence the OL events drop, plus the `schedules`/`depends_on` edges that never reach OL). It rebuilds the typed bipartite graph, reattaches per-edge confidence/evidence, and emits two honest abstraction views plus a single `views.json` render payload: - **L1 — file interaction (JOB-RETAINED).** Filter to `kind=file` datasets but KEEP the job node (dataset → job → dataset). It does NOT collapse jobs and NEVER fabricates a file→file cross-product edge (locked user decision). Every edge keeps its CSV confidence + evidence. - **L2 — table/data + column.** Filter to `kind ∈ {table,topic,queue}`, keep job nodes. Column lineage is read from the `columnLineage` 1-2-0 facet (when present) and nested under its parent table edge, surfaced as an expand-on-click `
` table — never graph clutter. Tolerant of an absent facet (table-level L2 still works). - **L3 — functional.** Deferred to v1.1 (a separate forge cycle); the report reserves a hidden L3 tab. `render_report.py` builds ONE self-contained `report.html` with ONE Cytoscape instance and a 3-tab `.view-tabs` switcher that swaps `cy.json({elements})` between the pre-built L1/L2 sets from `views.json`. XSS-safe throughout: every embedded element set goes through the same `html.escape` + `<`/`>`/`&`/U+2028/ U+2029 escape chain and is consumed only via `cy.json` / `textContent` — the template assigns NO HTML-string sink (`innerHTML` / `outerHTML` / `insertAdjacentHTML` / `document.write`) anywhere. Air-gap safe: with no vendor and no network, `report.html` is skipped and `report.md` carries the L1 Mermaid. **`sourceCodeLocation.contentSha256` JOB facet (v1.1 join key, shipped now).** `merge_into_ol.py` threads the already-computed `file_sha256` (`chunk_file.sha256_of_file`) into each JobEvent's `sourceCodeLocation` facet. The byte definition is **`contentSha256 = sha256 of the RAW on-disk source file bytes`** — a streaming whole-file hash with NO encoding normalization, NO copybook/symbol expansion, NO chunk scoping. This definition is IDENTICAL to the `mainframe-lineage-parsers` engine's `sourceCodeLocation.contentSha256` so the two streams join on the same key (the lineage JOB ↔ legacy-code-intel artifact join the deferred L3 view will use). The OL core spec stays pinned at 2.0.2 — the facet is purely additive and self-describing. ## Anti-patterns — STOP if you catch yourself - **Emitting `Run` by default** — canonical OL spec for static lineage is JobEvent + DatasetEvent ONLY. RunEvent is opt-in (`--with-static-run`). - **Inferring `grounded` confidence when evidence has f-string / `.format()` / `%`-format** — these are forced `speculative`. HARD-RULE 2. - **Merging datasets by basename alone** — basename-only merge is OFF by default. When `--merge-by-basename` is enabled, the match still gets `confidence: speculative` + `possible_alias` facet. - **Writing to `/tmp` for the run cache** — `~/.cache/lineage-extract-static/runs//` mode 0700 ONLY. HARD-RULE 5. - **Emitting partial output after a redaction error** — `scripts/redact.py` is fail-closed. Any error aborts the run. HARD-RULE 4. - **Synthesising a `Run` "for compatibility" without the user asking** — `--with-static-run` is OPT-IN. Synthesising on every run pollutes catalogs. - **Building per-format parsers in `scripts/`** — the LLM is the parser. v1 has NO `sqlglot` / `tree-sitter` / `xml.sax` plug-ins. If accuracy is insufficient for a specific format, add a deterministic plug-in in v1.1 against a frozen contract. - **Hard-coding file extensions** — chunk_file.py is format-agnostic (just I/O); the LLM decides which formats it can extract from. Unsupported formats produce `gap: language_unsupported` entries. ## Cross-tool portability Prompts in `prompts/*.md` are tested against all four AI CLIs: - Claude Code (Claude Opus 4.7) - Codex CLI (GPT-5.4) - Antigravity CLI (agy) - Copilot CLI (GPT-5.4 backend) Run `~/.claude/skills/research-for-skills/cross-tool-portability/scripts/verify-skill-portability.sh` to confirm. HARD-RULE 8. ## Dependencies & environments (CLI / VS Code end users) The **core is pure stdlib and always runs** — chunking, accumulation, the OpenLineage ndjson/CSV merge, redaction, and the L1/L2 view projection need **zero** third-party packages. The libraries in `requirements-optional.txt` are **OPTIONAL enhancers**; this skill never pip-installs at runtime. **See your situation first — the doctor never installs anything:** ```bash python3 scripts/check_deps.py ``` It prints the **active interpreter**, which enhancers are present/missing, what degrades when one is absent, a **PEP-668-safe** install recipe, and a sibling full-deps interpreter if one exists on the host. | Enhancer | Unlocks | If absent | |---|---|---| | `jinja2` | the self-contained `report.html` + the 3-tab L1/L2 view-switcher | `report.html` is skipped; `report.md` (Mermaid) is the air-gap fallback (L2 column detail is HTML-only) | | `jsonschema` | write-time OL 2.0.2 schema validation | OL is written unvalidated | **Install (only if you want the HTML report / validation).** Modern distros mark the system `python3` **PEP-668 externally-managed**, so a bare `pip install` is BLOCKED — use one of: ```bash # Recommended — PEP-668-safe project venv (CLI or VS Code): python3 -m venv .venv && .venv/bin/pip install -r requirements-optional.txt # then run the skill's scripts with .venv/bin/python # Per-user (where allowed): python3 -m pip install --user -r requirements-optional.txt # Last resort (system py): python3 -m pip install --break-system-packages -r requirements-optional.txt # Air-gapped (offline wheels): python3 -m pip install --no-index --find-links= -r requirements-optional.txt ``` **VS Code:** select the venv interpreter as the workspace interpreter — the skill's scripts run under whatever `python3` the CLI/extension invokes, so an env without `jinja2` silently produces the Mermaid `report.md` instead of `report.html`. Run `check_deps.py` in the integrated terminal to confirm which interpreter is active. ## Composition with other skills - `wiring-extract-static` — produces source-code call graph; lineage-extract-static can use wiring snapshots to enrich job-to-job depends_on edges (v1.1 enhancement). - `intent-extract` — adds per-job functional intent annotations (post-v1 enrichment for business / functional lineage). - `intent-map-render` — dual Mermaid + Cytoscape rendering precedent reused for `scripts/render_report.py`. - `visual-companion` — Cytoscape vendor templates extended at WP-7 (`scripts/install_vendor.sh`). - `dep-currency-check` — confidence_level pattern precedent (grounded / inferred / speculative). ## See also - `references/openlineage-spec-2.0.2.md` — what to emit, with worked examples - `references/confidence-classifier.md` — bright-line grounded/inferred/speculative rules - `references/dataset-identity.md` — 3-step waterfall codified pre-code - `references/output-formats.md` — OL JSON + CSV + HTML + Mermaid contract - `references/chunking-strategy.md` — two-phase + boundary_status reconciliation - `references/anti-patterns.md` — synthetic Run, basename merge, scope creep ## Security — XML / XHTML parsing When parsing any XML or XHTML payload from a remote API, untrusted file, or user-supplied source, NEVER use stdlib `xml.etree.ElementTree`, `xml.dom.minidom`, or `lxml.etree.fromstring` without XXE protection. Use `defusedxml` (`pip install defusedxml`) and replace `xml.etree.ElementTree` → `defusedxml.ElementTree`, `lxml.etree` → `defusedxml.lxml`. Stdlib XML parsers expand external entities by default and are vulnerable to billion-laughs / XXE / DTD-retrieval / SSRF-via-entity attacks (CWE-611). Local skill applicability: - API payloads that may legitimately be XML (storage format, error responses) - Imported / exported workflow files - Bulk import / migration paths For HTML/XHTML rendering of downstream output (storage format → display), sanitise with `bleach` or `nh3` BEFORE inserting into a browser context — never raw-render API-returned XHTML. See `llm-security` SKILL.md §4.4 for context-appropriate escaping rules.