Data Catalog Workflow
Goal
Produce a customer-readable Markdown catalog whose YAML frontmatter is a valid DS_CATALOG_V1 machine contract. Preserve uncertainty explicitly so inferred or assumed relationships never appear confirmed.
Flow
- Confirm the engagement name and the caller-approved durable output path.
- Inventory entities at business grain. Record source access, tier, volume, profile pointer, classification, lineage, and open questions without copying column-level profile data.
- Assign every relationship a stable
rel-* identifier. Record endpoints, maximum cardinality, both endpoint minimums, one or more paired join-key fields, confidence, and evidence basis.
- Reconcile coverage counts with the entity and relationship records.
- Render the human-readable sections from the YAML facts, ending with the canonical Data Science and Engineering Coaching disclaimer footer. Narrative can explain facts but cannot redefine them.
- Validate the artifact with
scripts/validate_catalog.py before treating it as ready for review.
Inputs
- Engagement context and a caller-approved output path
- Data source inventory and access status
- Business entity names, grain, and declared relationships
- Existing per-dataset profile paths, when available
- Privacy classifications or standards citations produced by the owning privacy workflow
Success criteria
- The frontmatter declares exactly
catalog_version: DS_CATALOG_V1 and validates against assets/ds-catalog-v1.schema.json.
- Entity IDs and relationship IDs are unique and stable. Every endpoint and lineage reference resolves.
- Every relationship declares
cardinality as its maximum multiplicity plus from_minimum and to_minimum as zero or one.
- Join keys use one string on both sides or paired arrays of equal length, and record field names only without primary-key, foreign-key, or uniqueness roles.
- Relationship confidence is one of
confirmed, inferred, or assumed, and every relationship records its basis.
- Classification uses the
privacy-standards citation-field names. The catalog does not invent standards identifiers.
- Column statistics and feature metadata remain behind
profile_ref rather than being copied into the catalog.
- Every customer-facing catalog ends with the canonical Data Science and Engineering Coaching disclaimer footer.
Constraints
- Treat catalog enrichment as user-driven. Offer enrichment when a source is missing, but do not change the catalog silently.
- Record source locations as paths or connection references, never embedded credentials.
- Keep DCAT alignment non-binding. The catalog is native YAML, not RDF, and makes no DCAT conformance claim.
- Keep tier recording separate from tier behavior.
data-catalog records the tier; dataops owns what that tier means.
- Keep rendering separate from semantic authority. Diagram tools consume declared relationships and do not infer new ones.
- End every customer-facing catalog with the canonical Data Science and Engineering Coaching disclaimer from
disclaimer-language.instructions.md.
Stop rules
- Stop and request clarification when an entity grain, relationship endpoint, join-key pairing, or endpoint minimum is ambiguous.
- Stop and retain
inferred or assumed confidence when evidence does not support confirmed.
- Stop before a durable write when the caller has not confirmed the destination.
- Stop and route privacy interpretation to
privacy-standards or the Privacy Planner when citation values or DPIA status are unknown.
Package resources
| Resource |
Use |
| catalog-contract.md |
Read for the authoritative field, identity, multiplicity, and body-generation rules |
| dcat-crosswalk.md |
Read when planning a DCAT, DCAT-AP, or catalog-platform export |
| provenance.md |
Read for standards selection, licensing, and non-conformance boundaries |
| ds-catalog-v1.md |
Copy when starting a new catalog |
| northwind-catalog.md |
Read as a complete valid example with scalar and composite join keys |
assets/ds-catalog-v1.schema.json |
Use as the structural JSON Schema for parsed frontmatter |
scripts/validate_catalog.py |
Execute with uv run python scripts/validate_catalog.py <catalog.md> to validate a catalog |
Attribution
The native contract, workflow, schema, template, examples, and validator are repository-original content licensed CC BY 4.0.
The crosswalk paraphrases selected concepts from W3C DCAT 3, W3C PROV-O, and Frictionless Table Schema v2. Standard names and term identifiers are factual citations, the crosswalk prose and the paired-array join-key contract are independently authored, and no upstream schema, example, table, or substantial excerpt is reproduced. Because no upstream expression is redistributed, the package remains solely CC BY 4.0. See provenance.md.
1---2name: data-catalog3description: Create and enrich durable data catalogs using the native DS_CATALOG_V1 Markdown contract, declared entity relationships, privacy citation fields, and stable relationship IDs. Use when inventorying engagement data, recording semantic relationships, or preparing a catalog for ERD rendering.4license: CC-BY-4.05---6
7# Data Catalog Workflow
8
9## Goal
10
11Produce a customer-readable Markdown catalog whose YAML frontmatter is a valid `DS_CATALOG_V1` machine contract. Preserve uncertainty explicitly so inferred or assumed relationships never appear confirmed.
12
13## Flow
14
151. Confirm the engagement name and the caller-approved durable output path.
162. Inventory entities at business grain. Record source access, tier, volume, profile pointer, classification, lineage, and open questions without copying column-level profile data.
173. Assign every relationship a stable `rel-*` identifier. Record endpoints, maximum cardinality, both endpoint minimums, one or more paired join-key fields, confidence, and evidence basis.
184. Reconcile coverage counts with the entity and relationship records.
195. Render the human-readable sections from the YAML facts, ending with the canonical Data Science and Engineering Coaching disclaimer footer. Narrative can explain facts but cannot redefine them.
206. Validate the artifact with `scripts/validate_catalog.py` before treating it as ready for review.
21
22## Inputs
23
24* Engagement context and a caller-approved output path
25* Data source inventory and access status
26* Business entity names, grain, and declared relationships
27* Existing per-dataset profile paths, when available
28* Privacy classifications or standards citations produced by the owning privacy workflow
29
30## Success criteria
31
32* The frontmatter declares exactly `catalog_version: DS_CATALOG_V1` and validates against `assets/ds-catalog-v1.schema.json`.
33* Entity IDs and relationship IDs are unique and stable. Every endpoint and lineage reference resolves.
34* Every relationship declares `cardinality` as its maximum multiplicity plus `from_minimum` and `to_minimum` as `zero` or `one`.
35* Join keys use one string on both sides or paired arrays of equal length, and record field names only without primary-key, foreign-key, or uniqueness roles.
36* Relationship confidence is one of `confirmed`, `inferred`, or `assumed`, and every relationship records its basis.
37* Classification uses the `privacy-standards` citation-field names. The catalog does not invent standards identifiers.
38* Column statistics and feature metadata remain behind `profile_ref` rather than being copied into the catalog.
39* Every customer-facing catalog ends with the canonical Data Science and Engineering Coaching disclaimer footer.
40
41## Constraints
42
43* Treat catalog enrichment as user-driven. Offer enrichment when a source is missing, but do not change the catalog silently.
44* Record source locations as paths or connection references, never embedded credentials.
45* Keep DCAT alignment non-binding. The catalog is native YAML, not RDF, and makes no DCAT conformance claim.
46* Keep tier recording separate from tier behavior. `data-catalog` records the tier; `dataops` owns what that tier means.
47* Keep rendering separate from semantic authority. Diagram tools consume declared relationships and do not infer new ones.
48* End every customer-facing catalog with the canonical Data Science and Engineering Coaching disclaimer from `disclaimer-language.instructions.md`.
49
50## Stop rules
51
52* Stop and request clarification when an entity grain, relationship endpoint, join-key pairing, or endpoint minimum is ambiguous.
53* Stop and retain `inferred` or `assumed` confidence when evidence does not support `confirmed`.
54* Stop before a durable write when the caller has not confirmed the destination.
55* Stop and route privacy interpretation to `privacy-standards` or the Privacy Planner when citation values or DPIA status are unknown.
56
57## Package resources
58
59| Resource | Use |
60|-------------------------------------------------------|---------------------------------------------------------------------------------------------|
61| [catalog-contract.md](references/catalog-contract.md) | Read for the authoritative field, identity, multiplicity, and body-generation rules |
62| [dcat-crosswalk.md](references/dcat-crosswalk.md) | Read when planning a DCAT, DCAT-AP, or catalog-platform export |
63| [provenance.md](references/provenance.md) | Read for standards selection, licensing, and non-conformance boundaries |
64| [ds-catalog-v1.md](templates/ds-catalog-v1.md) | Copy when starting a new catalog |
65| [northwind-catalog.md](examples/northwind-catalog.md) | Read as a complete valid example with scalar and composite join keys |
66| `assets/ds-catalog-v1.schema.json` | Use as the structural JSON Schema for parsed frontmatter |
67| `scripts/validate_catalog.py` | Execute with `uv run python scripts/validate_catalog.py <catalog.md>` to validate a catalog |
68
69## Attribution
70
71The native contract, workflow, schema, template, examples, and validator are repository-original content licensed CC BY 4.0.
72
73The crosswalk paraphrases selected concepts from W3C DCAT 3, W3C PROV-O, and Frictionless Table Schema v2. Standard names and term identifiers are factual citations, the crosswalk prose and the paired-array join-key contract are independently authored, and no upstream schema, example, table, or substantial excerpt is reproduced. Because no upstream expression is redistributed, the package remains solely CC BY 4.0. See [provenance.md](references/provenance.md).
gabrielmoreira