Fabric Metadata Creation

Fabric Metadata Creation - Reference

Generates non-destructive metadata proposals for Microsoft Fabric lakehouse tables, files, views, and semantic models. The result is a steward-review package that can later be mapped into a data catalog such as Microsoft Purview, but it must not apply catalog changes automatically.

The agent's core principle is traceability: every suggestion must distinguish metadata explicitly found in the source from metadata inferred by the agent, include a confidence level, and carry assumptions or open questions when evidence is incomplete.

Global Rules

• Proposal only. Never modify Fabric assets, Purview assets, glossary terms, labels, classifications, owners, stewards, retention policies, or access controls.
• Human review required. Governance outcomes are suggestions, not decisions.
• No invented accountability. Do not invent real owners, stewards, contacts, responsible teams, legal requirements, or compliance decisions. Use Unknown, TBD, or an open question when not explicit.
• Evidence labeling. Every metadata object must include metadata_source with one of: explicit, inferred, or mixed.
• Confidence required. Every proposal must include confidence: high, medium, or low.
• Assumptions required. Every inferred proposal should include assumptions; use an empty list only when there are no assumptions.
• Open questions required. Every asset-level and column-level proposal must include review questions when evidence is incomplete.
• Conservative language. Use candidate, suggested, likely, possible, or requires human review for inferred classifications and sensitivity labels.
• Structured output. Generate only the four final artifacts documented below: proposal JSON/PDF and standalone glossary Markdown/PDF.
• Spanish review documents. Human-readable artifacts must be written in Spanish. Keep original workspace, lakehouse, table, column, measure, semantic model, and glossary candidate names exactly as they appear in the source; do not translate source identifiers.

Helper Script

Use ./scripts/fabric_metadata.py for Fabric discovery/export, PDF rendering, and output validation. The agent must activate .venv/ before running it.

Install dependencies:

test -d .venv || python3 -m venv .venv
source .venv/bin/activate
pip install azure-identity requests markdown weasyprint

WeasyPrint requires system-level Cairo/Pango libraries. They are pre-installed on many Linux environments; on macOS install Pango with brew install pango.

Commands:

python .github/skills/fabric-metadata-creation/scripts/fabric_metadata.py list-workspaces
python .github/skills/fabric-metadata-creation/scripts/fabric_metadata.py list-items <workspace_id> [<item_type>]
python .github/skills/fabric-metadata-creation/scripts/fabric_metadata.py list-lakehouse-tables <workspace_id> <lakehouse_id>
python .github/skills/fabric-metadata-creation/scripts/fabric_metadata.py get-lakehouse-table <workspace_id> <lakehouse_id> <table_name>
python .github/skills/fabric-metadata-creation/scripts/fabric_metadata.py list-semantic-models <workspace_id> [<model_name>]
python .github/skills/fabric-metadata-creation/scripts/fabric_metadata.py export-semantic-model <workspace_id> <semantic_model_id> <output_dir>
python .github/skills/fabric-metadata-creation/scripts/fabric_metadata.py render-pdf <metadata_proposal.json> <metadata_proposal.pdf>
python .github/skills/fabric-metadata-creation/scripts/fabric_metadata.py render-glossary-md <metadata_proposal.json> <glossary_terms.md>
python .github/skills/fabric-metadata-creation/scripts/fabric_metadata.py render-glossary-pdf <metadata_proposal.json> <glossary_terms.pdf>
python .github/skills/fabric-metadata-creation/scripts/fabric_metadata.py render-markdown-pdf <input.md> <output.pdf>
python .github/skills/fabric-metadata-creation/scripts/fabric_metadata.py validate <output_dir>

Use MCP tools first when they are available for interactive discovery. Use the script when MCP does not expose the needed metadata, when exporting semantic model definitions, or when rendering/validating local outputs.

Output Directory

Save each run to:

./metadata_proposals/{SOURCE_NAME}/{YYYY-MM-DD_HHmmss}/

Required files:

Do not generate metadata_proposal.md, metadata_proposal.yaml, evidence.json, or other final artifacts. Temporary discovery/export files may be used while working, but they must not remain in the final output directory.

Default Proposal PDF Sections

The proposal PDF must present these simplified review sections in order. The JSON is the source of truth; the PDF is the human-readable rendering.

1. Resumen ejecutivo
2. Entrada interpretada
3. Activos y columnas
4. Glosario
5. Sugerencias de gobierno
6. Relaciones, linaje y calidad de datos
7. Supuestos y preguntas abiertas

Both PDFs must include an index/table of contents. Sections and subsections must be numbered. Asset sections must be visually isolated with page breaks, and the asset name must appear in the asset table header. Source names that contain Markdown-looking characters, such as # Case Updates, must be rendered as literal names rather than Markdown syntax.

Metadata Categories

Asset-Level Metadata

Use these compact fields for databases, schemas, tables, views, files, semantic models, measures, dimensions, and relationship assets when applicable:

asset_name, asset_type, fully_qualified_name, platform, source_system,
database_name, schema_name, object_type, data_domain, data_product,
environment, lifecycle_status, business_description, technical_description,
data_owner, data_steward, glossary_terms, upstream_lineage,
downstream_lineage, source_of_truth, security_notes, quality_notes,
confidence

Additional required traceability fields: metadata_source, assumptions, open_questions.

Column-Level Metadata

Use these compact fields for columns, semantic model columns, calculated columns, measures, and dimensional attributes when applicable:

column_name, business_name, data_type, nullable, key_role,
measure_or_dimension, business_description, technical_description,
semantic_type, classification, sensitivity_label, pii_indicator,
critical_data_element_candidate, derivation_logic, allowed_values,
example_values, quality_rules, relationship_hints, lineage_hints,
masking_recommendation, confidence

Additional required traceability fields: metadata_source, assumptions, open_questions.

Business Glossary Term Metadata

Use these compact fields for suggested business glossary terms:

term_name, definition, synonyms, related_terms, domain, status,
owner_or_steward, associated_assets, associated_columns, confidence,
metadata_source, open_questions

Use status: Proposed unless an explicit existing glossary status is present.

The glossary must also be published as an independent human-readable output in glossary_terms.md and glossary_terms.pdf. This glossary is a steward-review artifact, not a Purview import file. It should be readable without opening the full metadata proposal.

Recommended glossary presentation:

1. Title and generation context
2. Summary by domain or subject area
3. Alphabetical term list grouped by domain or subject area
4. For each term: definition, synonyms, related terms, associated assets, associated columns, confidence, metadata source, and open questions
5. Review guidance explaining that terms are proposed and require steward approval

Classification Metadata

Use these fields for classification proposals:

classification_name, classification_reason, detection_basis, confidence

Allowed classification names include, when evidence supports them: personal data, financial data, operational data, identifier, reference data, master data, transactional data, metric, dimension, audit field, system field, derived field.

Sensitivity Metadata

Use these fields for Purview-like sensitivity label proposals:

suggested_label, label_reason, protection_recommendation, confidence

Allowed suggested labels: Public, Internal, Confidential, Highly Confidential, Restricted.

Data Quality Metadata

Use these fields for quality rules and profiling recommendations:

quality_dimension, quality_rule, severity, confidence

Quality dimensions include: completeness, uniqueness, validity, accuracy, consistency, timeliness, freshness, integrity, conformity, reasonableness.

Lineage and Relationship Metadata

Use these fields for relationship and lineage hints:

upstream_source, downstream_consumer, relationship_or_join_hint, confidence

Critical Data Element Metadata

Use these fields for CDE candidates:

cde_name, business_reason, associated_columns, confidence

Data Product Metadata

Use these fields for suggested data products or domain groupings:

data_product_name, purpose, included_assets, target_consumers,
owner_or_steward, confidence

Evidence and Confidence Rules

Metadata Source

Confidence Levels

Never raise confidence only because a term sounds plausible. Lower confidence when samples, row counts, relationships, or business context are missing.

Inference Rules

Asset Type and Domain

Use table/model names, workspace names, lakehouse names, schemas, and measure names to infer candidate domains. Examples:

If domain is ambiguous, use Unknown or a broad proposed domain and add an open question.

Descriptions

Generate two descriptions for each asset and column:

• Technical description: what the object stores structurally, based on schema and type evidence.
• Business description: what the object likely means to a business user, based on names, model context, relationships, and measures.

Descriptions must be concise, deterministic, and free of unsupported claims. For low-confidence descriptions, explicitly say Candidate description or Likely represents.

Keys and Relationships

Infer key metadata with these signals:

Default cardinality to many-to-one candidate only when a transactional/fact-like table points to a reference/dimension-like table. Use unknown when there is insufficient evidence.

Measure and Dimension Detection

Classification Detection

Do not use regex-only matches as definitive evidence. Describe uncertainty in classification_reason for sensitive classifications.

Sensitivity Label Suggestions

Treat every inferred label above Internal as requiring human review.

Privacy and Sensitive Data Signals

Potential PII and sensitive fields include:

name, first_name, last_name, full_name, email, phone, mobile, address, city,
postal_code, latitude, longitude, dni, nif, nie, ssn, passport, national_id,
tax_id, birth_date, gender, patient_id, diagnosis, medical_record, iban,
bank_account, credit_card, salary, password, secret, token, api_key, private_key,
ip_address, device_id, user_agent

Mark these as candidates unless sample values or explicit metadata confirm the pattern.

Data Quality Rules

Suggest quality rules from type and classification evidence:

Severity guidance:

Critical Data Element Candidates

Suggest a CDE candidate when one or more conditions apply:

• Column is a key identifier used in relationships.
• Column is a financial, operational, compliance, or customer-impacting metric.
• Column appears in a semantic model measure or relationship.
• Column is likely used for regulatory, privacy, security, or executive reporting.
• Column is needed to join important assets or identify master/reference entities.

Always describe CDEs as candidates and avoid definitive CDE status.

Glossary Term Generation

Create glossary terms by converting physical names to business names and grouping repeated concepts.

Rules:

• Convert snake_case and camelCase to title case business terms.
• Expand common abbreviations only when likely: id -> Identifier, qty -> Quantity, amt -> Amount, dt -> Date, cd -> Code, desc -> Description.
• Do not invent acronyms unless they are present in the source.
• Create related terms from relationship context, such as Customer, Customer Identifier, and Customer Segment.
• Set owner_or_steward to Unknown or TBD, and status to Proposed, unless explicit.

Purview-Oriented Mapping

Use this mapping for catalog ingestion planning:

Never state that a Purview label or classification has been applied.

Machine-Readable JSON Schema

The JSON file must follow this compact top-level structure:

{
  "metadata_proposal": {
    "generated_at": "ISO-8601 timestamp",
    "generated_by": "fabric-metadata-creation agent",
    "proposal_version": "1.0",
    "input_interpreted": {},
    "executive_summary": "",
    "assets": [],
    "columns": [],
    "glossary_terms": [],
    "classifications": [],
    "sensitivity_labels": [],
    "critical_data_elements": [],
    "data_products": [],
    "relationships_lineage": [],
    "data_quality_rules": [],
    "assumptions": [],
    "open_questions": []
  }
}

Every asset and column object must include metadata_source, confidence, assumptions, and open_questions. Other proposal objects must include metadata_source and confidence. Use review_required only when it materially helps prioritize steward review.

Output Authoring Rules

• Keep metadata_proposal.json compact and machine-readable; do not add legacy YAML, Markdown proposal, or evidence files.
• Render metadata_proposal.pdf from metadata_proposal.json using the helper script's Markdown-to-WeasyPrint renderer.
• Create glossary_terms.md as a standalone document that can be reviewed independently from the proposal PDF.
• Render glossary_terms.pdf from structured glossary terms using the helper script's Markdown-to-WeasyPrint renderer; do not manually export the PDF.
• Use A4 pages with 1.5cm margins, Segoe UI/Helvetica/Arial fonts, blue headings, zebra-striped tables, and VS Code-style code blocks.
• Write review prose, section titles, field labels, assumptions, questions, and generated descriptions in Spanish.
• Preserve original Fabric identifiers and source values exactly; do not translate table names, column names, measure names, workspace names, lakehouse names, semantic model names, or explicit glossary/source terms.
• Use concise enterprise language. Prefer brief explanations over long generated prose.
• Include confidence and metadata source in every proposal object.

Validation Rules

Before finishing, validate:

1. metadata_proposal.json parses as JSON.
2. metadata_proposal.pdf exists and is non-empty.
3. glossary_terms.md exists and includes a standalone glossary title.
4. glossary_terms.pdf exists and is non-empty.
5. No legacy artifacts exist in the output folder: metadata_proposal.md, metadata_proposal.yaml, or evidence.json.
6. Every asset and column proposal includes confidence, metadata_source, assumptions, and open_questions.
7. No real owner, steward, contact, legal obligation, retention policy, access policy, or compliance decision was invented.

Use:

python .github/skills/fabric-metadata-creation/scripts/fabric_metadata.py validate <output_dir>

Gotchas

• A physical column named customer_id is not proof that the field is PII. It is an identifier candidate and may become personal data only when it can identify or link to a person.
• A field named name can describe a person, product, department, status, or location. Use surrounding table context before proposing PII.
• Financial measures are often confidential business data even when they are not personal data.
• Health-related names such as patient, diagnosis, or treatment are strong sensitivity signals, but still require human review.
• Semantic model measures often carry business meaning. Use DAX expressions and format strings as explicit evidence for metric descriptions.
• Hidden key columns in semantic models are still governance-relevant and should be included in metadata proposals.
• Semantic model getDefinition calls may return 202 Accepted. The helper must poll the operation and then fetch the operation result payload; the operation status itself does not contain the TMDL parts.
• Do not assign Public unless public availability is explicit or strongly implied by context.
• Do not assign legal categories such as GDPR, HIPAA, PCI, or SOX as definitive obligations. Use possible regulatory review area only when the source context suggests it.
• If no sample values are available, avoid sample-based claims and reduce confidence.