jediv Data Catalog Patterns
Reference patterns for managing the Dataiku data catalog via the Python API.
Key Concepts
| Concept | What it is | Scope |
|---|---|---|
| Data Collection | A curated group of datasets, visible across projects | Instance-level (client) |
| Metadata | Label, description, tags, checklists, custom key-value pairs | Per dataset or project |
| Meaning | A semantic type for columns (e.g., "Email", "Country Code") | Instance-level (client) |
| Tags | Freeform labels on datasets or projects | Per dataset or project |
Data Collections
List Collections
collections = client.list_data_collections(as_type="dict")
for c in collections:
print(f"{c['displayName']} ({c['id']}) — {c['itemCount']} items")
Create a Collection
dc = client.create_data_collection(
displayName="Customer Data",
description="All customer-related datasets",
tags=["customers", "production"]
)
Add Datasets to a Collection
dc = client.get_data_collection("collection_id")
# Add by dataset handle
ds = project.get_dataset("MY_DATASET")
dc.add_object(ds)
# Add by dict (for cross-project datasets)
dc.add_object({"type": "DATASET", "projectKey": "PROJECT_A", "id": "DATASET_NAME"})
List and Remove Objects
dc = client.get_data_collection("collection_id")
objects = dc.list_objects()
for obj in objects:
raw = obj.get_raw()
print(f" {raw['projectKey']}.{raw['id']}")
# Get as a dataset handle
ds = obj.get_as_dataset()
# Remove from collection
obj.remove()
Update Collection Settings
dc = client.get_data_collection("collection_id")
settings = dc.get_settings()
settings.display_name = "Renamed Collection"
settings.description = "Updated description"
settings.tags = ["new-tag", "production"]
settings.save()
Dataset Metadata
Get and Set Metadata
ds = project.get_dataset("MY_DATASET")
metadata = ds.get_metadata()
# Metadata structure:
# {
# "label": "...",
# "description": "...",
# "tags": ["tag1", "tag2"],
# "checklists": {"checklists": [...]},
# "custom": {"kv": {"key1": "value1"}}
# }
metadata["tags"] = ["cleaned", "production"]
metadata["custom"]["kv"]["owner"] = "data-team"
ds.set_metadata(metadata)
AI-Generated Descriptions
# Generate descriptions for dataset and columns (requires AI Services enabled)
result = ds.generate_ai_description(language="english", save_description=True)
Rate-limited: 1000 requests/day, then throttled to ~60s per call.
Meanings (Semantic Column Types)
List and Create Meanings
# List existing meanings
meanings = client.list_meanings()
# Create a values-list meaning
client.create_meaning(
id="country_code",
label="Country Code",
type="VALUES_LIST",
values=["US", "UK", "FR", "DE", "JP"],
normalizationMode="EXACT",
detectable=True
)
# Create a pattern-based meaning
client.create_meaning(
id="email_address",
label="Email Address",
type="PATTERN",
pattern=r"^[\w.-]+@[\w.-]+\.\w+$",
detectable=True
)
Update a Meaning
meaning = client.get_meaning("country_code")
definition = meaning.get_definition()
definition["entries"].append({"value": "CA"})
meaning.set_definition(definition)
Catalog Indexing
Trigger re-indexing of connections so new tables appear in the catalog:
# Index specific connections
client.catalog_index_connections(connection_names=["my_snowflake", "my_postgres"])
# Index all connections
client.catalog_index_connections(all_connections=True)
Detailed References
- references/data-collections.md — Permissions, completeness checks, cross-project patterns
- references/metadata-and-tags.md — Full metadata structure, project tags, custom metadata
- references/meanings.md — All meaning types, normalization modes, detectable meanings