# Genie Space Export Import API

> Comprehensive patterns for Databricks Genie Space Export/Import API - JSON schema, serialization format, and programmatic deployment. Use when programmatically creating, exporting, or importing Genie Spaces via REST API, troubleshooting API deployment errors, or implementing CI/CD for Genie Spaces. Includes complete GenieSpaceExport schema, API endpoints (List, Get, Create, Update, Delete), JSON format requirements, ID generation, variable substitution, inventory-driven generation patterns, and production deployment checklists.

- Skill: `databricks-solutions/genie-space-export-import-api` (Agent Skill, multi-file: 11 files)
- Install (CLI): `npx skillmds@latest add databricks-solutions/genie-space-export-import-api`
- Raw SKILL.md: https://api.skillmd.com/api/skills/databricks-solutions/genie-space-export-import-api/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: databricks-solutions (https://skillmd.com/u/databricks-solutions)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/databricks-solutions/genie-space-export-import-api

---


# Genie Space Export/Import API

## Overview

This skill provides comprehensive patterns for programmatically creating, exporting, and importing Databricks Genie Spaces via the REST API. It covers the complete `GenieSpaceExport` JSON schema, API endpoints, common deployment errors, and production-ready workflows including variable substitution and asset inventory-driven generation.

> **Terminology (2026):** Databricks renamed **Genie Spaces** to **Genie Agents**. The REST endpoints are unchanged (`/api/2.0/genie/spaces`), and this skill uses "Genie Space" and "Genie Agent" interchangeably. The API now also exposes **Agent-mode** endpoints (stream reasoning + SQL + final reports with citations) alongside the Chat-mode and Management APIs covered here. See the [Genie Agents API](https://docs.databricks.com/aws/en/genie/conversation-api).

## When to Use This Skill

Use this skill when you need to:

- **Programmatically deploy Genie Spaces** via REST API (CI/CD pipelines, environment promotion)
- **Export Genie Space configurations** for version control, backup, or migration
- **Troubleshoot API deployment errors** (`BAD_REQUEST`, `INVALID_PARAMETER_VALUE`, `INTERNAL_ERROR`)
- **Implement cross-workspace deployment** with template variable substitution
- **Generate Genie Spaces from asset inventories** to prevent non-existent table errors
- **Validate Genie Space JSON structure** before deployment
- **Understand the complete GenieSpaceExport schema** (config, data_sources, instructions, benchmarks)

## Start From Templates (Mandatory)

**NEVER write the deployment notebook or job YAML from scratch.** Writing these from scratch is the #1 source of deployment failures — copy the template, then customize:

- `assets/templates/deploy_genie_spaces.py` → copy to `src/{project}_semantic/deploy_genie_spaces.py`
- `assets/templates/genie-deployment-job-template.yml` → copy to `resources/semantic/genie_deploy_job.yml`

The templates encode the correct notebook cell separators, `extract_space_config()` (wrapped vs raw format handling), `validate_genie_json_structure()`, array sorting, and `base_parameters` wiring. Hand-written versions routinely miss one of these and fail in deploy cycles 2–9 (see the retrospective in `references/` or the project's retrospectives directory).

## End-to-End Deployment? Use the Orchestrator

If you are deploying **TVFs + Metric Views + Genie Spaces together** (not just a standalone Genie Space), **STOP and read `semantic-layer/00-semantic-layer-setup/SKILL.md` first.** That orchestrator:
- Mandates a Gold schema inventory query before artifact creation (prevents phantom table errors)
- Coordinates skill loading across 6 phases with validation gates
- Provides combined job templates with `depends_on` chains

This skill handles **individual Genie Space API operations.** The orchestrator handles the **end-to-end semantic layer lifecycle.**

## Quick Reference

### API Operations

| Operation | Method | Endpoint | Use Case |
|-----------|--------|----------|----------|
| **List Spaces** | GET | `/api/2.0/genie/spaces` | Discover existing spaces |
| **Get Space** | GET | `/api/2.0/genie/spaces/{space_id}?include_serialized_space=true` | Export config, backup |
| **Create Space** | POST | `/api/2.0/genie/spaces` | New deployment, CI/CD |
| **Update Space** | PATCH | `/api/2.0/genie/spaces/{space_id}` | Modify config, add benchmarks |
| **Delete Space** | DELETE | `/api/2.0/genie/spaces/{space_id}` | Cleanup, teardown |

### API Limits

| Resource | Limit | Enforcement |
|----------|-------|-------------|
| `instructions.sql_functions` | **Max 50** | Truncate in generation script |
| `benchmarks.questions` | **Max 50** | Truncate in generation script |
| `data_sources.tables` | No hard limit | Keep ~25-30 for performance |
| `data_sources.metric_views` | No hard limit | Keep ~5-10 per space |

### Required Root Field

Every Genie Space JSON MUST include an explicit `"version"` at the root of `serialized_space`. The API accepts **versions 1 and 2**; **use version 2** — it is the richer schema that supports `benchmarks`, `data_sources.metric_views`, `example_question_sqls[].parameters`, and `join_specs`:

```json
{"version": 2, "config": {...}, "data_sources": {...}, "instructions": {...}, "benchmarks": {...}}
```

Omitting `version` (or sending `0`) causes silent failures or API rejection (`ExportConverter supports versions 1 and 2, but got 0`). The API does NOT default the version — set it explicitly.

### Required `serialized_space` Invariants (MANDATORY — validate BEFORE every POST / PATCH)

Genie Space creation silently succeeds but produces a broken room when these invariants are violated. The API's validation surface is narrow — once the space exists, the only way to diagnose is inspecting the UI. Always run the validator below *before* calling `POST /api/2.0/genie/spaces` or `PATCH /api/2.0/genie/spaces/{id}`.

| Field path (inside `serialized_space`) | Required type | Non-negotiable invariants |
|---|---|---|
| `version` | `int` | Must be `1` or `2`; prefer `2`. Missing or `0` is rejected. |
| `config.title` | `str` | Non-empty. Mirrors top-level `title` in the POST envelope. |
| `config.description` | `str` | Non-empty. |
| `config.semantic_warehouse_id` | `str` (16+ hex chars) | Must be the **deploy-time** warehouse id (see Action S10). Never a template placeholder. |
| `data_sources.tables` | `list[object]` | Each entry: `{identifier: "catalog.schema.table", description?: List[str], column_configs?: [...] }`. **NO `id` field** (adding one fails with `Cannot find field: id`). **Sorted by `identifier`**; any `column_configs` sorted by `column_name`. |
| `data_sources.metric_views` | `list[object]` | Each entry: `{identifier: "catalog.schema.mv_name", description?: List[str], column_configs?: [...] }`. **NO `id` field**. Sorted by `identifier`; any `column_configs` sorted by `column_name`. |
| `instructions.general_instructions` | `list[str]` | List of plain strings — no nested objects. |
| `instructions.sql_functions` | `list[object]` | Each: `{id: uuid4.hex, identifier: "catalog.schema.fn_name"}`. No other fields. Max 50 entries. |
| `instructions.example_question_sqls` | `list[object]` | Each: `{id: uuid4.hex, question: List[str], sql: List[str]}` plus optional `parameters: [{name, type_hint, description?: List[str], default_value?}]` and `usage_guidance?: List[str]`. **`question` and `sql` MUST be `List[str]`**, never single strings. |
| `instructions.join_specs` | `list[object]` | (v2) Each: `{id: uuid4.hex, left: {identifier, alias}, right: {identifier, alias}, sql: List[str], comment?: List[str], instruction?: List[str]}`. **`sql` MUST contain exactly two elements**: the backtick-quoted join condition AND a relationship annotation `"--rt=FROM_RELATIONSHIP_TYPE_<CARDINALITY>--"`. Sorted by `id`. |
| `benchmarks.questions` | `list[object]` | Each: `{id: uuid4.hex, question: List[str], answer: [{format: "SQL"\|"INSTRUCTIONS", content: List[str]}]}`. SQL lives inside `answer[].content`, never as a top-level `sql` field. Max 50 entries. |
| Every `id` field | `str` (32 hex chars) | `uuid.uuid4().hex` — lowercase, no dashes. Regenerate on every new deploy (never copy/paste IDs across environments). Time-ordered 32-hex UUIDs also work and self-satisfy sort order. |

**The #1 silent-failure mode observed in production:** `example_question_sqls[].sql`, `example_question_sqls[].question`, or `benchmarks.questions[].answer[].content` submitted as a single string instead of `List[str]`. The API accepts it, but the resulting space has empty/broken example queries because Genie serializes only the first character and then errors internally on render. There is no error surfaced on POST.

#### Validator — `_assert_sql_arrays`

Run this against every `serialized_space` dict BEFORE `json.dumps(...)` and BEFORE the POST/PATCH. Failing loud here saves ~10–15 minutes of UI-round-trip debugging per iteration.

```python
import re
import uuid as _uuid_mod
from typing import Any, List

_UUID_HEX_RE = re.compile(r"^[0-9a-f]{32}$")
_WAREHOUSE_ID_RE = re.compile(r"^[0-9a-f]{16,}$")  # Databricks warehouse ids are hex


def _is_uuid_hex(value: Any) -> bool:
    return isinstance(value, str) and bool(_UUID_HEX_RE.match(value))


def _assert_sql_arrays(space: dict) -> None:
    """
    Validate serialized_space invariants before POST / PATCH.
    Raises RuntimeError on the FIRST violation — never returns False / warns.

    Schema reference: https://docs.databricks.com/aws/en/genie/conversation-api
    Enforces:
      - version is 1 or 2 (prefer 2); missing / 0 is rejected.
      - data_sources.tables / metric_views entries use `identifier` only (NO `id`);
        any column_configs are sorted by `column_name`.
      - instructions.sql_functions entries: {id, identifier} only.
      - instructions.example_question_sqls entries: {id, question: List[str], sql: List[str]}
        (optional `parameters` / `usage_guidance` are allowed).
      - instructions.join_specs[].sql has exactly two elements (condition + --rt=...-- annotation).
      - benchmarks.questions[].answer[].content must be List[str] (SQL lives there,
        not as a top-level `sql` field).
    """
    errors: List[str] = []

    if space.get("version") not in (1, 2):
        errors.append(
            "serialized_space.version must be 1 or 2 (prefer 2); got %r" % space.get("version")
        )

    cfg = space.get("config") or {}
    if not isinstance(cfg.get("title"), str) or not cfg.get("title"):
        errors.append("config.title must be a non-empty string")
    if not isinstance(cfg.get("description"), str) or not cfg.get("description"):
        errors.append("config.description must be a non-empty string")

    wh = cfg.get("semantic_warehouse_id")
    if not isinstance(wh, str) or not _WAREHOUSE_ID_RE.match(wh or ""):
        errors.append(
            "config.semantic_warehouse_id must be a concrete warehouse id baked at deploy time; "
            f"got {wh!r}. Template placeholders like '${{warehouse_id}}' are never acceptable."
        )

    # Data sources — sorted by `identifier`, NEVER include `id` on these entries.
    ds = space.get("data_sources") or {}
    for key in ("tables", "metric_views"):
        items = ds.get(key) or []
        if not isinstance(items, list):
            errors.append(f"data_sources.{key} must be a list")
            continue
        idents = [it.get("identifier", "") for it in items]
        if idents != sorted(idents):
            errors.append(f"data_sources.{key} must be sorted by identifier (got {idents})")
        for it in items:
            if "id" in it:
                errors.append(
                    f"data_sources.{key} entry MUST NOT include `id` — the API rejects with "
                    f"`Cannot find field: id`. Use only `identifier` and optional `description`. Got: {it}"
                )
            ident = it.get("identifier")
            if not isinstance(ident, str) or ident.count(".") != 2:
                errors.append(
                    f"data_sources.{key} entry `identifier` must be 'catalog.schema.name': {it}"
                )
            # column_configs (optional) must be sorted by column_name.
            ccs = it.get("column_configs")
            if ccs is not None:
                if not isinstance(ccs, list):
                    errors.append(f"data_sources.{key} entry `column_configs` must be a list: {it}")
                else:
                    names = [c.get("column_name", "") for c in ccs]
                    if names != sorted(names):
                        errors.append(
                            f"data_sources.{key}[{ident}].column_configs must be sorted by column_name (got {names})"
                        )

    instr = space.get("instructions") or {}

    # instructions.sql_functions — {id, identifier} only.
    sqlfns = instr.get("sql_functions") or []
    if not isinstance(sqlfns, list):
        errors.append("instructions.sql_functions must be a list")
    else:
        for idx, it in enumerate(sqlfns):
            if not isinstance(it, dict):
                errors.append(f"instructions.sql_functions[{idx}] must be an object")
                continue
            if not _is_uuid_hex(it.get("id")):
                errors.append(f"instructions.sql_functions[{idx}].id must be uuid4.hex (32 hex chars)")
            ident = it.get("identifier")
            if not isinstance(ident, str) or ident.count(".") != 2:
                errors.append(
                    f"instructions.sql_functions[{idx}].identifier must be 'catalog.schema.fn_name'"
                )

    # instructions.example_question_sqls — {id, question: List[str], sql: List[str]}.
    eqs = instr.get("example_question_sqls") or []
    if not isinstance(eqs, list):
        errors.append("instructions.example_question_sqls must be a list")
    else:
        for idx, it in enumerate(eqs):
            if not isinstance(it, dict):
                errors.append(f"instructions.example_question_sqls[{idx}] must be an object")
                continue
            if not _is_uuid_hex(it.get("id")):
                errors.append(f"instructions.example_question_sqls[{idx}].id must be uuid4.hex")
            for arr_field in ("question", "sql"):
                arr = it.get(arr_field)
                if not isinstance(arr, list) or not all(isinstance(s, str) and s.strip() for s in arr):
                    errors.append(
                        f"instructions.example_question_sqls[{idx}].{arr_field} must be a non-empty "
                        f"List[str] — single strings cause silent breakage. Wrap as [\"...\"]."
                    )

    # instructions.join_specs (v2) — {id, left, right, sql}. sql MUST be exactly two
    # elements: the join condition AND a "--rt=FROM_RELATIONSHIP_TYPE_<CARDINALITY>--" annotation.
    joins = instr.get("join_specs") or []
    if not isinstance(joins, list):
        errors.append("instructions.join_specs must be a list")
    else:
        join_ids = [j.get("id", "") for j in joins if isinstance(j, dict)]
        if join_ids != sorted(join_ids):
            errors.append(f"instructions.join_specs must be sorted by id (got {join_ids})")
        for idx, it in enumerate(joins):
            if not isinstance(it, dict):
                errors.append(f"instructions.join_specs[{idx}] must be an object")
                continue
            if not _is_uuid_hex(it.get("id")):
                errors.append(f"instructions.join_specs[{idx}].id must be uuid4.hex")
            for side in ("left", "right"):
                node = it.get(side)
                if not isinstance(node, dict) or not isinstance(node.get("identifier"), str):
                    errors.append(f"instructions.join_specs[{idx}].{side} must be {{identifier, alias}}")
            sql = it.get("sql")
            if not isinstance(sql, list) or len(sql) != 2:
                errors.append(
                    f"instructions.join_specs[{idx}].sql must be a 2-element List[str]: "
                    f"[join condition, '--rt=FROM_RELATIONSHIP_TYPE_<CARDINALITY>--']"
                )
            elif not str(sql[1]).startswith("--rt=FROM_RELATIONSHIP_TYPE_"):
                errors.append(
                    f"instructions.join_specs[{idx}].sql[1] must be a relationship annotation "
                    f"'--rt=FROM_RELATIONSHIP_TYPE_<CARDINALITY>--'; omitting it is rejected by the API."
                )

    # benchmarks.questions — SQL lives inside answer[].content, NOT a top-level sql field.
    bench = (space.get("benchmarks") or {}).get("questions") or []
    if not isinstance(bench, list):
        errors.append("benchmarks.questions must be a list")
    else:
        for idx, it in enumerate(bench):
            if not isinstance(it, dict):
                errors.append(f"benchmarks.questions[{idx}] must be an object")
                continue
            if not _is_uuid_hex(it.get("id")):
                errors.append(f"benchmarks.questions[{idx}].id must be uuid4.hex")
            q = it.get("question")
            if not isinstance(q, list) or not all(isinstance(s, str) and s.strip() for s in q):
                errors.append(f"benchmarks.questions[{idx}].question must be List[str]")
            answers = it.get("answer") or []
            if not isinstance(answers, list):
                errors.append(f"benchmarks.questions[{idx}].answer must be a list")
                continue
            for aidx, ans in enumerate(answers):
                if not isinstance(ans, dict):
                    errors.append(f"benchmarks.questions[{idx}].answer[{aidx}] must be an object")
                    continue
                if ans.get("format") not in ("SQL", "INSTRUCTIONS"):
                    errors.append(
                        f"benchmarks.questions[{idx}].answer[{aidx}].format must be 'SQL' or 'INSTRUCTIONS'"
                    )
                content = ans.get("content")
                if not isinstance(content, list) or not all(isinstance(s, str) and s.strip() for s in content):
                    errors.append(
                        f"benchmarks.questions[{idx}].answer[{aidx}].content must be List[str] — "
                        f"this is the #1 silent-failure mode for benchmark answers."
                    )

    # Limits
    if len(sqlfns) > 50:
        errors.append("instructions.sql_functions exceeds 50-entry limit — truncate before POST")
    if len(bench) > 50:
        errors.append("benchmarks.questions exceeds 50-entry limit — truncate before POST")

    gi = instr.get("general_instructions")
    if gi is not None:
        if not isinstance(gi, list) or not all(isinstance(x, str) for x in gi):
            errors.append("instructions.general_instructions must be List[str]")

    if errors:
        joined = "\n  - ".join(errors)
        raise RuntimeError(
            f"serialized_space validation failed — refusing to POST/PATCH:\n  - {joined}"
        )


# Usage:
#   _assert_sql_arrays(space_dict)
#   payload = {"title": ..., "warehouse_id": ..., "serialized_space": json.dumps(space_dict)}
#   ws.api_client.do("POST", "/api/2.0/genie/spaces", body=payload)
```

**Where to wire this in:**
- `scripts/import_genie_space.py` — immediately before `json.dumps(space)` in every path (initial POST and incremental PATCH).
- `assets/templates/deploy_genie_spaces.py` — at the top of each per-space loop iteration.
- Local Phase 0.5 pre-flight (see Action S7 in `00-semantic-layer-setup/SKILL.md`) — enumerated against the rendered config BEFORE `bundle validate`.

### `semantic_warehouse_id` MUST be baked at deploy time (NOT a runtime `--var`)

`serialized_space.config.semantic_warehouse_id` is consumed by the Genie runtime when it renders example queries and routes natural-language questions. The id is **embedded in the POSTed JSON body** — once the space exists, the Genie service does not re-read it from the Asset Bundle.

That means it must be a concrete 16+ character hex warehouse id at the moment `deploy_genie_spaces.py` calls `POST /api/2.0/genie/spaces`. Any of the following produces a broken room:

| Value at POST time | Outcome |
|---|---|
| `abc0123def456789` (real id) | ✅ Space works. |
| `${var.warehouse_id}` (unrendered) | ❌ Space created; every query fails with "warehouse not found". |
| `${warehouse_id}` (shell-style placeholder) | ❌ Same failure mode. |
| Empty string / missing field | ❌ API rejects with `INVALID_PARAMETER_VALUE`. |

**Rule:** Resolve `warehouse_id` at `databricks bundle deploy` time (deploy-time baking — see `skills/databricks-asset-bundles/SKILL.md` §Pitfall: `--var` at run time does NOT override deploy-time-baked values) and pass the resolved value into the notebook task via `base_parameters.warehouse_id`. The `deploy_space` helper in `assets/templates/deploy_genie_spaces.py` stamps this value into `serialized_space.config.semantic_warehouse_id` automatically, so you only need to ensure the value reaching the notebook widget is concrete.

**Validation:** `_assert_sql_arrays` enforces a hex-only `semantic_warehouse_id` on every POST/PATCH. If you see `config.semantic_warehouse_id must be a concrete deploy-time warehouse id` in a pre-flight error, the root cause is always that `warehouse_id` reached the notebook as a template placeholder — fix it in the bundle YAML, not in the Genie config.

**Post-deploy rotation:** If the workspace's semantic warehouse id changes (e.g. migration to serverless), the fix is NOT a runtime override. It's a `bundle deploy` with the new id + a PATCH of every existing space via `deploy_genie_spaces.py` (which will now take the update path because the space ids are persisted per Action S9).

### Core Workflow

**Initial Deployment:**
1. List spaces (check if already exists)
2. Load configuration from JSON file
3. Substitute template variables (`${catalog}`, `${gold_schema}`, etc.)
4. Create space with full configuration
5. Get space to verify deployment

**Incremental Updates:**
1. Get current space configuration (capture its `etag`)
2. Modify specific sections (e.g., add benchmarks)
3. Update space with PATCH (partial update), passing the captured `etag`

> **Optimistic concurrency (`etag`):** The GET response includes an `etag`. Pass it back on the PATCH
> (`update_genie_space(..., etag=...)`) and the update fails safely with `409 Conflict` if the space
> was edited by someone else since your GET — preventing a blind overwrite. Omit `etag` for a
> last-writer-wins update. Re-GET to fetch the new `etag` and retry on conflict.

**Migration/Backup:**
1. Get space with `include_serialized_space=true`
2. Save JSON to version control
3. Create space in new environment (with variable substitution)

## Key Patterns

### 1. JSON Structure Requirements

**CRITICAL:** The `serialized_space` field must be a JSON string (escaped), not a nested object:

```python
payload = {
    "title": "My Space",
    "warehouse_id": "abc123",
    "serialized_space": json.dumps(genie_config)  # ✅ String, not dict
}
```

### Section 4: ID Generation

**All IDs MUST be `uuid.uuid4().hex`** — a 32-character lowercase hex string with no dashes.

```python
import uuid

def generate_id() -> str:
    """Generate a Genie Space compatible ID (32 hex chars, no dashes)."""
    return uuid.uuid4().hex  # e.g., "a1b2c3d4e5f6789012345678abcdef01"
```

**Required ID fields** (every one must be a fresh `uuid.uuid4().hex`).

Use the canonical nested-schema field paths below. Any older guidance that listed flat `space.tables[].id` / `space.materialized_views[].id` / `space.sql_functions[].id` / `space.example_question_sqls[].id` as required was for a deprecated flat schema and is superseded by this list:

- `config.sample_questions[].id`
- `instructions.sql_functions[].id`
- `instructions.text_instructions[].id`
- `instructions.example_question_sqls[].id`
- `instructions.join_specs[].id`
- `instructions.sql_snippets.measures[].id`
- `instructions.sql_snippets.filters[].id`
- `instructions.sql_snippets.expressions[].id`
- `benchmarks.questions[].id`

> **Tip:** `uuid.uuid4().hex` is always valid. The Genie docs also suggest a **time-ordered** 32-hex UUID so IDs sort alphabetically in creation order and self-satisfy the sort requirements below.

**❌ Arrays that MUST NOT have an `id`** (adding one causes `Cannot find field: id in message ...` errors — see Common Errors):

- `data_sources.tables[]` — use only `identifier` and optional `description`
- `data_sources.metric_views[]` — use only `identifier` and optional `description`
- `benchmarks.questions[].answer[]` — use only `format` and `content`

This is the single source of truth for ID placement. The no-id list later in this section and in Section 7 intentionally restates it for retrieval during debugging — keep both lists consistent if editing.

**❌ WRONG IDs (will cause import failures):**
```python
"genie_" + uuid.uuid4().hex[:24]    # ❌ Prefixed, wrong length
"aaaa" * 8                           # ❌ Not random
str(uuid.uuid4())                    # ❌ Contains dashes (36 chars)
hashlib.md5(name.encode()).hexdigest()  # ❌ Deterministic, not UUID4
```

**✅ CORRECT: Always use `uuid.uuid4().hex`** — nothing else.

**Arrays that do NOT have `id` fields — NEVER add one:**
- `data_sources.tables[]` — uses `identifier` only
- `data_sources.metric_views[]` — uses `identifier` only
- `benchmarks.questions[].answer[]` — uses `format` + `content` only

A common agent error is applying `regenerate_ids()` universally across all arrays. The function must SKIP `data_sources.tables` and `data_sources.metric_views`.

### Section 5: Array Format Requirements

**ALL string-content fields in the Genie Space JSON MUST be single-element arrays**, not plain strings.

| Field | ❌ Wrong | ✅ Correct |
|-------|---------|-----------|
| `question` | `"What is revenue?"` | `["What is revenue?"]` |
| `content` (in answer) | `"SELECT ..."` | `["SELECT ..."]` |
| `description` (tables) | `"Orders table"` | `["Orders table"]` |
| `description` (MVs) | `"Revenue metrics"` | `["Revenue metrics"]` |
| `description` (TVFs) | `"Date range query"` | `["Date range query"]` |

**Rule:** If a field contains human-readable text or SQL, wrap it in a single-element array `["value"]`.

**Exception:** `format` in answer objects is a plain string: `"SQL"` or `"INSTRUCTIONS"`.

### 4. Template Variable Substitution

**NEVER hardcode schema paths.** Use template variables:

```json
{
  "data_sources": {
    "tables": [
      {"identifier": "${catalog}.${gold_schema}.dim_store"}  // ✅ Template
    ]
  }
}
```

Substitute at runtime:

```python
def substitute_variables(data: dict, variables: dict) -> dict:
    json_str = json.dumps(data)
    json_str = json_str.replace("${catalog}", variables.get('catalog', ''))
    json_str = json_str.replace("${gold_schema}", variables.get('gold_schema', ''))
    return json.loads(json_str)
```

### 5. Asset Inventory-Driven Generation

**Step 0 — Verify assets exist before referencing them:**

The schema(s) to inspect come from the planning manifest, not a hard-coded `gold_schema`. Production spaces inspect Gold; some manifests also expose `semantic_schema` (Metric Views/TVFs may be deployed there). When a manifest provides `unified_genie_space.assets` or `domains[].genie_spaces[].assets`, treat those identifiers as the **allowlist** — only verify schemas that contain those allowlisted assets.

```sql
-- Run this BEFORE creating or editing any Genie Space JSON.
-- {schema} is one of: ${gold_schema}, ${semantic_schema} (from manifest).
SELECT table_name, table_type
FROM {catalog}.information_schema.tables
WHERE table_schema = '{schema}'
ORDER BY table_type, table_name;

SELECT routine_name
FROM {catalog}.information_schema.routines
WHERE routine_schema = '{schema}';
```

**Only include assets that appear in these results AND are listed in the manifest's `assets` allowlist.** A Genie Space that references a non-existent table fails with `Table '...' does not exist` during space creation. This is the #1 cause of deployment failures. Do NOT trust a pre-generated manifest as ground truth — query the live catalog using the schemas declared by `planning_source` (`gold_schema` for production / Gold-based runs, or `silver_schema` / `bronze_schema` for workshop deployments). The semantic-layer orchestrator only stops before this skill runs when `planning_source.selected_layer = source_csv` (no live tables exist).

**NEVER manually edit `data_sources`.** Generate from verified inventory:

```python
# Load inventory
with open('actual_assets_inventory.json') as f:
    inventory = json.load(f)

# Generate data_sources from inventory
genie_config['data_sources']['tables'] = [
    {"identifier": table_id}
    for table_id in inventory['genie_space_mappings']['cost_intelligence']['tables']
]
```

**Benefits:**
- ✅ Prevents "table doesn't exist" errors
- ✅ Enforces API limits automatically
- ✅ Single source of truth for assets

### 6. Column Configs (optional, richer context)

`column_configs` is a first-class, per-column list on `data_sources.tables[]` and `data_sources.metric_views[]` that gives Genie extra retrieval/formatting hints. Supported per-column fields:

| Field | Purpose |
|-------|---------|
| `column_name` | Target column (required within a config entry). |
| `description` | `List[str]` column description. |
| `synonyms` | `List[str]` alternate names for entity matching. |
| `get_example_values` | Sample distinct values so Genie learns the domain. |
| `build_value_dictionary` | Build a value dictionary for high-cardinality categorical filters. |
| `enable_entity_matching` | Match user-mentioned entities to this column. |
| `enable_format_assistance` | Help Genie format values (dates, regions, etc.). |
| `exclude` | Hide the column from Genie. |

```json
{
  "data_sources": {
    "tables": [
      {
        "identifier": "catalog.schema.orders",
        "column_configs": [
          {"column_name": "order_date", "enable_format_assistance": true},
          {"column_name": "region", "enable_entity_matching": true, "build_value_dictionary": true}
        ]
      }
    ]
  }
}
```

**Rules / trade-offs:**
- `column_configs` entries MUST be **sorted by `column_name`** (the API validates this — see Section 8).
- Richer configs add LLM context but increase the risk of `INTERNAL_ERROR` on very complex spaces. If a large space fails to deploy, drop back to `identifier`-only entries first, then reintroduce column_configs incrementally.

### 7. Field Validation Rules

**config.sample_questions:**
- ✅ Array of objects (not strings)
- ✅ Each object: `{id: string, question: string[]}`
- ❌ NO `name`, `description` fields

**data_sources.metric_views:**
- ✅ `identifier` field (full 3-part UC name)
- ✅ Optional: `description`, `column_configs`
- ❌ NO `id`, `name`, `full_name` fields

**instructions.sql_functions:**
- ✅ `id` field (32 hex chars) - REQUIRED
- ✅ `identifier` field (full 3-part function name) - REQUIRED
- ❌ NO other fields (`name`, `signature`, `description`)

### Section 8: Array Sorting Requirements

**CRITICAL: All arrays in the Genie Space JSON MUST be sorted before any PATCH request.** The Genie API uses protobuf serialization which requires deterministic ordering. Unsorted arrays produce: `Invalid export proto: data_sources.tables must be sorted by identifier`.

**Sort keys by array path:**

| Array Path | Sort Key | Direction |
|------------|----------|-----------|
| `data_sources.tables` | `identifier` | Ascending |
| `data_sources.tables[].column_configs` | `column_name` | Ascending |
| `data_sources.metric_views` | `identifier` | Ascending |
| `data_sources.metric_views[].column_configs` | `column_name` | Ascending |
| `instructions.sql_functions` | `(id, identifier)` | Ascending |
| `instructions.text_instructions` | `id` | Ascending |
| `instructions.example_question_sqls` | `id` | Ascending |
| `instructions.join_specs` | `id` | Ascending |
| `instructions.sql_snippets.measures` | `id` | Ascending |
| `instructions.sql_snippets.filters` | `id` | Ascending |
| `instructions.sql_snippets.expressions` | `id` | Ascending |
| `config.sample_questions` | `id` | Ascending |
| `benchmarks.questions` | `id` | Ascending |

**Implementation — `sort_genie_config()`:**
```python
def sort_genie_config(config: dict) -> dict:
    """Sort all arrays in Genie config — API rejects unsorted data."""
    if "data_sources" in config:
        for key in ["tables", "metric_views"]:
            if key in config["data_sources"]:
                config["data_sources"][key] = sorted(
                    config["data_sources"][key],
                    key=lambda x: x.get("identifier", ""),
                )
                # column_configs (if present) must be sorted by column_name.
                for entry in config["data_sources"][key]:
                    if isinstance(entry.get("column_configs"), list):
                        entry["column_configs"] = sorted(
                            entry["column_configs"],
                            key=lambda c: c.get("column_name", ""),
                        )
    if "instructions" in config:
        if "sql_functions" in config["instructions"]:
            config["instructions"]["sql_functions"] = sorted(
                config["instructions"]["sql_functions"],
                key=lambda x: (x.get("id", ""), x.get("identifier", "")),
            )
        for key in ["text_instructions", "example_question_sqls", "join_specs"]:
            if key in config["instructions"]:
                config["instructions"][key] = sorted(
                    config["instructions"][key],
                    key=lambda x: x.get("id", ""),
                )
        if "sql_snippets" in config["instructions"]:
            for key in ["measures", "filters", "expressions"]:
                if key in config["instructions"]["sql_snippets"]:
                    config["instructions"]["sql_snippets"][key] = sorted(
                        config["instructions"]["sql_snippets"][key],
                        key=lambda x: x.get("id", ""),
                    )
    if "config" in config and "sample_questions" in config["config"]:
        config["config"]["sample_questions"] = sorted(
            config["config"]["sample_questions"],
            key=lambda x: x.get("id", ""),
        )
    if "benchmarks" in config and "questions" in config["benchmarks"]:
        config["benchmarks"]["questions"] = sorted(
            config["benchmarks"]["questions"],
            key=lambda x: x.get("id", ""),
        )
    return config
```

**Always call `sort_genie_config()` BEFORE submitting to the API.** The canonical implementation lives in `04-genie-optimization-applier/scripts/optimization_applier.py`.

### Section 9: Idempotent Deployment (Update-or-Create)

To prevent duplicate Genie Spaces on re-deployment, implement an update-or-create pattern:

1. **Store space IDs in `databricks.yml` variables:**
```yaml
variables:
  genie_space_id_<space_name>:
    description: "Existing Genie Space ID (empty for first deployment)"
    default: ""
```

2. **Deployment logic:**
```python
space_id = dbutils.widgets.get("genie_space_id_<space_name>")

if space_id:
    # UPDATE existing space (PATCH without title to avoid " (updated)" suffix)
    payload = {"serialized_space": json.dumps(space_json)}
    # Do NOT include "title" in PATCH to avoid title mutation
    response = requests.patch(f"{base_url}/api/2.0/genie/spaces/{space_id}", ...)
else:
    # CREATE new space
    response = requests.post(f"{base_url}/api/2.0/genie/spaces", ...)
    new_space_id = response.json()["space"]["id"]
    print(f"Created new space: {new_space_id}")
    print(f"Set variable: genie_space_id_<space_name> = {new_space_id}")
```

3. **⚠️ PATCH without title:** Including `title` in a PATCH request causes the API to append " (updated)" to the title. Omit `title` from PATCH payload to preserve the original name.

4. **After first deployment:** Record the returned space IDs and set them as `databricks.yml` variable defaults for subsequent deployments.

## Common Errors & Quick Fixes

| Error | Cause | Quick Fix |
|-------|-------|-----------|
| `BAD_REQUEST: Invalid JSON` | sample_questions as strings | Convert to objects with `id` and `question[]` |
| `BAD_REQUEST: Invalid JSON` | metric_views with `full_name` | Use `identifier` instead |
| `INTERNAL_ERROR: Failed to retrieve schema` | Missing `id` in sql_functions | Add `id` field (32 hex chars) |
| `INVALID_PARAMETER_VALUE: Expected array` | `question` is string | Wrap in array: `["question"]` |
| `Exceeded maximum number (50)` | Too many TVFs/benchmarks | Truncate to 50 in generation script |
| `expected_sql` field not recognized | Used `expected_sql` instead of `answer` | Use `answer: [{format: "SQL", content: ["SELECT ..."]}]` |
| `Invalid export proto: data_sources.tables must be sorted by identifier` | Arrays not sorted — sort key is `identifier` (not `table_name`) for tables/metric_views, `id` for all others | Call `sort_genie_config()` before every PATCH (see Section 8) |
| Invalid ID format | ID is not 32-char hex, contains dashes, or is prefixed | Use `uuid.uuid4().hex` exclusively |
| `Cannot find field: id in message ...MetricView` | Added `id` to `data_sources.metric_views[]` | Remove `id` — use only `identifier` and `description` (see Section 4) |
| `Cannot find field: id in message ...BenchmarkAnswer` | Added `id` to `benchmarks.questions[].answer[]` | Remove `id` — use only `format` and `content` |
| `Invalid export proto: ExportConverter supports versions 1 and 2, but got 0` | Missing top-level `version` field in `serialized_space` | Add `"version": 2` at the root before `json.dumps()` (see "Required Root Field" above) |

See [Troubleshooting Guide](references/troubleshooting.md) for detailed fix scripts.

## Reference Files

- **[API Reference](references/api-reference.md)**: Complete API endpoint documentation, request/response schemas, authentication details, Databricks CLI usage
- **[Workflow Patterns](references/workflow-patterns.md)**: Detailed GenieSpaceExport schema (config, data_sources, instructions, benchmarks), ID generation, serialization patterns, variable substitution, asset inventory-driven generation, complete examples
- **[Troubleshooting](references/troubleshooting.md)**: Common production errors with Python fix scripts, validation checklists, deployment checklist, error recovery patterns, field-level format requirements

## Implementation: Start from Templates (MANDATORY)

**NEVER write deployment notebooks or job YAMLs from scratch.** The templates below handle pre-flight JSON validation, correct ID field scoping, `extract_space_config()` for wrapped/raw formats, array sorting (via the canonical `sort_genie_config()`), and `version: 2` injection. Writing from scratch bypasses these safeguards.

**Step 1 — Copy the notebook template into your project:**

```bash
cp data_product_accelerator/skills/semantic-layer/04-genie-space-export-import-api/assets/templates/deploy_genie_spaces.py \
   src/{project}_semantic/deploy_genie_spaces.py
```

**Step 2 — Copy the job YAML template:**

```bash
cp data_product_accelerator/skills/semantic-layer/04-genie-space-export-import-api/assets/templates/genie-deployment-job-template.yml \
   resources/semantic/genie_deploy_job.yml
```

**Step 3 — Customize:**
- In the notebook: populate `GENIE_SPACE_METADATA` with your `{space_name: genie_space_id_<name>}` mapping
- In the job YAML: update `notebook_path` and `base_parameters` to match your bundle layout

**Available templates:**
- **`assets/templates/deploy_genie_spaces.py`** — Databricks notebook for Asset Bundle `notebook_task` deployment (parameters via `dbutils.widgets.get()`)
- **`assets/templates/genie-deployment-job-template.yml`** — Standalone Asset Bundle job YAML (for combined deployment, the orchestrator provides `semantic-layer-job-template.yml`)

> **CLI vs Notebook:** `scripts/import_genie_space.py` is the CLI tool (`argparse`) for local/CI use. The notebook template (`dbutils.widgets.get()`) is for Asset Bundle `notebook_task` deployment.

## Scripts

- **[export_genie_space.py](scripts/export_genie_space.py)**: Export Genie Space configurations
  ```bash
  python scripts/export_genie_space.py --host <workspace> --token <token> --list
  python scripts/export_genie_space.py --host <workspace> --token <token> --space-id <id> --output space.json
  ```

- **[import_genie_space.py](scripts/import_genie_space.py)**: Create/update Genie Spaces from JSON
  ```bash
  python scripts/import_genie_space.py --host <workspace> --token <token> create \
    --config space.json --title "My Space" --description "..." --warehouse-id <id>
  
  python scripts/import_genie_space.py --host <workspace> --token <token> update \
    --space-id <id> --title "Updated Title"
  ```

## Production Deployment Checklist

1. **Validate JSON Structure**

…(truncated)