Simple Agent Scaffold

Simple Agent Scaffold

Shortest reliable path from "I have Genie Spaces" to "I have a deployed agent endpoint." Follows the OpenAI MCP Tool Calling Agent notebook pattern verbatim.

Step 1         Step 2          Step 3          Step 4            Step 5
Write ──► Test locally ──► Log MLflow ──► Register UC ──► Deploy Serving
agent.py                                                       │
                                                               ▼
                                                    AI Playground (default)
                                                               │
                                                    ┌──────────┴──────────┐
                                                    ▼                     ▼
                                           06-appkit-serving       Go Further
                                           -wiring (optional)      (optional)

When to Use

• Creating a simple tool-calling agent with Genie Spaces
• Workshop quick-start: zero to deployed endpoint
• Prototyping an agent before adding evaluation, memory, or monitoring

Not for production-grade multi-agent systems. Use the full 00-course-orchestrator for evaluation pipelines, Lakebase memory, prompt registries, and multi-domain orchestration.

Prerequisites

Critical: do not skip the Genie Space test — a space that exists but can't answer questions produces an agent that only greets and never exercises the tool-calling path.

Genie Space fails to answer questions? Three remediation options:

1. Use a different space. Run databricks genie list-spaces --profile $PROFILE and test each one. Pick the first that returns data.
2. Create a new space. If all existing spaces reference tables you can't access, create a new Genie Space pointing to tables in YOUR gold schema (Workspace → Genie → New Space).
3. Fix permissions on the existing space. Ask a catalog admin to grant you SELECT on the tables the Genie Space references. Check the space's "Tables" tab to see which tables it uses.

Do not proceed to Step 1 until the Genie Space answers a data question successfully. An agent wired to a broken Genie Space will deploy but fail every data query, wasting the entire build-test-deploy cycle.

Decision Defaults

Step 1: Write agent.py

Copy the template and its config file to your project directory:

cp references/agent-template.py agent.py
cp references/agent-config.yaml agent-config.yaml

Open agent-config.yaml and resolve the three TODO blocks:

1. llm_endpoint — Verify the Foundation Model API endpoint name exists in your workspace.
2. system_prompt — Write domain-specific instructions for your agent.
3. genie_spaces — Replace each TODO_REPLACE_WITH_SPACE_ID with a real Genie Space ID. Add or remove entries as needed.

Finding Genie Space IDs:

Workspace → Genie → open a space → the ID is in the URL:
https://<workspace>.databricks.com/spaces/<SPACE_ID>/...

The MCP server URL format for Genie Spaces is:

{host}/api/2.0/mcp/genie/{space_id}

What the template contains

The template is the notebook's MCPToolCallingAgent class with one addition: ModelConfig for parameterization via agent-config.yaml. The class structure is identical to the canonical notebook:

At the bottom: mlflow.openai.autolog() enables automatic tracing and mlflow.models.set_model(AGENT) binds the model for logging.

Critical rules (from 01-responses-agent-patterns)

• ResponsesAgent is mandatory — not ChatAgent, not PythonModel.
• Never pass a signature parameter to log_model() — MLflow auto-infers it.
• Use input key, not messages — {"input": [{"role": "user", "content": "..."}]}.
• nest_asyncio is required — MCP servers use async internally; nest_asyncio.apply() avoids event loop conflicts in notebook environments.
• Build the McpServerToolkit per request, NOT at module load. A toolkit built at import time hard-binds whatever identity existed then (the system SP) and defeats OBO. predict_stream() constructs _obo_client() and the toolkit on every call so the Genie MCP call runs as the invoking user. See references/obo-authentication.md.

Gate: agent.py exists with all TODOs in agent-config.yaml resolved. No TODO_REPLACE strings remain.

Step 2: Test locally

Deploy Steps 2–5 as a bundle job (canonical — same spine as Bronze/Silver/Gold)

Workshop workspaces usually have no interactive cluster, and — more importantly — the agent endpoint should be created the same versioned way as every other artifact: by running a bundle job, not by an ad-hoc agents.deploy() in a loose notebook or a one-off jobs submit. Combine Steps 2–5 into one notebook (deploy_agent) and run it as a serverless bundle job:

1. Copy the templates into your bundle (keep them beside agent.py/agent-config.yaml so bundle deploy syncs them):
   • references/agent-deploy-notebook.py → <bundle>/agents/deploy_agent.py — the notebook-task body: Step 0 schema creation (direct SQL), Steps 2–5, Step 5b auto-grant, Step 5c checkpoint.
   • references/agent_deploy_job.yml → <bundle>/resources/agent_deploy_job.yml.
2. Wire the variables: block in databricks.yml (catalog, agent_schema, agent_model_name, gold_schema, semantic_warehouse_id, genie_space_id, agents_folder_ws_path) — see the YAML header.
3. Validate → deploy → run:

databricks bundle validate -t dev
databricks bundle deploy   -t dev
databricks bundle run      -t dev agent_deploy_job

The job's notebook creates the UC agent schema with direct SQL (CREATE SCHEMA IF NOT EXISTS — the schema exception, not a bundle resource), then logs, registers, deploys, auto-grants the endpoint system SP (Step 5b), and writes DEPLOY_CHECKPOINT.md (Step 5c).

Always use serverless (environment_key), never classic clusters (new_cluster) — workshop workspaces block classic clusters with NETWORK_CONFIGURATION_FAILURE. When run as a job the working directory is NOT the notebook's directory, which is why model_config="agent-config.yaml" is required in Step 3.

Genie Code: run bundle deploy/bundle run through runDatabricksCli from the bundle-editor page (open <bundle>/databricks.yml → "Open in bundle editor"); omit --profile (pre-authenticated) and do NOT databricks sync (deploy syncs the source). If a bundle command is blocked you are not on the bundle page — navigate there; do not fall back to jobs submit or a direct agents.deploy(). Keep the agent source under {REPO_ROOT}, never a bare relative path. See skills/genie-code-environment §2, §8.

Fallback (one-off, IDE only, no bundle): if you explicitly need a non-bundle run, references/job-submission.md documents a standalone databricks jobs submit + polling loop. It is a convenience escape hatch — the bundle job above is the canonical, version-controlled path.

Running Steps 2–5 interactively

In a notebook or Python REPL, import and test both prediction paths:

# Cell 1: Restart Python if you edited agent.py
# dbutils.library.restartPython()  # (uncomment in Databricks notebook)

# Cell 2: Test non-streaming
from agent import AGENT

result = AGENT.predict(
    {"input": [{"role": "user", "content": "What were total sales last month?"}]}
)
print(result.model_dump(exclude_none=True))

# Cell 3: Test streaming
for chunk in AGENT.predict_stream(
    {"input": [{"role": "user", "content": "What were total sales last month?"}]}
):
    print(chunk.model_dump(exclude_none=True))

Replace the test question with something your Genie Space can actually answer.

Gate: Both predict and predict_stream return valid, non-empty responses. MLflow traces are visible in the experiment UI (check the Traces tab).

Step 3: Log with MLflow (dual auth_policy — OBO-first)

Log the agent as code. This captures the agent.py file, its dependencies, and a dual auth_policy so the deployed endpoint supports BOTH the system SP (for the LLM and evaluation) and On-Behalf-Of the calling user (for the Genie MCP call).

import mlflow
from agent import LLM_ENDPOINT_NAME
from mlflow.models.auth_policy import AuthPolicy, SystemAuthPolicy, UserAuthPolicy
from mlflow.models.resources import (
    DatabricksGenieSpace,
    DatabricksServingEndpoint,
    DatabricksSQLWarehouse,
)
from pkg_resources import get_distribution

GENIE_SPACE_ID = "<GENIE_SPACE_ID>"
WAREHOUSE_ID = "<WAREHOUSE_ID>"  # the warehouse the Genie Space runs its SQL on

# Dual policy:
#   SystemAuthPolicy.resources → system SP gets CAN_QUERY (LLM), Can Run (Genie),
#       CAN USE (warehouse) automatically — used by the LLM call and by evaluation.
#   UserAuthPolicy.api_scopes  → forwards the caller's token for OBO. The Managed
#       MCP path needs "mcp.genie" (NOT "dashboards.genie", which is the
#       Conversation API) plus "sql".
auth_policy = AuthPolicy(
    system_auth_policy=SystemAuthPolicy(
        resources=[
            DatabricksServingEndpoint(endpoint_name=LLM_ENDPOINT_NAME),
            DatabricksGenieSpace(genie_space_id=GENIE_SPACE_ID),  # one per space
            DatabricksSQLWarehouse(warehouse_id=WAREHOUSE_ID),    # MANDATORY for Genie
        ]
    ),
    user_auth_policy=UserAuthPolicy(api_scopes=["mcp.genie", "sql"]),
)

with mlflow.start_run():
    logged_agent_info = mlflow.pyfunc.log_model(
        name="agent",
        python_model="agent.py",
        model_config="agent-config.yaml",  # REQUIRED — see note below
        auth_policy=auth_policy,
        pip_requirements=[
            f"mlflow[databricks]=={get_distribution('mlflow').version}",
            f"mcp=={get_distribution('mcp').version}",
            f"databricks-openai=={get_distribution('databricks-openai').version}",
            "databricks-ai-bridge",  # REQUIRED for OBO (ModelServingUserCredentials)
            "databricks-sdk",
        ],
    )

Key points:

• NO signature parameter. ResponsesAgent auto-infers it.
• python_model="agent.py" logs as "models from code" — MLflow loads the file, not a pickled object.
• model_config="agent-config.yaml" is required — MLflow copies agent.py to a temp dir for validation where the yaml isn't present. Without this parameter you get FileNotFoundError: Config file is not provided. Fixing __file__ / path tricks won't help; the parameter bypasses the file lookup.
• Use mlflow[databricks], not bare mlflow. On Azure the [databricks] extra ships azure-core and related storage SDKs required by register_model(). On AWS/GCP it adds harmless extras. Matches the pip install line in Prerequisites.
• auth_policy and resources= are mutually exclusive. Use auth_policy; put every resource inside SystemAuthPolicy.resources. Passing both raises a parameter conflict.
• DatabricksSQLWarehouse is mandatory — the Genie Space executes its SQL on that warehouse; omitting it is a common silent failure (Genie fails while the LLM works).
• databricks-ai-bridge MUST be in pip_requirements — without it ModelServingUserCredentials cannot be imported in the serving container and OBO silently degrades to the system SP.
• Add one DatabricksGenieSpace(genie_space_id="...") per Genie Space.

Why OBO instead of granting the system SP? A dual auth_policy deploys the endpoint as EMBEDDED_AND_USER_CREDENTIALS: the Genie MCP call runs as the calling user, so it respects their existing UC grants, row filters, and column masks with zero post-deploy grants. The system-SP fallback (Step 5b) is best-effort and only matters for true machine-to-machine callers. See references/post-deploy-permissions.md and genai-agents/.../references/obo-authentication.md.

Pre-deployment validation

Before registering, run the pre-deployment check:

mlflow.models.predict(
    model_uri=f"runs:/{logged_agent_info.run_id}/agent",
    input_data={"input": [{"role": "user", "content": "Hello!"}]},
    env_manager="uv",
)

This loads the agent in an isolated environment and runs a prediction, catching dependency or serialization issues.

Gate: logged_agent_info returned successfully. mlflow.models.predict() returns a valid response.

Step 4: Register in Unity Catalog

mlflow.set_registry_uri("databricks-uc")

# TODO: Set your catalog, schema, and model name
catalog = "my_catalog"
schema = "my_schema"
model_name = "my_genie_agent"
UC_MODEL_NAME = f"{catalog}.{schema}.{model_name}"

uc_registered_model_info = mlflow.register_model(
    model_uri=logged_agent_info.model_uri,
    name=UC_MODEL_NAME,
)

Gate: uc_registered_model_info returned with a version number. Verify in the Unity Catalog UI: Catalog → Models → your model.

Step 5: Deploy to Model Serving

import time
from databricks import agents
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
ENDPOINT_NAME = "<your-stable-endpoint-name>"  # explicit, <=63 chars

# Idempotency: never deploy onto an endpoint that is mid-update (raises
# ResourceConflict). If it exists, wait until NOT_UPDATING first.
try:
    while "NOT_UPDATING" not in str(w.serving_endpoints.get(ENDPOINT_NAME).state.config_update):
        print("endpoint busy; waiting…")
        time.sleep(20)
except Exception:
    pass  # endpoint doesn't exist yet — nothing to wait on

agents.deploy(
    UC_MODEL_NAME,
    uc_registered_model_info.version,
    endpoint_name=ENDPOINT_NAME,  # EXPLICIT — never rely on auto-naming
    tags={"endpointSource": "simple-agent-scaffold"},
)

This creates (or updates) a Model Serving endpoint with:

• EMBEDDED_AND_USER_CREDENTIALS from the dual auth_policy (system SP + OBO)
• OBO authentication for the Genie MCP call (runs with the caller's permissions)
• AI Playground integration

Always pass endpoint_name explicitly. agents.deploy() auto-naming prepends agents_ and truncates to 63 characters, which silently mismatches anything downstream (AppKit wiring, the checkpoint). Pick a short, stable name and reuse it everywhere.

Verify deployment

# Check endpoint status + run a query via the SDK (no PAT needed — the call is
# forwarded On-Behalf-Of you, exercising the OBO + Genie MCP path)
print(w.serving_endpoints.get(ENDPOINT_NAME).state)

r = w.serving_endpoints.query(
    name=ENDPOINT_NAME,
    inputs={"input": [{"role": "user", "content": "What were total sales last month?"}]},
)
print(r.as_dict() if hasattr(r, "as_dict") else r)

Step 5a — First data question hits PERMISSION_DENIED? Disambiguate first.

PERMISSION_DENIED: No access to table X has THREE orthogonal causes with the same error surface. Run this three-minute probe BEFORE granting anything — the cheapest fix differs per cause.

ONLY after both probes pass should you proceed (Step 5b is best-effort; the gate is the OBO query in Step 5).

Step 5b — Best-effort system-SP grants (fallback only — NOT the gate)

With the dual auth_policy from Step 3, the Genie MCP call runs On-Behalf-Of the caller — it needs zero SP grants. Step 5b only matters for true machine-to-machine callers that have no user token (an app SP token, a scheduled job) and therefore fall back to the endpoint system SP. Apply it as a best-effort top-up, never as a gate.

What probing established about this SP path:

• agents.deploy() creates a system service principal per endpoint, and it rotates — multiple distinct SPs can exist across deploys/config updates. Grant all of them.
• System SPs are NOT in workspace SCIM, so databricks service-principals list will not find them and PATCH /api/2.0/permissions/... with service_principal_name=<uuid> returns 200 but silently drops.
• SHOW GRANTS \` ON SCHEMA …returns **empty** for system SPs even after aGRANT … SUCCEEDED— they are invisible to SCIM. **Do not useSHOW GRANTSto verify**; aSUCCEEDED` statement is the best signal available, and the real proof is the OBO query in the gate.

Discover all system SPs from the endpoint events (backticks stripped to avoid malformed SQL) and apply grants best-effort:

from databricks.sdk import WorkspaceClient

def discover_endpoint_sps(w: WorkspaceClient, endpoint_name: str) -> list[str]:
    """Return ALL system SP UUIDs ever created for this endpoint (rotation-aware)."""
    resp = w.api_client.do(
        "GET", f"/api/2.0/serving-endpoints/{endpoint_name}/events", query={"limit": 200}
    )
    marker = "System service principal creation with ID "
    sps = [
        e["message"].split(marker, 1)[1].split(" ", 1)[0].strip().strip("`")
        for e in resp.get("events", [])
        if marker in e.get("message", "")
    ]
    return list(dict.fromkeys(sps))  # de-dup, preserve order


w = WorkspaceClient()
ENDPOINT_NAME = "<your-stable-endpoint-name>"
CATALOG       = "<your catalog>"
GOLD_SCHEMA   = "<your gold schema>"
WAREHOUSE_ID  = "<your warehouse id>"   # the semantic warehouse the Genie Space uses

for sp in discover_endpoint_sps(w, ENDPOINT_NAME):
    for stmt in [
        f"GRANT USE CATALOG ON CATALOG `{CATALOG}` TO `{sp}`",
        f"GRANT USE SCHEMA, SELECT, EXECUTE ON SCHEMA `{CATALOG}`.`{GOLD_SCHEMA}` TO `{sp}`",
    ]:
        try:
            w.statement_execution.execute_statement(
                warehouse_id=WAREHOUSE_ID, statement=stmt, wait_timeout="30s"
            )
            print(f"best-effort OK: {stmt}")
        except Exception as e:  # never fail the deploy on the fallback path
            print(f"best-effort SKIP ({type(e).__name__}): {stmt}")

EXECUTE is required if your Genie Space exposes TVFs as certified answers. The grants are idempotent.

For AppKit and any other user-facing caller, forward the user token (x-forwarded-access-token) so the app→agent hop stays OBO and never depends on this fallback. Full OBO-vs-SP matrix and the silent-drop traps: see references/post-deploy-permissions.md.

Step 5 verification gate — OBO query via the SDK (the real gate)

A greeting in AI Playground does NOT verify the tool-calling path. Query the deployed endpoint with a domain-specific data question via the SDK — the call is forwarded On-Behalf-Of you, so it exercises the OBO + Genie MCP path end-to-end. No PAT, no curl, no databricks auth token (hard-blocked on Genie Code):

r = w.serving_endpoints.query(
    name=ENDPOINT_NAME,
    inputs={"input": [{"role": "user", "content": "<domain-specific data question>"}]},
)
payload = r.as_dict() if hasattr(r, "as_dict") else r
out = payload.get("output", [])
assert any(o.get("type") == "function_call" for o in out), "no Genie tool call — greeting only"
assert any(o.get("type") == "message" for o in out), "no message with data"
print("PASS — OBO tool-calling path returned data")

IDE convenience: if you prefer a shell check on IDE, curl + PAT against /invocations works too (a PAT call is also forwarded OBO). It is NOT available on Genie Code (databricks auth token is hard-blocked) — use the SDK query above there.

The original curl + PAT form (for reference on IDE):

ENDPOINT="<your endpoint>"
HOST="$(databricks auth env --profile $PROFILE | jq -r .env.DATABRICKS_HOST)"
TOKEN="$(databricks auth token --profile $PROFILE | jq -r .access_token)"

curl -sS -X POST "$HOST/serving-endpoints/$ENDPOINT/invocations" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"input":[{"role":"user","content":"<domain-specific data question>"}]}' \
  | jq '.output[] | select(.type=="function_call" or .type=="message")'

• PASS — at least one function_call to <tool>__query_space_<space_id> appears in .output, followed by a message with real numbers.
• FAIL (greeting only) — no function_call means the tool wasn't exercised. Either the system prompt is too generic (add a domain nudge) or the Genie Space has no content (run Step 5a probes).
• FAIL (PERMISSION_DENIED) — under OBO this means YOUR own UC grants are missing on the space's tables (the query runs as you). Run Step 5a first; if the space is healthy, grant yourself SELECT/EXECUTE. The system-SP grant (Step 5b) only affects M2M callers, not this OBO gate.

Gate: ALL of the following:

1. Endpoint reaches READY state (EMBEDDED_AND_USER_CREDENTIALS).
2. Step 5a probes pass (space is healthy, serialized_space is non-empty).
3. The SDK OBO query with a domain-specific data question returns at least one function_call in .output followed by a message with real data.
4. Agent is visible in AI Playground (a smoke test, NOT the gate).

Step 5b (system-SP grant) is best-effort and explicitly not part of the gate — OBO does not require it.

Step 5c — Emit DEPLOY_CHECKPOINT.md for Step 17 handoff

After the verification gate passes, write a structured checkpoint file that Step 17 (06-appkit-serving-wiring) reads instead of rederiving values. Without this file, downstream AppKit wiring hand-recomputes the endpoint name, substitutes it in app.yaml, and often picks the wrong casing for DATABRICKS_SERVING_ENDPOINT_NAME's valueFrom — a common, silent failure mode.

Write to apps_lakebase/$APP_NAME/agents/DEPLOY_CHECKPOINT.md:

import pathlib, textwrap

APP_NAME = "<your app>"                         # same as Step 17's $APP_NAME
checkpoint_dir = pathlib.Path(f"apps_lakebase/{APP_NAME}/agents")
checkpoint_dir.mkdir(parents=True, exist_ok=True)

SPS = discover_endpoint_sps(w, ENDPOINT_NAME)  # from Step 5b

checkpoint = textwrap.dedent(f"""
    # Agent Deploy Checkpoint (Step 16)

    Structured handoff to Step 17 (`apps_lakebase/skills/06-appkit-serving-wiring`).
    Do NOT rederive these values by hand — read them from this file.

    | Field                        | Value |
    |---                           |---    |
    | Endpoint name                | `{ENDPOINT_NAME}` |
    | Auth model                   | `EMBEDDED_AND_USER_CREDENTIALS` (OBO-first) |
    | UC model name                | `{UC_MODEL_NAME}` |
    | UC model version             | `{registered.version}` |
    | Genie Space ID               | `{GENIE_SPACE_ID}` |
    | Warehouse ID                 | `{WAREHOUSE_ID}` |
    | System SP(s) (best-effort)   | `{', '.join(SPS) or '(none discovered)'}` |
    | Gold schema (best-effort)    | `{CATALOG}.{GOLD_SCHEMA}` |

    ## How the agent authenticates to Genie

    - **Primary (proven):** On-Behalf-Of the caller (`UserAuthPolicy` scope
      `mcp.genie`). Genie runs as the invoking user — zero SP grants needed.
    - **Fallback (best-effort):** true M2M callers use the system SP, which needs
      UC `SELECT`/`EXECUTE` on `{CATALOG}.{GOLD_SCHEMA}` (Step 5b, not guaranteed).
    - **AppKit:** forward the user token (`x-forwarded-access-token`) to keep the
      app→agent hop OBO.

    ## Verify (SDK, no PAT)

    ```python
    from databricks.sdk import WorkspaceClient
    w = WorkspaceClient()
    r = w.serving_endpoints.query(
        name="{ENDPOINT_NAME}",
        inputs={{"input": [{{"role": "user", "content": "<domain-specific data question>"}}]}},
    )
    print(r.as_dict() if hasattr(r, "as_dict") else r)
    ```

    PASS = at least one `function_call` to `<tool>__query_space_{GENIE_SPACE_ID}` in `.output`, then a `message` with real numbers.
""").strip()

(checkpoint_dir / "DEPLOY_CHECKPOINT.md").write_text(checkpoint + "\n")

The reference notebook references/agent-deploy-notebook.py writes this file automatically at the end of the run.

Checkpoint gate: DEPLOY_CHECKPOINT.md exists at apps_lakebase/$APP_NAME/agents/DEPLOY_CHECKPOINT.md with endpoint name, auth model, UC model name/version, Genie Space ID, warehouse ID, best-effort SP(s), and the SDK verify snippet. Step 17 reads this file on entry.

What's Next

Wire to AppKit UI (recommended)

The deployed endpoint from Step 5 is ready to be consumed by an AppKit application.

1. Read apps_lakebase/$APP_NAME/agents/DEPLOY_CHECKPOINT.md — Step 5c emitted this file with the endpoint name, SP UUID, UC model version, and the verified curl block. Step 17 reads these values from disk rather than rederiving them.
2. Read 06-appkit-serving-wiring — start at Step 2 (Configure app.yaml). The wiring skill covers Serving plugin registration, resource binding, streaming chat hooks, and server-side proxy patterns. Use the endpoint name from DEPLOY_CHECKPOINT.md verbatim — do NOT retype it.
3. If you haven't registered the Serving plugin yet, first read 04-appkit-plugin-add with references/plugin-serving.md.

Contract the UI layer must honor (when wiring this endpoint into an AppKit app via 06-appkit-serving-wiring):

1. Payload: send {"input": [{"role":"user","content":"..."}]} to the endpoint — not {"messages": [...]}. Sending messages produces 400: Model is missing inputs ['input'].
2. Streaming chunks: this agent emits the Databricks Responses API format — { type: "response.output_text.delta", delta: "..." } — not OpenAI chat completion. Parsers that read only chunk.choices[0].delta.content will return empty strings.
3. Forward the user token (OBO): this endpoint is EMBEDDED_AND_USER_CREDENTIALS, so the Genie MCP call runs On-Behalf-Of whoever's token reaches /invocations. Have the app forward x-forwarded-access-token so each user queries Genie as themselves — no system-SP grants needed. If the app instead calls with only its own SP token, it falls back to the endpoint system SP, which needs UC SELECT/EXECUTE (best-effort Step 5b) — see references/post-deploy-permissions.md.

If wiring via the AppKit serving() plugin, items 1 and 2 are handled automatically by the plugin. If building a custom proxy (plugin unavailable, older AppKit version), see 06-appkit-serving-wiring/references/custom-proxy-fallback.md and 06-appkit-serving-wiring/references/sse-format-patterns.md for the transformation + dual parser.

Add capabilities (optional)

Each add-on is an independent worker skill. Pick only what you need:

Full production agent

For the complete 9-phase implementation (foundation through monitoring), use 00-course-orchestrator. It routes to the appropriate foundation, track, SDLC, and monitoring skills in the recommended order.

Gotchas

Debugging decision tree for log_model() and register_model() errors

When log_model() or register_model() fails, match the error to a branch before retrying:

log_model() / register_model() failed
│
├── FileNotFoundError / "Config file is not provided"
│   │
│   ├── Trace shows _load_model_code_path / exec_module / MLflow internals
│   │     → Framework lifecycle error, not a path error.
│   │       Fix: add model_config="agent-config.yaml" to log_model().
│   │       Do NOT touch __file__, os.getcwd(), or Path() juggling.
│   │
│   └── Trace shows your own code reading a file
│         → Real missing file. Check path and CWD.
│
├── ModuleNotFoundError (azure.core, boto3, google.cloud, etc.)
│     → Bare `mlflow` is missing cloud storage SDKs.
│       Fix: install and declare `mlflow[databricks]` (covers all clouds).
│
├── "Model not found" / UC registration failure
│     → Catalog/schema missing or caller lacks CREATE MODEL.
│       Fix: verify `registered_model_name = "<catalog>.<schema>.<name>"`,
│       confirm CREATE MODEL on the schema, create catalog/schema if needed.
│
└── Timeout / endpoint stuck in PENDING
      → Dependency resolution or networking.
        Check the endpoint Events tab. Ensure `uv` is in pip_requirements.
        In restricted workspaces, resubmit as a serverless job.

Key principle: when the error comes from a framework's internal loader (not your code), the fix is almost never a path change — it's using the framework's parameter that passes context into the loading step.

Anti-Patterns

When a run fails, check whether you're falling into one of these reasoning traps before trying another variant of the same fix:

• FileNotFoundError from a framework loader ≠ a path problem. When the trace comes from _load_model_code_path, exec_module, or any framework-internal loader, the fix is almost never a path change — it's passing context into the loading step (e.g., model_config= for MLflow).
• "Genie Space exists" ≠ "Genie Space works." Listing a space proves it was created, not that its tables are queryable. Always ask a real question from Step 0 before wiring it into an agent.
• Classic cluster is not the default. In restricted/workshop workspaces, serverless is the safe default. Check prior compute patterns in .vibecoding-state.md before picking a cluster type.
• READY endpoint ≠ working agent. An endpoint that only answers greetings has never exercised the tool-calling path. Verify with a domain-specific data question.
• Same error class after retry ≠ try another variant — change approach. If two path-style fixes produce the same framework-loader error, the category of fix is wrong. Step back and understand the framework's lifecycle.
• Playground greeting ≠ Step 5 gate. A greeting proves the LLM is live but never exercises the Genie tool path. Use the Step 5 gate: an SDK serving_endpoints.query(...) with a domain question and assert a function_call + a message with data. (Playground and the SDK both forward the caller's identity OBO on an EMBEDDED_AND_USER_CREDENTIALS endpoint.)
• databricks service-principals list ≠ SP discovery for agents.deploy() endpoints. The endpoint's system SP is NOT in SCIM (and rotates across deploys). Use GET /api/2.0/serving-endpoints/{name}/events with the marker "System service principal creation with ID" and grant ALL discovered SPs. See Step 5b discover_endpoint_sps().
• PATCH /permissions/warehouses/{id} with service_principal_name = <uuid> ≠ granting a system SP. Returns 200 OK but silently drops the entry. Rely on users-group inheritance for workspace ACLs; grant UC privileges directly by UUID for the database layer.

Do NOT PATCH /api/2.0/data-rooms/{id} with partial payloads

/api/2.0/data-rooms/{id} is an internal API surface. PATCHing it with anything other than a full serialized_space payload (e.g. to flip run_as_type, display_name, or warehouse_id) will silently wipe the space's serialized_space. Symptoms after the wipe:

• GET /api/2.0/genie/spaces/{id} returns serialized_space = {} (empty).
• databricks genie start-conversation $SPACE_ID --content "..." returns "no tables or functions are available in this Genie space schema".
• Agent endpoint /invocations calls return PERMISSION_DENIED: No access to table X — misleading, because the real cause is an empty space, not a permission gap.

The ONLY supported mutation endpoint for a Genie Space's content is:

PATCH /api/2.0/genie/spaces/{id}  with  {"serialized_space": "<full JSON string>"}

If you hit this wipe, recover with references/restore-genie-space.py — it reads the source-of-truth genie_configs/*.json, substitutes template vars, s

…(truncated)