05 Knowledge Assistant

Foundation Step 5: Knowledge Assistant Lifecycle

Track-neutral. This skill is consumed by Track A (Custom Agent on Apps) as a document-Q&A tool, and by Track B (Supervisor API) as a hosted knowledge_assistant tool. The output ka_endpoint_name is the canonical handle.

Agent Bricks Knowledge Assistant (KA) is a managed document Q&A agent with citations. Use this skill to create, configure, and sync a KA so its serving endpoint can be consumed downstream as a tool by:

• Track A custom agents (call the KA endpoint as a function tool from your agent server, see tracks/A-custom-agent-apps/03-tools-and-mcp/).
• Track B Supervisor API agents (declare the KA as a hosted knowledge_assistant tool — see Track B Step 2).
• AppKit apps directly (via the serving() plugin against the KA endpoint).

Beta. Knowledge Assistants API is in Beta. Refer to the official REST and SDK references linked below for the latest field and endpoint shape.

When to Use

Use this skill when any of the following apply:

• Your agent needs document Q&A with citations (policies, runbooks, playbooks, FAQs).
• You want Databricks to manage chunking, embeddings, and retrieval quality instead of building a custom RAG stack.
• You want a managed endpoint you can call from a Supervisor API agent, a Databricks App, or AI Playground.
• You expect subject-matter experts to improve quality via guidelines and labeled examples rather than code changes.

Do not use this skill when:

• You need full control over the retrieval pipeline (custom chunking, re-ranking, dense+sparse fusion, etc.) — use the Vector Search MCP path described in F3: Tools and Data Access.
• Your knowledge source is structured data (tables) — use Genie instead.

Prerequisites

Source data is NOT a prerequisite. This skill stages source markdown into the knowledge_sources volume that F0 already created — pulling from a pre-staged volume, a local ka_source directory, or auto-generating from the resolved spec. Examples that ship a DAB which uploads markdown into the volume during Module 0 (e.g. example/skyloyalty/) merely make this step's "branch (A): volume already non-empty" path a fast no-op.

Schema and volume creation are NOT this skill's responsibility. They belong to F0: UC Resources Foundation, which is invoked once near the top of the prompt sequence and creates resources for every consuming skill (KA, agent memory, tool exports, benchmark tables). If F0 has not run, fail fast with a clear error pointing the caller at F0 — do not retry the schema/volume DDL here.

Source Mode Decision

Choose exactly one source mode per knowledge source. A single KA can attach up to 10 knowledge sources.

Recommendation: Start with UC Files when in doubt. Switch to a VS index only when you need custom chunking or to share the index across multiple agents.

Lifecycle Workflow

flowchart TD
  f0["F0: Provision agent_schema +<br/>knowledge_sources volume<br/>(invoked earlier — NOT this skill)"] --> stage["Step 5_0: Stage source markdown<br/>(this skill, idempotent)"]
  stage --> createKA["Step 5a: Get-or-create KA<br/>(idempotent on display_name)"]
  createKA --> attachSource["Step 5b: Get-or-create Knowledge Source(s)<br/>(idempotent on display_name)"]
  attachSource --> syncSources["Step 5c: Sync Knowledge Sources<br/>(incremental, safe to re-run)"]
  syncSources --> pollReady["Step 5c: Poll endpoint readiness"]
  pollReady --> captureId["Step 5d: Capture knowledge_assistant_id"]
  captureId --> handoff["Hand off ka_endpoint_name + knowledge_assistant_id"]

Read references/knowledge-assistant-api-sdk.md for complete SDK and REST examples for each step, including migration between workspaces and labeled-data import/export.

Step 5_0: Stage Source Markdown (KA-Specific)

This step assumes F0 has already created the agent UC schema and the knowledge_sources MANAGED volume. It does one thing: ensure that volume contains at least one .md file ready for KA ingestion. Three deterministic branches, taken in order; the first that applies wins.

Inputs:

Pre-flight assertion: w.volumes.read(volume_path) must succeed. If it raises NotFound, fail with RuntimeError("F0 has not run — invoke 00-uc-resources-foundation before this skill") rather than retrying schema/volume DDL.

Branch logic:

1. (A) Pre-staged volume — if w.files.list_directory_contents(volume_path) returns at least one .md file (filtering out names starting with _/. per Agent Bricks ingestion rules), record the file count and skip to Step 5a. No re-upload.
2. (B) Local source directory — else if ka_source points to an existing local directory containing one or more .md files, walk it recursively and w.files.upload(volume_path + "/" + relpath, content, overwrite=True) each file. Skip files > 50 MB or names starting with _/.. Record the upload count.
3. (C) PRD-derived auto-generation — else render a minimal corpus from the resolved spec (one file per state://DataSpec.glossary[] term, one capabilities.md, one faq.md of capability-scoped Q&A pairs), write to a temp dir, then upload as in (B).

Capture handoff values:

• knowledge_source_origin — pre_staged | local_dir | prd_generated
• knowledge_source_file_count — integer >= 1
• knowledge_source_path — the /Volumes/... URI consumed by Step 5b

See references/knowledge-assistant-api-sdk.md for the full reference implementation (stage_ka_sources(...), _render_prd_corpus(...)).

Step 5a: Create the Knowledge Assistant (idempotent)

Use the Databricks SDK for Python for all lifecycle operations. This step must be idempotent: re-running the prompt with the same display_name must reuse the existing KA, never create a duplicate.

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.knowledgeassistants import KnowledgeAssistant

w = WorkspaceClient()

def get_or_create_ka(
    *,
    display_name: str,
    description: str,
    instructions: str,
) -> KnowledgeAssistant:
    """Return existing KA matching display_name, else create one. Idempotent."""
    for existing in w.knowledge_assistants.list_knowledge_assistants():
        if getattr(existing, "display_name", None) == display_name:
            return existing
    return w.knowledge_assistants.create_knowledge_assistant(
        knowledge_assistant=KnowledgeAssistant(
            display_name=display_name,
            description=description,
            instructions=instructions,
        )
    )

ka = get_or_create_ka(
    display_name="loyalty-policy-assistant",
    description="Answers questions about loyalty program rules, FAQ, campaign playbook.",
    instructions=(
        "Always cite the specific document and section. "
        "If information is not in the documents, say so explicitly. "
        "Do not fabricate policy details."
    ),
)
print(ka.name)  # "knowledge-assistants/<id>"

Key points:

• display_name is the idempotency key — pin it to the user-scoped KA_DISPLAY_NAME from the resolved spec (e.g. ${firstname}-${last_initial}-${use_case_slug}-ka) so re-runs reattach to the same KA instead of creating a new one.
• If the SDK surfaces endpoint_status="ONLINE" for the existing KA on re-run, treat the step as a no-op and proceed straight to Step 5d.
• instructions shape answer style and citation behavior — keep them source-agnostic (don't mention a specific source mode). On re-run, if instructions or description differ from the desired values, optionally call w.knowledge_assistants.update_knowledge_assistant(...) instead of recreating; never delete-then-recreate just to change metadata.
• Record ka.name (the full resource name including the id). It is required for every downstream operation.

For the REST equivalent, see the Create a Knowledge Assistant API. Use the List endpoint as the get-step in a get-or-create REST loop.

Step 5b: Attach a Knowledge Source (idempotent)

Attach at least one source. You may attach up to 10. Each source is either UC Files or a VS index. This step must be idempotent — re-runs with the same display_name must reuse the existing source.

from databricks.sdk.service.knowledgeassistants import (
    KnowledgeSource, FilesSpec, IndexSpec,
)

def get_or_create_knowledge_source(
    *,
    parent: str,
    spec: KnowledgeSource,
) -> KnowledgeSource:
    """Return existing source matching display_name on this KA, else create. Idempotent."""
    for existing in w.knowledge_assistants.list_knowledge_sources(parent=parent):
        if getattr(existing, "display_name", None) == spec.display_name:
            # Optional: drift-detect path / index_name and call update_knowledge_source.
            return existing
    return w.knowledge_assistants.create_knowledge_source(
        parent=parent, knowledge_source=spec,
    )

UC Files

files_source = get_or_create_knowledge_source(
    parent=ka.name,
    spec=KnowledgeSource(
        display_name="loyalty-program-rules",
        description="Loyalty program rules, tier benefits, and policy FAQ.",
        source_type="files",
        files=FilesSpec(path=knowledge_source_path),  # from F0
    ),
)

Vector Search index

index_source = get_or_create_knowledge_source(
    parent=ka.name,
    spec=KnowledgeSource(
        display_name="loyalty-program-rules",
        description="Loyalty program rules indexed with databricks-gte-large-en.",
        source_type="index",
        index=IndexSpec(
            index_name="main.skyloyalty.loyalty_docs_index",
            text_col="content",
            doc_uri_col="doc_uri",
        ),
    ),
)

DO — keep one topic per source so retrieval stays focused.

DO — pin display_name to a deterministic, user-scoped value (${use_case_slug}-${topic}) so it is the stable idempotency key across re-runs.

DON'T — mix policy docs and marketing copy in the same source; the description-based routing will degrade.

DON'T — change display_name between runs; it is the only key available to the get-or-create lookup.

Step 5c: Sync and Poll Readiness

Syncing triggers ingestion for file sources and refresh for index sources. It is incremental — only new or changed files are processed.

w.knowledge_assistants.sync_knowledge_sources(name=ka.name)

Poll readiness until the KA is ready to serve:

import time

def wait_for_ready(name: str, timeout_s: int = 1800, interval_s: int = 30):
    deadline = time.time() + timeout_s
    while time.time() < deadline:
        current = w.knowledge_assistants.get_knowledge_assistant(name=name)
        status = getattr(current, "endpoint_status", None) or getattr(current, "state", None)
        if status and str(status).upper() in {"ONLINE", "READY"}:
            return current
        time.sleep(interval_s)
    raise TimeoutError(f"KA {name} not ready within {timeout_s}s")

ready = wait_for_ready(ka.name)

Constraints:

• Only the creator of the KA can sync or mutate knowledge sources.
• Ingestion time scales with volume size; budget up to a few hours for large corpora.
• Re-run sync_knowledge_sources after adding or updating files.

Step 5d: Capture the Endpoint Name and Knowledge Assistant ID

The KA produces two downstream handles. Capture both — different consumers use different handles:

# ka.name is "knowledge-assistants/<id>" — the trailing segment is the id.
knowledge_assistant_id = ka.name.split("/")[-1]

# Prefer the SDK-surfaced values where available:
refreshed = w.knowledge_assistants.get_knowledge_assistant(name=ka.name)
knowledge_assistant_id = (
    getattr(refreshed, "knowledge_assistant_id", None) or knowledge_assistant_id
)
ka_endpoint_name = (
    getattr(refreshed, "endpoint_name", None)
    or f"ka-{knowledge_assistant_id}"
)

print(ka_endpoint_name, knowledge_assistant_id)

Persist both in config.yml so downstream skills (Track A tools and MCP, Track B Hosted Tools, AppKit serving() plugin) can wire them:

# config.yml (excerpt — keep both handles)
ka_endpoint_name: "ka-<captured id>"
knowledge_assistant_id: "<captured id>"

Step 5e: Manage Sources Over Time

from databricks.sdk.service.knowledgeassistants import KnowledgeSource

w.knowledge_assistants.update_knowledge_source(
    name=index_source.name,
    knowledge_source=KnowledgeSource(
        display_name="loyalty-program-rules-v2",
        description="Loyalty program rules, tier benefits, and policy FAQ (updated quarterly).",
    ),
    update_mask="display_name,description",
)

w.knowledge_assistants.delete_knowledge_source(name=files_source.name)

Rules:

• You cannot remove the last remaining source — a KA must always have at least one.
• Only the creator can add/remove/update sources; workspace admins can manage permissions but not sources.

For quality improvement (guidelines, labeled examples, import/export of a UC table with guidelines), see references/knowledge-assistant-operations.md.

Delegation Map

This skill does not duplicate infrastructure or client patterns covered elsewhere. Delegate to:

Limitations to Plan For

• English only.
• UC tables are not supported as a source.
• Files > 50 MB are skipped at ingest.
• Filenames starting with _ or . are skipped at ingest.
• VS indexes must use databricks-gte-large-en with AI Guardrails and rate limits disabled on the embedding endpoint.
• Sync is creator-restricted. Plan service-account ownership accordingly.

Validation Gate

All must pass before this KA is consumed by an agent skill:

• F0 prerequisite: state://Resources.uc_volumes.knowledge_sources is set and the volume is reachable
• At least one source markdown file is present in the volume (Step 5_0; from pre-staged bundle, local ka_source dir, or PRD auto-generation)
• KA created or reused via SDK get-or-create on display_name; ka.name recorded (Step 5a, idempotent)
• At least one knowledge source attached or reused via SDK get-or-create on display_name (Step 5b, idempotent)
• sync_knowledge_sources called successfully (Step 5c — incremental, safe to re-run on a warm KA)
• Endpoint reports ready/online via get_knowledge_assistant (Step 5c)
• Both ka_endpoint_name and knowledge_assistant_id captured in config.yml (Step 5d)
• Creator/service-account constraint documented for future syncs
• Idempotency check: running this skill a second time without changing display_name produces zero new KAs and zero new knowledge sources (verified via list_knowledge_assistants count before/after)

Notes to Carry Forward

agent_schema and knowledge_source_volume are F0 outputs, not F5 outputs. Read them from state://Resources.agent_schema and state://Resources.uc_volumes.knowledge_sources respectively.

Next Step

After passing this gate, choose your downstream path:

Related Skills

References

• Knowledge Assistant (Agent Bricks)
• Knowledge Assistants REST API
• Databricks SDK for Python — knowledge_assistants
• Configure AI Gateway on model serving endpoints
• On-demand references:
  • references/knowledge-assistant-api-sdk.md
  • references/knowledge-assistant-operations.md

Upstream Lineage

This skill builds on, and delegates to, existing Databricks Agent Skills:

• databricks/databricks-agent-skills — skills/ (auth, CLI basics, Model Serving endpoint management)
• databricks/databricks-agent-skills — databricks-agent-bricks / knowledge-assistants (KA lifecycle, provisioning timeline, Supervisor integration patterns)

Version History