04 Authentication

Agent authentication (Databricks)

Doc-driven skill for how AI agents authenticate on Databricks Apps and Model Serving. Pair with Track A Step 2 (agent_class) when wiring an agent implementation to the right client and grants.

Primary sources:

• Authentication for AI agents (Databricks Apps)
• Authentication for AI agents (Model Serving)

When to Use

Use this skill whenever you are:

• Configuring permissions for a deployed agent (SP vs user, scopes, resources).
• Implementing user authorization (OBO) in agent handlers.
• Declaring MLflow resources for automatic authentication passthrough on Model Serving.
• Supporting dual deployment (Apps + Model Serving + local) with one codebase.
• Connecting to Databricks MCP servers or external MCP from an agent.

Cross-step impact: The auth patterns you establish here affect both Track A evaluation and deployment (A6/A7) and the SDLC pipeline. Environment detection (local vs Apps vs Model Serving) must work in all three contexts — especially during A6: Evaluation (which runs under your CLI credentials, not OBO) and S6: Deployment & Automation (which provisions a system SP for production).

Two authentication methods overview

You can combine both in one agent (for example, app SP for a shared Vector Search index and user client for user-specific tables).

App authorization (default for Databricks Apps)

• Databricks automatically creates a service principal when the app is created; that SP is the app identity.
• All users of the app share the same permissions granted to that SP.
• Grant the SP access via the Databricks Apps UI (Edit app → Configure → resources) or declaratively in databricks.yml / bundle app resources. See Apps in bundles.

MLflow experiment

The agent needs an MLflow experiment for traces and evaluation. Grant the SP Can Edit on that experiment and add it as an app resource. See Add an MLflow experiment resource.

Minimum permissions (common resources)

Align grants with official tables; typical minimum expectations for agents:

For the complete permission matrix (all resource types, grant methods, dependency chains), see: references/resource-permissions.md.

Bundle-kind selection: In databricks.yml, warehouses, serving endpoints, and UC connections have their own top-level kinds (sql_warehouse, serving_endpoint, connection). UC functions/tables/schemas/models use the uc_securable kind with the right securable_type. Another Databricks App (e.g. when an AppKit frontend proxies to an Agent App backend) uses the app kind with CAN_USE. See 03-tools-and-mcp for the full kind-selection table and the databricks bundle schema verification command.

For Model Serving automatic passthrough, permission names and minimum MLflow versions per resource type differ; see the Model Serving authentication page.

Prompt Registry (UC): grant CREATE FUNCTION, EXECUTE, and MANAGE on the UC schema used to store prompts.

Downstream dependencies: If you grant access to a composite resource (for example a Genie space), you must also grant access to dependent objects (underlying tables, warehouses, UC functions, etc.). Missing downstream grants surface as permission denied at runtime.

Who can attach resources: The user adding a resource needs Can Manage on both the resource and the app (per product docs).

Security hardening: follow App authorization for least privilege and credential handling.

For complete app authorization patterns (SP permission granting via UI/CLI/SDK, databricks.yml declarations, least privilege patterns, audit and rotation), see: references/app-authorization.md.

User authorization (OBO) for Databricks Apps

User authorization is Public Preview as of product documentation; workspace admins may need to enable it.

How it works

1. Add API scopes in the app so Databricks knows which APIs the app may call on behalf of users. See Add scopes to an app.
2. Databricks downscopes the user's credentials to those scopes.
3. The downscoped token is forwarded via the x-forwarded-access-token HTTP header.
4. MLflow Agent Server stores the token per request for use in agent code.

Resources reachable with user authorization (per docs) include SQL warehouses, Genie spaces, files/directories, Model Serving endpoints, Vector Search indexes, UC connections, and UC tables.

Canonical Databricks Apps user headers

Databricks Apps sets the following canonical headers on every inbound request. Read these directly — there is no JSON-blob "user info" header.

• x-forwarded-email — the end user's email (canonical attribute used as MLflow AssessmentSource.source_id).
• x-forwarded-preferred-username — the user's preferred username (often equals the email).
• x-forwarded-user — the user's stable workspace user id.
• x-forwarded-access-token — the downscoped OBO token for user-scoped Databricks API calls (only present when the App declares user_api_scopes).

x-forwarded-user-info is not a canonical Databricks Apps header and must not be used. Earlier internal patterns assumed it existed; it does not.

Pathway-C: outbound x-app-user-email from AppKit to the Agent App

When the AppKit App (frontend) is a separate Databricks App from the Agent App (backend) — the 2-Apps / Variant 4 / Pathway-C deployment — the AppKit proxy must set x-app-user-email on outbound requests to the Agent App. The agent uses this value as AssessmentSource.source_id for MLflow feedback (see SDLC 04c-end-user-feedback).

Do not infer originating user identity from the app-to-app OAuth token. The Authorization: Bearer header on the inbound agent request represents the AppKit App service principal hop between the two Databricks Apps, not the originating end user. If the agent calls WorkspaceClient().current_user.me() on that Bearer, it gets the AppKit SP — not the user — and every assessment will be attributed to the same SP UUID.

Concretely, in the Agent App's @invoke / @stream handler:

from mlflow.genai import agent_server
import mlflow

@agent_server.invoke
async def handle_invoke(request: dict, http_request) -> dict:
    user_email = http_request.headers.get("x-app-user-email")  # set by AppKit proxy
    # ... run the agent ...
    mlflow.log_feedback(
        trace_id=trace_id,
        name="user_feedback",
        value=is_upvoted,
        source=mlflow.entities.AssessmentSource(
            source_type="HUMAN",
            source_id=user_email,  # NOT the SP from the Bearer
        ),
    )

The proxy-side wire-up (TypeScript) lives in apps_lakebase/skills/06d-appkit-agent-app-proxy/SKILL.md under "Header Contract".

Declare API scopes in app.yaml (canonical)

Declare the scopes your agent needs in the App's app.yaml under user_api_scopes. The set should be least privilege — list only the APIs the agent actually calls on behalf of the user.

# app.yaml — Agent App
command:
  - mlflow
  - genai
  - agent-server
  - serve
  - --module
  - agent

user_api_scopes:
  - serving.serving-endpoints      # call LLM + KA endpoints as the user
  - dashboards.genie                # call Genie spaces as the user
  - sql.statement-execution         # run SQL with user warehouse permissions
  - vectorsearch.vector-search-endpoints  # if using Vector Search
  - catalog.connections             # if calling external MCP via UC connection

When a Databricks App declares scopes, the user must consent the first time they open the app. Token downscoping is enforced by Databricks — your code just reads x-forwarded-access-token from the incoming request.

Implement in code (Option B canonical)

Import the helper that matches your wrapping pattern:

# Option B (canonical) — module-level @invoke / @stream on Apps:
from databricks_app.utils import get_user_workspace_client

# Option A (legacy) — class-based ResponsesAgent on Model Serving:
# from agent_server.utils import get_user_workspace_client

CRITICAL: Call get_user_workspace_client() inside the request handler (@invoke / @stream for Option B, or predict / predict_stream for Option A), not in __init__ or app startup. There is no user context at startup; initializing early fails or uses the wrong identity.

# Option B canonical — receive the http_request, hand it to the helper
from mlflow.genai import agent_server
from databricks_app.utils import get_user_workspace_client

@agent_server.invoke
async def handle_invoke(request: dict, http_request) -> dict:
    ws = get_user_workspace_client(http_request)   # user-scoped
    response = ws.serving_endpoints.query(
        name="my-endpoint",
        messages=request["input"],
    )
    return {"output": response.choices[0].message.content}

For scope design and escalation considerations, see Scope-based security and privilege escalation.

For complete OBO implementation patterns (full agent class, scope walkthrough, token lifecycle, testing locally vs deployed, combining app + user auth), see: references/user-authorization.md.

Automatic auth passthrough (Model Serving)

For agents logged and served on Model Serving, declare dependencies at mlflow.pyfunc.log_model time. Databricks provisions and rotates short-lived credentials (M2M OAuth-style) for a system SP tied to the model version so the agent can call declared resources—no manual PAT wiring for those resources.

Notes from product docs:

• Endpoint creator must be able to access all logged resources.
• The system SP for the model version may not appear in normal SP listings; it is removed if the model version is removed from the endpoint.
• Log all downstream dependencies (same principle as Apps).

Example pattern (trim resources to what your agent actually uses):

from mlflow.models.resources import (
    DatabricksServingEndpoint,
    DatabricksVectorSearchIndex,
    DatabricksLakebase,
)

mlflow.pyfunc.log_model(
    python_model="/path/to/agent.py",
    resources=[
        DatabricksServingEndpoint(endpoint_name="my-llm"),
        DatabricksVectorSearchIndex(index_name="prod.docs.index"),
        DatabricksLakebase(database_instance_name="my-lakebase"),
    ],
)

See log agent and mlflow.pyfunc.log_model for full resources types, AuthPolicy / OBO on Model Serving, and MLflow version minimums per resource.

For complete Model Serving auth patterns (all resource types, AuthPolicy, system SP lifecycle, version requirements), see: references/model-serving-auth-passthrough.md.

Environment detection for dual deployment

Use environment signals to pick passthrough resources vs Apps OBO vs local development:

import os

if os.environ.get("IS_IN_DB_MODEL_SERVING_ENV"):
    # Model Serving: rely on resources declared at log_model time (and optional OBO policies)
    pass
elif os.environ.get("DATABRICKS_APP_NAME"):
    # Databricks Apps: user auth via Agent Server helper when using OBO
    from agent_server.utils import get_user_workspace_client
    client = get_user_workspace_client()
else:
    # Local / notebook: default workspace client (user or configured auth)
    from databricks.sdk import WorkspaceClient
    client = WorkspaceClient()

Treat this as a pattern; exact env vars and client construction should match your template and MLflow/Databricks versions.

For complete OBO authentication patterns (environment detection, error handling, token refresh, production learnings), see: references/obo-authentication.md.

MCP server authentication

Databricks managed MCP

• Declare all required resources in databricks.yml / bundle and in MLflow resources when applicable.

• Grant the app's service principal (app authorization) or users (user authorization) access to every downstream UC object the MCP paths touch (for example vector indexes and UC functions under the schemas used in MCP URLs).

• Product docs illustrate URL patterns under /api/2.0/mcp/...; ensure grants cover each index/schema/function namespace your agent uses.

• Auth header source. When constructing MCPServerSse inside an @invoke / @stream handler, pull the bearer token from the user-scoped client to preserve OBO end-to-end:

  ws = get_user_workspace_client(http_request)
  ai_search_mcp = MCPServerSse(
      # AI Search (formerly Vector Search); legacy /mcp/vector-search/ prefix still works
      url=f"https://{ws.config.host}/api/2.0/mcp/ai-search/prod/docs/idx",
      headers={"Authorization": f"Bearer {ws.config.token}"},
  )
  

  Building MCP clients at module load uses the SP token instead — fine for app-authorization, wrong for user-authorization.

For managed MCP, consider databricks_mcp.DatabricksMCPClient().get_databricks_resources() (from the databricks-mcp package) to enumerate dependencies—see Authentication for AI agents (Model Serving).

External MCP servers

• Databricks Apps / agents calling external MCP servers typically use Bearer tokens or OAuth client credentials (M2M) supplied by the external provider, stored and rotated per your platform’s secret management (not PATs for Databricks identity).
• Keep least privilege: scope tokens to the MCP server audience only; do not reuse workspace admin tokens for third-party MCP.

Common mistakes

Validation checklist

• Chosen app authorization, user authorization, or a documented combination for each resource class.
• SP or user has minimum privilege on experiments, warehouses, endpoints, indexes, UC objects, Lakebase, volumes, connections as required.
• Downstream dependencies for Genie, tools, and MCP URLs are granted.
• OBO: user_api_scopes declared in app.yaml (least privilege) or added via the Apps UI; get_user_workspace_client(http_request) is only called inside @invoke / @stream handlers.
• Model Serving: resources (and AuthPolicy if using OBO) complete at log_model time.
• MCP: bundle/databricks.yml resources and UC grants align with every MCP path used.
• Dual-env code paths tested in App, serving, and local where applicable.

Next Step

After passing this gate, load and execute tracks/A-custom-agent-apps/05-lakebase-memory/SKILL.md (A5: Lakebase Memory).

Related Skills

References

• Authentication for AI agents (Databricks Apps)
• Authentication for AI agents (Model Serving)
• Author an AI agent and deploy it on Databricks Apps
• Configure authorization in a Databricks app
• Add resources to a Databricks app
• App authorization (deep dive)
• Apps in bundles
• MLflow Agent Server
• mlflow.pyfunc.log_model