databricks-solutions
- 221 skills
- 0 followers
- 1 day ago last updated
- ▌ Databricks Isv Go SQL Driver · databricks-solutions bundlePWAF-compliant Databricks SQL Driver for Go (databricks-sql-go): PAT, OAuth M2M, OAuth U2M (browser + token-env + custom OAuth app PKCE), WithUserAgentEntry. Use when building or testing integrations that run SQL queries via a Databricks SQL warehouse.
- ▌ Databricks Isv Python Dbconnect · databricks-solutions bundlePWAF-compliant Databricks Connect for Python: PAT, OAuth M2M, OAuth U2M (PKCE + token-env), userAgent telemetry. Use when building Spark applications that run on Databricks serverless or classic compute.
- ▌ Pwaf Connector Testing · databricks-solutions bundleHow to build and use a PWAF connector test runner (tests/run_all_tests.sh). Covers env isolation, auth types, single connector or single auth mode, parallel execution, browser tests, and report generation.
- ▌ Databricks Isv Nodejs SQL Driver · databricks-solutions bundlePWAF-compliant Databricks SQL Driver for Node.js (@databricks/sql): PAT, OAuth M2M, OAuth U2M (custom OAuth app PKCE + token-env), userAgentEntry telemetry. Use when building or testing integrations that run SQL queries via a Databricks SQL warehouse.
- ▌ Databricks Isv Python Sqlalchemy · databricks-solutions bundleComprehensive SQLAlchemy + Databricks patterns: PAT/M2M/U2M authentication, User-Agent telemetry, connection URL format, error handling, and retry logic. Use when building or testing Python integrations using databricks-sqlalchemy dialect.
- ▌ Databricks Isv Databricks Connect · databricks-solutions bundlePWAF-compliant Databricks Connect (Python): PAT, OAuth M2M, OAuth U2M; serverless and classic compute. Use when building or testing Spark-over-Connect integrations.
- ▌ Databricks Isv Connector Structure · databricks-solutionsHow to structure a Databricks connector (REST or Python SDK): config, connect, operations, validation. Use when designing or building a new connector.
- ▌ Databricks Isv Python SQL Connector · databricks-solutions bundlePWAF-compliant Python SQL Connector (databricks-sql-connector): PAT, OAuth M2M, OAuth U2M (custom OAuth app PKCE + token-env), credentials_provider patterns, error handling, retry logic. Use when building Python integrations that run SQL queries via a Databricks SQL warehouse.
- ▌ Databricks Isv Telemetry Attribution · databricks-solutionsPWAF User-Agent telemetry: format, per-driver configuration (Python SDK, Java SDK/JDBC, Go SDK/SQL, Node.js, REST, Databricks Connect). Use when setting up User-Agent attribution in any connector.
- ▌ Databricks Isv Adding Databricks Connector · databricks-solutionsAdd a Databricks connector to an existing project that has no Databricks integration. Use when your product already exists and you want to add Databricks as a new data source or backend.
- ▌ Deploy · databricks-solutionsDeploy agent to Databricks Apps using DAB (Databricks Asset Bundles). Use when: (1) User says 'deploy', 'push to databricks', or 'bundle deploy', (2) 'App already exists' error occurs, (3) Need to bind/unbind existing apps, (4) Debugging deployed apps, (5) Querying deployed app endpoints.
- ▌ Add Tools · databricks-solutions bundleAdd tools to your agent and grant required permissions in databricks.yml. Use when: (1) Adding MCP servers, Genie spaces, vector search, or UC functions to agent, (2) Permission errors at runtime, (3) User says 'add tool', 'connect to', 'grant permission', (4) Configuring databricks.yml resources.
- ▌ Quickstart · databricks-solutionsSet up Databricks agent development environment. Use when: (1) First time setup, (2) Configuring Databricks authentication, (3) User says 'quickstart', 'set up', 'authenticate', or 'configure databricks', (4) No .env file exists.
- ▌ Run Locally · databricks-solutionsRun and test the agent locally. Use when: (1) User says 'run locally', 'start server', 'test agent', or 'localhost', (2) Need curl commands to test API, (3) Troubleshooting local development issues, (4) Configuring server options like port or hot-reload.
- ▌ Agent Memory · databricks-solutions bundleAdd memory capabilities to your agent. Use when: (1) User asks about 'memory', 'state', 'remember', 'conversation history', (2) Want to persist conversations or user preferences, (3) Adding checkpointing or long-term storage.
- ▌ Create Tools · databricks-solutions bundleCreate Databricks resources that agents connect to as tools. Use when: (1) User needs to create a Genie space, vector search index, UC function, or UC connection, (2) User says 'create tool', 'set up genie', 'create vector search', 'register MCP server', (3) Before add-tools when the resource doesn't exist yet, (4) User asks 'what do I need to create before adding this tool'.
- ▌ Load Testing · databricks-solutions bundleLoad test a Databricks App to find its maximum QPS. Use when: (1) User says 'load test', 'benchmark', 'QPS', 'throughput', or 'performance test', (2) User wants to find how many queries per second their app can handle, (3) User wants to set up load testing scripts for their agent, (4) User wants to view load test results/dashboard.
- ▌ Modify Agent · databricks-solutionsModify agent code, add tools, or change configuration. Use when: (1) User says 'modify agent', 'add tool', 'change model', or 'edit agent.py', (2) Adding MCP servers to agent, (3) Changing agent instructions, (4) Understanding SDK patterns.
- ▌ Discover Tools · databricks-solutionsDiscover available tools and resources in Databricks workspace. Use when: (1) User asks 'what tools are available', (2) Before writing agent code, (3) Looking for MCP servers, Genie spaces, UC functions, or vector search indexes, (4) User says 'discover', 'find resources', or 'what can I connect to'.
- ▌ Lakebase Setup · databricks-solutionsConfigure Lakebase for agent memory storage. Use when: (1) Adding memory capabilities to the agent, (2) 'Failed to connect to Lakebase' errors, (3) Permission errors on checkpoint/store tables, (4) User says 'lakebase', 'memory setup', or 'add memory'.
- ▌ Databricks Apps · databricks-solutions bundleBuild apps on Databricks Apps platform. Use when asked to create dashboards, data apps, analytics tools, or visualizations. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.
- ▌ Databricks Core · databricks-solutions bundleDatabricks CLI operations: auth, profiles, data exploration, and bundles. Contains up-to-date guidelines for Databricks-related CLI tasks.
- ▌ Databricks Dabs · databricks-solutions bundleCreate, configure, validate, deploy, run, and manage Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). Use when working with Databricks resources via DABs including dashboards, jobs, pipelines, alerts, volumes, and apps.
- ▌ Agent Evaluation · databricks-solutions bundleUse this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving tool selection accuracy, answer quality, reducing costs, or fixing issues where the agent gives wrong/incomplete responses. Evaluates agents systematically using MLflow evaluation with datasets, scorers, and tracing. IMPORTANT - Always also load the instrumenting-with-mlflow-tracing skill before starting any work. Covers end-to-end evaluation workflow or individual components (tracing setup, dataset creation, scorer definition, evaluation execution).
- ▌ Integration Test · databricks-solutionsRun the full cold-start end-to-end deploy test in an isolated git worktree with always-safe teardown. Use when: (1) verifying the one-shot `make deploy` works on a FRESH workspace/catalog (cold-start), (2) the user says 'integration test', 'cold-start test', 'test the deploy end to end', or 'does deploy still work on a new workspace', (3) validating a change to deploy.sh / databricks.yml / the seed before opening a PR. PROJECT-AUTHORED (not a vendored skill).
- ▌ Databricks Lakebase · databricks-solutions bundleDatabricks Lakebase Postgres: projects, scaling, connectivity, Lakebase synced tables, and Data API. Use when asked about Lakebase databases, OLTP storage, or connecting apps to Postgres on Databricks.
- ▌ Sync Architecture Docs · databricks-solutionsKeep this repo's architecture diagrams and prose docs in sync with the code as it evolves. Use this whenever the user asks to update/refresh the architecture diagrams, sync docs with the code, audit docs for staleness, or check that docs/architecture.md still matches the implementation — and proactively after any significant change (a new graph node, tool, Lakebase schema, data-pipeline step, auth change, or frontend swap), since the diagrams and READMEs drift silently. Covers refreshing the four Mermaid diagrams in docs/architecture.md, reconciling the known stale-doc patterns, and verifying with greps + Mermaid validity.
- ▌ Databricks Model Serving · databricks-solutions bundleManage Databricks Model Serving endpoints via CLI. Use when asked to create, configure, query, or manage model serving endpoints for LLM inference, custom models, or external models.
- ▌ Databricks Vector Search · databricks-solutions bundleDatabricks Vector Search endpoints and indexes for RAG and semantic search; covers index types, search modes, end-to-end RAG patterns
- ▌ Migrate From Model Serving · databricks-solutionsMigrate an MLflow ResponsesAgent from Databricks Model Serving to Databricks Apps. Use when: (1) User wants to migrate from Model Serving to Apps, (2) User has a ResponsesAgent with predict()/predict_stream() methods, (3) User wants to convert to @invoke/@stream decorators.
- ▌ Instrumenting With Mlflow Tracing · databricks-solutions bundleInstruments Python and TypeScript code with MLflow Tracing for observability. Must be loaded when setting up tracing as part of any workflow including agent evaluation. Triggers on adding tracing, instrumenting agents/LLM apps, getting started with MLflow tracing, tracing specific frameworks (LangGraph, LangChain, OpenAI, Gemini, DSPy, CrewAI, AutoGen), or when another skill references tracing setup. Examples - "How do I add tracing?", "Instrument my agent", "Trace my LangChain app", "Set up tracing for evaluation"
- ▌
- ▌
- ▌ Genie Code Environment · databricks-solutions bundleSession-start behavioral manifest for running this workshop inside Databricks Genie Code. Read FIRST when the client is Genie Code (the coding agent embedded in the Databricks workspace) so you begin knowing how it behaves — surface/page tool-scoping, the three execution paths (runDatabricksCli → SDK → native tools) and the "blocked ≠ impossible, try the next path" discipline, the runDatabricksCli allow-list tiers, bundle-deploy reality (--target dev mandatory, CWD pinned to the page's bundle root, FUSE create-then-validate gap), AppKit/Node reality (apps init --output-dir, no local npm but server-side build, SDK app deploy), agent-skills git-clone install, the Genie-Space deploy tiers, and how to verify a deployed app (3-hop OAuth session). Detection of WHICH client is active lives in vibecoding-state; this skill explains HOW the Genie Code client behaves. Not needed for the IDE+CLI client.
- ▌ Databricks Expert Agent · databricks-solutions bundleTransforms the assistant into a Senior Databricks Solutions Architect Agent that designs, implements, and reviews production-grade Databricks solutions following official best practices. Enforces Unity Catalog governance, Delta Medallion architecture, DLT expectations, Predictive Optimization, automatic liquid clustering, UC Metric Views, Genie TVFs, Serverless Workflows, and Asset Bundles. Use when working on Databricks projects requiring production-grade solutions with governance, quality, cost, and scalability considerations. Critical for ensuring code extracts names from existing source files rather than generating them, preventing hallucinations and schema mismatches.
- ▌ Databricks Asset Bundles · databricks-solutions bundleStandard patterns for Databricks Asset Bundles configuration files for serverless jobs, DLT pipelines, dashboards, alerts, apps, and workflows. Use when creating, configuring, or deploying DABs for infrastructure-as-code deployments. Covers mandatory serverless environment configuration, hierarchical job architecture (atomic/composite/orchestrator), DLT pipeline patterns, dashboard resources with dataset_catalog/dataset_schema, SQL Alerts v2 API schema, Apps lifecycle, Python notebook parameter passing (dbutils.widgets.get vs argparse), deployment error prevention, and pre-deployment validation.
- ▌ 00 Course Orchestrator · databricks-solutions bundleGenAI agent skill navigator for the `genai-agents/` folder. Start here to route agent-building, MLflow GenAI, tracing, Databricks Apps Track A, SDLC, evaluation, deployment, monitoring, prompt iteration, and vibecoding runtime tasks to the correct specialized skill. Uses tiered loading and keyword routing so agents read only the skills needed for the current task.
- ▌ 02 Appkit Build · databricks-solutions bundleBuild full-stack UI and backend features on a Databricks AppKit project from a PRD or feature spec. Covers SQL query design, type generation, React frontend with AppKit UI components, backend plugin wiring, and distinctive visual design. Use when asked to implement a UI, build features from a PRD, create pages or dashboards, add components, develop the frontend/backend of an existing AppKit app, build with mock data, or use static data for visual prototyping. Triggers on "build UI", "implement PRD", "create dashboard", "add page", "build features", "implement design", "create components", "build app from PRD", "develop frontend", "mock data", "static data", "two-phase data".
- ▌
- ▌
- ▌ 03 Appkit Deploy · databricks-solutions bundleDeploy a Databricks AppKit application to Databricks Apps. Covers config validation, build verification, deployment, UI verification, error diagnosis with fix loop, and workspace app limit handling. Use when asked to deploy an AppKit app, push to production, ship the app, or troubleshoot a failed deploy. Triggers on "deploy app", "push to databricks", "ship app", "deploy appkit", "databricks apps deploy", "fix deploy error", "app won't start".
- ▌ 04 AI Gateway · databricks-solutions bundleUse when you need central governance, observability, guardrails, or rate limits across LLM endpoints and MCP servers used by agents. Covers creating and configuring Databricks AI Gateway endpoints, enabling usage tracking, inference tables (UC Delta audit), rate limits, and guardrails, plus how deployed agent serving endpoints and Apps consume a gateway URL. Foundation Step 4 — ideal as a governance layer BEFORE Module 7 production deployment. Complements in-code guardrails and MLflow tracing; does NOT replace MLflow trace tables but adds a provider-level audit trail.
- ▌ 01 Appkit Scaffold · databricks-solutions bundleScaffold new Databricks AppKit applications using the Databricks CLI and Agent Skills. Creates blank or plugin-enabled AppKit projects (Lakebase, Analytics, Genie, Files). Use when asked to create a Databricks app, scaffold an AppKit project, bootstrap a new app, or set up a full-stack TypeScript Databricks application. Triggers on "create app", "new app", "scaffold", "AppKit", "databricks app", "blank app", "bootstrap app", "init app". To add a plugin to an existing app, use the 04-appkit-plugin-add skill instead.
- ▌ 08 Appkit Feedback · databricks-solutions bundleAdd user feedback (thumbs up/down) to an AppKit chat application, linked to MLflow assessments via the Databricks Assessments REST API. Covers the Vote table, feedback API routes (with AppKit-native auth via `getExecutionContext().client.config.authenticate()`), MLflow trace integration, and feedback UI components. Use when asked to add feedback, thumbs up/down, ratings, or link user judgments to MLflow traces. Triggers on "feedback", "thumbs up", "thumbs down", "rate response", "MLflow assessment", "user rating", "vote on message".
- ▌
- ▌
- ▌ 00 Appkit Navigator · databricks-solutions bundleEntry-point navigator for Databricks AppKit + Lakebase development. Routes tasks to the correct skill based on keyword detection: scaffolding, plugin integration, feature building, or deployment. This is a routing skill -- it does not generate code. It directs the agent to the correct specialized skill. Use this skill as the starting point for any AppKit-related task. Triggers on "AppKit", "Lakebase app", "Databricks app", "build app", "deploy app", "scaffold app", "add plugin", "AppKit project", "agent endpoint", "wire agent", "serving plugin", "agent app proxy", "chat history", "feedback".
- ▌
- ▌ 08 Prompt Optimization · databricks-solutions bundlePrompt Optimization (MLflow `optimize_prompts` + GEPA)
- ▌ 04 Appkit Plugin Add · databricks-solutions bundleAdd plugins to an existing Databricks AppKit project. Covers Lakebase (PostgreSQL), Analytics (SQL queries + dashboards), Genie (natural language AI/BI), Files (UC Volumes), and Serving (Model Serving / Agent endpoints). Guides through plugin registration, environment variables, app.yaml configuration, and frontend integration. Use when asked to add a plugin to an existing app, integrate Lakebase, add analytics, connect a Genie space, enable file uploads, add a serving endpoint, or extend an AppKit project with new capabilities. Triggers on "add plugin", "add lakebase", "add analytics", "add genie", "add files plugin", "add serving", "add agent endpoint", "integrate postgres", "add database", "add dashboards", "add file browser", "extend app", "connect genie space", "model serving plugin". Not for creating a Lakebase project or managing Lakebase via CLI -- use the databricks-lakebase agent skill for that.
- ▌ 04b Stakeholder Signoff · databricks-solutionsUse after evaluation (Step 4) passes but BEFORE deployment (Step 6). Covers translating technical eval metrics to business-meaningful terms, running a structured review with business and compliance stakeholders, capturing sign-off, and blocking promotion if review-blocking issues remain. Mirrors the "Align with stakeholders before production" phase of the Databricks agents development workflow. SDLC Step 4b. Do NOT use this skill in place of Step 4 eval — this is the gate layered on top of it.
- ▌
- ▌ 08b Prompt Handauthoring · databricks-solutionsDefault reusable MLflow workshop path for instruction-shaped prompt iteration. Hand-authored prompt revisions guided by failing-scorer rationales from the first scored eval, with full-dataset re-eval and alias-gated promotion. Use when Step 4 evaluation reveals specific prompt-related failure modes (guideline misses, missing structure, wrong tone) AND failure_shape is `instruction` AND there are no L1 scorer failures. Cheaper, more transparent, and lower-risk than automated optimization. SDLC Step 8b — runs AFTER Step 4. Do NOT use for L1 scorer failures (route to architecture review) or when the quality gap is tool/retrieval-shaped.
- ▌ 07 Appkit Chat History · databricks-solutions bundleAdd persistent chat history to an AppKit application using the Lakebase plugin and the Serving plugin (both AppKit-native, no Vercel AI SDK). Covers database schema, session auth middleware, a server-side proxy route that streams from `AppKit.serving("agent").asUser(req).stream()` while persisting assistant messages and capturing MLflow `trace_id`, the history API routes, a history sidebar, and ephemeral-mode fallback. Use when asked to persist conversations, add chat history, save messages, or build a conversation sidebar. Triggers on "chat history", "save conversations", "persistent chat", "conversation sidebar", "message storage", "save messages", "ephemeral chat".
- ▌ 06 Appkit Serving Wiring · databricks-solutions bundleWire a Databricks Model Serving or Agent endpoint into an existing AppKit project using the Serving plugin. Covers endpoint registration, app.yaml resource binding, streaming and invoke React hooks, conversation state management, agent response mapping, and server-side proxy patterns. PRD-independent patterns that apply to any AppKit + Serving app. Use after registering the Serving plugin via 04-appkit-plugin-add. Triggers on "wire agent", "agent endpoint", "serving plugin", "agent UI", "chat interface", "connect agent", "model serving", "useServingStream", "useServingInvoke", "agent chat", "wire serving", "serving wiring".
- ▌ 05 Appkit Lakebase Wiring · databricks-solutions bundleWire a Lakebase PostgreSQL backend into an existing AppKit project. Covers database schema design from a PRD, idempotent DDL, Express API routes with mock fallback, React data hooks, and local testing. PRD-independent patterns that apply to any AppKit + Lakebase app. Use after registering the Lakebase plugin via 04-appkit-plugin-add. Triggers on "wire lakebase", "lakebase backend", "CRUD API", "lakebase tables", "DDL", "database schema design", "useLakebaseData", "mock fallback", "ConnectionStatus", "replace mock data", "connect frontend to backend", "API-backed data", "replace static data with database".
- ▌ 02b Typescript Tracing · databricks-solutionsUse when adding MLflow tracing to a Node/TypeScript GenAI agent. Covers the official `mlflow-tracing` and `mlflow-openai` npm packages: `mlflow.init()`, `tracedOpenAI`, manual `mlflow.trace`/`@mlflow.trace`/`mlflow.withSpan`, nested spans, sessions/users, and when to choose a custom OTLP fallback. Foundation Step 2b. Sibling to the Python tracing skill (Foundation Step 2). Consumes the MLflow environment from Step 1.
- ▌ 05 Knowledge Assistant · databricks-solutions bundleUse when creating, syncing, or managing a Databricks Agent Bricks Knowledge Assistant (KA) as a document Q&A source for an agent. Covers source mode selection (UC Files vs Vector Search index), Python SDK lifecycle (create/update/sync/delete), REST API equivalents, readiness polling, and emitting the ka_endpoint_name handoff that downstream agent skills wire as a knowledge_assistant tool. Track-neutral foundation skill. Optional but recommended whenever document Q&A is part of the agent.
- ▌
- ▌ 06d Appkit Agent App Proxy · databricks-solutions bundleWire an AppKit frontend (Databricks App) to a separate Agent App backend (Databricks App hosting mlflow.genai.agent_server or ResponsesAgent) with OBO token forwarding via x-forwarded-access-token, app-to-app service principal auth, dual-format SSE streaming, and the /api/chat contract. Use this skill when the agent is deployed as its own Databricks App — not as a Model Serving endpoint. Triggers on "agent app backend", "two apps", "appkit plus agent app", "proxy agent app", "agent-backend resource", "x-forwarded-access-token", "migrate agent to apps", "agent app proxy".
- ▌ Skill Navigator · databricks-solutions bundleIntelligent skill navigation system with tiered loading and orchestrator-first routing for context-efficient agent operation. Routes tasks to the correct domain skill based on keyword detection with orchestrator priority. Each skill uses progressive disclosure with references/, scripts/, and assets/ directories. Use this skill as the entry point for any Databricks-related task to determine which specialized skills to load.
- ▌ 03 Tools And Data Access · databricks-solutions bundleUse when connecting any agent to data and external capabilities. Covers Databricks managed MCP servers (Vector Search, Genie, SQL, UC Functions), the databricks-mcp client library, _meta parameters, external MCP servers, code interpreter, non-MCP connections, resource grants, custom retriever schemas, tool cookbook (web search, SQL, file generation, HTTP), runtime guardrails, and end-to-end external MCP examples. Foundation Step 3. Consumed by all agent tracks (A, B, C).
- ▌ 00 Uc Resources Foundation · databricks-solutionsUse when any agent or downstream skill needs Unity Catalog schemas and managed volumes (knowledge_sources, agent_outputs, memory tables, benchmark tables, etc.) provisioned idempotently. Foundation Step 0 — runs before MLflow tracing setup and before any track-specific skill. Owned by no track. Reads the resolved spec to discover which volumes a use case needs and creates them alongside the canonical agent + ops UC schemas.
- ▌ 01 Mlflow Genai Foundation · databricks-solutions bundleUse when starting any new GenAI agent project on Databricks. Covers MLflow 3.x GenAI setup: install dependencies, enable autolog, configure connection pooling, understand model signatures, detect runtime environment. Foundation Step 1. Produces the configured MLflow environment consumed by all subsequent skills.
- ▌ Foundation Model Agent Loop · databricks-solutionsCanonical pattern for building an agentic tool-calling loop using a Databricks-hosted Foundation Model. Use when creating agent_loop.py or any backend that calls a Foundation Model with function-calling to orchestrate tools.
- ▌ 00b Agent Spec And Tool Plan · databricks-solutions bundleUse when deriving a Databricks Agent Spec and Agent Tool Plan from docs/design_prd.md before building Track A agents. Covers web-researched MCP recommendations, Databricks managed MCP choices, dynamic SQL MCP catalog/schema inputs, optional Knowledge Assistant selection, and validation of docs/agent_spec.yaml plus docs/agent_tool_plan.yaml.
- ▌ 08 Debugging · databricks-solutionsUse when a deployed Databricks Apps agent is failing, returning errors, or behaving unexpectedly. Covers systematic debugging of local dev, bundle configuration, deployment, runtime errors, authentication, resource permissions, and Lakebase memory. Track A Step 8. Consumes a deployed app from Step 7. Produces a resolved, healthy agent deployment.
- ▌ 05 Logged Model And Uc Registration · databricks-solutions bundleLogged Model and Unity Catalog Registration
- ▌ 06 Evaluation · databricks-solutionsUse when evaluating an agent's response quality and safety before deployment. Covers running agent-evaluate, evaluation dataset format, built-in judges (relevance, groundedness, safety), interpreting results, and customizing eval datasets. Track A Step 6. Consumes a working agent with tools from Steps 1-5. Produces evaluation results and confidence to deploy.
- ▌ Self Improvement · databricks-solutions bundleEnables agent self-reflection and learning from mistakes through systematic skill updates. Prioritizes updating existing skills over creating new ones - always searches existing skills first, creates new skills only when justified. Includes upstream source sync workflow for tracking and updating skills from databricks/databricks-agent-skills. Use after encountering errors, completing complex tasks, or when asked to reflect, learn, or document a mistake. Triggers on "learn from this", "don't repeat", "remember this pattern", "what went wrong", "update skills", "capture this learning", "document mistake", "prevent this error", "databricks-agent-skills updated", "upstream changed", "sync with databricks-agent-skills".
- ▌ Dqx Patterns · databricks-solutions bundleDatabricks DQX framework patterns for advanced data quality validation with detailed failure insights and flexible quarantine strategies. Use when implementing Silver/Gold layer validation, needing richer diagnostics than DLT expectations, or requiring pre-merge validation with detailed failure tracking. Supports YAML configuration, Delta table storage, and serverless compute compatibility.
- ▌ Ml Pipeline Setup · databricks-solutions bundleMLflow and ML Model patterns for Databricks including experiment creation, model training, batch inference, and Unity Catalog integration. Use when implementing ML pipelines, training models with Feature Store, or deploying batch inference jobs. Includes 19 non-negotiable rules covering experiment paths, dataset logging, UC model registration, NaN handling, label binarization, feature engineering workflows, and signature-driven preprocessing.
- ▌ Create Agent Skill · databricks-solutions bundleGuides the creation of new Agent Skills following the AgentSkills.io specification. Covers required frontmatter (name, description), optional metadata, directory structure (SKILL.md, references/, scripts/, assets/), progressive disclosure principles, and naming conventions. Use when creating a new skill from scratch, adding project-specific agent knowledge, defining reusable patterns for AI coding assistants, or structuring domain expertise as a skill. Triggers on "create skill", "new skill", "add skill", "write a skill", "skill template", "SKILL.md", "agent skill".
- ▌ Gold Layer Setup · databricks-solutions bundleEnd-to-end orchestrator for implementing Gold layer tables, merge scripts, FK constraints, and Asset Bundle jobs from YAML schema definitions. Guides users through Silver contract validation, YAML-driven table creation, Silver-to-Gold MERGE operations (SCD Type 1/2 dimensions, aggregated/transaction facts, accumulating snapshots, factless facts, periodic snapshots, junk dimensions), foreign key constraint application, Asset Bundle job configuration, and post-deployment validation. Orchestrates pipeline-workers (01-yaml-table-setup, 02-merge-patterns, 03-deduplication, 04-grain-validation, 05-schema-validation) and common skills (databricks-asset-bundles, databricks-table-properties, databricks-python-imports, schema-management-patterns, unity-catalog-constraints, databricks-expert-agent). Use when implementing Gold layer from YAML designs, creating table setup scripts, writing merge scripts, deploying Gold layer jobs, or troubleshooting Gold layer implementation errors.
- ▌ 01 Clone And Run · databricks-solutions bundleUse when starting a new custom agent project on Databricks Apps. Clone the agent-openai-agents-sdk template, run quickstart, start the local dev server, verify the built-in chat UI. Track A Step 1. Produces a running local agent project ready for customization.
- ▌ 03 Tools And MCP · databricks-solutionsUse when wiring tools into an OpenAI Agents SDK agent: local Python function tools, DatabricksMCPClient connections, and MCPServerSse for MCP servers. Track A Step 3. Builds on shared concepts from F3 (Tools and Data Access).
- ▌ Gold Layer Design · databricks-solutions bundleEnd-to-end orchestrator for designing complete Gold layer schemas with ERDs, YAML files, lineage tracking, and comprehensive business documentation. Guides users through dimensional modeling, ERD creation (master/domain/summary based on table count), YAML schema generation, column-level lineage documentation, business onboarding guide creation, source table mapping, and design validation. Orchestrates design-workers (05-erd-diagrams, 06-table-documentation, 01-grain-definition, 07-design-validation, 02-dimension-patterns, 03-fact-table-patterns, 04-conformed-dimensions, 08-industry-alignment). Use when designing a Gold layer from scratch, creating dimensional models, documenting business processes, aligning to Databricks Industry Vibe Data Models, or preparing for Gold layer implementation.
- ▌ 04 Authentication · databricks-solutions bundleUse when configuring agent permissions, implementing OBO, or granting access to Databricks resources. Covers App Authorization (service principal), User Authorization (OBO), auth passthrough for Model Serving, and MCP server authentication. Track A Step 4.
- ▌ 02c Trace Context And Environments · databricks-solutionsUse when adding user, session, environment, or version context to MLflow traces — Python or TypeScript. Covers the reserved metadata fields (`mlflow.trace.user`, `mlflow.trace.session`), auto-populated environment metadata (`mlflow.source.*`, Git provenance, model id), the `APP_ENVIRONMENT` override pattern for production, custom deployment metadata, and how this composes with `client_request_id` and `gen_ai.*`. Foundation Step 2c. Sibling to F2 (Python tracing) and F2b (TS tracing).
- ▌ 02 Agent Framework · databricks-solutions bundleUse when building a custom agent with the OpenAI Agents SDK and MLflow ResponsesAgent. Covers Agent class, Runner, @function_tool, handoffs, streaming, ModelConfig, mlflow.models.set_model(), autolog, and manual tracing. Track A Step 2. Produces a customized, MLflow-compatible agent class ready for tools and deployment.
- ▌ 05 Lakebase Memory · databricks-solutions bundleUse when implementing stateful agents that need conversation memory. Covers short-term continuity via AsyncDatabricksSession (prescribed for OpenAI Agents SDK), long-term preferences via DatabricksStore, graceful degradation, and thread_id resolution. Track A Step 5. Consumes authenticated client from Step 4.
- ▌ Skill Freshness Audit · databricks-solutions bundleSystematic skill freshness auditing with verification anchors, volatility classification, staleness detection, and upstream lineage tracking across every Agent Skill in the repository (data_product_accelerator/, genai-agents/, apps_lakebase/). Fetches official documentation URLs embedded in skills, compares patterns against live docs, tracks lineage to databricks/databricks-agent-skills (the single upstream registry), and reports drift. Use when auditing skill currency, verifying skills against latest Databricks/MLflow docs, checking for stale skills, syncing with the upstream registry, or after a platform release. Triggers on "audit skills", "check freshness", "stale skills", "verify skills", "skill audit", "update check", "Databricks released", "new MLflow version", "upstream sync", "databricks-agent-skills lineage", "upstream lineage".
- ▌ 07 Deploy And Query · databricks-solutionsUse when deploying an agent to Databricks Apps and querying the deployed endpoint. Covers databricks apps create, sync, deploy, OAuth token auth, PAT limitations, curl and Python SDK query methods, and the redeploy workflow. Track A Step 7. Consumes evaluation results from Step 6. Produces a running deployed agent on Databricks Apps.
- ▌ Bronze Layer Setup · databricks-solutions bundleEnd-to-end Bronze layer creation for testing and demos. Creates table DDLs, generates fake data with Faker, copies from existing sources, and configures Asset Bundle jobs. Covers Unity Catalog compliance, Change Data Feed, automatic liquid clustering, and governance metadata. Use when setting up Bronze layer tables, creating test/demo data, rapid prototyping Medallion Architecture, or bootstrapping a new Databricks project. For Faker-specific patterns (corruption rates, function signatures, provider examples), load the faker-data-generation skill.
- ▌ Project Planning · databricks-solutions bundleCreate multi-phase project plans for Databricks data platform solutions with Agent Domain Framework and Agent Layer Architecture. Includes interactive Quick Start with key decisions, industry-specific domain patterns, complete phase document templates (Use Cases, Agents, Frontend), Genie Space integration patterns, deployment order requirements, and worked examples. Supports both acceleration mode (plan on a completed Gold layer) and workshop mode (`planning_mode: workshop`) that plans from the best available layer with hard artifact caps. Use when planning any Databricks solution post-Gold layer — observability, analytics, agent-based frameworks, or multi-artifact projects.
- ▌ Silver Layer Setup · databricks-solutions bundleEnd-to-end orchestrator for creating Silver layer pipelines using Spark Declarative Pipelines (SDP, formerly DLT) with Delta table-based data quality rules, quarantine patterns, and monitoring views. Orchestrates mandatory dependencies on common skills (databricks-table-properties, databricks-python-imports, databricks-asset-bundles, schema-management-patterns, unity-catalog-constraints, databricks-expert-agent) and Silver-domain skills (dlt-expectations-patterns, dqx-patterns). Use when creating a Silver layer from scratch, setting up Bronze-to-Silver pipelines, or implementing Silver SDP/DLT with streaming ingestion and runtime-updateable DQ rules.
- ▌ 02 Experiment Tracing And Uc Storage · databricks-solutions bundleUse when setting up MLflow experiments, tracing, or UC OTEL trace storage for a GenAI agent. Covers structured experiment paths, tracing decorators, manual spans, tags, connection pooling, and Unity Catalog OTEL storage for SQL-queryable trace retention. Foundation Step 2. Consumes MLflow environment from Step 1.
- ▌ Faker Data Generation · databricks-solutions bundleGenerate synthetic data with Faker for Bronze layer testing with configurable data corruption. Use when creating test data for data quality validation, testing DLT expectations, or simulating production-like datasets. Supports realistic data generation with intentional corruption patterns mapped to specific DQ expectations.
- ▌ Naming Tagging Standards · databricks-solutions bundleEnforces enterprise naming conventions (snake_case, table prefixes, approved abbreviations), dual-purpose COMMENT formats for tables/columns/TVFs/metric views/dashboards/Genie Spaces, and config-aware tagging standards. Scans context/ for customer tagging standards in any format (YAML, CSV, Markdown, JSON, TXT); derives meaningful smart defaults when none supplied. Uses Databricks Data Classification class.* system governed tags for PII (always inferred from column names + customer declarations). Ensures tag consistency across all project assets. Triggers on "naming", "comment", "COMMENT", "schema COMMENT", "tag", "PII", "cost_center", "snake_case", "dim_", "fact_", "governed tag", "budget policy", "class.*", "data classification", "dashboard", "metric view", "Genie Space".
- ▌ Anomaly Detection · databricks-solutions bundleSchema-level anomaly detection for Databricks Unity Catalog using the Data Quality API (Public Preview). Automatically monitors table freshness and completeness using ML models. Use when setting up schema-wide data reliability monitoring, detecting stale or incomplete tables, configuring anomaly detection alerts, or querying the system results table. **Auto-triggered by Silver and Gold layer setup workflows** to ensure every new schema has baseline freshness/completeness monitoring from day one.
- ▌ Documentation Organization · databricks-solutions bundleEnforces documentation organization standards and provides comprehensive framework documentation authoring for Databricks projects. Operates in two modes: (1) Organizational Enforcement — validates file placement (root directory rules, docs/ hierarchy), naming conventions (kebab-case, date-prefixed), and proactively suggests cleanup for misorganized files. (2) Framework Documentation — orchestrates creation of complete technical documentation sets using fill-in-the-blank templates (index, introduction, architecture, component deep dives, implementation guides, operations guides, troubleshooting guides) with requirements gathering, quality checklists, and worked examples. Use when creating any .md file, organizing existing documentation, maintaining project structure, or creating comprehensive framework documentation sets.
- ▌ Databricks Python Imports · databricks-solutions bundlePatterns for sharing code between Databricks notebooks using pure Python files and standard imports. Covers Asset Bundle path setup (rsplit canonical pattern), notebook-to-module conversion, import patterns vs %run magic commands, job submission context vs notebook context, MLflow model packaging path requirements, and troubleshooting ModuleNotFoundError. Use when creating shared modules, deploying jobs that import local code, or packaging MLflow models with local dependencies.
- ▌ Unity Catalog Constraints · databricks-solutions bundleUnity Catalog Primary Key and Foreign Key constraint patterns for proper relational modeling in Databricks. Use when implementing star schema dimensional models with PK/FK relationships in Gold layer tables. Covers surrogate keys as PRIMARY KEYS (not business keys), facts referencing surrogate PKs via FOREIGN KEY constraints, NOT NULL requirements for PK columns, proper dimensional modeling patterns (SCD Type 1/2, date dimensions), production deployment error prevention (never define FK inline in CREATE TABLE, DATE type casting from DATE_TRUNC, avoid module imports, widget parameter naming consistency), and validation checklists. Critical for ensuring proper relational modeling and preventing constraint application errors during Gold layer deployment.
- ▌ Schema Management Patterns · databricks-solutions bundleSchema Management Patterns for Databricks Asset Bundles
- ▌ Observability Setup · databricks-solutionsEnd-to-end orchestrator for setting up Databricks observability including Lakehouse Monitoring, Anomaly Detection, AI/BI Dashboards, and SQL Alerts. Guides users through monitor creation for Gold tables, schema-level anomaly detection for freshness/completeness, dashboard design with monitoring widgets, and config-driven alerting. Orchestrates mandatory dependencies on monitoring skills (lakehouse-monitoring-comprehensive, anomaly-detection, databricks-aibi-dashboards, sql-alerting-patterns) and common skills (databricks-asset-bundles, databricks-expert-agent, databricks-python-imports). Use when setting up observability end-to-end, creating Lakehouse Monitors, enabling anomaly detection, building dashboards, or configuring SQL alerts.
- ▌ Databricks Table Properties · databricks-solutions bundleProvides standard TBLPROPERTIES and metadata patterns for Unity Catalog Delta tables across Bronze, Silver, and Gold medallion layers. Ensures governance compliance, performance optimization, and proper metadata tagging for all table creation operations. Covers required TBLPROPERTIES by layer (Bronze, Silver DLT, Gold), mandatory CLUSTER BY AUTO configuration, Change Data Feed (CDF) enablement, auto-optimize settings, table and column comment patterns (LLM-friendly for Bronze/Silver, dual-purpose for Gold), domain values, data classification tags, and validation checklists. Use when creating Delta tables, configuring table properties, enabling CDF, setting up auto-optimize, or ensuring governance metadata consistency. Critical for preventing missing properties, incorrect clustering configurations, and governance compliance issues.
- ▌ 05 Erd Diagrams · databricks-solutions bundlePatterns for creating clean, professional Mermaid ERD diagrams for data modeling documentation. Use when documenting Gold layer data models, creating master ERDs for complete models, domain-specific ERDs for focused views, or summary ERDs for large models (20+ tables). Includes organization strategies, syntax standards, relationship patterns, and cross-domain reference patterns.
- ▌ SQL Alerting Patterns · databricks-solutions bundleComprehensive guide for Databricks SQL Alerts V2 - config-driven alerting framework with SDK deployment, hierarchical job architecture (5 atomic + 1 composite), proactive EXPLAIN-based query validation, and partial success patterns. Use when setting up SQL alerts, creating alert configuration tables, deploying alerts via Databricks SDK (V2 dict-based or typed classes), or troubleshooting alert failures. Includes config-driven patterns, fully qualified table names (no parameters), severity-based routing, alert ID conventions, SQL query patterns (threshold, percentage change, anomaly detection), DataFrame-based config seeding, DAB job configuration, custom notification templates, Quartz cron schedules, and troubleshooting patterns.
- ▌ Dlt Expectations Patterns · databricks-solutions bundleSpark Declarative Pipeline (SDP, formerly DLT) expectations patterns for data quality with Unity Catalog Delta table storage. Use when implementing Silver layer SDP/DLT pipelines, creating portable data quality rules, or needing runtime-updateable expectations without code deployment. Supports severity-based filtering (critical vs warning) and quarantine patterns. Standardizes on `import dlt` for the DQ-rules framework; the modern `dp` API (`from pyspark import pipelines as dp`) also supports expectations (`dp.expect_all_or_drop`) and is Databricks' recommended forward path.
- ▌ Simple Agent Scaffold · databricks-solutions bundleScaffold a minimal MCP tool-calling agent with Genie Spaces and deploy it to Databricks Model Serving in 5 steps, following the canonical OpenAI MCP Tool Calling Agent notebook verbatim. Produces a working endpoint testable in AI Playground and consumable by 06-appkit-serving-wiring. No evaluation, memory, or prompt registry — add those later via the existing worker skills. Use when creating a simple agent, scaffolding a new agent, building a quick tool-calling agent, or connecting an agent to Genie Spaces. Triggers on "simple agent", "scaffold agent", "MCP agent", "quick agent", "create agent", "tool calling agent", "Genie agent", "basic agent".