all publishers

databricks-solutions

@databricks-solutions source repo

266 published skills · page 3 of 3

  1. ▌
    00 Appkit Navigator · databricks-solutions bundle
    Entry-point navigator for Databricks AppKit + Lakebase development. Routes tasks to the correct skill based on keyword detection: scaffolding, plugin integration, feature building, or deployment. This is a routing skill -- it does not generate code. It directs the agent to the correct specialized skill. Use this skill as the starting point for any AppKit-related task. Triggers on "AppKit", "Lakebase app", "Databricks app", "build app", "deploy app", "scaffold app", "add plugin", "AppKit project", "agent endpoint", "wire agent", "serving plugin", "agent app proxy", "chat history", "feedback".
    6 repo stars
  2. ▌
    02 Evaluation Datasets · databricks-solutions bundle
    Evaluation Dataset Creation
    6 repo stars
  3. ▌
    08 Prompt Optimization · databricks-solutions bundle
    Prompt Optimization (MLflow `optimize_prompts` + GEPA)
    6 repo stars
  4. ▌
    04 Appkit Plugin Add · databricks-solutions bundle
    Add plugins to an existing Databricks AppKit project. Covers Lakebase (PostgreSQL), Analytics (SQL queries + dashboards), Genie (natural language AI/BI), Files (UC Volumes), and Serving (Model Serving / Agent endpoints). Guides through plugin registration, environment variables, app.yaml configuration, and frontend integration. Use when asked to add a plugin to an existing app, integrate Lakebase, add analytics, connect a Genie space, enable file uploads, add a serving endpoint, or extend an AppKit project with new capabilities. Triggers on "add plugin", "add lakebase", "add analytics", "add genie", "add files plugin", "add serving", "add agent endpoint", "integrate postgres", "add database", "add dashboards", "add file browser", "extend app", "connect genie space", "model serving plugin". Not for creating a Lakebase project or managing Lakebase via CLI -- use the databricks-lakebase agent skill for that.
    6 repo stars
  5. ▌
    04b Stakeholder Signoff · databricks-solutions
    Use after evaluation (Step 4) passes but BEFORE deployment (Step 6). Covers translating technical eval metrics to business-meaningful terms, running a structured review with business and compliance stakeholders, capturing sign-off, and blocking promotion if review-blocking issues remain. Mirrors the "Align with stakeholders before production" phase of the Databricks agents development workflow. SDLC Step 4b. Do NOT use this skill in place of Step 4 eval — this is the gate layered on top of it.
    6 repo stars
  6. ▌
    07 Production Monitoring · databricks-solutions bundle
    Production Monitoring (MLflow 3.10+)
    6 repo stars
  7. ▌
    08b Prompt Handauthoring · databricks-solutions
    Default reusable MLflow workshop path for instruction-shaped prompt iteration. Hand-authored prompt revisions guided by failing-scorer rationales from the first scored eval, with full-dataset re-eval and alias-gated promotion. Use when Step 4 evaluation reveals specific prompt-related failure modes (guideline misses, missing structure, wrong tone) AND failure_shape is `instruction` AND there are no L1 scorer failures. Cheaper, more transparent, and lower-risk than automated optimization. SDLC Step 8b — runs AFTER Step 4. Do NOT use for L1 scorer failures (route to architecture review) or when the quality gap is tool/retrieval-shaped.
    6 repo stars
  8. ▌
    07 Appkit Chat History · databricks-solutions bundle
    Add persistent chat history to an AppKit application using the Lakebase plugin and the Serving plugin (both AppKit-native, no Vercel AI SDK). Covers database schema, session auth middleware, a server-side proxy route that streams from `AppKit.serving("agent").asUser(req).stream()` while persisting assistant messages and capturing MLflow `trace_id`, the history API routes, a history sidebar, and ephemeral-mode fallback. Use when asked to persist conversations, add chat history, save messages, or build a conversation sidebar. Triggers on "chat history", "save conversations", "persistent chat", "conversation sidebar", "message storage", "save messages", "ephemeral chat".
    6 repo stars
  9. ▌
    06 Appkit Serving Wiring · databricks-solutions bundle
    Wire a Databricks Model Serving or Agent endpoint into an existing AppKit project using the Serving plugin. Covers endpoint registration, app.yaml resource binding, streaming and invoke React hooks, conversation state management, agent response mapping, and server-side proxy patterns. PRD-independent patterns that apply to any AppKit + Serving app. Use after registering the Serving plugin via 04-appkit-plugin-add. Triggers on "wire agent", "agent endpoint", "serving plugin", "agent UI", "chat interface", "connect agent", "model serving", "useServingStream", "useServingInvoke", "agent chat", "wire serving", "serving wiring".
    6 repo stars
  10. ▌
    02b Typescript Tracing · databricks-solutions
    Use when adding MLflow tracing to a Node/TypeScript GenAI agent. Covers the official `mlflow-tracing` and `mlflow-openai` npm packages: `mlflow.init()`, `tracedOpenAI`, manual `mlflow.trace`/`@mlflow.trace`/`mlflow.withSpan`, nested spans, sessions/users, and when to choose a custom OTLP fallback. Foundation Step 2b. Sibling to the Python tracing skill (Foundation Step 2). Consumes the MLflow environment from Step 1.
    6 repo stars
  11. ▌
    05 Knowledge Assistant · databricks-solutions bundle
    Use when creating, syncing, or managing a Databricks Agent Bricks Knowledge Assistant (KA) as a document Q&A source for an agent. Covers source mode selection (UC Files vs Vector Search index), Python SDK lifecycle (create/update/sync/delete), REST API equivalents, readiness polling, and emitting the ka_endpoint_name handoff that downstream agent skills wire as a knowledge_assistant tool. Track-neutral foundation skill. Optional but recommended whenever document Q&A is part of the agent.
    6 repo stars
  12. ▌
    06 Deployment And Automation · databricks-solutions bundle
    Deployment and Automation
    6 repo stars
  13. ▌
    06d Appkit Agent App Proxy · databricks-solutions bundle
    Wire an AppKit frontend (Databricks App) to a separate Agent App backend (Databricks App hosting mlflow.genai.agent_server or ResponsesAgent) with OBO token forwarding via x-forwarded-access-token, app-to-app service principal auth, dual-format SSE streaming, and the /api/chat contract. Use this skill when the agent is deployed as its own Databricks App — not as a Model Serving endpoint. Triggers on "agent app backend", "two apps", "appkit plus agent app", "proxy agent app", "agent-backend resource", "x-forwarded-access-token", "migrate agent to apps", "agent app proxy".
    6 repo stars
  14. ▌
    Skill Navigator · databricks-solutions bundle
    Intelligent skill navigation system with tiered loading and orchestrator-first routing for context-efficient agent operation. Routes tasks to the correct domain skill based on keyword detection with orchestrator priority. Each skill uses progressive disclosure with references/, scripts/, and assets/ directories. Use this skill as the entry point for any Databricks-related task to determine which specialized skills to load.
    6 repo stars
  15. ▌
    03 Tools And Data Access · databricks-solutions bundle
    Use when connecting any agent to data and external capabilities. Covers Databricks managed MCP servers (Vector Search, Genie, SQL, UC Functions), the databricks-mcp client library, _meta parameters, external MCP servers, code interpreter, non-MCP connections, resource grants, custom retriever schemas, tool cookbook (web search, SQL, file generation, HTTP), runtime guardrails, and end-to-end external MCP examples. Foundation Step 3. Consumed by all agent tracks (A, B, C).
    6 repo stars
  16. ▌
    01 Mlflow Genai Foundation · databricks-solutions bundle
    Use when starting any new GenAI agent project on Databricks. Covers MLflow 3.x GenAI setup: install dependencies, enable autolog, configure connection pooling, understand model signatures, detect runtime environment. Foundation Step 1. Produces the configured MLflow environment consumed by all subsequent skills.
    6 repo stars
  17. ▌
    Foundation Model Agent Loop · databricks-solutions
    Canonical pattern for building an agentic tool-calling loop using a Databricks-hosted Foundation Model. Use when creating agent_loop.py or any backend that calls a Foundation Model with function-calling to orchestrate tools.
    6 repo stars
  18. ▌
    00b Agent Spec And Tool Plan · databricks-solutions bundle
    Use when deriving a Databricks Agent Spec and Agent Tool Plan from docs/design_prd.md before building Track A agents. Covers web-researched MCP recommendations, Databricks managed MCP choices, dynamic SQL MCP catalog/schema inputs, optional Knowledge Assistant selection, and validation of docs/agent_spec.yaml plus docs/agent_tool_plan.yaml.
    6 repo stars
  19. ▌
    08 Debugging · databricks-solutions
    Use when a deployed Databricks Apps agent is failing, returning errors, or behaving unexpectedly. Covers systematic debugging of local dev, bundle configuration, deployment, runtime errors, authentication, resource permissions, and Lakebase memory. Track A Step 8. Consumes a deployed app from Step 7. Produces a resolved, healthy agent deployment.
    6 repo stars
  20. ▌
    05 Logged Model And Uc Registration · databricks-solutions bundle
    Logged Model and Unity Catalog Registration
    6 repo stars
  21. ▌
    06 Evaluation · databricks-solutions
    Use when evaluating an agent's response quality and safety before deployment. Covers running agent-evaluate, evaluation dataset format, built-in judges (relevance, groundedness, safety), interpreting results, and customizing eval datasets. Track A Step 6. Consumes a working agent with tools from Steps 1-5. Produces evaluation results and confidence to deploy.
    6 repo stars
  22. ▌
    Self Improvement · databricks-solutions bundle
    Enables agent self-reflection and learning from mistakes through systematic skill updates. Prioritizes updating existing skills over creating new ones - always searches existing skills first, creates new skills only when justified. Includes upstream source sync workflow for tracking and updating skills from databricks/databricks-agent-skills. Use after encountering errors, completing complex tasks, or when asked to reflect, learn, or document a mistake. Triggers on "learn from this", "don't repeat", "remember this pattern", "what went wrong", "update skills", "capture this learning", "document mistake", "prevent this error", "databricks-agent-skills updated", "upstream changed", "sync with databricks-agent-skills".
    6 repo stars
  23. ▌
    Dqx Patterns · databricks-solutions bundle
    Databricks DQX framework patterns for advanced data quality validation with detailed failure insights and flexible quarantine strategies. Use when implementing Silver/Gold layer validation, needing richer diagnostics than DLT expectations, or requiring pre-merge validation with detailed failure tracking. Supports YAML configuration, Delta table storage, and serverless compute compatibility.
    6 repo stars
  24. ▌
    Ml Pipeline Setup · databricks-solutions bundle
    MLflow and ML Model patterns for Databricks including experiment creation, model training, batch inference, and Unity Catalog integration. Use when implementing ML pipelines, training models with Feature Store, or deploying batch inference jobs. Includes 19 non-negotiable rules covering experiment paths, dataset logging, UC model registration, NaN handling, label binarization, feature engineering workflows, and signature-driven preprocessing.
    6 repo stars
  25. ▌
    Create Agent Skill · databricks-solutions bundle
    Guides the creation of new Agent Skills following the AgentSkills.io specification. Covers required frontmatter (name, description), optional metadata, directory structure (SKILL.md, references/, scripts/, assets/), progressive disclosure principles, and naming conventions. Use when creating a new skill from scratch, adding project-specific agent knowledge, defining reusable patterns for AI coding assistants, or structuring domain expertise as a skill. Triggers on "create skill", "new skill", "add skill", "write a skill", "skill template", "SKILL.md", "agent skill".
    6 repo stars
  26. ▌
    Gold Layer Setup · databricks-solutions bundle
    End-to-end orchestrator for implementing Gold layer tables, merge scripts, FK constraints, and Asset Bundle jobs from YAML schema definitions. Guides users through Silver contract validation, YAML-driven table creation, Silver-to-Gold MERGE operations (SCD Type 1/2 dimensions, aggregated/transaction facts, accumulating snapshots, factless facts, periodic snapshots, junk dimensions), foreign key constraint application, Asset Bundle job configuration, and post-deployment validation. Orchestrates pipeline-workers (01-yaml-table-setup, 02-merge-patterns, 03-deduplication, 04-grain-validation, 05-schema-validation) and common skills (databricks-asset-bundles, databricks-table-properties, databricks-python-imports, schema-management-patterns, unity-catalog-constraints, databricks-expert-agent). Use when implementing Gold layer from YAML designs, creating table setup scripts, writing merge scripts, deploying Gold layer jobs, or troubleshooting Gold layer implementation errors.
    6 repo stars
  27. ▌
    01 Clone And Run · databricks-solutions bundle
    Use when starting a new custom agent project on Databricks Apps. Clone the agent-openai-agents-sdk template, run quickstart, start the local dev server, verify the built-in chat UI. Track A Step 1. Produces a running local agent project ready for customization.
    6 repo stars
  28. ▌
    03 Tools And MCP · databricks-solutions
    Use when wiring tools into an OpenAI Agents SDK agent: local Python function tools, DatabricksMCPClient connections, and MCPServerSse for MCP servers. Track A Step 3. Builds on shared concepts from F3 (Tools and Data Access).
    6 repo stars
  29. ▌
    04 Authentication · databricks-solutions bundle
    Use when configuring agent permissions, implementing OBO, or granting access to Databricks resources. Covers App Authorization (service principal), User Authorization (OBO), auth passthrough for Model Serving, and MCP server authentication. Track A Step 4.
    6 repo stars
  30. ▌
    02c Trace Context And Environments · databricks-solutions
    Use when adding user, session, environment, or version context to MLflow traces — Python or TypeScript. Covers the reserved metadata fields (`mlflow.trace.user`, `mlflow.trace.session`), auto-populated environment metadata (`mlflow.source.*`, Git provenance, model id), the `APP_ENVIRONMENT` override pattern for production, custom deployment metadata, and how this composes with `client_request_id` and `gen_ai.*`. Foundation Step 2c. Sibling to F2 (Python tracing) and F2b (TS tracing).
    6 repo stars
  31. ▌
    02 Agent Framework · databricks-solutions bundle
    Use when building a custom agent with the OpenAI Agents SDK and MLflow ResponsesAgent. Covers Agent class, Runner, @function_tool, handoffs, streaming, ModelConfig, mlflow.models.set_model(), autolog, and manual tracing. Track A Step 2. Produces a customized, MLflow-compatible agent class ready for tools and deployment.
    6 repo stars
  32. ▌
    05 Lakebase Memory · databricks-solutions bundle
    Use when implementing stateful agents that need conversation memory. Covers short-term continuity via AsyncDatabricksSession (prescribed for OpenAI Agents SDK), long-term preferences via DatabricksStore, graceful degradation, and thread_id resolution. Track A Step 5. Consumes authenticated client from Step 4.
    6 repo stars
  33. ▌
    Skill Freshness Audit · databricks-solutions bundle
    Systematic skill freshness auditing with verification anchors, volatility classification, staleness detection, and upstream lineage tracking across every Agent Skill in the repository (data_product_accelerator/, genai-agents/, apps_lakebase/). Fetches official documentation URLs embedded in skills, compares patterns against live docs, tracks lineage to databricks/databricks-agent-skills (the single upstream registry), and reports drift. Use when auditing skill currency, verifying skills against latest Databricks/MLflow docs, checking for stale skills, syncing with the upstream registry, or after a platform release. Triggers on "audit skills", "check freshness", "stale skills", "verify skills", "skill audit", "update check", "Databricks released", "new MLflow version", "upstream sync", "databricks-agent-skills lineage", "upstream lineage".
    6 repo stars
  34. ▌
    07 Deploy And Query · databricks-solutions
    Use when deploying an agent to Databricks Apps and querying the deployed endpoint. Covers databricks apps create, sync, deploy, OAuth token auth, PAT limitations, curl and Python SDK query methods, and the redeploy workflow. Track A Step 7. Consumes evaluation results from Step 6. Produces a running deployed agent on Databricks Apps.
    6 repo stars
  35. ▌
    Bronze Layer Setup · databricks-solutions bundle
    End-to-end Bronze layer creation for testing and demos. Creates table DDLs, generates fake data with Faker, copies from existing sources, and configures Asset Bundle jobs. Covers Unity Catalog compliance, Change Data Feed, automatic liquid clustering, and governance metadata. Use when setting up Bronze layer tables, creating test/demo data, rapid prototyping Medallion Architecture, or bootstrapping a new Databricks project. For Faker-specific patterns (corruption rates, function signatures, provider examples), load the faker-data-generation skill.
    6 repo stars
  36. ▌
    Project Planning · databricks-solutions bundle
    Create multi-phase project plans for Databricks data platform solutions with Agent Domain Framework and Agent Layer Architecture. Includes interactive Quick Start with key decisions, industry-specific domain patterns, complete phase document templates (Use Cases, Agents, Frontend), Genie Space integration patterns, deployment order requirements, and worked examples. Supports both acceleration mode (plan on a completed Gold layer) and workshop mode (`planning_mode: workshop`) that plans from the best available layer with hard artifact caps. Use when planning any Databricks solution post-Gold layer — observability, analytics, agent-based frameworks, or multi-artifact projects.
    6 repo stars
  37. ▌
    Silver Layer Setup · databricks-solutions bundle
    End-to-end orchestrator for creating Silver layer pipelines using Spark Declarative Pipelines (SDP, formerly DLT) with Delta table-based data quality rules, quarantine patterns, and monitoring views. Orchestrates mandatory dependencies on common skills (databricks-table-properties, databricks-python-imports, databricks-asset-bundles, schema-management-patterns, unity-catalog-constraints, databricks-expert-agent) and Silver-domain skills (dlt-expectations-patterns, dqx-patterns). Use when creating a Silver layer from scratch, setting up Bronze-to-Silver pipelines, or implementing Silver SDP/DLT with streaming ingestion and runtime-updateable DQ rules.
    6 repo stars
  38. ▌
    02 Experiment Tracing And Uc Storage · databricks-solutions bundle
    Use when setting up MLflow experiments, tracing, or UC OTEL trace storage for a GenAI agent. Covers structured experiment paths, tracing decorators, manual spans, tags, connection pooling, and Unity Catalog OTEL storage for SQL-queryable trace retention. Foundation Step 2. Consumes MLflow environment from Step 1.
    6 repo stars
  39. ▌
    Faker Data Generation · databricks-solutions bundle
    Generate synthetic data with Faker for Bronze layer testing with configurable data corruption. Use when creating test data for data quality validation, testing DLT expectations, or simulating production-like datasets. Supports realistic data generation with intentional corruption patterns mapped to specific DQ expectations.
    6 repo stars
  40. ▌
    Naming Tagging Standards · databricks-solutions bundle
    Enforces enterprise naming conventions (snake_case, table prefixes, approved abbreviations), dual-purpose COMMENT formats for tables/columns/TVFs/metric views/dashboards/Genie Spaces, and config-aware tagging standards. Scans context/ for customer tagging standards in any format (YAML, CSV, Markdown, JSON, TXT); derives meaningful smart defaults when none supplied. Uses Databricks Data Classification class.* system governed tags for PII (always inferred from column names + customer declarations). Ensures tag consistency across all project assets. Triggers on "naming", "comment", "COMMENT", "schema COMMENT", "tag", "PII", "cost_center", "snake_case", "dim_", "fact_", "governed tag", "budget policy", "class.*", "data classification", "dashboard", "metric view", "Genie Space".
    6 repo stars
  41. ▌
    Anomaly Detection · databricks-solutions bundle
    Schema-level anomaly detection for Databricks Unity Catalog using the Data Quality API (Public Preview). Automatically monitors table freshness and completeness using ML models. Use when setting up schema-wide data reliability monitoring, detecting stale or incomplete tables, configuring anomaly detection alerts, or querying the system results table. **Auto-triggered by Silver and Gold layer setup workflows** to ensure every new schema has baseline freshness/completeness monitoring from day one.
    6 repo stars
  42. ▌
    Documentation Organization · databricks-solutions bundle
    Enforces documentation organization standards and provides comprehensive framework documentation authoring for Databricks projects. Operates in two modes: (1) Organizational Enforcement — validates file placement (root directory rules, docs/ hierarchy), naming conventions (kebab-case, date-prefixed), and proactively suggests cleanup for misorganized files. (2) Framework Documentation — orchestrates creation of complete technical documentation sets using fill-in-the-blank templates (index, introduction, architecture, component deep dives, implementation guides, operations guides, troubleshooting guides) with requirements gathering, quality checklists, and worked examples. Use when creating any .md file, organizing existing documentation, maintaining project structure, or creating comprehensive framework documentation sets.
    6 repo stars
  43. ▌
    Databricks Python Imports · databricks-solutions bundle
    Patterns for sharing code between Databricks notebooks using pure Python files and standard imports. Covers Asset Bundle path setup (rsplit canonical pattern), notebook-to-module conversion, import patterns vs %run magic commands, job submission context vs notebook context, MLflow model packaging path requirements, and troubleshooting ModuleNotFoundError. Use when creating shared modules, deploying jobs that import local code, or packaging MLflow models with local dependencies.
    6 repo stars
  44. ▌
    Unity Catalog Constraints · databricks-solutions bundle
    Unity Catalog Primary Key and Foreign Key constraint patterns for proper relational modeling in Databricks. Use when implementing star schema dimensional models with PK/FK relationships in Gold layer tables. Covers surrogate keys as PRIMARY KEYS (not business keys), facts referencing surrogate PKs via FOREIGN KEY constraints, NOT NULL requirements for PK columns, proper dimensional modeling patterns (SCD Type 1/2, date dimensions), production deployment error prevention (never define FK inline in CREATE TABLE, DATE type casting from DATE_TRUNC, avoid module imports, widget parameter naming consistency), and validation checklists. Critical for ensuring proper relational modeling and preventing constraint application errors during Gold layer deployment.
    6 repo stars
  45. ▌
    Schema Management Patterns · databricks-solutions bundle
    Schema Management Patterns for Databricks Asset Bundles
    6 repo stars
  46. ▌
    Observability Setup · databricks-solutions
    End-to-end orchestrator for setting up Databricks observability including Lakehouse Monitoring, Anomaly Detection, AI/BI Dashboards, and SQL Alerts. Guides users through monitor creation for Gold tables, schema-level anomaly detection for freshness/completeness, dashboard design with monitoring widgets, and config-driven alerting. Orchestrates mandatory dependencies on monitoring skills (lakehouse-monitoring-comprehensive, anomaly-detection, databricks-aibi-dashboards, sql-alerting-patterns) and common skills (databricks-asset-bundles, databricks-expert-agent, databricks-python-imports). Use when setting up observability end-to-end, creating Lakehouse Monitors, enabling anomaly detection, building dashboards, or configuring SQL alerts.
    6 repo stars
  47. ▌
    Databricks Table Properties · databricks-solutions bundle
    Provides standard TBLPROPERTIES and metadata patterns for Unity Catalog Delta tables across Bronze, Silver, and Gold medallion layers. Ensures governance compliance, performance optimization, and proper metadata tagging for all table creation operations. Covers required TBLPROPERTIES by layer (Bronze, Silver DLT, Gold), mandatory CLUSTER BY AUTO configuration, Change Data Feed (CDF) enablement, auto-optimize settings, table and column comment patterns (LLM-friendly for Bronze/Silver, dual-purpose for Gold), domain values, data classification tags, and validation checklists. Use when creating Delta tables, configuring table properties, enabling CDF, setting up auto-optimize, or ensuring governance metadata consistency. Critical for preventing missing properties, incorrect clustering configurations, and governance compliance issues.
    6 repo stars
  48. ▌
    05 Erd Diagrams · databricks-solutions bundle
    Patterns for creating clean, professional Mermaid ERD diagrams for data modeling documentation. Use when documenting Gold layer data models, creating master ERDs for complete models, domain-specific ERDs for focused views, or summary ERDs for large models (20+ tables). Includes organization strategies, syntax standards, relationship patterns, and cross-domain reference patterns.
    6 repo stars
  49. ▌
    SQL Alerting Patterns · databricks-solutions bundle
    Comprehensive guide for Databricks SQL Alerts V2 - config-driven alerting framework with SDK deployment, hierarchical job architecture (5 atomic + 1 composite), proactive EXPLAIN-based query validation, and partial success patterns. Use when setting up SQL alerts, creating alert configuration tables, deploying alerts via Databricks SDK (V2 dict-based or typed classes), or troubleshooting alert failures. Includes config-driven patterns, fully qualified table names (no parameters), severity-based routing, alert ID conventions, SQL query patterns (threshold, percentage change, anomaly detection), DataFrame-based config seeding, DAB job configuration, custom notification templates, Quartz cron schedules, and troubleshooting patterns.
    6 repo stars
  50. ▌
    Dlt Expectations Patterns · databricks-solutions bundle
    Spark Declarative Pipeline (SDP, formerly DLT) expectations patterns for data quality with Unity Catalog Delta table storage. Use when implementing Silver layer SDP/DLT pipelines, creating portable data quality rules, or needing runtime-updateable expectations without code deployment. Supports severity-based filtering (critical vs warning) and quarantine patterns. Standardizes on `import dlt` for the DQ-rules framework; the modern `dp` API (`from pyspark import pipelines as dp`) also supports expectations (`dp.expect_all_or_drop`) and is Databricks' recommended forward path.
    6 repo stars
  51. ▌
    Simple Agent Scaffold · databricks-solutions bundle
    Scaffold a minimal MCP tool-calling agent with Genie Spaces and deploy it to Databricks Model Serving in 5 steps, following the canonical OpenAI MCP Tool Calling Agent notebook verbatim. Produces a working endpoint testable in AI Playground and consumable by 06-appkit-serving-wiring. No evaluation, memory, or prompt registry — add those later via the existing worker skills. Use when creating a simple agent, scaffolding a new agent, building a quick tool-calling agent, or connecting an agent to Genie Spaces. Triggers on "simple agent", "scaffold agent", "MCP agent", "quick agent", "create agent", "tool calling agent", "Genie agent", "basic agent".
    6 repo stars
  52. ▌
    03 Deduplication · databricks-solutions bundle
    Delta MERGE deduplication pattern for Gold layer to prevent duplicate source key errors. Use when merging from Silver to Gold tables where Silver may contain duplicate business keys due to incremental DLT streaming, CDC patterns, SCD Type 2 tracking, or multiple batch loads. Prevents DELTA_MULTIPLE_SOURCE_ROW_MATCHING_TARGET_ROW_IN_MERGE errors by deduplicating before merge.
    6 repo stars
  53. ▌
    Genie Space Patterns · databricks-solutions bundle
    Patterns for setting up Databricks Genie Spaces with comprehensive agent instructions, data assets, SQL expressions, and benchmark questions. Use when creating Genie Spaces, configuring agent behavior, selecting data assets, defining SQL expressions (measures, filters, dimensions), or validating benchmark questions. Includes mandatory 8-section deliverable structure, General Instructions (≤20 lines), data asset organization (Metric Views → TVFs → Tables), SQL expressions (sql_snippets) for structured KPI/filter/dimension definitions, benchmark questions with exact SQL, Serverless warehouse mandate, table/column comment requirements for Genie SQL quality, pre-creation table inspection, Conversation API programmatic validation, follow-up vs new conversation patterns, deployment checklists, post-deployment configuration audit for drift detection, cross-consumer design considerations (Genie + dashboards), and benchmark regression testing patterns.
    6 repo stars
  54. ▌
    01 Grain Definition · databricks-solutions bundle
    Grain definition patterns for fact tables during the Gold layer design phase. Use when choosing grain types (transaction, aggregated, snapshot), documenting grain in YAML schemas, and applying the PK-grain decision tree to determine whether a fact table needs aggregation. Prevents costly table rewrites by catching grain ambiguity before implementation.
    6 repo stars
  55. ▌
    02 Merge Patterns · databricks-solutions bundle
    Provides production-grade patterns for Gold layer MERGE operations from Silver to Gold tables. Covers column mapping, schema evolution, SCD Type 1/2 patterns, fact table aggregation, and preventing variable naming conflicts with PySpark functions. Use when creating Gold layer MERGE operations, handling column name differences between Silver and Gold, implementing SCD Type 1/2 dimensions, aggregating fact tables, or troubleshooting MERGE errors. Triggers on "Gold merge", "MERGE operation", "upsert Gold", "SCD Type 1", "SCD Type 2", "fact table merge", "Silver to Gold", "column mapping", "schema evolution".
    6 repo stars
  56. ▌
    02 Dimension Patterns · databricks-solutions bundle
    Dimension design patterns for Gold layer modeling. Covers role-playing dimensions, degenerate dimensions, junk dimensions, mini-dimensions, outrigger dimensions, denormalized hierarchies, null handling, and flags as textual attributes. Use when designing dimension tables, choosing between dimension patterns, handling NULLs in dimensions, or flattening hierarchies. Triggers on "dimension pattern", "role-playing", "junk dimension", "degenerate dimension", "mini-dimension", "hierarchy", "NULL handling".
    6 repo stars
  57. ▌
    08 Industry Alignment · databricks-solutions bundle
    Active, source-bounded alignment of a Gold dimensional design to Databricks Industry Vibe Data Models (and other canonical industry reference models — TM Forum SID, ARTS, ACORD, HL7, BIAN). Use when the customer operates in a recognizable vertical (retail, banking, healthcare, telecom, insurance, hospitality) and wants the Gold layer to cover industry-standard entities and use industry terminology, when checking industry coverage/gaps, or when the customer has a Vibe-generated Silver business model to align to. Overlays the parsed source schema with industry entities in Phase 0, steers domain assignment, conformed dimensions, grain/measure completeness, and naming in Phase 2, and emits an INDUSTRY_CROSSWALK the advisory Validation 6 re-scores. Does NOT invent tables absent from the source.
    6 repo stars
  58. ▌
    01 YAML Table Setup · databricks-solutions bundle
    Patterns for creating Gold layer tables dynamically from YAML schema definitions at runtime. Use when managing 10+ Gold layer tables across multiple domains, when schema evolves frequently, or when you want to avoid embedded SQL DDL strings in Python. Includes YAML schema structure, setup script implementation, Asset Bundle configuration, and workflow patterns for schema changes.
    6 repo stars
  59. ▌
    04 Grain Validation · databricks-solutions bundle
    Pre-merge grain validation patterns for Gold layer fact tables. Use when creating fact table merge scripts to validate that DataFrame grain matches DDL PRIMARY KEY, prevent transaction vs aggregated confusion, and catch grain mismatches before MERGE. Includes grain inference functions, pre-merge validation, validation SQL, and common grain mismatch errors.
    6 repo stars
  60. ▌
    Adhoc Exploration Notebooks · databricks-solutions bundle
    Create dual-format ad-hoc exploration notebooks for Databricks workspace (.py) and local Jupyter (.ipynb) with Databricks Connect. Use when building data exploration tools, debugging data quality issues, or creating interactive analysis notebooks. Supports widget fallback patterns, helper functions for table discovery, and proper Spark session initialization for both environments.
    6 repo stars
  61. ▌
    03 Fact Table Patterns · databricks-solutions bundle
    Advanced fact table design patterns for Gold layer modeling. Covers measure additivity classification (additive, semi-additive, non-additive), factless fact tables, accumulating snapshot facts, consolidated fact tables, header/line fact patterns, late-arriving facts and dimensions, and NULL handling in measures. Use when designing fact tables beyond basic transaction/aggregate patterns, classifying measure types, or handling complex fact scenarios. Triggers on "fact pattern", "factless", "accumulating snapshot", "measure additivity", "semi-additive", "late arriving", "consolidated fact".
    6 repo stars
  62. ▌
    06 Table Documentation · databricks-solutions bundle
    Comprehensive documentation standards for Gold layer tables including naming conventions, column descriptions, and metadata requirements. Use when creating Gold layer tables, columns, or writing documentation to ensure dual-purpose descriptions that serve both business users and technical users (including LLMs like Genie). Includes YAML schema consultation patterns, surrogate key patterns, SCD Type 2 documentation, and implementation guidance for Silver table naming conventions.
    6 repo stars
  63. ▌
    05 Schema Validation · databricks-solutions bundle
    Runtime schema validation patterns for Gold layer merge scripts. Use when creating merge scripts to ensure DataFrame columns match target DDL schemas, validate column mappings before MERGE operations, and catch schema issues before deployment. Includes the DDL-first workflow, validate_merge_schema() helper, explicit column mapping, DDL schema reader, and pre-deployment validation script. Addresses the "three sources of truth" problem by enforcing DDL as runtime truth.
    6 repo stars
  64. ▌
    04 Conformed Dimensions · databricks-solutions bundle
    Enterprise integration patterns for Gold layer dimensional models. Covers conformed dimensions, the enterprise data warehouse bus matrix, shrunken/rollup dimensions, conformed facts, and drill-across query patterns. Use when planning dimensions shared across multiple fact tables, creating a bus matrix for enterprise integration, designing rollup dimensions, or enabling cross-process analytics. Triggers on "conformed dimension", "bus matrix", "drill-across", "shrunken dimension", "rollup", "enterprise integration", "cross-process".
    6 repo stars
  65. ▌
    Lakehouse Monitoring Comprehensive · databricks-solutions bundle
    Comprehensive guide for Databricks Lakehouse Monitoring (Data Profiling) with quick-start workflow (2 hours), fill-in-the-blank requirements template, concrete fact/dimension monitor examples, and complete deployment patterns. Uses the new Data Quality API (`databricks.sdk.service.dataquality`). Use when setting up Lakehouse Monitoring for Gold layer tables, creating custom business metrics, designing monitoring strategy, querying monitoring tables for dashboards, or troubleshooting monitor initialization failures. Includes setup patterns with graceful degradation, custom metric syntax (AGGREGATE, DERIVED, DRIFT), table-level business KPIs with `input_columns=[":table"]`, query patterns for dashboards, async operations handling, monitor cleanup, Genie documentation integration, and production deployment workflow.
    6 repo stars
  66. ▌
    Databricks Table Valued Functions · databricks-solutions bundle
    End-to-end guide for planning, creating, deploying, and validating Table-Valued Functions (TVFs) in Databricks optimized for Genie Space natural language queries. Use when creating TVFs for Genie Spaces, planning TVF requirements from business questions, troubleshooting TVF compilation errors, or ensuring Genie compatibility. Includes requirements gathering templates, schema validation patterns, SQL requirements (STRING parameters, parameter ordering, LIMIT workarounds), v3.0 bullet-point comment format, null safety, SCD2 handling, cartesian product prevention, 5 complete domain-adaptable examples, Asset Bundle deployment patterns, and post-deployment validation queries.
    6 repo stars