databrickslabs
- 71 skills
- 0 followers
- 14 hours ago last updated
- ▌ Bdd Run · databrickslabsThis skill should be used when the user asks to "run BDD tests", "execute Behave", "run Gherkin tests", "run my feature files", "behave test results", "run smoke tests", "BDD test report", or needs to execute Behave test suites with specific options like tag filtering, parallel execution, or CI reporting.
- ▌ Bdd Steps · databrickslabs bundleThis skill should be used when the user asks to "write step definitions", "implement BDD steps", "generate step code", "create Behave steps", "implement Given When Then", "write Python steps for Gherkin", "step definitions for Databricks", or needs to create Python step implementations for existing Gherkin feature files.
- ▌ Bdd Features · databrickslabs bundleThis skill should be used when the user asks to "write Gherkin", "create feature files", "generate BDD scenarios", "write acceptance tests in Gherkin", "create Behave features", "write Given When Then tests", "BDD test cases for my pipeline", "Gherkin for Unity Catalog", or wants to translate requirements into Gherkin feature files for Databricks.
- ▌ Bdd Scaffold · databrickslabs bundleThis skill should be used when the user asks to "set up BDD", "create a Behave project", "scaffold BDD tests", "initialize Behave", "add BDD to my project", "set up Gherkin testing", "create test structure for Behave", or mentions setting up behavior-driven development testing. Generates a complete Behave project structure wired to Databricks SDK.
- ▌ Databricks Python Sdk · databrickslabs bundleDatabricks development guidance including Python SDK, Databricks Connect, CLI, and REST API. Use when working with databricks-sdk, databricks-connect, or Databricks APIs.
- ▌ Refresh Databricks Skills · databrickslabsUse when Databricks skills need updating, user asks to refresh or sync skills from upstream, or skills seem outdated compared to the ai-dev-kit repo
- ▌ Databricks Unity Catalog · databrickslabs bundleUnity Catalog system tables and volumes. Use when querying system tables (audit, lineage, billing) or working with volume file operations (upload, download, list files in /Volumes/).
- ▌ Sdp Meta · databrickslabs bundleUse this skill for metadata-driven Databricks pipelines built with SDP-META (the framework in this repo; formerly DLT-META, package databricks-labs-sdp-meta, CLI `databricks labs sdp-meta`). Triggers: (1) 'sdp-meta', 'sdp meta', 'dlt-meta', 'dlt meta', or 'metadata-driven pipeline'; (2) onboarding a dataflowspec / writing an onboarding JSON/YAML that describes bronze/silver tables as metadata; (3) generating many bronze/silver Lakeflow SDP pipelines from config instead of hand-writing pipeline code; (4) `sdp-meta onboard`, `sdp-meta deploy`, or `sdp-meta bundle-init/bundle-add-flow/bundle-validate`; (5) scaffolding an sdp-meta Asset Bundle (DAB). This is the config-driven LAYER ON TOP OF Lakeflow Spark Declarative Pipelines — for hand-authored streaming tables / materialized views, that is plain SDP, not this framework.
- ▌ Dax Translation · databrickslabs bundleDAX to SQL/UC-metric-view translation patterns. Use when converting Power BI DAX measures to SQL aggregations or metric view YAML measure expressions. Covers CALCULATE, FILTER, DIVIDE, SWITCH, ALLSELECTED, time intelligence, SELECTEDVALUE, and common untranslatable patterns.
- ▌ Ip Acl Migration · databrickslabsMigrate a Databricks workspace's existing IP access list (ACL) into a context-based ingress (CBI) account network policy, as-is, using the dbx-migrate-ip-acls CLI. Use when the user wants to convert / migrate an existing IP access list to a network policy, recreate their IP ACL as CBI, or stand up a network policy from the current ACL without audit-log analysis. Runs `dbx-migrate-ip-acls`, which reads this workspace's ACL (ALLOW->allow rules, BLOCK->deny rules) and recreates it verbatim — nothing added — then creates the account network policy (enforce or dry-run) and optionally auto-assigns it to the current workspace.
- ▌ Dqx Storage · databrickslabsLoad and save DQX checks (quality rules) to a file, workspace path, Unity Catalog volume, Delta table, Lakebase, or the DQX installation folder. Use when the user asks to "load DQX checks from YAML", "save checks to a Delta table", "read checks from a volume", "share checks across notebooks", or "use the DQX workspace install's default checks location". Covers every *ChecksStorageConfig and the matching load/save calls.
- ▌ Dqx End To End · databrickslabsRun DQX validation end-to-end — read an input table or path, apply checks, and write valid and quarantined rows to output locations — in a single call. Use when the user asks for "apply and save", "quality-check a table and split the output", "DQX on a whole table", "save valid and invalid rows", or wants to drop DQX into a Lakeflow / workflow that runs on a table or path. Covers apply_checks_and_save_in_table, the by_metadata variant, InputConfig / OutputConfig, and incremental streaming mode.
- ▌ Dqx Apply Checks · databrickslabsValidate a PySpark DataFrame or Delta table against a set of DQX quality rules using DQEngine. Use when the user asks to "run data quality checks", "apply DQX rules to a DataFrame/table", "split valid and invalid rows", "quarantine bad records", or "integrate DQX into a streaming pipeline". Covers apply_checks, apply_checks_and_split, the by_metadata variants, and the shape of the result columns.
- ▌ Dqx Define Checks · databrickslabsCreate DQX quality rules (checks) for a PySpark DataFrame or Delta table. Use when the user asks to "add a DQX check", "define a data quality rule", "validate that column X is not null / unique / in a set", or wants checks expressed in YAML/JSON for storage. Covers DQRowRule, DQDatasetRule, DQForEachColRule, built-in check_funcs, filters, user_metadata, custom SQL/Python checks, and the declarative metadata form.
- ▌ Dqx Profile And Generate · databrickslabsProfile a DataFrame or table and generate DQX quality rule candidates with summary statistics. Use when the user asks to "profile a table", "generate DQX rules from data", "suggest data quality checks", "bootstrap a checks.yml", or "generate DLT expectations". Covers DQProfiler, DQGenerator, DQDltGenerator, the profiler workflow, sampling / filter options, and AI-assisted variants.
- ▌ Deploy · databrickslabsUse when the user asks to deploy, ship, release, or push OntoBricks to Databricks. Wraps the Databricks Asset Bundle deploy for the FastAPI app and the MCP server, with the bootstrap-perms safety net described in README.md.
- ▌ Changelog · databrickslabsUse after any code change (feature, fix, refactor, review fixup) to update /changelogs/YYYY-MM-DD.log and run the test suite. Mandatory post-change routine — see .cursorrules.
- ▌ AI Feature · databrickslabs bundleUse when the user adds, changes, or refactors an LLM agent under src/agents/ — or anything that goes through Foundation Model API or an MLflow-traced LLM call. Mandatory under CNS §3.5 and .cursor/12-ai-feature-lifecycle.mdc. Walks the SPEC → dataset → eval-harness → impl → re-eval sequence.
- ▌ Code Review · databrickslabsUse when the user asks for a code review, asks to "review the code", or requests review of a feature/PR/branch. Runs the OntoBricks review checklist defined in .cursorrules.
- ▌ Refactoring · databrickslabsUse when the user asks to "refactor", restructure, clean up, simplify, deduplicate, extract, or reorganize code. Enforces the Martin Fowler discipline defined in src/.coding_rules.md and .cursor/08-testing-and-deployment.mdc.
- ▌ Frontend Design · databrickslabsUse when creating or changing OntoBricks templates, page layouts, tabs, cards, forms, or responsive UI behavior.
- ▌ Adding Subpackage · databrickslabsUse when adding a new subpackage under back/core/, back/objects/, or agents/ — e.g. a new graph DB engine, W3C parser, industry importer, reasoning module, or domain class. Enforces the checklist defined in .cursor/07-project-conventions.mdc.
- ▌ Impulse · databrickslabsEntry point for the Impulse framework — the Databricks Labs library for analyzing large-scale time-series measurement data (automotive testing, industrial IoT sensor recordings) on Spark and Delta. Use when the user mentions "Impulse", "TSAL", measurement/sensor/telemetry channels, test drives, or wants to build histograms, event windows, or statistics over time-series recordings and isn't sure where to start. Explains the core concepts (container, channel, event, aggregation), the three usage modes (reporting, ad-hoc analysis, ML), how to set up `spark` + `WorkspaceClient`, and routes to the right sibling skill.
- ▌ Impulse Ml · databrickslabsTurn Impulse measurement recordings into a machine-learning feature matrix. Use when the user wants to "make ML features from measurement data", "train a model on containers/recordings", build a per-drive / per-recording feature vector from sensor statistics, or feed Impulse output to MLflow or AutoML. Covers computing event-scoped statistics with a sinkless Report, reading them from `report.aggregation_dfs`, and pivoting to one row per container.
- ▌ Impulse Tsal · databrickslabsWrite TSAL (Time Series Analytics Language) expressions in Impulse — select measurement channels by metadata tags, derive virtual signals with arithmetic, express conditions as time windows, detect edges, resample, integrate, and build histograms. Use when the user wants to "select a channel", "build a virtual/derived signal", "detect when signal > X", "compute distance from speed", "find rising edges", or asks what a TSAL expression evaluates to. Covers `QueryBuilder.channel()`, operators, all signal methods, and the four result types (SampleSeries, Intervals, PointsInTime, PointsInTimeSeries).
- ▌ Impulse Config · databrickslabsWrite the Impulse report configuration (`ImpulseConfig`) — the JSON/dict that points Impulse at its silver input tables, the gold output location, and tunes the solver. Use when the user asks to "configure an Impulse report", set the source/sink tables, filter which containers are processed, choose RLE vs RAW, turn on incremental processing, run without writing (sinkless), remap column names, or scope by project. Covers source, unity_sink, container_filters, query_engine, solver_config, incremental, full_recalculation, measurement_dimensions, and calculated_channels, all validated by Pydantic.
- ▌ Impulse Events · databrickslabsDefine event windows in Impulse — the time spans that scope aggregations. Use when the user wants to "define an event", segment recordings into intervals (e.g. "engine RPM between 2000 and 5000"), aggregate over the whole recording, capture state transitions/sequences, or mark instants like rising edges. Covers BasicEvent, ContainerEvent, SequenceOfEvents, and PointsInTimeEvent — which TSAL result type each requires, their constructor parameters, and the event fact/dimension output.
- ▌ Impulse Analyze · databrickslabsRun ad-hoc Impulse analysis in a notebook — evaluate TSAL expressions directly through the query engine and get a Spark or pandas DataFrame back, with no gold-layer write and no reporting setup. Use when the user wants to "explore signals interactively", "get a DataFrame from Impulse", compute a quick per-container mean/histogram, or prototype expressions before wiring them into a report. Covers building a query on a MeasurementDB, `select()`, `solve()` / `toPandas()`, and choosing the solver.
- ▌ Impulse Channels · databrickslabsCompute calculated (derived) channels in Impulse — new time-series signals built from existing channels and materialized at the same per-sample grain. Use when the user wants to "add a calculated channel", derive/persist a signal (e.g. "speed in km/h", "power = rpm × torque"), materialize a virtual signal into a queryable table, or run `solve_calculated_channels`. Covers the reporting-layer CalculatedChannel, the ad-hoc `QueryBuilder.solve_calculated_channels` endpoint, the calculated_channel_fact/dimension gold output, and the optional calculated_channel_metrics table.
- ▌ Impulse Reporting · databrickslabsBuild and run an Impulse reporting pipeline that persists events and aggregations to the gold-layer star schema. Use when the user wants to "build an Impulse report", compute histograms/statistics across all recordings and write them to Delta tables for a dashboard or scheduled job, or run incrementally so only new/changed data is reprocessed. Covers the Report/Page lifecycle (`add_event`, `add_page`, `determine_report`, `persist_results`), incremental processing, and sinkless mode.
- ▌ Impulse Data Model · databrickslabsUnderstand and prepare the data Impulse reads and writes. Use when the user asks "what tables does Impulse need", how to land / ingest measurement data into the silver layer, what the gold-layer output looks like, how fact and dimension tables join, or how to point Impulse at existing tables whose column names differ (via SolverConfig column mappings). Covers the three required silver tables, the optional tag/mapping/unit tables, RLE vs RAW channel formats, the gold star schema, and the SolverConfig / custom-solver escape hatches.
- ▌ Impulse Aggregations · databrickslabsCompute results over channels in Impulse, optionally scoped to an event. Use when the user wants a "histogram" or "2D histogram / heatmap" of a signal, duration- or distance-weighted binning, per-event descriptive statistics (min/max/mean/median), or channel values sampled at instants. Covers HistogramDuration / HistogramDistance / HistogramCustomWeights, the Histogram2D variants, StatsAggregator, PointValueAggregator, and the Page that groups them, plus their gold-layer output.
- ▌ Deploy Connector · databrickslabsGuide the user through creating or updating a pipeline for a source connector — read the docs, build a pipeline spec interactively, and run create_pipeline or update_pipeline.
- ▌ Write Back Testing · databrickslabsImplement test utilities that write test data to the source system and validate end-to-end read cycles.
- ▌ Authenticate Source · databrickslabsSet up authentication for a source connector — generate connector spec, collect credentials interactively, and validate auth.
- ▌ Collect Credentials · databrickslabsRun the authenticate script to collect credentials from the user via a browser form.
- ▌ Implement Connector · databrickslabsSingle step only: implement the connector in Python when the API doc already exists. Do NOT use for full connector creation — use the create-connector agent instead.
- ▌ Research Source API · databrickslabsSingle step only: research and document a source API when explicitly asked. Do NOT use for full connector creation — use the create-connector agent instead.
- ▌ Self Review Connector · databrickslabsSingle step only: audit a completed connector — implementation, testing & simulator validation, artifacts, security smells, cross-doc consistency — and produce a scored markdown review report. Read-mostly; does not modify connector code.
- ▌ Test And Fix Connector · databrickslabsSingle step only: run the per-source pytest suite, diagnose failures, and fix the connector or simulator until everything passes. Branches on mode={simulate|record}. Do NOT use for full connector creation — use the create-connector agent instead.
- ▌ Build Connector Package · databrickslabsCreate a pyproject.toml for a source connector and build it as an independent Python package.
- ▌ Generate Connector Spec · databrickslabsGenerate the connector spec YAML file defining connection parameters and external options allowlist.
- ▌ Validate Connector Auth · databrickslabsGenerate and run an auth verification test to confirm that collected credentials are valid.
- ▌ Create Connector Document · databrickslabsGenerate public-facing documentation for a connector targeted at end users.
- ▌ Validate Incremental Sync · databrickslabsValidate that a connector's CDC/incremental sync implementation correctly tracks offsets and filters records.
- ▌ Research Write API Of Source · databrickslabsResearch and document write/create APIs of a source system to enable write-back testing functionality.
- ▌ Migrate Legacy Implementation · databrickslabsMigrate a legacy source connector from the old sources/ directory to the new package structure under src/databricks/labs/community_connector/sources/.
- ▌ Implement Partitioned Connector · databrickslabsSingle step only: implement a partitioned connector that extends SupportsPartitionedStream (or SupportsPartition) alongside LakeflowConnect. Do NOT use for full connector creation — use the create-connector agent instead.
- ▌ Databricks Docs · databrickslabsDatabricks documentation reference via llms.txt index. Use when other skills do not cover a topic, looking up unfamiliar Databricks features, or needing authoritative docs on APIs, configurations, or platform capabilities.
- ▌ Databricks Jobs · databrickslabs bundleUse this skill proactively for ANY Databricks Jobs task - creating, listing, running, updating, or deleting jobs. Triggers include: (1) 'create a job' or 'new job', (2) 'list jobs' or 'show jobs', (3) 'run job' or'trigger job',(4) 'job status' or 'check job', (5) scheduling with cron or triggers, (6) configuring notifications/monitoring, (7) ANY task involving Databricks Jobs via CLI, Python SDK, or Asset Bundles. ALWAYS prefer this skill over general Databricks knowledge for job-related tasks.
- ▌ Databricks Genie · databrickslabs bundleCreate and query Databricks Genie Spaces for natural language SQL exploration. Use when building Genie Spaces, exporting and importing Genie Spaces, migrating Genie Spaces between workspaces or environments, or asking questions via the Genie Conversation API.
- ▌ Verify Coda Live · databrickslabsUse after merging CoDA PRs, before a release, or when asked whether coda-main still works. Drives coda-main through an authenticated Chrome session and CoDA's JSON terminal API, runs the structured scripts/verify_coda_live.py smoke test, and returns evidence for SP-authenticated Claude Code/Pi/OpenCode inference, live workspace model-catalog parity, GitHub CLI access, and Databricks CLI workspace read/write access. Can switch coda-main's git-linked deployment branch and always restores it.
- ▌ Databricks Config · databrickslabsManage Databricks workspace connections: check current workspace, switch profiles, list available workspaces, or authenticate to a new workspace. Use when the user mentions "switch workspace", "which workspace", "current profile", "databrickscfg", "connect to workspace", or "databricks auth".
- ▌ Databricks Bundles · databrickslabs bundleCreate and configure Declarative Automation Bundles (formerly Asset Bundles) with best practices for multi-environment deployments (CICD). Use when working with: (1) Creating new DAB projects, (2) Adding resources (dashboards, pipelines, jobs, alerts), (3) Configuring multi-environment deployments, (4) Setting up permissions, (5) Deploying or running bundle resources
- ▌ Databricks Iceberg · databrickslabs bundleApache Iceberg tables on Databricks — Managed Iceberg tables, External Iceberg Reads (fka Uniform), Compatibility Mode, Iceberg REST Catalog (IRC), Iceberg v3, Snowflake interop, PyIceberg, OSS Spark, external engine access and credential vending. Use when creating Iceberg tables, enabling External Iceberg Reads (uniform) on Delta tables (including Streaming Tables and Materialized Views via compatibility mode), configuring external engines to read Databricks tables via Unity Catalog IRC, integrating with Snowflake catalog to read Foreign Iceberg tables
- ▌ Databricks Apps Python · databrickslabs bundleBuilds Databricks applications. Prefers AppKit (TypeScript + React SDK) for new apps; falls back to Python frameworks (Dash, Streamlit, Gradio, Flask, FastAPI, Reflex) when Python is required. Handles OAuth authorization, app resources, SQL warehouse and Lakebase connectivity, model serving, foundation model APIs, and deployment. Use when building web apps, dashboards, ML demos, or REST APIs for Databricks, or when the user mentions AppKit, Streamlit, Dash, Gradio, Flask, FastAPI, Reflex, or Databricks app.
- ▌ Databricks Agent Bricks · databrickslabs bundleCreate and manage Databricks Agent Bricks: Knowledge Assistants (KA) for document Q&A, Genie Spaces for SQL exploration, and Supervisor Agents (MAS) for multi-agent orchestration. Use when building conversational AI applications on Databricks.
- ▌ Databricks AI Functions · databrickslabs bundleUse Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).
- ▌ Databricks Metric Views · databrickslabs bundleUnity Catalog metric views: define, create, query, and manage governed business metrics in YAML. Use when building standardized KPIs, revenue metrics, order analytics, or any reusable business metrics that need consistent definitions across teams and tools.
- ▌ Databricks Model Serving · databrickslabs bundleDeploy and query Databricks Model Serving endpoints. Use when (1) deploying MLflow models or AI agents to endpoints, (2) creating ChatAgent/ResponsesAgent agents, (3) integrating UC Functions or Vector Search tools, (4) querying deployed endpoints, (5) checking endpoint status. Covers classical ML models, custom pyfunc, and GenAI agents.
- ▌ Databricks Vector Search · databrickslabs bundlePatterns for Databricks Vector Search: create endpoints and indexes, query with filters, manage embeddings. Use when building RAG applications, semantic search, or similarity matching. Covers both storage-optimized and standard endpoints.
- ▌ Spark Python Data Source · databrickslabs bundleBuild custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.
- ▌ Databricks Zerobus Ingest · databrickslabs bundleBuild Zerobus Ingest clients for near real-time data ingestion into Databricks Delta tables via gRPC. Use when creating producers that write directly to Unity Catalog tables without a message bus, working with the Zerobus Ingest SDK in Python/Java/Go/TypeScript/Rust, generating Protobuf schemas from UC tables, or implementing stream-based ingestion with ACK handling and retry logic.
- ▌ Databricks Aibi Dashboards · databrickslabs bundleCreate Databricks AI/BI dashboards. Use when creating, updating, or deploying Lakeview dashboards. CRITICAL: You MUST test ALL SQL queries via execute_sql BEFORE deploying. Follow guidelines strictly.
- ▌ Databricks Execution Compute · databrickslabs bundleExecute code and manage compute on Databricks. Use this skill when the user mentions: "run code", "execute", "run on databricks", "serverless", "no cluster", "run python", "run scala", "run sql", "run R", "run file", "push and run", "notebook run", "batch script", "model training", "run script on cluster", "create cluster", "new cluster", "resize cluster", "modify cluster", "delete cluster", "terminate cluster", "create warehouse", "new warehouse", "resize warehouse", "delete warehouse", "node types", "runtime versions", "DBR versions", "spin up compute", "provision cluster".
- ▌ Databricks Lakebase Autoscale · databrickslabs bundlePatterns and best practices for Lakebase Autoscaling (next-gen managed PostgreSQL). Use when creating or managing Lakebase Autoscaling projects, configuring autoscaling compute or scale-to-zero, working with database branching for dev/test workflows, implementing reverse ETL via synced tables, or connecting applications to Lakebase with OAuth credentials.
- ▌ Databricks Synthetic Data Gen · databrickslabs bundleGenerate realistic synthetic data using Spark + Faker (strongly recommended). Supports serverless execution, multiple output formats (Parquet/JSON/CSV/Delta), and scales from thousands to millions of rows. For small datasets (<10K rows), can optionally generate locally and upload to volumes. Use when user mentions 'synthetic data', 'test data', 'generate data', 'demo dataset', 'Faker', or 'sample data'.
- ▌ Databricks Lakebase Provisioned · databrickslabs bundlePatterns and best practices for Lakebase Provisioned (Databricks managed PostgreSQL) for OLTP workloads. Use when creating Lakebase instances, connecting applications or Databricks Apps to PostgreSQL, implementing reverse ETL via synced tables, storing agent or chat memory, or configuring OAuth authentication for Lakebase.
- ▌ Databricks Spark Structured Streaming · databrickslabs bundleComprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.
- ▌ Databricks Spark Declarative Pipelines · databrickslabs bundleCreates, configures, and updates Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP) using serverless compute. Handles data ingestion with streaming tables, materialized views, CDC, SCD Type 2, and Auto Loader ingestion patterns. Use when building data pipelines, working with Delta Live Tables, ingesting streaming data, implementing change data capture, or when the user mentions SDP, LDP, DLT, Lakeflow pipelines, streaming tables, or bronze/silver/gold medallion architectures.
- ▌ Databricks Unstructured PDF Generation · databrickslabsGenerate PDF documents from HTML and upload to Unity Catalog volumes. Use for creating test PDFs, demo documents, reports, or evaluation datasets.