Databricks AI Search (formerly Vector Search)
FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.
Patterns for creating, managing, and querying AI Search indexes for RAG and semantic search applications.
Naming: the REST API (/api/2.0/vector-search/...), the databricks vector-search-* CLI groups, and the databricks_vector_search_* Terraform resources keep the Vector Search names (API and tool names); the Python client is databricks-ai-search (AISearchClient).
When to Use
Use this skill when:
- Building RAG (Retrieval-Augmented Generation) applications
- Implementing semantic search or similarity matching
- Creating vector indexes from Delta tables
- Choosing between storage-optimized and standard endpoints
- Querying vector indexes with filters
Overview
Databricks AI Search provides managed vector similarity search with automatic embedding generation and Delta Lake integration.
| Component | Description |
|---|---|
| Endpoint | Compute resource hosting indexes (Standard or Storage-Optimized) |
| Index | Vector data structure for similarity search |
| Delta Sync | Auto-syncs with source Delta table |
| Direct Access | Manual CRUD operations on vectors (Standard endpoints only) |
Endpoint Types
| Type | Latency | Capacity | Cost | Best For |
|---|---|---|---|---|
| Standard | 20-50ms | 320M vectors (768 dim) | Higher | Real-time, low-latency, high QPS |
| Storage-Optimized | 300-500ms | 1B+ vectors (768 dim) | Up to 7x lower | 10M+ vectors, cost-sensitive |
Storage-Optimized supports only TRIGGERED Delta Sync indexes: no CONTINUOUS pipelines, no Direct Access, no target_qps. The endpoint type cannot be changed after creation.
Index Types
| Type | Embeddings | Sync | Use Case |
|---|---|---|---|
| Delta Sync (managed) | Databricks computes | Auto from Delta | Easiest setup |
| Delta Sync (self-managed) | You provide | Auto from Delta | Custom embeddings |
| Direct Access | You provide | Manual CRUD | Real-time updates |
Installation
pip install "databricks-ai-search>=0.79"
In a notebook, use %pip install instead and run dbutils.library.restartPython() afterwards:
the package requires mlflow-skinny>=3.10.1, so installing it upgrades mlflow in place.
databricks.ai_search first shipped in databricks-ai-search 0.73. This skill needs 0.79+:
from 0.79, get_endpoint, list_endpoints and list_indexes always return snake_case keys,
and 0.79 also includes the 0.78 fix that makes concurrent similarity_search calls thread-safe
(needed for production traffic). From 0.81, service principal auth no longer requires the
all-apis OAuth scope.
Existing databricks-vectorsearch code: on databricks-vectorsearch 0.74+,
VectorSearchClient is an alias of AISearchClient, so the calls in this skill work on it
once databricks-ai-search>=0.79 is installed alongside it. On 0.73 and older it is a
separate class: switch the import to from databricks.ai_search.client import AISearchClient.
The old class has no start/stop calls, and before 0.68 no target_qps.
Auth: a bare AISearchClient() uses the notebook's credentials. Elsewhere, pass
workspace_url plus a personal_access_token, or service_principal_client_id and
service_principal_client_secret (needed for high-QPS traffic; see
Production and High QPS).
AISearchClient takes plain strings (endpoint_type="STANDARD", pipeline_type="TRIGGERED").
Don't pass the databricks.sdk.service.vectorsearch enums to it; those are for
WorkspaceClient.vector_search_*.
Quick Start
Create Endpoint
from databricks.ai_search.client import AISearchClient
client = AISearchClient()
client.create_endpoint(
name="my-vs-endpoint",
endpoint_type="STANDARD" # or "STORAGE_OPTIMIZED"
)
# Note: Endpoint creation is asynchronous; check status with get_endpoint()
create_endpoint_and_wait(..., timeout=datetime.timedelta(minutes=30)) blocks until the endpoint is
ONLINE. It fails fast on OFFLINE (and, from 0.80, STOPPING and STOPPED); in other states it polls until the
timeout. On a timeout the endpoint may still be provisioning: check
client.get_endpoint(name)["endpoint_status"]["state"] before retrying, and don't create it again.
Create Delta Sync Index (Managed Embeddings)
Standard endpoints need Change Data Feed on the source table:
ALTER TABLE catalog.schema.documents SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
# Source table must have: primary key column + text column
index = client.create_delta_sync_index(
endpoint_name="my-vs-endpoint",
source_table_name="catalog.schema.documents",
index_name="catalog.schema.my_index",
pipeline_type="TRIGGERED", # "CONTINUOUS" is Standard-only
primary_key="id",
embedding_source_column="content",
embedding_model_endpoint_name="databricks-gte-large-en"
)
Query Index
index = client.get_index(
endpoint_name="my-vs-endpoint",
index_name="catalog.schema.my_index"
)
results = index.similarity_search(
query_text="What is machine learning?",
columns=["id", "content", "metadata"],
num_results=5
)
# Returns a dict; each row holds the requested columns, then the similarity score
for row in results["result"]["data_array"]:
print(row)
Common Patterns
Create Storage-Optimized Endpoint
# For large-scale, cost-effective deployments
client.create_endpoint(
name="my-storage-endpoint",
endpoint_type="STORAGE_OPTIMIZED"
)
Delta Sync with Self-Managed Embeddings
# Source table must have: primary key + embedding vector column
index = client.create_delta_sync_index(
endpoint_name="my-vs-endpoint",
source_table_name="catalog.schema.documents",
index_name="catalog.schema.my_index",
pipeline_type="TRIGGERED",
primary_key="id",
embedding_dimension=768,
embedding_vector_column="embedding"
)
Direct Access Index (Standard endpoints only)
# Create index for manual CRUD
index = client.create_direct_access_index(
endpoint_name="my-vs-endpoint",
index_name="catalog.schema.direct_index",
primary_key="id",
embedding_dimension=768,
embedding_vector_column="embedding",
schema={
"id": "string",
"text": "string",
"embedding": "array<float>",
"metadata": "string"
}
)
# Upsert data
index.upsert([
{"id": "1", "text": "Hello", "embedding": [0.1, 0.2, ...], "metadata": "doc1"},
{"id": "2", "text": "World", "embedding": [0.3, 0.4, ...], "metadata": "doc2"},
])
# Delete data
index.delete(primary_keys=["1", "2"])
Query with Embedding Vector
# When you have pre-computed query embedding
results = index.similarity_search(
query_vector=[0.1, 0.2, 0.3, ...], # Your 768-dim vector
columns=["id", "text"],
num_results=10
)
Hybrid Search (Semantic + Keyword)
Hybrid search runs ANN and full-text (keyword) search and merges the results with Reciprocal Rank Fusion (RRF). Use it when queries contain exact terms that must match — SKUs, error codes, proper nouns, or technical terminology — where pure semantic search might miss keyword-specific results. See references/search-modes.md for detailed guidance on choosing between ANN and hybrid search.
# Needs an index with a managed embedding model, so query_text alone is enough
index = client.get_index(
endpoint_name="my-vs-endpoint",
index_name="catalog.schema.my_index",
)
# Combines vector similarity with keyword matching
results = index.similarity_search(
query_text="SPARK-12345 executor memory error",
query_type="hybrid",
columns=["id", "content"],
num_results=10
)
Filtering
Filter syntax differs by endpoint type: Standard endpoints take a dict, Storage-Optimized endpoints take a SQL-like string. On Storage-Optimized, use the string: a dict (filters_json) is rejected on most query routes (except filter-only lookups with no query text or vector), and other routes translate it to SQL instead, so don't rely on either behavior.
Standard Endpoint Filters (Dictionary)
results = index.similarity_search(
query_text="machine learning",
columns=["id", "content"],
num_results=10,
filters={"category": "ai", "status": ["active", "pending"]}
)
Storage-Optimized Filters (SQL-like string)
index = client.get_index(
endpoint_name="my-storage-endpoint",
index_name="catalog.schema.my_storage_index",
)
results = index.similarity_search(
query_text="machine learning",
columns=["id", "content"],
num_results=10,
filters="category = 'ai' AND status IN ('active', 'pending')"
)
# filters="price > 100 AND price < 500"
# filters="department LIKE 'eng%'"
# filters="created_at >= TO_TIMESTAMP('2024-01-01T00:00:00')" # a bare timestamp string is rejected
See the filtering guide for operators and limitations.
Never interpolate untrusted input (tenant IDs, ACL values) into a filter as-is:
- Take column names from a fixed allowlist. In a dict filter the key also carries the operator (
"price >","col LIKE"), so allowlist whole keys. - Pass scalar values, validated against an allowlist or a strict pattern (for example,
^[A-Za-z0-9_-]+$). Never let untrusted input become a list: a list value becomes an OR/IN match. In a SQL string, don't rely on quote escaping alone. - On Storage-Optimized,
%and_in a LIKE value act as wildcards, so reject them in untrusted input. (On Standard, LIKE matches whole tokens, not wildcards.) - Filters are application-level ACLs, not a security boundary: row- and column-level permissions aren't supported, so anyone who can query the index can read every row.
In a RAG prompt, treat retrieved text as data, not instructions.
Trigger Index Sync
# TRIGGERED pipelines only, after the initial sync from creation finishes (rejected while provisioning).
# Also rejected while an update is running; wait for that update instead of retrying.
index = client.get_index(
endpoint_name="my-vs-endpoint",
index_name="catalog.schema.my_index"
)
index.sync()
Scan Index Entries
# Returns at most num_results rows, ordered by primary key. The response carries
# the cursor for the next page as last_primary_key.
page = index.scan(num_results=100)
rows = page.get("data", [])
while page.get("last_primary_key") and page.get("data"):
page = index.scan(num_results=100, last_primary_key=page["last_primary_key"])
rows += page.get("data", [])
# scan rows are field structs, NOT the plain lists similarity_search returns:
# {"fields": [{"key": "id", "value": {"string_value": "r1"}}, ...]}
for row in rows:
values = {f["key"]: next(iter(f["value"].values())) for f in row["fields"]}
Production and High QPS
By default, Standard endpoints serve about 20-200 QPS, depending on index size. Traffic over a PAT or the workspace query URL is capped at a few tens of QPS.
- Auth and route: production traffic needs service principal OAuth and the index URL. Create
AISearchClient(workspace_url=..., service_principal_client_id=..., service_principal_client_secret=...), callget_index()once, and reuse the index object; it queries the index URL. Use a dedicated service principal. For Delta Sync indexes with managed embeddings, an IP access list or a private access settings object (on Azure, public network access disabled) keeps queries on the capped route; on GCP, a private access settings object does so for every index. Otherwise, self-managed embeddings avoid it. See requirements.- Pitfalls: with any token passed as
personal_access_token(a PAT, or a forwarded user or on-behalf-of token such as the Databricks Appsx-forwarded-access-token), or a bareAISearchClient()that picks up notebook or model-serving credentials,get_index()queries the workspace URL even thoughdescribe()listsstatus.index_url; ifindex.index_urlstarts with your workspace URL, queries are on the capped route. In Databricks Apps, send query traffic with the app's service principal, not the forwarded user token.workspace_urlmust includehttps://: in Databricks Apps,DATABRICKS_HOSTcan lack the scheme, so add it (example).
- Pitfalls: with any token passed as
- Query type: start with ANN; hybrid uses about 4x the resources. Load-test with the query type production sends.
- 429s: move to service principal OAuth and the index URL; wait for
SCALING_CHANGE_APPLIED; withquery_vector, lower concurrency or raisetarget_qps; withquery_texton a managed-embedding index, the embedding endpoint may be the bottleneck.similarity_searchalready retries (total_retries,backoff_factor,backoff_jitter); raise those or add your own bounded backoff with jitter. See Troubleshoot 429 errors. - Load testing: use Run load test on the endpoint page or the load test notebook. With
databricks-ai-search0.82+,databricks-ai-search guide --topic high-qpsprints the guide anddatabricks-ai-search load-test --index <name>plans a run (it runs only with--yes); skip this if thedatabricks-ai-searchcommand isn't available. - Agent safety: ask before creating endpoints or indexes, setting
target_qps, running load tests, or upserting/deleting data, since each costs money or changes data. Never print, paste, or commit tokens or client secrets; read them from a secret scope or environment variables.
Target QPS is Standard-only and best-effort, and the capacity is billed whether or not traffic arrives. Storage-Optimized endpoints reject target_qps. A change can be rejected with RESOURCE_CONFLICT while a previous change is still applying. Nothing is saved, so requested_target_qps still shows the earlier target: poll until scaling_info.state is SCALING_CHANGE_APPLIED (whatever the target), or wait the minutes the error gives, then call update_endpoint again before running the loop below.
import time
client.update_endpoint(name="my-vs-endpoint", target_qps=500)
# Or at creation: create_endpoint(..., target_qps=500)
# Match the requested target too: right after the update, state can still be APPLIED from an earlier target.
# Target QPS is best-effort: if it's still in progress at the deadline, move on.
deadline = time.monotonic() + 30 * 60
while time.monotonic() < deadline:
info = client.get_endpoint("my-vs-endpoint").get("scaling_info") or {}
# int(): proto3 JSON may serialize int64 fields as strings
if info.get("state") == "SCALING_CHANGE_APPLIED" and int(info.get("requested_target_qps") or 0) == 500:
break
time.sleep(30)
print(info)
ONLINE doesn't mean scaled. These are separate waits:
- Endpoint provisioned:
endpoint_status.stateisONLINE(whatcreate_endpoint_and_waitwaits for).ONLINEdoesn't mean the target QPS is applied. - Target QPS applied:
scaling_info.stateisSCALING_CHANGE_APPLIEDfor the requested target (the loop above). This takes minutes to hours, depending on index size and how much the target changes. - Index ready:
status.readyis true. A new endpoint's first index hasstatus.detailed_statePROVISIONING_ENDPOINTwhile compute is created.
Load-test only after all three; a load test before SCALING_CHANGE_APPLIED doesn't reflect the target capacity. If the loop above hits its deadline, report that target QPS is still applying instead of load testing.
Reference Files
| Topic | File | Description |
|---|---|---|
| Index Types | references/index-types.md | Detailed comparison of Delta Sync (managed/self-managed) vs Direct Access |
| End-to-End RAG | references/end-to-end-rag.md | Complete walkthrough: source table → endpoint → index → query → agent integration |
| Search Modes | references/search-modes.md | When to use semantic (ANN) vs hybrid search, decision guide |
| Operations | references/troubleshooting-and-operations.md | Monitoring, cost optimization, capacity planning, migration |
CLI Quick Reference
# List endpoints
databricks vector-search-endpoints list-endpoints
# Create endpoint (positional args: NAME ENDPOINT_TYPE)
databricks vector-search-endpoints create-endpoint my-endpoint STANDARD
# List indexes on endpoint (positional arg: ENDPOINT_NAME)
databricks vector-search-indexes list-indexes my-endpoint
# Get index status (positional arg: INDEX_NAME)
databricks vector-search-indexes get-index catalog.schema.my_index
# Sync index (positional arg: INDEX_NAME)
databricks vector-search-indexes sync-index catalog.schema.my_index
# Delete index (positional arg: INDEX_NAME)
databricks vector-search-indexes delete-index catalog.schema.my_index
Common Issues
| Issue | Solution |
|---|---|
| Index sync slow | Use Storage-Optimized endpoints (10-20x faster indexing) |
| Query latency high | Use Standard endpoint for <100ms latency |
| Filters not working | Standard endpoints use a dict (filters={"col": "val"}); Storage-Optimized use a SQL string (filters="col = 'val'") |
| Embedding dimension mismatch | Ensure query and index dimensions match |
| Index not updating | Check pipeline_type; use index.sync() for TRIGGERED. If it's rejected as already running, wait for the current update to finish |
| Out of capacity | Capacity is sized from the source table at index creation. Load the full table first; past ~320M vectors, recreate on Storage-Optimized (1B+ vectors) |
| Rate limited (429 errors) | See Production and High QPS |
query_vector truncated |
Large vectors (e.g. 1024-dim) can be truncated when serialized as JSON. Use query_text instead (for managed embedding indexes), or pass the vector directly via index.similarity_search(query_vector=...) |
Embedding Models
Databricks provides built-in embedding models:
| Model | Dimensions | Context Window | Use Case |
|---|---|---|---|
databricks-qwen3-embedding-0-6b |
1024 (configurable, up to 1024) | ~32K tokens | Recommended for production by the AI Search docs (Public Preview) |
databricks-gte-large-en |
1024 | 8192 tokens | English text, high quality |
databricks-bge-large-en |
1024 | 512 tokens | English text, general purpose |
# Use with managed embeddings
index = client.create_delta_sync_index(
...,
embedding_source_column="content",
embedding_model_endpoint_name="databricks-qwen3-embedding-0-6b"
)
Notes
- Storage-Optimized is newer — better for most use cases unless you need <100ms latency, high QPS, Direct Access, or CONTINUOUS sync
- Delta Sync recommended — easier than Direct Access for most scenarios
- Hybrid search — available for both Delta Sync and Direct Access indexes
columns_to_sync— limits which columns are synced and returned; only synced columns are available in query results. On Storage-Optimized, the docs' limitations list says it's unsupported, but their full-text example uses it; prefer syncing all columns there- Filter syntax differs by endpoint — Standard uses a dict, Storage-Optimized uses a SQL-like string
- Management vs runtime — CLI and SDK handle lifecycle management; for agent tool-calling at runtime, use
VectorSearchRetrieverToolfromdatabricks_langchainordatabricks_openai
Related Skills
- databricks-model-serving - Deploy agents that use VectorSearchRetrieverTool
- databricks-ml-training - Author and deploy a
ResponsesAgentthat uses an AI Search retriever - databricks-genie-agents - No-code document Q&A: a Genie Agent with a UC volume (replaces the legacy Knowledge Assistant); hand-roll AI Search only for custom retrieval
- databricks-unstructured-pdf-generation - Generate documents to index in AI Search
- databricks-unity-catalog - Manage the catalogs and tables that back Delta Sync indexes
- databricks-pipelines - Build Delta tables used as AI Search sources