# Databricks Vector Search

> Databricks AI Search (formerly Vector Search) endpoints and indexes for RAG and semantic search. Use when asked to: create Standard or Storage-Optimized endpoints and Delta Sync or Direct Access indexes with AISearchClient; query with ANN, hybrid, or full-text search and filters; sync, monitor, or migrate indexes; set a target QPS, triage 429s, or load-test an endpoint; wire an index into an agent as a retriever tool. NOT for: no-code document Q&A (databricks-genie-agents); serving-endpoint CRUD (databricks-model-serving); custom ResponsesAgent authoring and deployment (databricks-ml-training).

- Skill: `stanfish06/databricks-vector-search` (Agent Skill, multi-file: 8 files)
- Install (CLI): `npx get-skill add stanfish06/databricks-vector-search`
- Raw SKILL.md: https://api.skillmd.com/api/skills/stanfish06/databricks-vector-search/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: stanfish06 (https://skillmd.com/u/stanfish06)
- Updated: 2026-11-12
- Page: https://skillmd.com/skills/stanfish06/databricks-vector-search

---


# Databricks AI Search (formerly Vector Search)

**FIRST**: Use the parent `databricks-core` skill for CLI basics, authentication, and profile selection.

Patterns for creating, managing, and querying AI Search indexes for RAG and semantic search applications.

**Naming:** the REST API (`/api/2.0/vector-search/...`), the `databricks vector-search-*` CLI groups, and the `databricks_vector_search_*` Terraform resources keep the Vector Search names ([API and tool names](https://docs.databricks.com/aws/en/ai-search/ai-search#vector-search-names)); the Python client is `databricks-ai-search` (`AISearchClient`).

## When to Use

Use this skill when:
- Building RAG (Retrieval-Augmented Generation) applications
- Implementing semantic search or similarity matching
- Creating vector indexes from Delta tables
- Choosing between storage-optimized and standard endpoints
- Querying vector indexes with filters

## Overview

Databricks AI Search provides managed vector similarity search with automatic embedding generation and Delta Lake integration.

| Component | Description |
|-----------|-------------|
| **Endpoint** | Compute resource hosting indexes (Standard or Storage-Optimized) |
| **Index** | Vector data structure for similarity search |
| **Delta Sync** | Auto-syncs with source Delta table |
| **Direct Access** | Manual CRUD operations on vectors (Standard endpoints only) |

## Endpoint Types

| Type | Latency | Capacity | Cost | Best For |
|------|---------|----------|------|----------|
| **Standard** | 20-50ms | 320M vectors (768 dim) | Higher | Real-time, low-latency, high QPS |
| **Storage-Optimized** | 300-500ms | 1B+ vectors (768 dim) | Up to 7x lower | 10M+ vectors, cost-sensitive |

Storage-Optimized supports only `TRIGGERED` Delta Sync indexes: no `CONTINUOUS` pipelines, no Direct Access, no `target_qps`. The endpoint type cannot be changed after creation.

## Index Types

| Type | Embeddings | Sync | Use Case |
|------|------------|------|----------|
| **Delta Sync (managed)** | Databricks computes | Auto from Delta | Easiest setup |
| **Delta Sync (self-managed)** | You provide | Auto from Delta | Custom embeddings |
| **Direct Access** | You provide | Manual CRUD | Real-time updates |

## Installation

```bash
pip install "databricks-ai-search>=0.79"
```

In a notebook, use `%pip install` instead and run `dbutils.library.restartPython()` afterwards:
the package requires `mlflow-skinny>=3.10.1`, so installing it upgrades mlflow in place.

`databricks.ai_search` first shipped in `databricks-ai-search` 0.73. This skill needs 0.79+:
from 0.79, `get_endpoint`, `list_endpoints` and `list_indexes` always return snake_case keys,
and 0.79 also includes the 0.78 fix that makes concurrent `similarity_search` calls thread-safe
(needed for production traffic). From 0.81, service principal auth no longer requires the
`all-apis` OAuth scope.

**Existing `databricks-vectorsearch` code:** on `databricks-vectorsearch` 0.74+,
`VectorSearchClient` is an alias of `AISearchClient`, so the calls in this skill work on it
once `databricks-ai-search>=0.79` is installed alongside it. On 0.73 and older it is a
separate class: switch the import to `from databricks.ai_search.client import AISearchClient`.
The old class has no start/stop calls, and before 0.68 no `target_qps`.

**Auth:** a bare `AISearchClient()` uses the notebook's credentials. Elsewhere, pass
`workspace_url` plus a `personal_access_token`, or `service_principal_client_id` and
`service_principal_client_secret` (needed for high-QPS traffic; see
[Production and High QPS](#production-and-high-qps)).

`AISearchClient` takes plain strings (`endpoint_type="STANDARD"`, `pipeline_type="TRIGGERED"`).
Don't pass the `databricks.sdk.service.vectorsearch` enums to it; those are for
`WorkspaceClient.vector_search_*`.

## Quick Start

### Create Endpoint

```python
from databricks.ai_search.client import AISearchClient

client = AISearchClient()

client.create_endpoint(
    name="my-vs-endpoint",
    endpoint_type="STANDARD"  # or "STORAGE_OPTIMIZED"
)
# Note: Endpoint creation is asynchronous; check status with get_endpoint()
```

`create_endpoint_and_wait(..., timeout=datetime.timedelta(minutes=30))` blocks until the endpoint is
`ONLINE`. It fails fast on `OFFLINE` (and, from 0.80, `STOPPING` and `STOPPED`); in other states it polls until the
timeout. On a timeout the endpoint may still be provisioning: check
`client.get_endpoint(name)["endpoint_status"]["state"]` before retrying, and don't create it again.

### Create Delta Sync Index (Managed Embeddings)

Standard endpoints need Change Data Feed on the source table:

```sql
ALTER TABLE catalog.schema.documents SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
```

```python
# Source table must have: primary key column + text column
index = client.create_delta_sync_index(
    endpoint_name="my-vs-endpoint",
    source_table_name="catalog.schema.documents",
    index_name="catalog.schema.my_index",
    pipeline_type="TRIGGERED",  # "CONTINUOUS" is Standard-only
    primary_key="id",
    embedding_source_column="content",
    embedding_model_endpoint_name="databricks-gte-large-en"
)
```

### Query Index

```python
index = client.get_index(
    endpoint_name="my-vs-endpoint",
    index_name="catalog.schema.my_index"
)

results = index.similarity_search(
    query_text="What is machine learning?",
    columns=["id", "content", "metadata"],
    num_results=5
)

# Returns a dict; each row holds the requested columns, then the similarity score
for row in results["result"]["data_array"]:
    print(row)
```

## Common Patterns

### Create Storage-Optimized Endpoint

```python
# For large-scale, cost-effective deployments
client.create_endpoint(
    name="my-storage-endpoint",
    endpoint_type="STORAGE_OPTIMIZED"
)
```

### Delta Sync with Self-Managed Embeddings

```python
# Source table must have: primary key + embedding vector column
index = client.create_delta_sync_index(
    endpoint_name="my-vs-endpoint",
    source_table_name="catalog.schema.documents",
    index_name="catalog.schema.my_index",
    pipeline_type="TRIGGERED",
    primary_key="id",
    embedding_dimension=768,
    embedding_vector_column="embedding"
)
```

### Direct Access Index (Standard endpoints only)

```python
# Create index for manual CRUD
index = client.create_direct_access_index(
    endpoint_name="my-vs-endpoint",
    index_name="catalog.schema.direct_index",
    primary_key="id",
    embedding_dimension=768,
    embedding_vector_column="embedding",
    schema={
        "id": "string",
        "text": "string",
        "embedding": "array<float>",
        "metadata": "string"
    }
)

# Upsert data
index.upsert([
    {"id": "1", "text": "Hello", "embedding": [0.1, 0.2, ...], "metadata": "doc1"},
    {"id": "2", "text": "World", "embedding": [0.3, 0.4, ...], "metadata": "doc2"},
])

# Delete data
index.delete(primary_keys=["1", "2"])
```

### Query with Embedding Vector

```python
# When you have pre-computed query embedding
results = index.similarity_search(
    query_vector=[0.1, 0.2, 0.3, ...],  # Your 768-dim vector
    columns=["id", "text"],
    num_results=10
)
```

### Hybrid Search (Semantic + Keyword)

Hybrid search runs ANN and full-text (keyword) search and merges the results with Reciprocal Rank Fusion (RRF). Use it when queries contain exact terms that must match — SKUs, error codes, proper nouns, or technical terminology — where pure semantic search might miss keyword-specific results. See [references/search-modes.md](references/search-modes.md) for detailed guidance on choosing between ANN and hybrid search.

```python
# Needs an index with a managed embedding model, so query_text alone is enough
index = client.get_index(
    endpoint_name="my-vs-endpoint",
    index_name="catalog.schema.my_index",
)

# Combines vector similarity with keyword matching
results = index.similarity_search(
    query_text="SPARK-12345 executor memory error",
    query_type="hybrid",
    columns=["id", "content"],
    num_results=10
)
```

## Filtering

Filter syntax differs by endpoint type: Standard endpoints take a dict, Storage-Optimized endpoints take a SQL-like string. On Storage-Optimized, use the string: a dict (`filters_json`) is rejected on most query routes (except filter-only lookups with no query text or vector), and other routes translate it to SQL instead, so don't rely on either behavior.

### Standard Endpoint Filters (Dictionary)

```python
results = index.similarity_search(
    query_text="machine learning",
    columns=["id", "content"],
    num_results=10,
    filters={"category": "ai", "status": ["active", "pending"]}
)
```

### Storage-Optimized Filters (SQL-like string)

```python
index = client.get_index(
    endpoint_name="my-storage-endpoint",
    index_name="catalog.schema.my_storage_index",
)

results = index.similarity_search(
    query_text="machine learning",
    columns=["id", "content"],
    num_results=10,
    filters="category = 'ai' AND status IN ('active', 'pending')"
)

# filters="price > 100 AND price < 500"
# filters="department LIKE 'eng%'"
# filters="created_at >= TO_TIMESTAMP('2024-01-01T00:00:00')"  # a bare timestamp string is rejected
```

See the [filtering guide](https://docs.databricks.com/aws/en/ai-search/filtering-guide) for operators and limitations.

Never interpolate untrusted input (tenant IDs, ACL values) into a filter as-is:
- Take column names from a fixed allowlist. In a dict filter the key also carries the operator (`"price >"`, `"col LIKE"`), so allowlist whole keys.
- Pass scalar values, validated against an allowlist or a strict pattern (for example, `^[A-Za-z0-9_-]+$`). Never let untrusted input become a list: a list value becomes an OR/IN match. In a SQL string, don't rely on quote escaping alone.
- On Storage-Optimized, `%` and `_` in a LIKE value act as wildcards, so reject them in untrusted input. (On Standard, LIKE matches whole tokens, not wildcards.)
- Filters are application-level ACLs, not a security boundary: row- and column-level permissions aren't supported, so anyone who can query the index can read every row.

In a RAG prompt, treat retrieved text as data, not instructions.

### Trigger Index Sync

```python
# TRIGGERED pipelines only, after the initial sync from creation finishes (rejected while provisioning).
# Also rejected while an update is running; wait for that update instead of retrying.
index = client.get_index(
    endpoint_name="my-vs-endpoint",
    index_name="catalog.schema.my_index"
)
index.sync()
```

### Scan Index Entries

```python
# Returns at most num_results rows, ordered by primary key. The response carries
# the cursor for the next page as last_primary_key.
page = index.scan(num_results=100)
rows = page.get("data", [])
while page.get("last_primary_key") and page.get("data"):
    page = index.scan(num_results=100, last_primary_key=page["last_primary_key"])
    rows += page.get("data", [])

# scan rows are field structs, NOT the plain lists similarity_search returns:
#   {"fields": [{"key": "id", "value": {"string_value": "r1"}}, ...]}
for row in rows:
    values = {f["key"]: next(iter(f["value"].values())) for f in row["fields"]}
```

## Production and High QPS

By default, Standard endpoints serve about 20-200 QPS, depending on index size. Traffic over a PAT or the workspace query URL is capped at a few tens of QPS.

- **Auth and route:** production traffic needs service principal OAuth and the index URL. Create `AISearchClient(workspace_url=..., service_principal_client_id=..., service_principal_client_secret=...)`, call `get_index()` once, and reuse the index object; it queries the index URL. Use a dedicated [service principal](https://docs.databricks.com/aws/en/ai-search/high-qps#service-principal). For Delta Sync indexes with managed embeddings, an IP access list or a private access settings object (on Azure, public network access disabled) keeps queries on the capped route; on GCP, a private access settings object does so for every index. Otherwise, self-managed embeddings avoid it. See [requirements](https://docs.databricks.com/aws/en/ai-search/high-qps#requirements).
  - **Pitfalls:** with any token passed as `personal_access_token` (a PAT, or a forwarded user or on-behalf-of token such as the Databricks Apps `x-forwarded-access-token`), or a bare `AISearchClient()` that picks up notebook or model-serving credentials, `get_index()` queries the workspace URL even though `describe()` lists `status.index_url`; if `index.index_url` starts with your workspace URL, queries are on the capped route. In Databricks Apps, send query traffic with the app's service principal, not the forwarded user token. `workspace_url` must include `https://`: in Databricks Apps, `DATABRICKS_HOST` can lack the scheme, so add it ([example](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/vector-search)).
- **Query type:** start with ANN; hybrid uses about 4x the resources. Load-test with the query type production sends.
- **429s:** move to service principal OAuth and the index URL; wait for `SCALING_CHANGE_APPLIED`; with `query_vector`, lower concurrency or raise `target_qps`; with `query_text` on a managed-embedding index, the embedding endpoint may be the bottleneck. `similarity_search` already retries (`total_retries`, `backoff_factor`, `backoff_jitter`); raise those or add your own bounded backoff with jitter. See [Troubleshoot 429 errors](https://docs.databricks.com/aws/en/ai-search/high-qps#troubleshoot-429-errors).
- **Load testing:** use **Run load test** on the endpoint page or the [load test notebook](https://docs.databricks.com/aws/en/ai-search/endpoint-load-test#notebook). With `databricks-ai-search` 0.82+, `databricks-ai-search guide --topic high-qps` prints the guide and `databricks-ai-search load-test --index <name>` plans a run (it runs only with `--yes`); skip this if the `databricks-ai-search` command isn't available.
- **Agent safety:** ask before creating endpoints or indexes, setting `target_qps`, running load tests, or upserting/deleting data, since each costs money or changes data. Never print, paste, or commit tokens or client secrets; read them from a secret scope or environment variables.

**Target QPS** is Standard-only and best-effort, and the capacity is billed whether or not traffic arrives. Storage-Optimized endpoints reject `target_qps`. A change can be rejected with `RESOURCE_CONFLICT` while a previous change is still applying. Nothing is saved, so `requested_target_qps` still shows the earlier target: poll until `scaling_info.state` is `SCALING_CHANGE_APPLIED` (whatever the target), or wait the minutes the error gives, then call `update_endpoint` again before running the loop below.

```python
import time

client.update_endpoint(name="my-vs-endpoint", target_qps=500)
# Or at creation: create_endpoint(..., target_qps=500)

# Match the requested target too: right after the update, state can still be APPLIED from an earlier target.
# Target QPS is best-effort: if it's still in progress at the deadline, move on.
deadline = time.monotonic() + 30 * 60
while time.monotonic() < deadline:
    info = client.get_endpoint("my-vs-endpoint").get("scaling_info") or {}
    # int(): proto3 JSON may serialize int64 fields as strings
    if info.get("state") == "SCALING_CHANGE_APPLIED" and int(info.get("requested_target_qps") or 0) == 500:
        break
    time.sleep(30)
print(info)
```

**`ONLINE` doesn't mean scaled.** These are separate waits:
- **Endpoint provisioned:** `endpoint_status.state` is `ONLINE` (what `create_endpoint_and_wait` waits for). `ONLINE` doesn't mean the target QPS is applied.
- **Target QPS applied:** `scaling_info.state` is `SCALING_CHANGE_APPLIED` for the requested target (the loop above). This takes minutes to hours, depending on index size and how much the target changes.
- **Index ready:** `status.ready` is true. A new endpoint's first index has `status.detailed_state` `PROVISIONING_ENDPOINT` while compute is created.

Load-test only after all three; a load test before `SCALING_CHANGE_APPLIED` doesn't reflect the target capacity. If the loop above hits its deadline, report that target QPS is still applying instead of load testing.

## Reference Files

| Topic | File | Description |
|-------|------|-------------|
| Index Types | [references/index-types.md](references/index-types.md) | Detailed comparison of Delta Sync (managed/self-managed) vs Direct Access |
| End-to-End RAG | [references/end-to-end-rag.md](references/end-to-end-rag.md) | Complete walkthrough: source table → endpoint → index → query → agent integration |
| Search Modes | [references/search-modes.md](references/search-modes.md) | When to use semantic (ANN) vs hybrid search, decision guide |
| Operations | [references/troubleshooting-and-operations.md](references/troubleshooting-and-operations.md) | Monitoring, cost optimization, capacity planning, migration |

## CLI Quick Reference

```bash
# List endpoints
databricks vector-search-endpoints list-endpoints

# Create endpoint (positional args: NAME ENDPOINT_TYPE)
databricks vector-search-endpoints create-endpoint my-endpoint STANDARD

# List indexes on endpoint (positional arg: ENDPOINT_NAME)
databricks vector-search-indexes list-indexes my-endpoint

# Get index status (positional arg: INDEX_NAME)
databricks vector-search-indexes get-index catalog.schema.my_index

# Sync index (positional arg: INDEX_NAME)
databricks vector-search-indexes sync-index catalog.schema.my_index

# Delete index (positional arg: INDEX_NAME)
databricks vector-search-indexes delete-index catalog.schema.my_index
```

## Common Issues

| Issue | Solution |
|-------|----------|
| **Index sync slow** | Use Storage-Optimized endpoints (10-20x faster indexing) |
| **Query latency high** | Use Standard endpoint for <100ms latency |
| **Filters not working** | Standard endpoints use a dict (`filters={"col": "val"}`); Storage-Optimized use a SQL string (`filters="col = 'val'"`) |
| **Embedding dimension mismatch** | Ensure query and index dimensions match |
| **Index not updating** | Check pipeline_type; use `index.sync()` for TRIGGERED. If it's rejected as already running, wait for the current update to finish |
| **Out of capacity** | Capacity is sized from the source table at index creation. Load the full table first; past ~320M vectors, recreate on Storage-Optimized (1B+ vectors) |
| **Rate limited (429 errors)** | See [Production and High QPS](#production-and-high-qps) |
| **`query_vector` truncated** | Large vectors (e.g. 1024-dim) can be truncated when serialized as JSON. Use `query_text` instead (for managed embedding indexes), or pass the vector directly via `index.similarity_search(query_vector=...)` |

## Embedding Models

Databricks provides built-in embedding models:

| Model | Dimensions | Context Window | Use Case |
|-------|------------|----------------|----------|
| `databricks-qwen3-embedding-0-6b` | 1024 (configurable, up to 1024) | ~32K tokens | **Recommended for production** by the AI Search docs (Public Preview) |
| `databricks-gte-large-en` | 1024 | 8192 tokens | English text, high quality |
| `databricks-bge-large-en` | 1024 | 512 tokens | English text, general purpose |

```python
# Use with managed embeddings
index = client.create_delta_sync_index(
    ...,
    embedding_source_column="content",
    embedding_model_endpoint_name="databricks-qwen3-embedding-0-6b"
)
```

## Notes

- **Storage-Optimized is newer** — better for most use cases unless you need <100ms latency, high QPS, Direct Access, or CONTINUOUS sync
- **Delta Sync recommended** — easier than Direct Access for most scenarios
- **Hybrid search** — available for both Delta Sync and Direct Access indexes
- **`columns_to_sync`** — limits which columns are synced and returned; only synced columns are available in query results. On Storage-Optimized, the docs' limitations list says it's unsupported, but their full-text example uses it; prefer syncing all columns there
- **Filter syntax differs by endpoint** — Standard uses a dict, Storage-Optimized uses a SQL-like string
- **Management vs runtime** — CLI and SDK handle lifecycle management; for agent tool-calling at runtime, use `VectorSearchRetrieverTool` from `databricks_langchain` or `databricks_openai`

## Related Skills

- **databricks-model-serving** - Deploy agents that use VectorSearchRetrieverTool
- **[databricks-ml-training](../databricks-ml-training/SKILL.md)** - Author and deploy a `ResponsesAgent` that uses an AI Search retriever
- **[databricks-genie-agents](../databricks-genie-agents/SKILL.md)** - No-code document Q&A: a Genie Agent with a UC volume (replaces the legacy Knowledge Assistant); hand-roll AI Search only for custom retrieval
- **[databricks-unstructured-pdf-generation](../databricks-unstructured-pdf-generation/SKILL.md)** - Generate documents to index in AI Search
- **[databricks-unity-catalog](../databricks-unity-catalog/SKILL.md)** - Manage the catalogs and tables that back Delta Sync indexes
- **databricks-pipelines** - Build Delta tables used as AI Search sources

