Building LanceDB Pipelines
Use this skill to produce LanceDB pipelines that are portable between local and remote tables (for LanceDB Enterprise/Cloud) and idiomatic for the selected SDK.
LanceDB Table Modes
LanceDB has two common execution modes:
- Local table: embedded, open source, in-process LanceDB. The client opens data from a local path or object storage URI and executes queries in the application process.
- Remote table: LanceDB Enterprise/Cloud table opened through a
db://... URI. The data may be very large, commonly backed by object storage, and queried through a remote service.
Do NOT assume local-only table helpers exist on remote tables. If the user asks for LanceDB Enterprise, Cloud, db://..., production remote access, or a remote table, focus on the remote table path: use search() / query(), keep reads bounded with select() and limit(), and avoid table-level full materialization APIs.
Workflow
- Identify the SDK: Python, TypeScript, or both.
- Identify the table mode: local/embedded OSS, remote Enterprise/Cloud, or portable across both. If the user says "LanceDB Enterprise", choose the remote table path.
- Read the matching language branch before writing or changing code:
- Python patterns:
references/python/patterns.md
- Python API quick reference:
references/python/api_reference.md
- Python performance guidance:
references/python/performance.md
- TypeScript patterns:
references/typescript/patterns.md
- TypeScript API quick reference:
references/typescript/api_reference.md
- TypeScript performance guidance:
references/typescript/performance.md
- Start with
patterns.md for the selected SDK. Read api_reference.md when choosing method names or return collectors. Read performance.md when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets.
- Prefer
search() or query() builders with explicit select() and limit() for reads.
- Avoid table-level full materialization in remote or portable code. This is the main local-vs-remote pitfall.
- If reviewing an existing file or repo, run
scripts/check_materialization.py on the relevant paths and inspect each finding before editing.
- Cross-check unfamiliar or non-trivial API claims against the source tree instead of relying on memory.
Core Portability Rule
Do not write code that assumes a local table API will exist on a remote table. Remote tables can be very large, so whole-table materialization helpers are intentionally unavailable or unsafe.
This does not mean result conversion is forbidden. Bounded query/search result collection is normal:
- Python:
table.search(...).select([...]).limit(10).to_pandas()
- TypeScript:
await table.search(...).select([...]).limit(10).toArray()
The unsafe pattern is table-level or unbounded collection, plus local-only dataset escape hatches in remote code:
- Python:
table.to_pandas(), table.to_arrow(), table.to_polars(); table.to_lance() is local/OSS-only dataset access, not materialization
- TypeScript:
await table.toArrow(), await table.query().toArray() without limit()
Script
Run the scanner when reviewing or modifying an existing codebase:
python skills/lancedb/scripts/check_materialization.py path/to/file_or_dir
The script reports likely unsafe full-table materialization in Python and TypeScript. Treat results as review prompts, not automatic proof of a bug.
1---2name: lancedb-23description: Use when writing, reviewing, debugging, or documenting LanceDB pipelines in Python or TypeScript, especially code that should work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables. Helps avoid non-portable full-table materialization, choose idiomatic query/search patterns, and apply LanceDB performance defaults for ingestion, indexing, filtering, and diagnostics.4---56# Building LanceDB Pipelines78Use this skill to produce LanceDB pipelines that are portable between local and remote tables (for LanceDB Enterprise/Cloud) and idiomatic for the selected SDK.910## LanceDB Table Modes1112LanceDB has two common execution modes:1314- **Local table**: embedded, open source, in-process LanceDB. The client opens data from a local path or object storage URI and executes queries in the application process.15- **Remote table**: LanceDB Enterprise/Cloud table opened through a `db://...` URI. The data may be very large, commonly backed by object storage, and queried through a remote service.1617Do NOT assume local-only table helpers exist on remote tables. If the user asks for LanceDB Enterprise, Cloud, `db://...`, production remote access, or a remote table, focus on the remote table path: use `search()` / `query()`, keep reads bounded with `select()` and `limit()`, and avoid table-level full materialization APIs.1819## Workflow20211. Identify the SDK: Python, TypeScript, or both.222. Identify the table mode: local/embedded OSS, remote Enterprise/Cloud, or portable across both. If the user says "LanceDB Enterprise", choose the remote table path.233. Read the matching language branch before writing or changing code:24 - Python patterns: `references/python/patterns.md`25 - Python API quick reference: `references/python/api_reference.md`26 - Python performance guidance: `references/python/performance.md`27 - TypeScript patterns: `references/typescript/patterns.md`28 - TypeScript API quick reference: `references/typescript/api_reference.md`29 - TypeScript performance guidance: `references/typescript/performance.md`304. Start with `patterns.md` for the selected SDK. Read `api_reference.md` when choosing method names or return collectors. Read `performance.md` when the task involves ingestion, indexing, filtering, query tuning, diagnostics, or large datasets.315. Prefer `search()` or `query()` builders with explicit `select()` and `limit()` for reads.326. Avoid table-level full materialization in remote or portable code. This is the main local-vs-remote pitfall.337. If reviewing an existing file or repo, run `scripts/check_materialization.py` on the relevant paths and inspect each finding before editing.348. Cross-check unfamiliar or non-trivial API claims against the source tree instead of relying on memory.3536## Core Portability Rule3738Do not write code that assumes a local table API will exist on a remote table. Remote tables can be very large, so whole-table materialization helpers are intentionally unavailable or unsafe.3940This does **not** mean result conversion is forbidden. Bounded query/search result collection is normal:4142- Python: `table.search(...).select([...]).limit(10).to_pandas()`43- TypeScript: `await table.search(...).select([...]).limit(10).toArray()`4445The unsafe pattern is table-level or unbounded collection, plus local-only dataset escape hatches in remote code:4647- Python: `table.to_pandas()`, `table.to_arrow()`, `table.to_polars()`; `table.to_lance()` is local/OSS-only dataset access, not materialization48- TypeScript: `await table.toArrow()`, `await table.query().toArray()` without `limit()`4950## Script5152Run the scanner when reviewing or modifying an existing codebase:5354```bash55python skills/lancedb/scripts/check_materialization.py path/to/file_or_dir56```5758The script reports likely unsafe full-table materialization in Python and TypeScript. Treat results as review prompts, not automatic proof of a bug.
lancedb