# Duckdb Polars Boundaries

> Guides the choice between DuckDB and Polars for each stage of an analytical pipeline, covering Arrow transfer, lazy versus eager execution, registration lifetime, schema conversion, and result ownership.

- Skill: `schattenspiegel/duckdb-polars-boundaries` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds add schattenspiegel/duckdb-polars-boundaries`
- Raw SKILL.md: https://api.skillmd.com/api/skills/schattenspiegel/duckdb-polars-boundaries/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics, Data Analysis, SQL & Databases
- Tags: Arrow, Data Pipeline, Dataframe, Duckdb, Parquet, Polars, Sql
- Author: schattenspiegel (https://skillmd.com/u/schattenspiegel)
- Updated: 2026-08-22
- Page: https://skillmd.com/skills/schattenspiegel/duckdb-polars-boundaries

---


# DuckDB and Polars execution boundary

Choose one engine to own each stage. DuckDB owns relational SQL, multi-source
joins, and file scans that SQL expresses clearly. Polars owns reusable typed
expression pipelines and dataframe-native transformations. Cross once at a
deliberate consumer boundary rather than alternating after every operation.

## Workflow

1. Inspect DuckDB and Polars versions, source size/format, desired output grain,
   schema, null/timestamp/nested types, ordering, and final consumer.
2. Decide whether files remain the shared boundary or whether an in-memory
   Arrow-compatible transfer is justified. Prefer Parquet for reusable or
   process-crossing artifacts.
3. Keep source objects alive while DuckDB replacement scans or registered views
   refer to them. Use explicit registration when lifetime or naming must be
   obvious; unregister when a long-lived connection would retain stale state.
4. Materialize exactly once when crossing to an eager consumer. Do not call
   `.collect()`, `.pl()`, `.arrow()`, or dataframe conversion repeatedly in a
   loop merely to continue the pipeline in the other engine.
5. Validate row count, grain, column names/order, dtypes, nulls, timestamps,
   nested values, dictionary/categorical semantics, and ordering after transfer.

## Decision rules

- If DuckDB SQL can scan the original files and produce the final relational
  result, keep the work there and convert only for the Polars consumer.
- If Polars transformation is the reusable product, scan/transform in Polars
  and expose Arrow/Parquet to DuckDB only for a SQL stage that adds value.
- Treat conversion as potentially copying or coercing unless current APIs and
  buffers prove otherwise. Zero-copy is an optimization claim, not a default.
- SQL result order is undefined without `ORDER BY`; dataframe transfer does not
  create a durable ordering guarantee.

```python
import duckdb
import polars as pl

source = pl.scan_parquet("events/*.parquet")
prepared = source.filter(pl.col("status") == "ok").collect()

with duckdb.connect() as connection:
    connection.register("prepared", prepared)
    result = connection.sql(
        "select customer_id, count(*) as n from prepared group by customer_id"
    ).pl()
```

Verify the installed conversion APIs and add a round-trip contract test. Read
[ownership and lifetime](references/ownership.md), [schema transfer](references/schema.md),
and [verification](references/verification.md).