Polars Python
Produce version-grounded Polars code whose object types, row grain, schema,
missing-value policy, ordering, and execution boundary are deliberate and
tested.
Boundary
Use this skill when the project already uses Polars or the user explicitly
requests Polars or a pandas-to-Polars migration. Polars SQL is in scope only
inside a Polars pipeline. Do not introduce Polars into a library-neutral,
pandas-only, PySpark, Rust, standalone-SQL, Cloud, On-Prem, distributed, or GPU
task. Preserve the caller's public return type unless the task changes it.
Know the objects before choosing an API
| Object |
Runtime meaning |
Use it for |
DataFrame |
An eager, materialized two-dimensional table of uniquely named, equal-length typed columns. |
In-memory work, an eager interface, or a collected result whose final table fits memory. |
Series |
One eager, materialized, named one-dimensional column with one dtype. |
A one-column input/output boundary. |
Expr |
An unevaluated recipe for producing values from columns or literals; it is not data. |
Reusable computation inside an expression context. |
LazyFrame |
A logical query plan that produces a table only when executed. |
File-backed, multi-stage, large, or optimization-sensitive work. |
Schema |
The ordered mapping from column names to dtypes. |
The table interface contract. |
| Selector |
A schema-aware expression expander such as cs.numeric(). |
Apply one rule to columns selected by name or dtype. |
DataFrame columns are Series. Use frame methods to change table shape and
Expr objects to define column computations or predicates inside those methods.
Eager and lazy frames share most expression APIs: eager contexts execute now,
while lazy contexts extend a plan. GroupBy and LazyGroupBy are builders, not
result frames; finish them with a terminal group operation, normally .agg(...).
Polars has no pandas-style semantic row index—keep record labels as columns.
Read the object and shape model whenever object
type, dimensionality, positional selection, dtype namespaces, or return shape
is uncertain.
Ordered workflow
- Recover the contract from the request, callers, schema, and tests: public
return object, row grain, keys, dtypes, missing values, and required order.
- Classify the operation's shape effect, then choose its context or API family.
- Choose eager, lazy, streaming, or sink execution from the real input and
consumer boundaries.
- Apply the relevant schema, join, time-series, reshape, migration, or UDF rule.
- Keep native expressions in the pipeline; in lazy work, materialize only at a
required consumer or unsupported-operation boundary.
- Add the smallest falsifier: typed empty input, null/
NaN, duplicate keys,
order permutation, ambiguous rows, or dirty input.
- Run targeted tests. Inspect a plan or measure scale only for a performance
claim.
Rigorous analytical contract
Separate four stages and report which ones actually ran:
| Stage |
Required output |
| Frame |
Population, row grain, keys, metric formulas, units, missingness, and required order. |
| Transform |
Input/output objects, shape change, schema, execution boundary, and failure policy. |
| Validate |
Cardinality, reconciliation, invariants, adversarial fixtures, and execution evidence. |
| Interpret |
Evidence-supported result plus unresolved semantic, data, or numerical risk. |
Do not infer semantics from column names alone. Distinguish row count, entity
count, non-null count, numerator, denominator, and units before aggregation.
Compute a rate from aggregated numerator and denominator unless the contract
specifically defines a justified weighted alternative; never average row or
subgroup rates merely because they are available.
For consequential transformations, use at least one independent falsifier:
input-order permutation, batch partition/recombination for row-local work,
component-to-total reconciliation, eager/lazy equivalence, or a second query
formulation. Passing execution is not validation. Read rigorous Polars
practice for the full checklist and
evaluated anchors.
Choose by intent and output shape
| Required result |
Use |
Shape contract |
| Only selected/computed columns |
select |
Only requested outputs; expression lengths must be compatible. |
| Original columns plus additions/replacements |
with_columns |
Input height and unspecified columns remain. |
| Rows satisfying a predicate |
filter |
Same columns; only predicate True survives. |
| One materialized column |
get_column |
Returns Series; select("x") instead returns a one-column frame. |
| One row per group |
group_by(...).agg(...) |
Output grain is distinct group keys. |
| Group result aligned to every row |
expression .over(...) |
Default group_to_rows mapping preserves input grain; explode is shape-changing. |
| Rows chosen by position |
slice, head, tail, or gather |
Positions, not labels, define the result. |
| Records stacked by schema |
pl.concat |
Schema compatibility or union policy is explicit. |
| Tables combined by keys |
join |
Cardinality follows key multiplicity and join type. |
| Nested elements/fields expanded |
explode / unnest |
Row or column grain changes and is tested. |
| Columns converted to/from rows |
unpivot / pivot |
Identifier grain and generated-column policy are explicit. |
Read the DataFrame operation map for
construction, inspection, selectors, ordinary transformations, sorting,
deduplication, reshaping, and export.
Canonical object flow
import polars as pl
def normalized_email(column: str) -> pl.Expr:
return pl.col(column).str.strip_chars().str.to_lowercase()
def enrich(frame: pl.DataFrame) -> pl.DataFrame:
return (
frame.with_columns(
normalized_email("email").alias("email"),
net=pl.col("quantity") * pl.col("unit_price"),
)
.with_columns(tax=pl.col("net") * pl.lit(0.19))
.filter(pl.col("net").is_not_null())
.select("order_id", "email", "net", "tax")
)
The helper returns a symbolic Expr, so the caller chooses eager or lazy
execution. The second with_columns is required because sibling expressions
see the context's input schema, not aliases created by siblings. pl.lit
unambiguously represents a value; strings in expression-input positions often
mean column names.
Execution boundary
- Return
LazyFrame when the public contract requires a plan; never collect
inside that function or collect and re-lazify to imitate laziness.
- For file-backed or multi-stage work whose final table fits memory, start with
scan_*, keep one plan, and collect() once at the eager consumer.
- For a file or batch consumer, use an installed
sink_* or batch API when it
avoids materializing an oversized final DataFrame.
- For a small existing
DataFrame with an eager consumer, eager expressions
are valid; do not add .lazy().collect() ceremonially.
- If a required operation has no usable lazy form, materialize immediately
before it only when the public contract permits an eager result. Record the
optimization barrier.
- A streaming collect still materializes its final
DataFrame; it only can
bound intermediate memory. A LazyFrame is a plan, not a cached result.
Read lazy execution and performance before making
an engine, streaming, caching, plan, or memory claim.
High-risk semantic rules
Expressions and schema
- Use parenthesized predicates with
&, |, and ~, never Python boolean
operators on expressions. Every conditional branch must be independently
valid.
- Alias derived scalar outputs. Selector and dtype expressions can expand to
zero, one, or many columns; test expected names.
- Choose
.str, .dt, .list, .arr, .struct, .cat, or .bin from the
dtype rather than converting values to Python.
pl.len() counts rows; expression .count() counts non-null values; a bare
grouped column produces a List.
- Pin constructor or scan schemas when identifiers, temporal/nested values,
empty inputs, or late dirty rows make inference unsafe. For ambiguous
row-oriented constructor data, pass
orient="row".
- Keep casts strict unless invalid values becoming null is the declared policy,
then test or count introduced nulls. Null and floating
NaN are distinct.
Read expression and shape rules and schema and
missing-data rules for these branches.
Relational, ordered, and shape-changing work
Before a join, state preserved rows, key domains/dtypes, uniqueness on both
sides, null-key behavior, output-key behavior, and order. Validate known
cardinality; otherwise prove uniqueness on the constrained side. Never cast
keys unless both represent the same domain and conversion is lossless. Read
join rules.
After an important join, measure result rows, null fact keys, unmatched
non-null fact rows, unused dimension keys, and any conserved measure required
by the analysis. A cardinality declaration proves multiplicity, not semantic
coverage or reconciliation.
Do not rely on undocumented output order from group, join, unique, pivot, or
unpivot. Preserve input order only where the chosen API guarantees it; otherwise
encode the sort keys and tie-breakers that define a survivor, list, cumulative
result, or final table. Read join rules for as-of joins and
time-series rules for rolling, dynamic grouping,
calendar durations, and time zones.
Python escape hatch
Prefer native expressions, selectors, typed namespaces, folds, windows, joins,
and reshapes. If they cannot express the operation, choose the narrowest
installed batch UDF before scalar map_elements; use a whole-frame UDF only
under an explicit schema contract. Declare dtype/schema, purity, null/exception
behavior, and empty/all-null behavior. Claim streamability only if arbitrary
batch boundaries cannot change results. Read Python UDF
contracts before using any callback.
Version grounding and completion
Inspect the installed version when a signature, keyword, dtype, engine,
warning, or capability can drift. Run python scripts/inspect_polars.py from
the installed skill directory for JSON evidence and read API
grounding. Do not copy stale syntax from a prompt.
Translate pandas semantics rather than spellings; read pandas
migration before a migration. Use Polars
testing helpers and typed expected frames; read testing Polars
behavior.
Do not declare completion until the return object, grain, schema, cardinality,
missing-value policy, and required order match the contract; the relevant
dirty/empty/null/duplicate/permutation fixtures pass; no accidental
materialization, unjustified Python row path, arbitrary key cast, or stale API
remains; aggregate components reconcile where required; validation status says
what executed versus what was only generated; and project checks pass or
skipped evidence and its consequence are reported.
References
- Core solution recipes
- Lazy and temporal recipes
- Rigorous analytical practice and recipes
- Object and shape model
- DataFrame operation map
- Expression and shape rules
- Schema and missing-data rules
- Join rules
- Time-series rules
- Lazy execution and performance
- Python UDF contracts
- Pandas migration
- Testing Polars behavior
- API grounding
1---2name: polars-python3description: Use for writing, reviewing, debugging, testing, or optimizing Python Polars DataFrame, Series, LazyFrame, Expr, selector, and I/O code, or for an explicitly requested pandas-to-Polars migration. Trigger on table construction, operation selection, lazy plans, expression-context mistakes, schema or dtype failures, joins, nested columns, missing values, and Polars performance work. Do not use for pandas-only, PySpark, Rust Polars, standalone SQL unrelated to Polars, library-neutral data tasks, Polars Cloud or On-Prem, distributed Polars, or GPU execution.4argument-hint: "[Polars task, code, error, or migration]"5metadata:6 source: https://skillmd.com/skills/schattenspiegel/polars-python7---89# Polars Python1011Produce version-grounded Polars code whose object types, row grain, schema,12missing-value policy, ordering, and execution boundary are deliberate and13tested.1415## Boundary1617Use this skill when the project already uses Polars or the user explicitly18requests Polars or a pandas-to-Polars migration. Polars SQL is in scope only19inside a Polars pipeline. Do not introduce Polars into a library-neutral,20pandas-only, PySpark, Rust, standalone-SQL, Cloud, On-Prem, distributed, or GPU21task. Preserve the caller's public return type unless the task changes it.2223## Know the objects before choosing an API2425| Object | Runtime meaning | Use it for |26|---|---|---|27| `DataFrame` | An eager, materialized two-dimensional table of uniquely named, equal-length typed columns. | In-memory work, an eager interface, or a collected result whose final table fits memory. |28| `Series` | One eager, materialized, named one-dimensional column with one dtype. | A one-column input/output boundary. |29| `Expr` | An unevaluated recipe for producing values from columns or literals; it is not data. | Reusable computation inside an expression context. |30| `LazyFrame` | A logical query plan that produces a table only when executed. | File-backed, multi-stage, large, or optimization-sensitive work. |31| `Schema` | The ordered mapping from column names to dtypes. | The table interface contract. |32| Selector | A schema-aware expression expander such as `cs.numeric()`. | Apply one rule to columns selected by name or dtype. |3334`DataFrame` columns are `Series`. Use frame methods to change table shape and35`Expr` objects to define column computations or predicates inside those methods.36Eager and lazy frames share most expression APIs: eager contexts execute now,37while lazy contexts extend a plan. `GroupBy` and `LazyGroupBy` are builders, not38result frames; finish them with a terminal group operation, normally `.agg(...)`.39Polars has no pandas-style semantic row index—keep record labels as columns.4041Read [the object and shape model](references/object-model.md) whenever object42type, dimensionality, positional selection, dtype namespaces, or return shape43is uncertain.4445## Ordered workflow46471. Recover the contract from the request, callers, schema, and tests: public48 return object, row grain, keys, dtypes, missing values, and required order.492. Classify the operation's shape effect, then choose its context or API family.503. Choose eager, lazy, streaming, or sink execution from the real input and51 consumer boundaries.524. Apply the relevant schema, join, time-series, reshape, migration, or UDF rule.535. Keep native expressions in the pipeline; in lazy work, materialize only at a54 required consumer or unsupported-operation boundary.556. Add the smallest falsifier: typed empty input, null/`NaN`, duplicate keys,56 order permutation, ambiguous rows, or dirty input.577. Run targeted tests. Inspect a plan or measure scale only for a performance58 claim.5960## Rigorous analytical contract6162Separate four stages and report which ones actually ran:6364| Stage | Required output |65|---|---|66| Frame | Population, row grain, keys, metric formulas, units, missingness, and required order. |67| Transform | Input/output objects, shape change, schema, execution boundary, and failure policy. |68| Validate | Cardinality, reconciliation, invariants, adversarial fixtures, and execution evidence. |69| Interpret | Evidence-supported result plus unresolved semantic, data, or numerical risk. |7071Do not infer semantics from column names alone. Distinguish row count, entity72count, non-null count, numerator, denominator, and units before aggregation.73Compute a rate from aggregated numerator and denominator unless the contract74specifically defines a justified weighted alternative; never average row or75subgroup rates merely because they are available.7677For consequential transformations, use at least one independent falsifier:78input-order permutation, batch partition/recombination for row-local work,79component-to-total reconciliation, eager/lazy equivalence, or a second query80formulation. Passing execution is not validation. Read [rigorous Polars81practice](references/recipes-rigorous-analysis.md) for the full checklist and82evaluated anchors.8384## Choose by intent and output shape8586| Required result | Use | Shape contract |87|---|---|---|88| Only selected/computed columns | `select` | Only requested outputs; expression lengths must be compatible. |89| Original columns plus additions/replacements | `with_columns` | Input height and unspecified columns remain. |90| Rows satisfying a predicate | `filter` | Same columns; only predicate `True` survives. |91| One materialized column | `get_column` | Returns `Series`; `select("x")` instead returns a one-column frame. |92| One row per group | `group_by(...).agg(...)` | Output grain is distinct group keys. |93| Group result aligned to every row | expression `.over(...)` | Default `group_to_rows` mapping preserves input grain; `explode` is shape-changing. |94| Rows chosen by position | `slice`, `head`, `tail`, or `gather` | Positions, not labels, define the result. |95| Records stacked by schema | `pl.concat` | Schema compatibility or union policy is explicit. |96| Tables combined by keys | `join` | Cardinality follows key multiplicity and join type. |97| Nested elements/fields expanded | `explode` / `unnest` | Row or column grain changes and is tested. |98| Columns converted to/from rows | `unpivot` / `pivot` | Identifier grain and generated-column policy are explicit. |99100Read [the DataFrame operation map](references/dataframe-operations.md) for101construction, inspection, selectors, ordinary transformations, sorting,102deduplication, reshaping, and export.103104## Canonical object flow105106```python107import polars as pl108109110def normalized_email(column: str) -> pl.Expr:111 return pl.col(column).str.strip_chars().str.to_lowercase()112113114def enrich(frame: pl.DataFrame) -> pl.DataFrame:115 return (116 frame.with_columns(117 normalized_email("email").alias("email"),118 net=pl.col("quantity") * pl.col("unit_price"),119 )120 .with_columns(tax=pl.col("net") * pl.lit(0.19))121 .filter(pl.col("net").is_not_null())122 .select("order_id", "email", "net", "tax")123 )124```125126The helper returns a symbolic `Expr`, so the caller chooses eager or lazy127execution. The second `with_columns` is required because sibling expressions128see the context's input schema, not aliases created by siblings. `pl.lit`129unambiguously represents a value; strings in expression-input positions often130mean column names.131132## Execution boundary133134- Return `LazyFrame` when the public contract requires a plan; never collect135 inside that function or collect and re-lazify to imitate laziness.136- For file-backed or multi-stage work whose final table fits memory, start with137 `scan_*`, keep one plan, and `collect()` once at the eager consumer.138- For a file or batch consumer, use an installed `sink_*` or batch API when it139 avoids materializing an oversized final `DataFrame`.140- For a small existing `DataFrame` with an eager consumer, eager expressions141 are valid; do not add `.lazy().collect()` ceremonially.142- If a required operation has no usable lazy form, materialize immediately143 before it only when the public contract permits an eager result. Record the144 optimization barrier.145- A streaming collect still materializes its final `DataFrame`; it only can146 bound intermediate memory. A `LazyFrame` is a plan, not a cached result.147148Read [lazy execution and performance](references/performance.md) before making149an engine, streaming, caching, plan, or memory claim.150151## High-risk semantic rules152153### Expressions and schema154155- Use parenthesized predicates with `&`, `|`, and `~`, never Python boolean156 operators on expressions. Every conditional branch must be independently157 valid.158- Alias derived scalar outputs. Selector and dtype expressions can expand to159 zero, one, or many columns; test expected names.160- Choose `.str`, `.dt`, `.list`, `.arr`, `.struct`, `.cat`, or `.bin` from the161 dtype rather than converting values to Python.162- `pl.len()` counts rows; expression `.count()` counts non-null values; a bare163 grouped column produces a `List`.164- Pin constructor or scan schemas when identifiers, temporal/nested values,165 empty inputs, or late dirty rows make inference unsafe. For ambiguous166 row-oriented constructor data, pass `orient="row"`.167- Keep casts strict unless invalid values becoming null is the declared policy,168 then test or count introduced nulls. Null and floating `NaN` are distinct.169170Read [expression and shape rules](references/expressions.md) and [schema and171missing-data rules](references/schema-missing.md) for these branches.172173### Relational, ordered, and shape-changing work174175Before a join, state preserved rows, key domains/dtypes, uniqueness on both176sides, null-key behavior, output-key behavior, and order. Validate known177cardinality; otherwise prove uniqueness on the constrained side. Never cast178keys unless both represent the same domain and conversion is lossless. Read179[join rules](references/joins.md).180181After an important join, measure result rows, null fact keys, unmatched182non-null fact rows, unused dimension keys, and any conserved measure required183by the analysis. A cardinality declaration proves multiplicity, not semantic184coverage or reconciliation.185186Do not rely on undocumented output order from group, join, unique, pivot, or187unpivot. Preserve input order only where the chosen API guarantees it; otherwise188encode the sort keys and tie-breakers that define a survivor, list, cumulative189result, or final table. Read [join rules](references/joins.md) for as-of joins and190[time-series rules](references/time-series.md) for rolling, dynamic grouping,191calendar durations, and time zones.192193### Python escape hatch194195Prefer native expressions, selectors, typed namespaces, folds, windows, joins,196and reshapes. If they cannot express the operation, choose the narrowest197installed batch UDF before scalar `map_elements`; use a whole-frame UDF only198under an explicit schema contract. Declare dtype/schema, purity, null/exception199behavior, and empty/all-null behavior. Claim streamability only if arbitrary200batch boundaries cannot change results. Read [Python UDF201contracts](references/udf-contracts.md) before using any callback.202203## Version grounding and completion204205Inspect the installed version when a signature, keyword, dtype, engine,206warning, or capability can drift. Run `python scripts/inspect_polars.py` from207the installed skill directory for JSON evidence and read [API208grounding](references/api-grounding.md). Do not copy stale syntax from a prompt.209210Translate pandas semantics rather than spellings; read [pandas211migration](references/pandas-migration.md) before a migration. Use Polars212testing helpers and typed expected frames; read [testing Polars213behavior](references/testing.md).214215Do not declare completion until the return object, grain, schema, cardinality,216missing-value policy, and required order match the contract; the relevant217dirty/empty/null/duplicate/permutation fixtures pass; no accidental218materialization, unjustified Python row path, arbitrary key cast, or stale API219remains; aggregate components reconcile where required; validation status says220what executed versus what was only generated; and project checks pass or221skipped evidence and its consequence are reported.222223## References224225- [Core solution recipes](references/recipes-core.md)226- [Lazy and temporal recipes](references/recipes-scale.md)227- [Rigorous analytical practice and recipes](references/recipes-rigorous-analysis.md)228- [Object and shape model](references/object-model.md)229- [DataFrame operation map](references/dataframe-operations.md)230- [Expression and shape rules](references/expressions.md)231- [Schema and missing-data rules](references/schema-missing.md)232- [Join rules](references/joins.md)233- [Time-series rules](references/time-series.md)234- [Lazy execution and performance](references/performance.md)235- [Python UDF contracts](references/udf-contracts.md)236- [Pandas migration](references/pandas-migration.md)237- [Testing Polars behavior](references/testing.md)238- [API grounding](references/api-grounding.md)