Polars Python

Polars Python

Produce version-grounded Polars code whose object types, row grain, schema, missing-value policy, ordering, and execution boundary are deliberate and tested.

Boundary

Use this skill when the project already uses Polars or the user explicitly requests Polars or a pandas-to-Polars migration. Polars SQL is in scope only inside a Polars pipeline. Do not introduce Polars into a library-neutral, pandas-only, PySpark, Rust, standalone-SQL, Cloud, On-Prem, distributed, or GPU task. Preserve the caller's public return type unless the task changes it.

Know the objects before choosing an API

DataFrame columns are Series. Use frame methods to change table shape and Expr objects to define column computations or predicates inside those methods. Eager and lazy frames share most expression APIs: eager contexts execute now, while lazy contexts extend a plan. GroupBy and LazyGroupBy are builders, not result frames; finish them with a terminal group operation, normally .agg(...). Polars has no pandas-style semantic row index—keep record labels as columns.

Read the object and shape model whenever object type, dimensionality, positional selection, dtype namespaces, or return shape is uncertain.

Ordered workflow

1. Recover the contract from the request, callers, schema, and tests: public return object, row grain, keys, dtypes, missing values, and required order.
2. Classify the operation's shape effect, then choose its context or API family.
3. Choose eager, lazy, streaming, or sink execution from the real input and consumer boundaries.
4. Apply the relevant schema, join, time-series, reshape, migration, or UDF rule.
5. Keep native expressions in the pipeline; in lazy work, materialize only at a required consumer or unsupported-operation boundary.
6. Add the smallest falsifier: typed empty input, null/NaN, duplicate keys, order permutation, ambiguous rows, or dirty input.
7. Run targeted tests. Inspect a plan or measure scale only for a performance claim.

Rigorous analytical contract

Separate four stages and report which ones actually ran:

Do not infer semantics from column names alone. Distinguish row count, entity count, non-null count, numerator, denominator, and units before aggregation. Compute a rate from aggregated numerator and denominator unless the contract specifically defines a justified weighted alternative; never average row or subgroup rates merely because they are available.

For consequential transformations, use at least one independent falsifier: input-order permutation, batch partition/recombination for row-local work, component-to-total reconciliation, eager/lazy equivalence, or a second query formulation. Passing execution is not validation. Read rigorous Polars practice for the full checklist and evaluated anchors.

Choose by intent and output shape

Read the DataFrame operation map for construction, inspection, selectors, ordinary transformations, sorting, deduplication, reshaping, and export.

Canonical object flow

import polars as pl


def normalized_email(column: str) -> pl.Expr:
    return pl.col(column).str.strip_chars().str.to_lowercase()


def enrich(frame: pl.DataFrame) -> pl.DataFrame:
    return (
        frame.with_columns(
            normalized_email("email").alias("email"),
            net=pl.col("quantity") * pl.col("unit_price"),
        )
        .with_columns(tax=pl.col("net") * pl.lit(0.19))
        .filter(pl.col("net").is_not_null())
        .select("order_id", "email", "net", "tax")
    )

The helper returns a symbolic Expr, so the caller chooses eager or lazy execution. The second with_columns is required because sibling expressions see the context's input schema, not aliases created by siblings. pl.lit unambiguously represents a value; strings in expression-input positions often mean column names.

Execution boundary

• Return LazyFrame when the public contract requires a plan; never collect inside that function or collect and re-lazify to imitate laziness.
• For file-backed or multi-stage work whose final table fits memory, start with scan_*, keep one plan, and collect() once at the eager consumer.
• For a file or batch consumer, use an installed sink_* or batch API when it avoids materializing an oversized final DataFrame.
• For a small existing DataFrame with an eager consumer, eager expressions are valid; do not add .lazy().collect() ceremonially.
• If a required operation has no usable lazy form, materialize immediately before it only when the public contract permits an eager result. Record the optimization barrier.
• A streaming collect still materializes its final DataFrame; it only can bound intermediate memory. A LazyFrame is a plan, not a cached result.

Read lazy execution and performance before making an engine, streaming, caching, plan, or memory claim.

High-risk semantic rules

Expressions and schema

• Use parenthesized predicates with &, |, and ~, never Python boolean operators on expressions. Every conditional branch must be independently valid.
• Alias derived scalar outputs. Selector and dtype expressions can expand to zero, one, or many columns; test expected names.
• Choose .str, .dt, .list, .arr, .struct, .cat, or .bin from the dtype rather than converting values to Python.
• pl.len() counts rows; expression .count() counts non-null values; a bare grouped column produces a List.
• Pin constructor or scan schemas when identifiers, temporal/nested values, empty inputs, or late dirty rows make inference unsafe. For ambiguous row-oriented constructor data, pass orient="row".
• Keep casts strict unless invalid values becoming null is the declared policy, then test or count introduced nulls. Null and floating NaN are distinct.

Read expression and shape rules and schema and missing-data rules for these branches.

Relational, ordered, and shape-changing work

Before a join, state preserved rows, key domains/dtypes, uniqueness on both sides, null-key behavior, output-key behavior, and order. Validate known cardinality; otherwise prove uniqueness on the constrained side. Never cast keys unless both represent the same domain and conversion is lossless. Read join rules.

After an important join, measure result rows, null fact keys, unmatched non-null fact rows, unused dimension keys, and any conserved measure required by the analysis. A cardinality declaration proves multiplicity, not semantic coverage or reconciliation.

Do not rely on undocumented output order from group, join, unique, pivot, or unpivot. Preserve input order only where the chosen API guarantees it; otherwise encode the sort keys and tie-breakers that define a survivor, list, cumulative result, or final table. Read join rules for as-of joins and time-series rules for rolling, dynamic grouping, calendar durations, and time zones.

Python escape hatch

Prefer native expressions, selectors, typed namespaces, folds, windows, joins, and reshapes. If they cannot express the operation, choose the narrowest installed batch UDF before scalar map_elements; use a whole-frame UDF only under an explicit schema contract. Declare dtype/schema, purity, null/exception behavior, and empty/all-null behavior. Claim streamability only if arbitrary batch boundaries cannot change results. Read Python UDF contracts before using any callback.

Version grounding and completion

Inspect the installed version when a signature, keyword, dtype, engine, warning, or capability can drift. Run python scripts/inspect_polars.py from the installed skill directory for JSON evidence and read API grounding. Do not copy stale syntax from a prompt.

Translate pandas semantics rather than spellings; read pandas migration before a migration. Use Polars testing helpers and typed expected frames; read testing Polars behavior.

Do not declare completion until the return object, grain, schema, cardinality, missing-value policy, and required order match the contract; the relevant dirty/empty/null/duplicate/permutation fixtures pass; no accidental materialization, unjustified Python row path, arbitrary key cast, or stale API remains; aggregate components reconcile where required; validation status says what executed versus what was only generated; and project checks pass or skipped evidence and its consequence are reported.

References

• Core solution recipes
• Lazy and temporal recipes
• Rigorous analytical practice and recipes
• Object and shape model
• DataFrame operation map
• Expression and shape rules
• Schema and missing-data rules
• Join rules
• Time-series rules
• Lazy execution and performance
• Python UDF contracts
• Pandas migration
• Testing Polars behavior
• API grounding