# Polars Pyarrow Boundaries

> ---

- Skill: `schattenspiegel/polars-pyarrow-boundaries` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds add schattenspiegel/polars-pyarrow-boundaries`
- Raw SKILL.md: https://api.skillmd.com/api/skills/schattenspiegel/polars-pyarrow-boundaries/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: schattenspiegel (https://skillmd.com/u/schattenspiegel)
- Updated: 2026-08-19
- Page: https://skillmd.com/skills/schattenspiegel/polars-pyarrow-boundaries

---

---
        name: polars-pyarrow-boundaries
        description: >-
          Use when Python code crosses between Polars and PyArrow tables, arrays, schemas, record batches, Parquet, or Arrow C/Data interfaces, especially for timestamp, dictionary, nested, nullability, chunking, and zero-copy contracts. Do not use for Polars-only transformations or PyArrow-only dataset work with no interchange.
        argument-hint: "[Polars and PyArrow Boundaries task, code, contract, or failure]"
        ---

        # Polars and PyArrow Boundaries

        Use the smallest explicit execution contract that preserves semantics. Inspect
        the installed version before relying on a drifting signature. State ownership,
        input and output shapes, ordering, failure behavior, and the verification command
        before writing substantial code.

        ## Object and execution model

        | Object | Meaning | Boundary |
        |---|---|---|
        | `Polars DataFrame/Series` | Materialized columnar values with Polars dtypes. | Conversion can rechunk, normalize, or reject Arrow representations. |
| `Arrow Table/Array` | Immutable chunked columnar values plus Arrow schema. | Chunking, offsets, dictionary encoding, and metadata are observable. |
| `schema` | Field names, logical types, nullability, and metadata. | Value equality alone cannot prove a boundary contract. |
| `buffer ownership` | The lifetime and sharing of underlying memory. | Zero-copy is conditional and must not be promised without buffer evidence. |
| `storage boundary` | Parquet/IPC or consumer protocol after conversion. | Test the final consumer semantics, not just an intermediate type. |

        Read [the object model](references/object-model.md) when the task mixes two
        objects or crosses an execution boundary.

        ## Workflow

        1. Inspect both installed versions and source/target schemas including field metadata.
2. Classify each dtype as exact, safely normalizable, lossy, unsupported, or version-sensitive.
3. Convert at one named boundary and avoid Python row objects.
4. Verify values, nulls, order, dtype/unit/timezone, nested shape, and consumer behavior.
5. Measure buffer sharing separately if zero-copy affects performance requirements.

        ## Decision rules

        - Declare names, order, Arrow and Polars dtypes, nullability, timezone, precision, nested shape, and metadata requirements before conversion.
- Use `pl.from_arrow` and `.to_arrow()` as explicit boundaries, then compare schemas and representative null/value behavior.
- Treat zero-copy as an optimization requiring compatible representation, stable ownership, and buffer-level evidence; correctness must not depend on it.
- Preserve timezone and timestamp unit deliberately; Python-object conversion can truncate nanoseconds and erase Arrow distinctions.
- Decide whether dictionary encoding and chunk boundaries are semantic, performance-only, or allowed to normalize.
- Version-ground unstable or evolving nested and extension-type behavior against both installed libraries.

        If a required fact is unknown, inspect the target code, installed signature,
        schema, shape, or lifecycle owner. Do not replace an unknown with a permissive
        fallback. See [the decision guide](references/decision-guide.md).

        ## Complex solution routes

        Load only the matching section of
        [the evaluated recipes](references/recipes-solutions.md):

        - `boundary.timestamp-schema-roundtrip` and `boundary.verify-nanosecond-timezone-values`: preserve timestamp units, timezone, nulls, and nanoseconds.
- `boundary.chunk-normalization` and `boundary.verify-nullability-and-values`: normalize chunks while preserving typed nullable values.
- `boundary.dictionary-nested-policy` and `boundary.verify-decoded-and-nested-values`: cross categorical and nested columns without silent semantic loss.

        Recipes are anchors, not blind templates. Preserve their named invariants and
        adapt types and names only after inspecting the actual boundary.

        ## Verification contract

        - Test observable behavior, not the presence of API tokens.
        - Exercise empty, singleton, malformed, and failure inputs when the operation
          accepts them.
        - Assert shape, dtype or type, ordering, ownership, and error semantics where
          they are part of the contract.
        - Keep external I/O deterministic with injected clocks, transports, processes,
          files, random state, or test doubles.
        - Run the narrow test first, then the relevant project suite. Do not declare
          completion when warnings, background failures, convergence flags, or cleanup
          errors remain unexplained.

        Use [the verification matrix](references/verification.md) for completion checks.

        ## Failure routing and adaptation

        Classify a failure before changing code: input-contract failures require a
        precise rejection; environment or version failures require inspection; execution
        failures require lifecycle, convergence, or cleanup evidence; invariant failures
        require a semantic correction. Do not relax a check, coerce a value, broaden a
        failure handler, or materialize data merely to make the symptom disappear.

        When adapting a recipe:

        1. Match its objects, ownership, execution timing, and output contract to the task.
        2. Preserve every branch condition and completion check while changing domain names.
        3. Add the project's real empty, malformed, duplicate, cancellation, precision, or
           boundary case before removing any guard.
        4. If the installed API differs, inspect the signature and primary documentation,
           then update implementation, test, and authoring evidence together.

        ## Version grounding

        Inspect the installed package and signature when editing an existing project.
        Treat examples here as verified anchors for the version recorded by the Foundry,
        not as permission to overwrite a repository's compatibility policy. When current
        behavior differs, preserve the project target and update tests and authoring
        evidence together.

        ## Completion

        Complete the task only when the implementation preserves the declared object
        model, no accidental materialization or lifetime extension was introduced, all
        failures are surfaced at the correct boundary, and deterministic tests prove the
        critical behavior. Report any environment or version fact that could not be
        verified.