PyArrow Python
Produce schema-explicit columnar code whose logical object, batch boundary,
null behavior, memory lifetime, and materialization point are deliberate.
Boundary
Use this skill when the implementation directly uses pyarrow or must expose an
Arrow-compatible boundary. Do not introduce PyArrow merely to perform a small
Python-list or pandas transformation. Keep storage-format questions in scope
only when PyArrow reads or writes them. Arrow Flight servers, Acero internals,
and non-Python implementations require their own guidance.
Classify the object first
| Object |
Meaning |
Use it for |
DataType / Field / Schema |
Immutable type and named-field contracts, including nullability and metadata. |
Pinning a boundary before constructing or scanning data. |
Scalar |
One typed value, possibly null. |
Kernel arguments and scalar results. |
Array |
One contiguous typed column backed by buffers. |
A single physical column segment. |
ChunkedArray |
One logical column made of zero or more same-typed arrays. |
Table columns and multi-source results; chunk count is not row count. |
RecordBatch |
Equal-length arrays under one schema. |
Bounded transport or processing batches. |
Table |
A logical table whose columns may be chunked. |
Materialized tabular results that fit the required memory boundary. |
RecordBatchReader |
A schema plus a consumable stream of batches. |
Streaming interchange when the consumer can process batches once. |
Dataset |
A logical collection of fragments with a unified schema. |
Discovering and querying multi-file or partitioned data. |
Scanner |
A dataset scan with projection, filter, and batching options bound. |
Deferred dataset execution and bounded batch iteration. |
Arrays and schemas are immutable. A Table is materialized but may be
physically chunked; combine_chunks() can allocate and is not routine cleanup.
A Dataset describes sources, while a Scanner describes the read. Calling
to_table() materializes all selected rows; to_batches() preserves a batch
boundary. Read the object and memory model when
chunking, buffers, ownership, dictionaries, nested types, or zero-copy claims
matter.
Ordered workflow
- Recover the boundary contract: input object, output object, schema, field
nullability, row order if required, batch size, and ownership lifetime.
- Choose
Array, RecordBatch, Table, RecordBatchReader, Dataset, or
Scanner from that contract; do not default every job to Table.
- Pin types where empty input, identifiers, timestamps, decimals, nested data,
dictionary encoding, or cross-language interchange make inference unsafe.
- Push dataset projection and predicates into the scan. Materialize only at a
consumer that requires all rows.
- Use
pyarrow.compute kernels for Arrow data. Do not convert to Python rows
or pandas solely to perform a supported kernel operation.
- Test schema equality separately from value equality, and include empty,
null, multi-chunk, and multi-batch inputs.
- Inspect the installed API before using a drifting option or claiming a
zero-copy, pushdown, or memory property.
Choose by required result
| Required result |
Use |
Critical condition |
| One typed column |
pa.array(..., type=...) |
Conversion and null policy are explicit. |
| One bounded set of equal-length columns |
pa.record_batch(..., schema=...) |
All arrays conform to the same schema and length. |
| One logical materialized table |
pa.table(..., schema=...) |
Total selected result fits the boundary. |
| Consumable batches with a common schema |
RecordBatchReader |
The consumer accepts a one-pass stream. |
| Multi-file discovery and pruning |
pyarrow.dataset.dataset |
Format, filesystem, partitioning, and schema are known or inspected. |
| Deferred filtered/projected scan |
dataset.scanner(...) |
Use dataset expressions such as ds.field, not eager masks. |
| Elementwise, vector, aggregate, or selection work |
pyarrow.compute |
Choose kernel null semantics and checked/safe behavior deliberately. |
| Portable Arrow stream/file serialization |
pyarrow.ipc |
Choose stream for sequential batches, file for random-access footer metadata. |
| Analytical columnar storage |
Parquet APIs or Dataset writer |
Schema, partition layout, row groups, overwrite policy, and reader compatibility are explicit. |
Read the operation map before choosing construction,
compute, dataset, serialization, or conversion APIs.
Canonical typed scan
from collections.abc import Iterator
from pathlib import Path
import pyarrow as pa
import pyarrow.dataset as ds
ORDER_SCHEMA = pa.schema(
[
pa.field("order_id", pa.uint64(), nullable=False),
pa.field("country", pa.string(), nullable=False),
pa.field("amount", pa.decimal128(18, 2), nullable=True),
]
)
def scan_orders(root: Path, country: str) -> Iterator[pa.RecordBatch]:
dataset = ds.dataset(
root,
format="parquet",
schema=ORDER_SCHEMA,
partitioning="hive",
)
scanner = dataset.scanner(
columns=["order_id", "country", "amount"],
filter=ds.field("country") == country,
batch_size=65_536,
)
yield from scanner.to_batches()
This returns bounded batches rather than concatenating the whole dataset. The
explicit schema prevents empty or heterogeneous fragments from silently
changing the boundary. Whether the filter is pruned by partitions/statistics or
evaluated after reading depends on the fragments and expression; verify a
performance claim instead of inferring it from the code.
High-risk rules
Types, nulls, and chunks
- Treat field nullability as a contract. A nullable type does not mean nulls
have been checked for domain validity.
- Do not assume
nullable=False validates constructed values. PyArrow table
construction can attach a non-nullable field to an array that still contains
nulls. Build typed arrays, reject a nonzero null_count for required fields,
then construct the batch or table under the exact schema.
- Use safe casts by default. Permit truncation, overflow, temporal-unit loss, or
invalid UTF-8 only when the contract names that loss and a test proves it.
- Arrow null and floating
NaN are distinct. Choose kernel options explicitly
when either affects filtering, aggregation, or equality.
- Compute functions commonly propagate nulls; Boolean kernels also have Kleene
variants. Select the truth table required by the domain.
- Do not assume one chunk. Test with a multi-chunk
Table, and call
combine_chunks() only for an API that requires contiguous buffers or after
measurement justifies the allocation.
- Dictionary, list, large-list, struct, map, decimal, and timestamp types carry
semantics that Python containers do not preserve automatically. Pin the exact
Arrow type at external boundaries.
Datasets and materialization
- Use dataset
Expression objects for scan filters and projections. A Python
function or eager Boolean array cannot provide file/row-group pruning.
- Include partition columns in the declared schema and specify partitioning
when directory names encode values. Do not infer a partition convention from
one path example.
to_table() loads the selected result; use to_batches() or a
RecordBatchReader when total size is not proven safe.
- Preserve scanner/reader lifetime until consumption completes. Do not return a
view over memory whose producer or foreign owner is already gone.
- “Zero-copy” is conditional on type, layout, alignment, mutability, and the
source/consumer protocol. Verify actual buffers or document the allowed copy;
never promise zero-copy from an API name alone.
Conversion and storage
Conversions to pandas, NumPy, or Python objects can allocate, change nullable
representations, or lose nested/dictionary metadata. State the consumer and
test round trips only for properties the contract requires. For Parquet or IPC,
test the persisted schema and representative values after reopening the
artifact; an in-memory pre-write assertion is insufficient. Run the installed-API inspector, read version and
API grounding, and use testing Arrow
boundaries.
Completion gate
Do not declare completion until the public object type and consumption model
match the caller; schema names, order, types, nullability, metadata requirements,
and time-zone/decimal semantics are asserted; empty, null, multi-chunk, and
multi-batch cases pass; dataset projection/filtering occurs before
materialization where required; conversions and writes are reopened or
round-tripped; and every copy, allocation, ordering, and pushdown claim is
measured or qualified. Report skipped checks and their consequence.
References
- Object and memory model
- Operation map
- Version and API grounding
- Testing Arrow boundaries
1---2name: pyarrow-python3description: Use for writing, reviewing, debugging, testing, or optimizing Python code that uses PyArrow arrays, schemas, Tables, RecordBatches, RecordBatchReaders, compute kernels, Dataset scans, Parquet, or Arrow IPC and C data interfaces. Trigger on schema drift, null/type semantics, chunking, batch streaming, predicate projection pushdown, conversion, and memory ownership. Do not use for pandas-only transformations, Polars-only expressions, generic SQL, Arrow Flight service design, C++/Rust Arrow, or Parquet work that does not use PyArrow.4argument-hint: "[PyArrow task, code, schema, error, or data boundary]"5metadata:6 source: https://skillmd.com/skills/schattenspiegel/pyarrow-python7---89# PyArrow Python1011Produce schema-explicit columnar code whose logical object, batch boundary,12null behavior, memory lifetime, and materialization point are deliberate.1314## Boundary1516Use this skill when the implementation directly uses `pyarrow` or must expose an17Arrow-compatible boundary. Do not introduce PyArrow merely to perform a small18Python-list or pandas transformation. Keep storage-format questions in scope19only when PyArrow reads or writes them. Arrow Flight servers, Acero internals,20and non-Python implementations require their own guidance.2122## Classify the object first2324| Object | Meaning | Use it for |25|---|---|---|26| `DataType` / `Field` / `Schema` | Immutable type and named-field contracts, including nullability and metadata. | Pinning a boundary before constructing or scanning data. |27| `Scalar` | One typed value, possibly null. | Kernel arguments and scalar results. |28| `Array` | One contiguous typed column backed by buffers. | A single physical column segment. |29| `ChunkedArray` | One logical column made of zero or more same-typed arrays. | Table columns and multi-source results; chunk count is not row count. |30| `RecordBatch` | Equal-length arrays under one schema. | Bounded transport or processing batches. |31| `Table` | A logical table whose columns may be chunked. | Materialized tabular results that fit the required memory boundary. |32| `RecordBatchReader` | A schema plus a consumable stream of batches. | Streaming interchange when the consumer can process batches once. |33| `Dataset` | A logical collection of fragments with a unified schema. | Discovering and querying multi-file or partitioned data. |34| `Scanner` | A dataset scan with projection, filter, and batching options bound. | Deferred dataset execution and bounded batch iteration. |3536Arrays and schemas are immutable. A `Table` is materialized but may be37physically chunked; `combine_chunks()` can allocate and is not routine cleanup.38A `Dataset` describes sources, while a `Scanner` describes the read. Calling39`to_table()` materializes all selected rows; `to_batches()` preserves a batch40boundary. Read [the object and memory model](references/object-model.md) when41chunking, buffers, ownership, dictionaries, nested types, or zero-copy claims42matter.4344## Ordered workflow45461. Recover the boundary contract: input object, output object, schema, field47 nullability, row order if required, batch size, and ownership lifetime.482. Choose `Array`, `RecordBatch`, `Table`, `RecordBatchReader`, `Dataset`, or49 `Scanner` from that contract; do not default every job to `Table`.503. Pin types where empty input, identifiers, timestamps, decimals, nested data,51 dictionary encoding, or cross-language interchange make inference unsafe.524. Push dataset projection and predicates into the scan. Materialize only at a53 consumer that requires all rows.545. Use `pyarrow.compute` kernels for Arrow data. Do not convert to Python rows55 or pandas solely to perform a supported kernel operation.566. Test schema equality separately from value equality, and include empty,57 null, multi-chunk, and multi-batch inputs.587. Inspect the installed API before using a drifting option or claiming a59 zero-copy, pushdown, or memory property.6061## Choose by required result6263| Required result | Use | Critical condition |64|---|---|---|65| One typed column | `pa.array(..., type=...)` | Conversion and null policy are explicit. |66| One bounded set of equal-length columns | `pa.record_batch(..., schema=...)` | All arrays conform to the same schema and length. |67| One logical materialized table | `pa.table(..., schema=...)` | Total selected result fits the boundary. |68| Consumable batches with a common schema | `RecordBatchReader` | The consumer accepts a one-pass stream. |69| Multi-file discovery and pruning | `pyarrow.dataset.dataset` | Format, filesystem, partitioning, and schema are known or inspected. |70| Deferred filtered/projected scan | `dataset.scanner(...)` | Use dataset expressions such as `ds.field`, not eager masks. |71| Elementwise, vector, aggregate, or selection work | `pyarrow.compute` | Choose kernel null semantics and checked/safe behavior deliberately. |72| Portable Arrow stream/file serialization | `pyarrow.ipc` | Choose stream for sequential batches, file for random-access footer metadata. |73| Analytical columnar storage | Parquet APIs or Dataset writer | Schema, partition layout, row groups, overwrite policy, and reader compatibility are explicit. |7475Read [the operation map](references/operations.md) before choosing construction,76compute, dataset, serialization, or conversion APIs.7778## Canonical typed scan7980```python81from collections.abc import Iterator82from pathlib import Path8384import pyarrow as pa85import pyarrow.dataset as ds868788ORDER_SCHEMA = pa.schema(89 [90 pa.field("order_id", pa.uint64(), nullable=False),91 pa.field("country", pa.string(), nullable=False),92 pa.field("amount", pa.decimal128(18, 2), nullable=True),93 ]94)959697def scan_orders(root: Path, country: str) -> Iterator[pa.RecordBatch]:98 dataset = ds.dataset(99 root,100 format="parquet",101 schema=ORDER_SCHEMA,102 partitioning="hive",103 )104 scanner = dataset.scanner(105 columns=["order_id", "country", "amount"],106 filter=ds.field("country") == country,107 batch_size=65_536,108 )109 yield from scanner.to_batches()110```111112This returns bounded batches rather than concatenating the whole dataset. The113explicit schema prevents empty or heterogeneous fragments from silently114changing the boundary. Whether the filter is pruned by partitions/statistics or115evaluated after reading depends on the fragments and expression; verify a116performance claim instead of inferring it from the code.117118## High-risk rules119120### Types, nulls, and chunks121122- Treat field nullability as a contract. A nullable type does not mean nulls123 have been checked for domain validity.124- Do not assume `nullable=False` validates constructed values. PyArrow table125 construction can attach a non-nullable field to an array that still contains126 nulls. Build typed arrays, reject a nonzero `null_count` for required fields,127 then construct the batch or table under the exact schema.128- Use safe casts by default. Permit truncation, overflow, temporal-unit loss, or129 invalid UTF-8 only when the contract names that loss and a test proves it.130- Arrow null and floating `NaN` are distinct. Choose kernel options explicitly131 when either affects filtering, aggregation, or equality.132- Compute functions commonly propagate nulls; Boolean kernels also have Kleene133 variants. Select the truth table required by the domain.134- Do not assume one chunk. Test with a multi-chunk `Table`, and call135 `combine_chunks()` only for an API that requires contiguous buffers or after136 measurement justifies the allocation.137- Dictionary, list, large-list, struct, map, decimal, and timestamp types carry138 semantics that Python containers do not preserve automatically. Pin the exact139 Arrow type at external boundaries.140141### Datasets and materialization142143- Use dataset `Expression` objects for scan filters and projections. A Python144 function or eager Boolean array cannot provide file/row-group pruning.145- Include partition columns in the declared schema and specify partitioning146 when directory names encode values. Do not infer a partition convention from147 one path example.148- `to_table()` loads the selected result; use `to_batches()` or a149 `RecordBatchReader` when total size is not proven safe.150- Preserve scanner/reader lifetime until consumption completes. Do not return a151 view over memory whose producer or foreign owner is already gone.152- “Zero-copy” is conditional on type, layout, alignment, mutability, and the153 source/consumer protocol. Verify actual buffers or document the allowed copy;154 never promise zero-copy from an API name alone.155156### Conversion and storage157158Conversions to pandas, NumPy, or Python objects can allocate, change nullable159representations, or lose nested/dictionary metadata. State the consumer and160test round trips only for properties the contract requires. For Parquet or IPC,161test the persisted schema and representative values after reopening the162artifact; an in-memory pre-write assertion is insufficient. Run [the installed-API inspector](scripts/inspect_pyarrow.py), read [version and163API grounding](references/version-grounding.md), and use [testing Arrow164boundaries](references/testing.md).165166## Completion gate167168Do not declare completion until the public object type and consumption model169match the caller; schema names, order, types, nullability, metadata requirements,170and time-zone/decimal semantics are asserted; empty, null, multi-chunk, and171multi-batch cases pass; dataset projection/filtering occurs before172materialization where required; conversions and writes are reopened or173round-tripped; and every copy, allocation, ordering, and pushdown claim is174measured or qualified. Report skipped checks and their consequence.175176## References177178- [Object and memory model](references/object-model.md)179- [Operation map](references/operations.md)180- [Version and API grounding](references/version-grounding.md)181- [Testing Arrow boundaries](references/testing.md)