eyadsibai Polars Fast DataFrame Library
Lightning-fast DataFrame library with lazy evaluation and parallel execution.
When to Use
- Pandas is too slow for your dataset
- Working with 1-100GB datasets that fit in RAM
- Need lazy evaluation for query optimization
- Building ETL pipelines
- Want parallel execution without extra config
Lazy vs Eager Evaluation
| Mode | Function | Executes | Use Case |
|---|---|---|---|
| Eager | read_csv() |
Immediately | Small data, exploration |
| Lazy | scan_csv() |
On .collect() |
Large data, pipelines |
Key concept: Lazy mode builds a query plan that gets optimized before execution. The optimizer applies predicate pushdown (filter early) and projection pushdown (select columns early).
Core Operations
Data Selection
| Operation | Purpose |
|---|---|
select() |
Choose columns |
filter() |
Choose rows by condition |
with_columns() |
Add/modify columns |
drop() |
Remove columns |
head(n) / tail(n) |
First/last n rows |
Aggregation
| Operation | Purpose |
|---|---|
group_by().agg() |
Group and aggregate |
pivot() |
Reshape wide |
melt() |
Reshape long |
unique() |
Distinct values |
Joins
| Join Type | Description |
|---|---|
| inner | Matching rows only |
| left | All left + matching right |
| outer | All rows from both |
| cross | Cartesian product |
| semi | Left rows with match |
| anti | Left rows without match |
Expression API
Key concept: Polars uses expressions (pl.col()) instead of indexing. Expressions are lazily evaluated and optimized.
Common Expressions
| Expression | Purpose |
|---|---|
pl.col("name") |
Reference column |
pl.lit(value) |
Literal value |
pl.all() |
All columns |
pl.exclude(...) |
All except |
Expression Methods
| Category | Methods |
|---|---|
| Aggregation | .sum(), .mean(), .min(), .max(), .count() |
| String | .str.contains(), .str.replace(), .str.to_lowercase() |
| DateTime | .dt.year(), .dt.month(), .dt.day() |
| Conditional | .when().then().otherwise() |
| Window | .over(), .rolling_mean(), .shift() |
Pandas Migration
| Pandas | Polars |
|---|---|
df['col'] |
df.select('col') |
df[df['col'] > 5] |
df.filter(pl.col('col') > 5) |
df['new'] = df['col'] * 2 |
df.with_columns((pl.col('col') * 2).alias('new')) |
df.groupby('col').mean() |
df.group_by('col').agg(pl.all().mean()) |
df.apply(func) |
df.map_rows(func) (avoid if possible) |
Key concept: Polars prefers explicit operations over implicit indexing. Use .alias() to name computed columns.
File I/O
| Format | Read | Write | Notes |
|---|---|---|---|
| CSV | read_csv() / scan_csv() |
write_csv() |
Human readable |
| Parquet | read_parquet() / scan_parquet() |
write_parquet() |
Fast, compressed |
| JSON | read_json() / scan_ndjson() |
write_json() |
Newline-delimited |
| IPC/Arrow | read_ipc() / scan_ipc() |
write_ipc() |
Zero-copy |
Key concept: Use Parquet for performance. Use scan_* for large files to enable lazy optimization.
Performance Tips
| Tip | Why |
|---|---|
| Use lazy mode | Query optimization |
| Use Parquet | Column-oriented, compressed |
| Select columns early | Projection pushdown |
| Filter early | Predicate pushdown |
| Avoid Python UDFs | Breaks parallelism |
| Use expressions | Vectorized operations |
| Set dtypes on read | Avoid inference overhead |
vs Alternatives
| Tool | Best For | Limitations |
|---|---|---|
| Polars | 1-100GB, speed critical | Must fit in RAM |
| Pandas | Small data, ecosystem | Slow, memory hungry |
| Dask | Larger than RAM | More complex API |
| Spark | Cluster computing | Infrastructure overhead |
| DuckDB | SQL interface | Different API style |
Resources
- Docs: https://pola.rs/
- User Guide: https://docs.pola.rs/user-guide/
- Cookbook: https://docs.pola.rs/user-guide/misc/cookbook/