Other Integrations

Other Integrations

Miscellaneous integrations that don't fit into other categories, including DataFrame libraries and utility tools.

DataFrame Libraries

Pandas

Package: dagster-pandas | Support: Dagster-supported

In-memory DataFrame library for data manipulation and analysis with type validation.

Use cases:

• Small to medium dataset transformations
• Data cleaning and preparation
• Exploratory data analysis
• CSV/Excel file processing

Quick start:

from dagster_pandas import PandasColumn, create_dagster_pandas_dataframe_type
import pandas as pd

# Define DataFrame schema
EventDataFrame = create_dagster_pandas_dataframe_type(
    name="EventDataFrame",
    columns=[
        PandasColumn.integer_column("user_id", min_value=0),
        PandasColumn.string_column("event_type"),
        PandasColumn.datetime_column("timestamp")
    ]
)

@dg.asset
def events() -> EventDataFrame:
    return pd.DataFrame({
        "user_id": [1, 2, 3],
        "event_type": ["click", "view", "click"],
        "timestamp": pd.date_range("2024-01-01", periods=3)
    })

Docs: https://docs.dagster.io/integrations/libraries/pandas

Polars

Package: dagster-polars | Support: Community-supported

Fast DataFrame library with columnar storage and lazy evaluation, often 5-10x faster than pandas.

Use cases:

• Fast in-memory transformations
• Processing medium to large datasets
• Lazy evaluation for query optimization
• Alternative to pandas with better performance

Quick start:

import polars as pl

@dg.asset
def polars_transform() -> pl.DataFrame:
    return pl.DataFrame({
        "a": [1, 2, 3],
        "b": [4, 5, 6]
    }).with_columns(
        (pl.col("a") * pl.col("b")).alias("product")
    )

# Use lazy evaluation for large datasets
@dg.asset
def lazy_polars() -> pl.LazyFrame:
    return (
        pl.scan_parquet("large-file.parquet")
        .filter(pl.col("value") > 100)
        .group_by("category")
        .agg(pl.col("amount").sum())
    )

Docs: https://docs.dagster.io/integrations/libraries/polars

DataFrame Library Comparison

Common Patterns

In-Memory DataFrame Processing

@dg.asset
def load_data() -> pd.DataFrame:
    return pd.read_csv("input.csv")

@dg.asset
def transform_data(load_data: pd.DataFrame) -> pd.DataFrame:
    return load_data[load_data["value"] > 100].groupby("category").sum()

@dg.asset
def save_data(transform_data: pd.DataFrame):
    transform_data.to_parquet("output.parquet")

Type-Safe DataFrames

# Using Dagster's pandas type system
UserDataFrame = create_dagster_pandas_dataframe_type(
    name="UserDataFrame",
    columns=[
        PandasColumn.integer_column("id"),
        PandasColumn.string_column("name"),
        PandasColumn.categorical_column("status", categories=["active", "inactive"])
    ]
)

@dg.asset
def validated_users() -> UserDataFrame:
    df = pd.read_csv("users.csv")
    # Type validation happens automatically
    return df

Lazy Evaluation with Polars

@dg.asset
def lazy_pipeline() -> pl.DataFrame:
    # Build query plan without executing
    lazy_df = (
        pl.scan_parquet("large_data.parquet")
        .filter(pl.col("date") >= "2024-01-01")
        .group_by("category")
        .agg([
            pl.col("amount").sum().alias("total"),
            pl.col("amount").mean().alias("average")
        ])
    )

    # Execute query plan efficiently
    return lazy_df.collect()

Migrating from Pandas to Polars

# Pandas version
@dg.asset
def pandas_transform() -> pd.DataFrame:
    df = pd.read_csv("data.csv")
    return df[df["value"] > 100].groupby("category")["amount"].sum()

# Polars equivalent (faster)
@dg.asset
def polars_transform() -> pl.DataFrame:
    df = pl.read_csv("data.csv")
    return (
        df.filter(pl.col("value") > 100)
        .group_by("category")
        .agg(pl.col("amount").sum())
    )

Tips

For Pandas:

• Memory: Monitor memory usage for datasets > 1GB
• Dtypes: Specify dtypes when reading CSVs to save memory
• Chunking: Use chunksize parameter for large CSV files
• Vectorization: Use vectorized operations instead of loops
• Copy vs View: Be aware of whether operations return copies or views

For Polars:

• Lazy evaluation: Use scan_* methods for large files
• Streaming: Enable streaming mode for larger-than-memory datasets
• Expressions: Learn Polars expression API for optimal performance
• Migration: Most pandas operations have direct Polars equivalents
• Type system: Polars has stricter types - embrace them for better performance

General:

• Start with Pandas: If you're familiar with pandas, start there
• Upgrade to Polars: When performance becomes an issue, consider migrating
• File formats: Use Parquet over CSV for better performance
• Profiling: Profile your code to identify bottlenecks
• Testing: Test with small datasets locally, scale up in production