Optimize Filesystem Performance

Speed up a dlt filesystem pipeline. Use when reading files (CSV, Parquet, JSONL) from local disk, S3, GCS, Azure, or SFTP is slow or memory-heavy and the user wants to optimize it — choose a faster reader, read files in parallel, narrow the glob, or stream in chunks. For first-time incremental/merge setup, or to skip files already loaded in previous runs, use add-incremental-loading instead.

dlt-hub Updated 65 repo stars

File contents

dlt-hub/dlthub-ai-workbench commit d743ad172e

Frequently asked questions

npx get-skill add dlt-hub/optimize-filesystem-performance-2