Add Incremental Loading

Add incremental loading to a dlt filesystem pipeline — filter files by modification date, optionally filter records by a timestamp column, and switch to merge with a primary key to deduplicate updated records, so each run only reads new or modified files. Use after create-filesystem-pipeline produces a working replace-mode pipeline, or when the user wants to set up merge/dedup or move a pipeline from replace to incremental. For speed/memory tuning (faster reader, chunked streaming, parallel reads, glob narrowing) use optimize-filesystem-performance instead.

dlt-hub Updated 65 repo stars

File contents

dlt-hub/dlthub-ai-workbench commit 54816124a0

Frequently asked questions

npx get-skill add dlt-hub/add-incremental-loading-2