Litdata · lightning-ai bundle Expert use of the LitData library and work on its codebase. Use when writing or reviewing code that calls litdata (StreamingDataset, StreamingDataLoader, StreamingRawDataset, optimize, map, CombinedStreamingDataset, ParallelStreamingDataset, TokensLoader, serializers, typed media wrappers Text/Audio/Video/Image/Jpeg/File/Tensor/Graph/…, list_media_folder, litdata_collate, train_test_split, complete_dataset, merge_datasets, index_parquet_dataset, index_hf_dataset), answering how-to questions, choosing raw vs optimize vs parquet/HF/MDS, keyed lookup (`key_fn`, `build_keys_index`, `dataset_update`, `get_by_key`), elastic mid-epoch resume, tuning cache/prefetch/shuffle/seed, resolving paths (s3/gs/r2/azure/hf/local:/teamspace via resolver.py), documenting or debugging optimize/map I/O (shared queue, async Downloader download/upload, LITDATA_OPTIMIZE_*), FsProvider vs Downloader, FUSE s3_connections/s3_folders, multi-node DATA_OPTIMIZER_* / num_nodes jobs, POSIX-fast mmap, litsim simulator, or when navigating/edit
