agentskillexchange Run high-volume S3 copy, sync, and delete jobs faster than ad hoc aws s3 loops with s5cmd
Execute large parallel object-store operations from command files or shell pipelines when agents need speed and repeatability beyond basic aws s3 loops.
Prerequisites
s5cmd plus credentials for Amazon S3 or another compatible object store
Installation
Use the upstream install or setup path that matches your environment:
- brew install peak/tap/s5cmd
- conda config --add channels conda-forge
- conda config --set channel_priority strict
- conda install s5cmd
Requirements and caveats from upstream:
Docker
- s5cmd uses official AWS SDK to access S3. SDK requires credentials to sign
Basic usage or getting-started notes:
Command file support to run commands in batches at very high execution speeds
Dry run support
Source: https://github.com/peak/s5cmd
Extracted from upstream docs: https://raw.githubusercontent.com/peak/s5cmd/HEAD/README.md