s5cmd

name

s5cmd

description

Blazing-fast parallel S3 and local-filesystem data movement with the `s5cmd` CLI (v2.3.0). Use when copying, uploading, downloading, moving, syncing, listing, or deleting objects in S3 or S3-compatible stores (MinIO, Ceph, R2, GCS, B2, Wasabi); when transfers via `aws s3` / `aws s3 cp` / `boto3` are too slow; when batching thousands of object operations; streaming objects to/from stdin/stdout (`cat`, `pipe`); running SQL `select` over CSV/JSON/Parquet objects; generating presigned URLs; reading object metadata/`head`; computing storage usage (`du`); managing buckets (`mb`, `rb`, `bucket-version`); or moving training data, checkpoints, datasets, and backups at scale. Trigger on mentions of 's5cmd', 'S3 sync', 'fast S3 copy/download/upload', 'parallel S3 transfer', or custom S3 endpoints.

s5cmd

Overview

s5cmd is a massively parallel S3 and local-filesystem command-line tool — typically 10–30x faster than aws s3 for bulk transfers because it parallelizes both across objects (--numworkers, default 256) and within large objects (--concurrency, multipart). Use it for any high-throughput S3 data movement.

Core mental model: s5cmd [GLOBAL OPTIONS] <command> [command options] <args>. Global options (auth, --dry-run, --endpoint-url, --json, --numworkers) come before the command; command options come after.

The #1 rule: always quote wildcards. s5cmd cp "s3://bucket/*" dir/ — without quotes the shell expands * locally and the command does the wrong thing or fails.

Auth & configuration

s5cmd uses the standard AWS credential chain (env vars, ~/.aws/credentials, IAM roles). Key global options:

Option	Purpose
--endpoint-url URL	Custom S3 host (MinIO/Ceph/R2/GCS/Wasabi). Also $S3_ENDPOINT_URL.
--profile NAME	Named profile from credentials file.
--credentials-file PATH	Non-default credentials file.
--no-sign-request	Anonymous access to public buckets (no creds loaded).
--request-payer requester	Access requester-pays buckets.
--no-verify-ssl	Skip TLS cert verification (self-signed endpoints).
--numworkers N	Parallel workers across objects (default 256).
--retry-count N, -r	Retries per request on failure (default 10).
--dry-run	Print what would run without executing. Global — put before the command.
--json	JSON-formatted output (machine parsing).
--log LEVEL	trace/debug/info/error.
--stat	Print execution stats at the end.

# MinIO / R2 / custom endpoint
s5cmd --endpoint-url https://minio.local:9000 ls s3://bucket/
export S3_ENDPOINT_URL=https://<accountid>.r2.cloudflarestorage.com   # Cloudflare R2
# Public bucket, no credentials
s5cmd --no-sign-request ls "s3://public-bucket/*"

Command quick reference

Command	Does
ls	List buckets and objects
cp	Copy objects (upload/download/S3→S3)
mv	Move/rename objects (copy then delete source)
rm	Remove objects
mb	Make bucket
rb	Remove bucket
sync	Sync source→dest (only changed objects)
du	Show object size usage
cat	Print remote object content to stdout
pipe	Stream stdin → remote object
select	Run SQL queries on csv/json/parquet objects
run	Run many commands in batch/parallel
head	Print object metadata (or check existence)
presign	Print presigned URL
du	Disk/storage usage
bucket-version	Get/set bucket versioning
version	Print s5cmd version

Every subcommand has s5cmd <cmd> --help with full flags and numbered examples. Details below cover the high-value flags; run --help for the long tail.

cp / mv — copy & move

cp source dest and mv source dest upload, download, or copy S3→S3. mv = cp + delete source. Direction is inferred from s3:// prefixes.

s5cmd cp s3://bucket/prefix/obj.gz .            # download one object
s5cmd cp "s3://bucket/*" target-dir/            # download all (quote the glob!)
s5cmd cp dir/ s3://bucket/                       # upload a directory recursively
s5cmd cp "dir/*.gz" s3://bucket/                 # upload matching files
s5cmd cp s3://src/obj s3://dst/prefix/obj        # server-side S3→S3 copy

Key cp/mv/sync flags:

• --flatten, -f — flatten directory structure from the first wildcard.
• --no-clobber, -n — don't overwrite existing destination.
• --if-size-differ, -s — overwrite only if size differs.
• --if-source-newer, -u — overwrite only if source is newer. (Combine -n -s -u for cheap incremental copies.)
• --exclude PATTERN / --include PATTERN — repeatable; filter by glob. --exclude removes, --include keeps only matches.
• --concurrency N, -c — concurrent multipart chunks per object (default 5). Raise for few large files.
• --part-size N, -p — multipart chunk size in MiB (default 50).
• --storage-class CLASS — STANDARD, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER, DEEP_ARCHIVE, REDUCED_REDUNDANCY.
• --sse aws:kms + --sse-kms-key-id ID — server-side encryption.
• --acl public-read — object ACL.
• --metadata 'k=v' (repeatable), --metadata-directive COPY|REPLACE.
• --content-type, --content-encoding, --content-disposition, --cache-control, --expires (RFC3339).
• --source-region / --destination-region — cross-region S3→S3 (auto-discovered if omitted).
• --version-id ID — operate on a specific object version.
• --force-glacier-transfer / --ignore-glacier-warnings — handle archived objects.
• --raw — disable wildcard expansion (filenames containing literal glob chars).
• --show-progress, --sp — progress bar.

sync — incremental sync

sync source dest transfers only objects that changed. By default compares size AND modification time.

s5cmd sync folder/ s3://bucket/                  # local → S3
s5cmd sync "s3://bucket/*" folder/               # S3 → local (quote glob)
s5cmd sync --delete folder/ s3://bucket/         # also delete dest objects missing from source
s5cmd sync --size-only "s3://bucket/*" folder/   # compare size only (ignore modtime)

Sync-specific flags (plus all cp flags): --delete (remove dest-only objects — destructive), --size-only, --exit-on-error. Dry-run first when using --delete: s5cmd --dry-run sync --delete src/ s3://bucket/.

ls / du / head — inspect

s5cmd ls                                  # list buckets
s5cmd ls "s3://bucket/prefix/*"           # list objects
s5cmd ls -H --etag "s3://bucket/*"        # human sizes + ETag
s5cmd ls --all-versions s3://bucket/obj   # all object versions
s5cmd du -H "s3://bucket/*"               # total size, human-readable
s5cmd du --group "s3://bucket/*"          # broken down by storage class
s5cmd head s3://bucket/prefix/obj         # object metadata
s5cmd head s3://bucket                    # check a bucket exists (exit code)

ls: --humanize/-H, --etag/-e, --storage-class/-s, --all-versions, --show-fullpath, --exclude.

rm — delete

s5cmd rm s3://bucket/prefix/obj.gz                       # one object
s5cmd rm "s3://bucket/prefix/*"                          # all under prefix (quote!)
s5cmd rm --exclude "*.keep" "s3://bucket/prefix/*"       # delete all except *.keep
s5cmd rm --all-versions "s3://bucket/prefix*"            # all versions

rm flags: --exclude/--include, --all-versions, --version-id, --raw. Wildcard deletes are irreversible — prefer s5cmd --dry-run rm ... first.

cat / pipe — streaming

s5cmd cat s3://bucket/prefix/obj | jq .                  # stream object to stdout
s5cmd cat "s3://bucket/prefix/*" > combined              # concatenate matching objects
echo "content" | gzip | s5cmd pipe s3://bucket/obj.gz    # stream stdin → object
curl -sL https://example.com/file | s5cmd pipe s3://bucket/file   # remote → S3, no disk

pipe accepts the upload/encryption flags (--storage-class, --sse, --metadata, --content-type, --no-clobber, --concurrency, --part-size).

run — batch / parallel command files

The power feature for thousands of operations. Each line is a normal s5cmd command (without the s5cmd prefix); all run in parallel respecting --numworkers.

# commands.txt
cp s3://bucket/a.gz .
cp s3://bucket/b.gz .
rm s3://bucket/old/obj1
mv s3://bucket/x s3://bucket/y

s5cmd run commands.txt          # run a file
cat commands.txt | s5cmd run    # or pipe via stdin
# Generate then execute (e.g. delete a computed list):
s5cmd ls "s3://bucket/*" | awk '{print "rm s3://bucket/"$NF}' | s5cmd run

select — SQL over objects

s5cmd select csv|json|parquet --query "SQL" s3://bucket/... runs S3 Select-style queries.

s5cmd select csv --query "SELECT * FROM s3object s WHERE s._1 = 'x'" s3://bucket/data.csv
s5cmd select parquet --query "SELECT col FROM s3object" --output-format json "s3://bucket/*.parquet"

Flags: --query/-e, --compression GZIP|BZIP2, --output-format json|csv, plus glacier/version flags.

presign / mb / rb / bucket-version

s5cmd presign --expire 24h s3://bucket/prefix/obj    # presigned URL (default 3h)
s5cmd mb s3://newbucket                               # make bucket
s5cmd rb s3://oldbucket                               # remove bucket
s5cmd bucket-version s3://bucket                      # get versioning status
s5cmd bucket-version --set Enabled s3://bucket        # enable (or Suspended)

Performance tuning

• Many small objects → raise --numworkers (global, default 256).
• Few large objects → raise per-object --concurrency/-c and --part-size/-p.
• --numworkers is global (before command); --concurrency/--part-size are per-command.
• Use run to batch heterogeneous operations into one parallel pass.
• Add --stat to measure, --log debug to diagnose throttling/retries.

Common mistakes

Mistake	Fix
Unquoted wildcard: s5cmd cp s3://b/* .	Quote it: s5cmd cp "s3://b/*" . — shell eats the * otherwise.
--dry-run placed after the command	It's a global flag: s5cmd --dry-run rm "s3://b/*".
--endpoint-url after the command	Also global — must precede the command.
Expecting cp to skip unchanged files	cp always re-transfers. Use sync, or cp -n -s -u.
sync --delete without preview	Run --dry-run first; --delete permanently removes dest objects.
Confusing --numworkers with --concurrency	numworkers = parallelism across objects; concurrency = multipart parallelism within one object.
Filenames with literal */? mis-handled	Use --raw to disable wildcard interpretation.
Public bucket "access denied"	Add --no-sign-request.
GLACIER objects fail to copy	--force-glacier-transfer (must already be restored) or --ignore-glacier-warnings.

Verify

s5cmd version          # confirm install (v2.3.0+)
s5cmd <cmd> --help     # authoritative flags + numbered examples for any subcommand