| name | tao-data-io |
| description | The data-mover for TAO jobs — decides the storage tier (A pre-positioned mount with zero fetch / B volume-from-S3 / C ephemeral in-compute fetch), stages inputs (bulk + annotation-selective + archive extract + HF/NGC PTM), maps credentials to env, routes outputs 3-way with upload-excludes, and runs the compute-frame verify gate. A support skill other platform skills (docker, kubernetes, slurm, brev, virtualenv) call to get data to and from the compute container without the TAO SDK. Trigger phrases include "stage inputs", "mount the dataset", "upload TAO results", "download only referenced files", "resolve results_dir", "verify the container can read the data". |
| license | Apache-2.0 |
| compatibility | Requires aws CLI or s5cmd on the staging host, plus Python 3.10+ with boto3 and pandas/pyarrow for annotation-selective download. No nvidia-tao-sdk, no fsspec/s3fs. Credentials are read from the process environment, whether exported in the user's shell or sourced from a user-approved env file. |
| metadata | {"author":"NVIDIA Corporation","version":"0.1.0"} |
| allowed-tools | Read Bash |
| tags | ["platform","storage"] |
tao-data-io
Get data to and from the compute container. Decide the storage tier first —
under strategy A (pre-positioned mount) no bytes move at all — and when a
fetch is needed, move it host-side with aws/s5cmd/boto3/huggingface-cli/ngc
directly — no nvidia-tao-sdk, no in-container runtime. Other platform skills
call this skill to stage inputs before launch and sync outputs after. It never
launches a container itself. The chosen tier is stamped into the job-record at
submit.
When NOT to invoke this skill: if the inputs are already readable from the
compute frame (a local path on the execution host, an existing Lustre/PVC/bind
mount), that IS tier A — record it and skip this skill entirely; there is
nothing to move. Air-gapped hosts: tier A is the only tier — never
attempt an S3/HF/NGC fetch; anything missing (datasets, checkpoints, and the
container images themselves) must be pre-positioned by the operator, and the
preflight's readability check is the only data step that runs.
Credentials (env vars; values never written to disk by this skill)
S3 credentials use the officially documented AWS env vars, read from the
session environment: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and (for
S3-compatible stores) AWS_ENDPOINT_URL, AWS_DEFAULT_REGION. The aws
CLI and boto3 pick the variables up natively — never run aws configure
and never write ~/.aws/credentials:
set -a; source /path/to/.env; set +a
aws s3 ls "s3://$S3_BUCKET_NAME/..."
If a session provides only the legacy TAO names (ACCESS_KEY, SECRET_KEY,
S3_ENDPOINT_URL, CLOUD_REGION), map them once, scoped to the command:
AWS_ACCESS_KEY_ID="$ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SECRET_KEY" aws s3 ...
HF_TOKEN / pass through unchanged for PTM pulls. Never pass a
credential as a CLI argument (, , ); use
or (no value).
