Use when working on cuVSLAM CI/CD - the GitHub Actions nightly and PR-verify pipelines, dataset provisioning and staging, the evaluation run and KPI reporting, the build/test/lint matrix, branch rulesets, or repository secrets and variables. Covers adding a dataset, changing dataset format or packing, and controlling which datasets run in PR versus nightly.
Use when working on cuVSLAM CI/CD - the GitHub Actions nightly and PR-verify pipelines, dataset provisioning and staging, the evaluation run and KPI reporting, the build/test/lint matrix, branch rulesets, or repository secrets and variables. Covers adding a dataset, changing dataset format or packing, and controlling which datasets run in PR versus nightly.
cuVSLAM CI/CD
CI/CD runs build, unit test, lint, dataset evaluation, dataset provisioning, and
nightly releases on GitHub Actions with self-hosted GPU and Jetson runners. This
file is the task entry point. Read reference.md for the
architecture, the secrets and variables, and the load-bearing constraints before
changing anything.
Component map
Workflows (.github/workflows/):
pr-verify.yml - lint, then build + unit test on x86, Orin, and Thor; eval on the x86 job (fork-gated); posts a KPI table to the PR comment.
nightly.yml - scheduled/manual build + test matrix; eval on the four x86 configs; writes per-config KPI history and versioned Actions artifacts. Scheduled runs never create a Release. A manual dispatch from a matching release/vX.Y.Z branch promotes the same distributable bytes to a protected draft GitHub Release.
provision-datasets.yml - manual workflow_dispatch on the default branch; downloads, converts, and uploads a dataset tarball to S3. The only writer of dataset storage.
sync-rulesets.yml - applies .github/rulesets/default-branch-ruleset.json through the API.
In scripts/datasets_config.sh, add the name to PROVISIONABLE_DATASETS and add its dataset_upload_subdir case (empty string means the converted root; otherwise the subdir under the converted output).
Add tools/python_tools/cuvslam_tools/dataset_preparation/<name>/prepare_<name>.sh (plus a download script) that converts raw data to the edex layout under --output-dir. dataset_prepare_script resolves tools/python_tools/cuvslam_tools/dataset_preparation/<name>/prepare_<name>.sh.
Add the dataset to the dataset choice input in provision-datasets.yml.
Run Provision dataset (workflow_dispatch) on the default branch. It writes <S3_DATASETS_BUCKET>/<name>.tar.
Add the name to EVAL_DATASET_NAMES in datasets_config.sh, and add a record to DATASETS[] in scripts/run_eval.sh: LABEL|link_name|subdir|test_config|app_flags.
Add expected KPI ranges for the dataset to scripts/kpi_baseline_ranges.json.
Task: change dataset format or packing
A dataset moves through four stages. Change the one that owns the format, and keep packing and extraction in sync.
Conversion (raw to stored layout, e.g. images vs mp4): tools/python_tools/cuvslam_tools/dataset_preparation/<name>/prepare_<name>.sh and the converter it calls.
Tarball packing: scripts/provision_dataset.sh creates an uncompressed .tar (tar -cf); s3_tarball_uri names it <name>.tar.
Extraction: scripts/stage_eval_datasets.sh runs tar -xf.
In-archive layout consumed at eval: tools/cuvslam_app/cuvslam_app.py and tools/cuvslam_app/edex_reader.py (already reads per-folder <folder>.tar archives).
Do not reintroduce gzip: provisioning uses uncompressed .tar to cap memory on the provisioning runner. If packing changes, change extraction in the same MR.
Task: control the PR vs nightly matrix
Nightly configs: nightly.ymlstrategy.matrix.include. Eval runs on entries flagged eval: true (currently the four x86 configs). Every eval-enabled config needs the RUNNER_STORAGE_ROOT mount and configured repo secrets/variables; the cuvslam-ci:local image supplies the AWS CLI.
PR config: pr-verify.yml runs eval only on build-test-x86 (fork-gated). EVAL_CONFIG is the static slug label for the PR table.
Active dataset set: DATASETS[] in run_eval.sh is global; PR and nightly run the same set. There is no per-pipeline dataset selection today. To run a different set in PR vs nightly, add an env-selected subset in run_eval.sh and have each workflow pass the selector.
Task: preserve nightly version provenance
VERSION controls the package filename, while get_version() is generated independently as
MAJOR.MINOR.PATCH+<short-git-sha>[-modified]. The -modified suffix means the build container's
tracked worktree differs from HEAD.
Keep git-lfs installed and configured system-wide in scripts/Dockerfile. Nightly pulls LFS objects on the host,
and Git without the LFS clean filter misidentifies the materialized files as source modifications.
Do not edit tracked files before the C++ build. Pass runner-specific settings such as the Ubuntu Ports mirror
through Docker build arguments instead.
Keep CUVSLAM_REQUIRE_CLEAN_SOURCE=1 on nightly C++ builds. It checks the source using the same image and Git/LFS
configuration that generate the version header.
Pass the expected package version and checked-out full Git SHA to verify_pycuvslam_wheel_in_docker.sh. The
verifier must reject -modified and a mismatched embedded revision before artifacts are uploaded.
When changing checkout, LFS, Docker build, or version logic, test both an LFS-materialized clean checkout and an
intentional tracked edit.
Task: build a draft release
Create or update a branch named release/vMAJOR.MINOR[.PATCH][-SUFFIX].
Manually dispatch Nightly Build & Test from that branch. Release dispatches always build, even without commits in the last 24 hours.
After every matrix job and evaluation succeeds, the workflow validates the branch version against VERSION, derives the tag from the branch (release/v17.0 -> v17.0), and creates a draft Release containing the already-built C++ archives, wheels, documentation, and permanent evaluation bundle.
Review the draft and publish it manually. A draft/published Release or Git tag with the same version is never overwritten.
To rebuild an unpublished release after fixes, explicitly delete the old draft and dispatch the updated release branch again.
Scheduled nightlies publish only versioned 30-day Actions artifacts. They never create or update a GitHub Release.
