scrub-physicsnemo-arm64-gpu-container

name	scrub-physicsnemo-arm64-gpu-container
description	Nightly local ARM GPU scrub that pulls the latest main PhysicsNeMo arm64 container and runs CUDA-marked tests outside GitHub Actions.
schedule_cron	0 17 3 ? * *
schedule_tz	America/New_York

scrub-physicsnemo-arm64-gpu-container

Use this skill for the nightly ARM GPU container validation scrub.

Purpose

• Validate the latest main ARM64 PhysicsNeMo container on a real local ARM machine with a modern NVIDIA GPU.
• Keep this outside GitHub Actions. Cloud ARM GPU instances currently expose unsupported NVIDIA T4G-class hardware for this stack.
• Exercise the GPU path that CI cannot cover cheaply: pull the latest 26.05-arm64-latest image and run CUDA-marked tests locally.

Required Host

• aarch64 Linux host.
• NVIDIA GPU visible through nvidia-smi.
• Docker usable by the current user, or through non-interactive sudo docker, with NVIDIA container runtime support.
• Repository checkout available with this script from main.

Run

From the repository root:

scripts/container/scrub_arm64_gpu_container_tests.sh

The script defaults to:

ghcr.io/m2lines/ocean-emulator-physicsnemo:26.05-arm64-latest

Useful overrides:

IMAGE_TAG=ghcr.io/m2lines/ocean-emulator-physicsnemo:26.05-arm64-<sha> \
PYTEST_ARGS="-x -k test_trainer" \
scripts/container/scrub_arm64_gpu_container_tests.sh

Expected Result

• Docker pulls the latest ARM64 image published from main.
• scripts/container/run_cuda_tests_in_image.sh runs the CUDA-marked tests inside the container, including the FOMO flash-attention coverage.
• A successful scrub exits 0 and ends with HARNESS_SCRUB_LOOP {"needs_followup_at":null}.

Failure Handling

• If pull fails, check whether main has published 26.05-arm64-latest and whether GHCR auth is available.
• If Docker/NVIDIA runtime is missing, capture the host setup blocker explicitly.
• If CUDA tests fail, inspect the failing test and either fix the regression in a PR or file/update a concrete issue with the image tag, commit, GPU model, and failing traceback.
• Use scrub follow-up only when waiting on an external image publish or another long-running validation; otherwise complete the run in one turn.