docker-environment-setup

# Docker Environment Setup Set up a Docker container environment for running Nextflow training examples. This skill handles all Docker configuration needed to match the Codespaces/Gitpod environment that learners use. ## Initial Questions Use `AskUserQuestion` to determine the setup type: **Which Docker setup do you need?** - **Basic setup (Recommended)** - For tutorials without containerized processes (e.g., hello_nextflow basics, plugin_development) - **DooD setup** - For tutorials with containerized processes (e.g., genomics, essential_scripting_patterns) - **Check/restart existing** - Verify or restart an existing nf-training container --- ## Determine NXF_VER and training image Before any Docker commands, read the Nextflow version and the training image reference from devcontainer.json: ```bash NXF_VER=$(grep -o '"NXF_VER":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4) TRAINING_IMAGE=$(grep -o '"image":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4) echo "Using NXF_VER=${NXF_VER}, TRAINING_IMAGE=${TRAINING_IMAGE}" ``` Always use `${TRAINING_IMAGE}` as read from the **currently checked-out branch's** `devcontainer.json` — do not hardcode `ghcr.io/nextflow-io/training:latest`. The tag a branch pins is the version that corresponds to what that branch's docs were written/tested against; pulling whatever the registry's `latest` tag happens to be right now can silently diverge from it. Pull that exact reference (`docker pull "${TRAINING_IMAGE}"`) before starting the container, so a stale local cache doesn't ship an outdated image under the same tag name — but never substitute a different tag than what the branch declares. --- ## Basic Setup For tutorials that don't use containerized processes: ```bash # Clean up any existing container docker stop nf-training 2>/dev/null; docker rm nf-training 2>/dev/null # Read the image reference pinned by this branch's devcontainer.json and pull it NXF_VER=$(grep -o '"NXF_VER":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4) TRAINING_IMAGE=$(grep -o '"image":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4) docker pull "${TRAINING_IMAGE}" # Start fresh container with UTF-8 locale support docker run -d --name nf-training \ -e NXF_VER=${NXF_VER} \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -v "${PWD}:/workspaces/training" \ -w /workspaces/training \ "${TRAINING_IMAGE}" \ sleep infinity ``` **Important**: The `LANG=C.UTF-8` and `LC_ALL=C.UTF-8` environment variables are critical for handling non-ASCII characters (like "Holà", "Grüß Gott") in file names and content. ### Running Commands ```bash docker exec -e LANG=C.UTF-8 -e LC_ALL=C.UTF-8 \ -w /workspaces/training/<working-dir> \ nf-training \ <command> ``` --- ## Docker-outside-of-Docker (DooD) Setup For tutorials with containerized processes (FASTP, BWA, SAMTOOLS, etc.): ```bash # Clean up any existing container docker stop nf-training 2>/dev/null; docker rm nf-training 2>/dev/null # Read the image reference pinned by this branch's devcontainer.json and pull it NXF_VER=$(grep -o '"NXF_VER":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4) TRAINING_IMAGE=$(grep -o '"image":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4) HOST_PATH="${PWD}" docker pull "${TRAINING_IMAGE}" # Start container with DooD support docker run -d --name nf-training \ -e NXF_VER=${NXF_VER} \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "${HOST_PATH}:${HOST_PATH}" \ -w "${HOST_PATH}" \ "${TRAINING_IMAGE}" \ sleep infinity # Create symlink for Codespaces paths docker exec nf-training bash -c "rm -rf /workspaces/training && mkdir -p /workspaces && ln -sf ${HOST_PATH} /workspaces/training" ``` **Critical differences from basic setup:** 1. **Docker socket mount** (`-v /var/run/docker.sock:/var/run/docker.sock`) - Allows Nextflow to spawn sibling containers 2. **Matching host paths** (`-v "${HOST_PATH}:${HOST_PATH}"`) - Work directories resolve correctly between containers 3. **Symlink** - Makes `/workspaces/training/...` paths work locally ### Running Commands with DooD ```bash docker exec -e LANG=C.UTF-8 -e LC_ALL=C.UTF-8 -e USER=testuser \ -w "${HOST_PATH}/<working-dir>" \ nf-training \ nextflow run <script.nf> [options] ``` ### When DooD is Needed Any tutorial where processes specify containers: - `hello_nextflow` (later lessons with containers) - `nf4_science/genomics` and other domain modules - Side quests: `essential_scripting_patterns`, `metadata`, etc. --- ## Apple Silicon (ARM) Macs Most bioinformatics containers are built for x86_64/amd64. On ARM Macs, create a platform config: ```bash docker exec nf-training bash -c 'cat > /tmp/platform.config << EOF docker.runOptions = "--platform linux/amd64" EOF' ``` Include when running: ```bash docker exec -e LANG=C.UTF-8 -e LC_ALL=C.UTF-8 -e USER=testuser \ -w "${HOST_PATH}/<working-dir>" \ nf-training \ nextflow run <script.nf> -c /tmp/platform.config ``` **Note**: Platform emulation uses more memory. For OOM errors (exit code 137), increase Docker Desktop memory in Preferences → Resources. --- ## Troubleshooting | Error | Cause | Solution | | ---------------------------------------- | ------------------- | ---------------------------------------------------- | | `Cannot connect to Docker daemon` | Socket not mounted | Add `-v /var/run/docker.sock:/var/run/docker.sock` | | `.command.sh: No such file or directory` | Path mismatch | Use matching paths: `-v "${HOST_PATH}:${HOST_PATH}"` | | `exec format error` | ARM/x86 mismatch | Add `--platform linux/amd64` to docker.runOptions | | Exit code 137 (OOM) | Insufficient memory | Increase Docker Desktop memory allocation | | `Malformed input or unmappable chars` | Missing UTF-8 | Add `-e LANG=C.UTF-8 -e LC_ALL=C.UTF-8` | | `Error: No such container: nf-training` | Container stopped | Restart container (see below) | --- ## Container Restart Procedure The container may stop during long sessions. To restart: ```bash # 1. Check if container is running docker ps | grep nf-training # 2. If not running, restart with DooD setup docker stop nf-training 2>/dev/null; docker rm nf-training 2>/dev/null HOST_PATH="${PWD}" NXF_VER=$(grep -o '"NXF_VER":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4) TRAINING_IMAGE=$(grep -o '"image":\s*"[^"]*"' .devcontainer/devcontainer.json | cut -d'"' -f4) docker pull "${TRAINING_IMAGE}" docker run -d --name nf-training \ -e NXF_VER=${NXF_VER} \ -e LANG=C.UTF-8 \ -e LC_ALL=C.UTF-8 \ -v /var/run/docker.sock:/var/run/docker.sock \ -v "${HOST_PATH}:${HOST_PATH}" \ -w "${HOST_PATH}" \ "${TRAINING_IMAGE}" \ sleep infinity # 3. Recreate symlink (critical!) docker exec nf-training bash -c "rm -rf /workspaces/training && mkdir -p /workspaces && ln -sf ${HOST_PATH} /workspaces/training" # 4. Recreate platform config if needed (ARM Macs) docker exec nf-training bash -c 'cat > /tmp/platform.config << EOF docker.runOptions = "--platform linux/amd64" EOF' ``` --- ## Cleanup When done with testing: ```bash docker stop nf-training && docker rm nf-training ``` --- ## Notes - Always verify you're in the repository root before starting (check for `docs/en/mkdocs.yml`) - The container uses `sleep infinity` so it persists across multiple command executions - Symlink must be recreated each time the container restarts - For long sessions, periodically check container is still running: `docker ps | grep nf-training` - The `docker run`/`docker exec` commands above only set `NXF_VER`, `LANG`, and `LC_ALL` inside the container, so it does not inherit the host shell's `CLAUDECODE` variable and Nextflow 26.04+'s agent-mode console output is not triggered here. See [Console Output Mode](../shared/repo-conventions.md#console-output-mode-critical) in repo-conventions — do not add `-e CLAUDECODE` or otherwise forward the host environment into this container, as that would reintroduce the problem.