| name | osdc-deployment |
| description | OSDC deployment workflow, just recipes, base deploy order, module deploy order, clusters.yaml configuration, Terraform architecture, smoke tests, justfile conventions. Applies to ~/meta/ci-infra/osdc. Load when deploying, adding modules, or modifying deploy scripts.
|
OSDC Deployment & Operations
How Deployment Works
clusters.yaml defines every cluster and its modules. The justfile reads it via scripts/cluster-config.py.
just setup
just clean
just list
just show <cluster>
just kubeconfig <cluster>
just bootstrap <cluster>
just bootstrap-all
just deploy <cluster>
just deploy-base <cluster>
just deploy-module <cluster> <module> [force]
just plan <cluster>
just deploy-history <cluster>
just deploy-status <cluster> [name]
just recycle-nodes <cluster>
just test
just test-compactor <cluster>
just test-janitor <cluster>
just smoke <cluster>
just integration-test <cluster> [args...]
just generate-arc-runners <cluster>
just lint
just lint-fix
just analyze-utilization
just simulate-cluster
just taint-nodes <cluster>
just untaint-nodes <cluster>
just drain-runners <cluster>
just resume-runners <cluster>
just remove-module <cluster> <module>
just heal-arc-check <cluster> [name]
just heal-arc <cluster> [name]
just load-test <cluster>
just workload-test <cluster>
Internal recipes (prefixed _, called from other recipes; documented here because they expose load-bearing behavior):
_deploy-harbor <cluster> — called from deploy-base. Reads harbor_role_arn, harbor_s3_bucket, harbor_s3_access_key_id, harbor_s3_secret_access_key from base tofu outputs; installs the Harbor Helm chart; applies the standalone Harbor PDB manifest (chart has no native PDB support); then calls _configure-harbor-projects._configure-harbor-projects — bootstraps Harbor proxy-cache projects via the Harbor REST API. Optionally consumes harbor-dockerhub-credentials and harbor-github-credentials secrets in harbor-system to authenticate proxy-cache pulls against Docker Hub / ghcr.io (avoids anonymous rate limits). Create those secrets out-of-band before deploy if needed.
Environment variables that control deploy behavior
OSDC_CONFIRM=yes|no|ask (default ask) — non-interactive confirmation for just deploy and just deploy-base plans.OSDC_TAINT_NODES=yes|no|ask (default ask) — auto-taint ARC runner nodes after just deploy. The deploy recipe consults this only when recycle_karpenter_nodes != true; under recycle, tainting is skipped unconditionally (nodes are already being destroyed).OSDC_SMOKE=yes|no|ask (default ask) — auto-run smoke tests at end of just deploy.OSDC_DRAIN_TIMEOUT_SECS=<seconds> (default 3600) — wait budget for just drain-runners while in-flight runner pods finish before declaring stragglers and exiting non-zero.OSDC_CONFIRM_BASE=destroy — required in addition to OSDC_CONFIRM=yes for scripts/destroy-cluster.sh to non-interactively destroy base infrastructure. Not consumed by just deploy; included here because it sits on the same operational surface.OSDC_RESOLVER_READONLY=1 — set by the smoke recipe around its generate-arc-runners preflight so the runner-image resolver reads the newest cached entry from the lock ConfigMap instead of mutating it or hitting GitHub. Anyone running generate-arc-runners standalone for testing should NOT set this unless they explicitly want the cached behavior.HELM_FORCE_UPGRADE=1 — passed by deploy-module when invoked with the force arg; modules' deploy.sh use this to force Helm upgrades.AGENT_ENVIRONMENT=true — agent mode for just lint, just test, just smoke, just generate-arc-runners: suppresses output on success, prints only on failure. Useful for sub-agents.OSDC_ROOT, OSDC_UPSTREAM, CLUSTERS_YAML — exported by deploy-module, deploy-base, drain-runners, resume-runners, smoke, etc., into the environment of every module deploy.sh. OSDC_ROOT is the consumer repo root (often equal to OSDC_UPSTREAM); OSDC_UPSTREAM is the upstream OSDC checkout providing shared scripts and modules. Module scripts that source shared helpers (mise-activate.sh, state-config.sh, kubectl-apply.sh, etc.) MUST use OSDC_UPSTREAM so they work both in standalone and consumer-override setups.
Base deploy order (always)
- Terraform — VPC, EKS, Harbor S3 bucket + IAM role + S3 access keys (all produced by
modules/eks/terraform, parameterized, no per-env dirs)
- Mirror images — Harbor bootstrap images to ECR (Harbor can't cache itself)
- Base k8s — StorageClass, NVIDIA plugin, performance tuning (
kubectl apply -k)
- node-taint-remover — Generates the
node-taint-remover-lib ConfigMap in kube-system from lib/taint_remover.py (generated at apply time to prevent drift). Mounted into DaemonSets (cache-enforcer, registry-mirror-config, node-performance-tuning) so they can remove their own startup taint after init.
- ENIConfigs — One AZ-named ENIConfig CR per AZ, rendered from
private_subnets_by_az (base terraform output). Inert by design under IPv6-only EKS — VPC CNI Custom Networking is unsupported in IPv6 mode. Retained on the deploy path as the rollback fallback for IPv4 + Custom Networking. Has its own deploy.sh because it must read the terraform output at apply time.
- Harbor — Pull-through cache (caches docker.io, ghcr.io, nvcr.io, registry.k8s.io, quay.io, public.ecr.aws).
_deploy-harbor reads harbor_role_arn, harbor_s3_bucket, harbor_s3_access_key_id, harbor_s3_secret_access_key from base outputs.
- Node compactor — Taints underutilized Karpenter nodes for consolidation (if enabled)
- Image Cache Janitor — Cleans up unused cached images on nodes
- NodeLocal DNSCache — Per-node CoreDNS DaemonSet (iptables-mode interception). Has its own
deploy.sh (NOT in base/kubernetes/kustomization.yaml — same pattern as image-cache-janitor) because it must resolve the live kube-dns Service ClusterIP at apply time and substitute it into Corefile + DaemonSet args.
Module deploy order (per clusters.yaml list order)
Each module may have:
| File | Purpose | Applied by justfile |
|---|
terraform/main.tf | AWS resources (optional) | tofu apply with cluster vars |
kubernetes/kustomization.yaml | K8s manifests (optional) | kubectl apply -k |
deploy.sh | Custom deploy logic (optional) | Called with (cluster-id, cluster-name, region) |
The justfile auto-detects which of these exist and runs them in order: terraform -> kubernetes -> deploy.sh.
deploy-module exports three env vars consumed by every module deploy.sh: OSDC_ROOT (consumer repo), OSDC_UPSTREAM (upstream OSDC checkout), and CLUSTERS_YAML (path to the active clusters.yaml). It also resolves the module directory with consumer-first precedence: if $OSDC_ROOT/modules/$MODULE exists, it wins over $OSDC_UPSTREAM/modules/$MODULE. This is how downstream consumers override upstream modules without forking the repo. See docs/modules.md for the full module contract.
clusters.yaml drives everything
There is no hardcoded staging/production concept. Each cluster is an independent installation with its own config. clusters.yaml controls everything: infrastructure sizing, component tuning (Harbor replicas, Karpenter PDB, ARC log level), module-specific settings (BuildKit replicas, runner GitHub config, max runner counts), and module list.
Adding a new cluster: add an entry to clusters.yaml with region, cluster_name, state_bucket, base config, component tuning, module config, and module list. Then just bootstrap <cluster-id> + just deploy <cluster-id>.
Adding a module to a cluster: append the module name to the cluster's modules list.
To create a new module: make a directory under modules/, add any of the files above, and list the module name in clusters.yaml for the target cluster.
Reading cluster config in scripts: uv run scripts/cluster-config.py <cluster-id> <dot.path> [default]. Supports nested paths (e.g., harbor.core_replicas). Falls back to defaults: section, then optional third argument.
recycle_karpenter_nodes: true (cluster-level config, e.g. meta-staging-aws-uw1, meta-staging-aws-ue1): when set, just deploy calls just recycle-nodes "$CLUSTER" automatically after modules deploy and skips the ARC node-tainting step (tainting is pointless when nodes are being destroyed).
proactive_capacity_max: <int> (cluster-level): caps the warm-pool placeholder pairs the runner-set scaler will provision ahead of demand. Staging clusters set this to 0 to disable pre-provisioning entirely. See docs/arc-fork-build-deploy.md for how this is plumbed into the controller (CAPACITY_AWARE_PROACTIVE_CAPACITY).
Justfile Kubeconfig Convention (MANDATORY)
Every just recipe that interacts with a live cluster MUST call just kubeconfig <cluster> before doing any kubectl/helm work. This prevents operators from accidentally running commands against the wrong cluster.
The kubeconfig recipe wraps aws eks update-kubeconfig via scripts/kubeconfig-lock.sh — a directory-based mutex (atomic mkdir) that serializes concurrent kubeconfig updates and prevents races between parallel deploys. The wrapper enforces a 90s acquisition timeout and breaks stale locks older than 45s.
Note: just plan does NOT call just kubeconfig — it is read-only tofu and does no k8s/helm work, so it is intentionally exempt from the kubeconfig contract.
The pattern (in shebang recipes):
export CLUSTERS_YAML="{{CLUSTERS_YAML}}"
just kubeconfig "$CLUSTER"
This applies to: deploy-base, deploy-module, recycle-nodes, taint-nodes, untaint-nodes, drain-runners, resume-runners, _deploy-harbor, smoke, test-compactor, test-janitor, integration-test, deploy-history, deploy-status, and any future recipe that calls kubectl, helm, or deploy scripts. When adding a new recipe that touches a cluster, always include the just kubeconfig step. Do NOT call aws eks update-kubeconfig directly — the locking wrapper is required.
Justfile + Mise Gotcha
Tool versions for the entire repo are pinned in mise.toml (tofu, kubectl, helm, crane, awscli, ruff, taplo, shellcheck, shfmt, hadolint, kube-linter, kubeconform, tflint, trivy, uv, python, etc.). The [settings] auto_install = true setting means mise exec installs missing pinned tools on first use — there is no separate "install tools" command; just setup only runs uv sync for Python deps.
The justfile uses set shell := ["mise", "exec", "--", "bash", "-euo", "pipefail", "-c"] so that mise-managed tools are on PATH for every recipe line.
Shebang recipes bypass this. A recipe starting with #!/usr/bin/env bash is executed as a standalone script by just, NOT through the configured shell. This means mise-managed tools may not be on PATH inside shebang recipes or scripts they call. Shebang recipes in this repo work around it by sourcing scripts/mise-activate.sh as their first line.
For recipes that need mise tools, use non-shebang style (line-by-line with @ prefix) so they run through mise exec. If a shebang recipe is required, source mise-activate.sh (or call subscripts via mise exec -- ./script.sh explicitly).
Terraform Architecture
Single parameterized root at modules/eks/terraform/. No per-environment directories.
- Variables (
cluster_name, aws_region, vpc_cidr, etc.) come from clusters.yaml -> justfile -> tofu -var=
- Backend configured at
tofu init via -backend-config (bucket, key, region)
- Each cluster gets its own state:
s3://<state_bucket>/<cluster-id>/base/terraform.tfstate
- Modules with their own terraform get:
s3://<state_bucket>/<cluster-id>/<module>/terraform.tfstate. Module terraform also receives state_bucket and cluster_id as vars.
Smoke Tests (MANDATORY)
Post-deploy validation that a cluster is actually healthy. Run via just smoke <cluster> or automatically at the end of just deploy (controlled by OSDC_SMOKE=yes|no|ask).
Smoke tests verify that what was deployed is actually working: Helm releases are deployed, pods are Running, DaemonSets have all pods ready, required secrets and service accounts exist, AWS resources (IAM roles, SQS queues) are present, and Karpenter NodePools / ARC runner scale sets match their definition files. They do NOT test application-level behavior — just that the deployment landed correctly.
Architecture: Each module owns its own smoke tests, co-located at <module>/tests/smoke/. A root tests/smoke/conftest.py provides shared fixtures (batch-fetched cluster state, cluster config, CLI helpers) that all module tests inherit via pytest's conftest discovery. just smoke <cluster> reads clusters.yaml, finds enabled modules, collects their tests/smoke/ directories, and runs pytest across all of them. The recipe also unconditionally pulls smoke tests from base/**/tests/smoke AND modules/eks/tests/smoke (EKS is always deployed as part of base, regardless of clusters.yaml.modules). Adding a module with smoke tests = they're discovered automatically. Moving a module = its tests move with it.
Preflight: ARC runner YAML pre-generation. Before any pytest worker starts, the smoke recipe iterates the cluster's enabled modules and runs just generate-arc-runners "$CLUSTER" for every arc-runners / arc-runners-* variant (canonical + per-GPU-arch), parameterized via ARC_RUNNERS_DEFS_DIR / ARC_RUNNERS_OUTPUT_DIR / ARC_RUNNERS_MODULE_NAME. This regeneration MUST happen once up front — previously a test fixture shelled out per-test and raced under pytest-xdist. Anyone debugging "why does smoke regenerate before tests run" is looking at this step.
When changing ANY module or base component, you MUST update its smoke tests to cover the change. If the component doesn't have smoke tests yet, add them. Smoke tests are the safety net that catches deployment regressions — they only work if they stay current.
- Smoke test files live in
<component>/tests/smoke/test_*.py
- Shared helpers live in the
tests/smoke/helpers/ package:
helpers/cli.py — run_kubectl, run_helm, run_aws (proxy-bypass-aware)helpers/filters.py — pure in-memory filters over batch fixtureshelpers/nodes.py — node-churn helpers (unstable nodes, disruption taints)helpers/retry.py — retry_with_backoff primitivehelpers/k8s_asserts.py — DaemonSet / Deployment readiness assertionshelpers/remote.py — Mimir / Loki query helpers with skip-on-network-failure semanticshelpers/__init__.py re-exports everything (from helpers import X works without naming the submodule)
- Each module's
tests/smoke/conftest.py re-exports the root conftest: from smoke_conftest import *
- DaemonSet checks use
assert_daemonset_healthy() which tolerates mismatches caused by nodes in transition (new, NotReady, or being deleted). Deployment checks use assert_deployment_ready() which has a 90-second rollout-completion window followed by the standard retry_with_backoff schedule once the rollout finishes but replicas still aren't ready
- Retry primitive:
retry_with_backoff(check, refresh=None, delays=None) is the canonical retry abstraction for smoke tests. Retries only on AssertionError — other exceptions propagate immediately. Optional refresh callable runs between attempts to re-fetch live state. Re-raises the LAST AssertionError after exhaustion.
- Backoff schedule (resolved at module load time from
os.environ.get("CI")):
- Local (default): 6 attempts total — 1 initial + 5 retries. Sleeps
[10, 17, 28.9, 49.1, 83.4] seconds. Worst-case sleep budget per assertion: ~188s.
- CI (
CI=true, set by Buildkite/GHA/etc.): 8 attempts total — extended to [10, 17, 28.9, 49.1, 83.4, 141.78, 241.03] to give slow recoveries more time. Worst-case sleep budget: ~571s per assertion.
- When writing new smoke tests: use
retry_with_backoff for any "wait for X to be ready" pattern — do NOT hand-roll for _ in range(N): time.sleep(D) loops.
Unit Tests & Coverage
just test discovers test_*.py files across the repo, runs them with pytest-xdist (-n auto), and enforces a per-file coverage threshold of 95%. Any source file under the test dirs whose coverage drops below 95% will fail the run, even if pytest itself passed. Add or extend tests when you change source files — bumping global coverage is not enough.
Exclusions and gotcha around smoke tests:
- Excluded entirely:
tests/e2e/, */base/*/tests/smoke/, */modules/*/tests/smoke/ (live-cluster suites — run via just smoke <cluster>).
- Included: the project-root
tests/smoke/ directory, because it hosts unit tests for the smoke-helpers package itself (e.g. tests/smoke/helpers/test_retry.py). The recipe runs those tests but exempts them from the 95% per-file coverage gate (the helpers' full coverage comes from live-cluster smoke runs).
Module-Specific Deployment Notes
EKS Terraform State Key Gotcha
The eks module uses <cluster-id>/base/terraform.tfstate as the state key (historical, predates module extraction). Don't assume the state key matches the module name eks. Modules reading eks outputs via terraform_remote_state (e.g., karpenter reads OIDC provider, node role, SG, subnets) depend on this key.
Base Node Group — Hidden Behaviors
The aws_eks_node_group "base" resource at osdc/modules/eks/terraform/modules/eks/main.tf has two non-obvious behaviors that matter when changing base_node_count or base_node_instance_type in clusters.yaml.
base_node_count is half-applied by tofu
lifecycle.ignore_changes = [scaling_config[0].desired_size] means tofu updates min and max but NOT desired. After bumping base_node_count, you MUST manually run:
NO_PROXY="$NO_PROXY,.eks.amazonaws.com" no_proxy="$no_proxy,.eks.amazonaws.com" \
aws eks update-nodegroup-config \
--cluster-name <cluster_name> \
--nodegroup-name <cluster_name>-base-nodes \
--region <region> \
--scaling-config minSize=N,maxSize=N,desiredSize=N
Skipping this leaves the ASG at the OLD desired_size. Symptom: pods go Pending despite the new count being "applied".
base_node_instance_type change triggers an in-place rolling replacement
EKS treats an instance_types change as a managed node group update: every node is cordoned, drained, and replaced.
- Honors PDBs for 15 minutes (the
update_config timeout)
- Then forces eviction because
force_update_version = true
- Blast radius controlled by
base_node_max_unavailable_percentage in clusters.yaml (default 33%, staging 100%)
Monitor the rolling update with:
aws eks list-updates --cluster-name <cluster_name> --nodegroup-name <cluster_name>-base-nodes --region <region>
aws eks describe-update --cluster-name <cluster_name> --nodegroup-name <cluster_name>-base-nodes --region <region> --update-id <ID>
For one-time migrations (e.g. switching to a larger instance family), follow the cluster-specific playbook if one exists at the repo root — disruption coordination, manual desired_size bump, and rollback steps live there, not here. If no playbook exists for your migration, write one before starting.
Cluster recreation runbook:
docs/cluster-recreation.md — operator-facing runbook for destroying and recreating an existing OSDC cluster (used for any change that touches an immutable aws_eks_cluster property: IP family, VPC, CIDR, encryption_config, etc.). Cluster recreates are required because EKS does not support in-place IPv4↔IPv6 conversion. Automation: scripts/destroy-cluster.sh <cluster-id> (requires OSDC_CONFIRM=yes + OSDC_CONFIRM_BASE=destroy for non-interactive destroy).
CRD Ordering — Monitoring Module
Monitoring deploy order is strict: (1) namespace + DCGM DaemonSet via kustomization, (2) kube-prometheus-stack Helm (installs CRDs + exporters), (3) monitors/ applied AFTER CRDs exist, (4) Alloy conditionally. Applying ServiceMonitor/PodMonitor resources before kube-prometheus-stack installs the CRDs will fail.
Cache-Enforcer Dependency
cache-enforcer depends on Harbor (base) and pypi-cache module. Without pypi-cache deployed, pip installs in runner jobs fail entirely (traffic blocked but no cache). Deploy pypi-cache before or alongside cache-enforcer.
clusters.yaml Config Keys Reference
clusters.yaml itself is the source of truth — the section below is a curated subset highlighting load-bearing or non-obvious keys. For the full schema (every defaults: and every per-cluster override), read clusters.yaml directly.
Cluster-level keys (set under clusters.<id>:, no fallback to defaults:):
Karpenter:
karpenter:
replicas: 4
log_level: info
pdb_enabled: true
pdb_min_available: 1
Harbor:
harbor:
core_replicas: 4
registry_replicas: 4
nginx_replicas: 6
pdb_max_unavailable: 1
Standalone PDB manifests are applied by _deploy-harbor after the Helm install (Harbor chart has no native PDB support). See osdc-harbor skill for selector details and known limits.
ARC controller:
arc:
chart_version: "0.14.1-jeanschmidt.16"
runner_image_tag: "2.334.0"
replica_count: 4
log_level: info
controller_cpu_request: "1"
controller_cpu_limit: "4"
controller_memory_request: "2Gi"
controller_memory_limit: "4Gi"
BuildKit: Defaults provide only replicas_per_arch: 12. All other knobs are per-cluster overrides:
defaults:
buildkit:
replicas_per_arch: 12
clusters:
my-cluster:
buildkit:
amd64_instance_type: m6id.24xlarge
amd64_replicas: 32
amd64_pods_per_node: 2
arm64_instance_type: m7gd.16xlarge
arm64_replicas: 8
arm64_pods_per_node: 4
Per-arch override keys are separate (amd64_* and arm64_*) — there is no flat pods_per_node key.
Node Compactor: All config under node_compactor: key. Knobs configurable via clusters.yaml: enabled, interval_seconds (20), dry_run, min_nodes (1), min_node_age_seconds (900), capacity_reservation_nodes (0), max_uptime_hours (48). Other knobs (taint_rate, fleet_cooldown, spare_capacity_nodes, spare_capacity_ratio, spare_capacity_threshold) are internal Python defaults in the compactor source, NOT configurable via clusters.yaml.
Other defaults blocks (curated — see clusters.yaml defaults: for the full list and current values):
coredns.replicas — CoreDNS replica count.nodepools.{gpu_consolidate_after,cpu_consolidate_after,baremetal_consolidate_after,gpu_disruption_budget,cpu_disruption_budget} — Karpenter NodePool churn knobs.monitoring.{grafana_cloud_url,grafana_cloud_read_url} — Grafana Cloud Mimir push/read endpoints.logging.grafana_cloud_loki_url — Grafana Cloud Loki push endpoint.alloy_chart_version — Alloy Helm chart version (shared by monitoring + logging Alloy DaemonSets).keda.chart_version — KEDA operator Helm chart version (used by the keda module; consumed by buildkit autoscaling).zombie_cleanup.{enabled,pending_max_age_hours,running_max_age_hours,dry_run} — orphan-pod GC tuning.pypi_cache.{instance_type,cuda_versions,python_versions,target_architectures,target_manylinux} — PyPI wheel cache matrix (see osdc-pypi-cache for slug semantics; cuda_versions and python_versions are mandatory for this module).
ARC Module — No Terraform
Unlike most modules, modules/arc/ has no terraform phase — it is pure k8s/helm. Don't look for terraform state or try just tf-* recipes for ARC.
Pypi-Cache — Two Terraform Roots
The pypi-cache module has TWO terraform roots; only one runs as part of just deploy:
modules/pypi-cache/terraform/main.tf — per-cluster IRSA role, IAM policies, etc. Auto-applied by just deploy-module pypi-cache (the standard deploy-module flow detects terraform/main.tf and runs tofu init && tofu apply on it).modules/pypi-cache/terraform/wheel-cache-bucket/ — standalone tofu root for the shared S3 bucket (pytorch-pypi-wheel-cache). This is a one-time, account-wide resource, NOT part of just deploy. Run tofu init && tofu apply manually in this sub-directory if the bucket needs to be created or updated.
Pypi-Cache — Required clusters.yaml Keys
cuda_versions and python_versions are NOT module defaults. They must be configured in clusters.yaml under defaults.pypi_cache or per-cluster clusters.<id>.pypi_cache. Omitting them produces zero CUDA slugs and zero deployments. cuda_slug() normalizes versions to major.minor only: "12.8.1" → cu128 (matches PyTorch convention).
CI Workflows
OSDC is driven from CI via GitHub Actions workflows living in the parent repo (ci-infra/.github/workflows/), NOT inside osdc/. There is no osdc/.github/ directory.
Key workflows (all callable as reusable workflows via workflow_call from the orgs that ship from this repo):
| Workflow | What it runs |
|---|
osdc-lint.yml | just lint |
osdc-test.yml | just test |
osdc-pre-merge.yml | Lint + test on PR |
osdc-plan-prod.yml | just plan <cluster> (read-only tofu plan; CI-safe) |
osdc-deploy-prod.yml | Per-cluster prod deploy. Calls _osdc-deploy.yml |
_osdc-deploy.yml | Reusable: just lint && just test → just deploy <cluster> → just smoke <cluster> → just integration-test <cluster> --skip-drain --skip-smoke --skip-compactor. Pre-flight checks can be skipped with the skip-checks input (firefighting only). |
_osdc-plan.yml | Reusable: just plan <cluster> and tee to plan.txt |
_osdc-slow-tests.yml | Reusable: just load-test, just test-compactor, just test-janitor |
osdc-capacity-report.yml | just simulate-cluster + just analyze-utilization, periodic |
When adding a new just recipe that CI should run, plumb it through the corresponding reusable workflow.
