| name | osdc-project-structure |
| description | OSDC platform architecture, project organization, directory tree, key design decisions, external knowledge base, key files, and docs index. Applies to ~/meta/ci-infra/osdc. Load when working on OSDC to understand project layout and architecture.
|
OSDC Project Structure & Architecture
What This Is
Modular Kubernetes infrastructure platform on AWS EKS. A shared base/ provides the cluster (VPC, EKS, Harbor, GPU plugins, NodeLocal DNS), and optional modules/ layer services on top (ARC, runners, BuildKit, future projects). One codebase drives multiple clusters across regions via clusters.yaml.
Working directory: osdc/. Run all commands from here.
Project Organization — Critical
Before making ANY changes, understand how this project is organized. This project MUST be well-organized at all times:
- Base vs modules: Shared infrastructure goes in
base/. Optional services go in modules/<name>/. If every cluster needs it, it's base.
- Folder separation is mandatory: Separate directories for
terraform/, kubernetes/, helm/, scripts/, docker/ within each component.
- Technology separation: Don't mix Python, Terraform, Bash, Helm, and Kubernetes manifests in the same directory.
- Cloud separation: Separate cloud-specific code (AWS/Kubernetes) from cloud-agnostic definitions (runner defs).
Before planning changes:
- FIRST: Examine the current project structure
- THEN: Plan changes to maintain or improve the organization
- NEVER: Mix unrelated files or technologies in the same directory
osdc/
├── clusters.yaml # THE source of truth — clusters + module lists
├── justfile # All operations (deploy, lint, test, setup)
├── mise.toml # Tool versions (tofu, kubectl, helm, crane, ruff, etc.)
├── pyproject.toml # Python deps + dev deps (managed by uv)
├── scripts/ # Orchestration helpers
│ ├── cluster-config.py # Reads clusters.yaml, outputs values for just/shell
│ ├── bootstrap-state.sh # Creates S3 + DynamoDB for tofu state
│ ├── mise-activate.sh # Sourceable helper for shebang recipes needing mise tools
│ └── python/
│ └── configure_harbor_projects.py # Harbor proxy cache project setup
├── base/ # Deployed to EVERY cluster
│ ├── kubernetes/ # StorageClass, NVIDIA plugin, Harbor NS, perf tuning, image-cache-janitor, nodelocaldns, node-taint-remover, registry-mirror config, git-cache, algif-mitigation (CVE-2026-31431, temporary), dirtyfrag-mitigation (CVE-2026-43284 + CVE-2026-43500, temporary)
│ │ ├── eniconfigs/ # AZ-named ENIConfig CRs (one per AZ from private_subnets_by_az terraform output); inert until VPC CNI Custom Networking is enabled. Not used under IPv6-only mode (Custom Networking is IPv4-only); retained as IPv4 rollback safety net
│ │ ├── harbor/ # PDB template (sed-substituted, applied by _deploy-harbor)
│ │ ├── image-cache-janitor/ # Cleans up unused cached container images on nodes
│ │ ├── nodelocaldns/ # NodeLocal DNSCache — per-node CoreDNS DaemonSet (iptables-mode interception); deployed via own deploy.sh (resolves kube-dns ClusterIP at apply time)
│ │ ├── node-taint-remover/ # Operator that strips startup taints once their gating DaemonSets are Ready; ConfigMap registry generated by lib/taint_remover.py, consumed by cache-enforcer, registry-mirror-config, node-performance-tuning
│ │ └── tests/ # Base-level Kubernetes test fixtures
│ ├── helm/ # Harbor values
│ ├── docker/ # Container images (runner-base)
│ ├── scripts/ # Bootstrap (EKS node setup)
│ └── node-compactor/ # Node consolidation controller (taints underutilized nodes)
├── modules/ # Optional, per clusters.yaml
│ ├── eks/ # AWS infrastructure (VPC, EKS, Harbor S3/IAM, image mirroring)
│ ├── karpenter/ # Karpenter controller + AWS infra (IAM, SQS, EventBridge)
│ ├── keda/ # KEDA operator (provides ScaledObject/TriggerAuthentication CRDs used by buildkit autoscaling)
│ ├── arc/ # ARC controller (GitHub Actions)
│ ├── nodepools/ # Karpenter NodePools (pure compute provisioning)
│ ├── nodepools-b200/ # B200 (Blackwell) GPU nodepools
│ ├── nodepools-h100/ # H100 (Hopper) GPU nodepools
│ ├── arc-runners/ # ARC runner scale sets (requires arc + nodepools)
│ ├── arc-runners-b200/ # B200 GPU runner scale sets
│ ├── arc-runners-h100/ # H100 GPU runner scale sets
│ ├── buildkit/ # BuildKit build service (arm64 + amd64, HAProxy LB)
│ ├── cache-enforcer/ # DaemonSet blocking direct registry/PyPI access (forces Harbor + pypi-cache)
│ ├── harbor-cache-recovery/ # Recovers Harbor cache state after pod restarts/failures
│ ├── logging/ # Centralized log collection (Alloy DaemonSet for logs + Deployment for events -> Grafana Cloud Loki)
│ ├── monitoring/ # Metrics pipeline: kube-prometheus-stack CRDs/exporters + Alloy -> Grafana Cloud
│ ├── pypi-cache/ # Self-learning PyPI wheel cache (nginx + pypiserver)
│ └── zombie-cleanup/ # Periodic cleanup of orphaned/zombie pods and resources
├── integration-tests/ # Top-level integration suites (load-test, workflows, workload-test, scripts, docker)
├── tests/smoke/ # Top-level smoke tests
└── docs/ # Architecture and operations docs
Key Design Decisions
- Harbor is always-on — baked into base, not a module. Every cluster needs a pull-through cache.
- NVIDIA device plugin in base — foundational infrastructure, not optional.
- Karpenter is a module — allows clusters to opt-in to Karpenter. The controller, IAM, SQS, and EventBridge are all in
modules/karpenter/. NodePools are separate in modules/nodepools/.
- Compute split: nodepools + arc-runners —
nodepools deploys pure Karpenter NodePools (compute provisioning). arc-runners deploys ARC runner scale sets (requires both arc and nodepools). Non-ARC clusters can use nodepools alone for compute.
- One terraform root, many clusters — no code duplication per environment.
- Modules are independent — each has its own terraform state, k8s resources, deploy script. No cross-module imports.
- Monitoring and logging are both modules — metrics collection (Alloy Deployment -> Grafana Cloud Mimir) is opt-in via the
monitoring module. Log collection (Alloy DaemonSet for pod/journal logs + Alloy Deployment for Kubernetes events -> Grafana Cloud Loki) is in modules/logging/. Both are secret-gated — no credentials, no Alloy.
- Three Alloy installations — separate Helm releases, namespaces, and RBAC for: monitoring metrics (Deployment,
modules/monitoring/helm/alloy-values.yaml), logging pod/journal logs (DaemonSet, modules/logging/helm/alloy-logging-values.yaml), and logging Kubernetes events (Deployment, modules/logging/helm/alloy-events-values.yaml). This avoids config/permission conflicts and lets each scale independently. See osdc-observability skill for details.
External Knowledge Base
The actions-knowledge-base/ directory contains source code, documentation, and detailed reference material for many of the open-source projects and tools used by this project — including Actions Runner Controller, Harbor, the Harbor Helm chart, the GitHub Actions runner, runner images, cloud-provider credential actions, and more. Each project lives as a git submodule under repos/ and the knowledge base's own AGENTS.md indexes every included repository with summaries and key paths.
Finding it: The directory is named actions-knowledge-base and lives somewhere above or beside this project. Walk upward from the current working directory, checking each ancestor and its children, until you find a directory named actions-knowledge-base. It is typically a sibling of the top-level repo (e.g., beside ci-infra/), but the exact location depends on the checkout layout. Do not hardcode a relative path — search for it.
The knowledge base has two key directories:
actions-knowledge-base/repos/ — Read-only upstream source code and docs (git submodules). Managed via sync.py.actions-knowledge-base/docs/ — Our own findings, workarounds, and gotchas discovered during development and operations (e.g., BuildKit OTEL crash, Grafana Alloy setup, CRD ordering issues, deploy phase ordering).
You should consult this knowledge base whenever you need to:
- Understand how a dependency works (e.g., ARC Helm chart values, Harbor configuration, runner lifecycle)
- Clarify configuration quirks, edge cases, or undocumented behavior
- Look up default values, API surfaces, or internal implementation details
- Verify correct usage of upstream Helm charts, container images, or action inputs
- Review operational learnings and previously discovered issues (
docs/)
In most cases, reading the relevant source or docs in the knowledge base first will produce more accurate results than guessing or relying on general knowledge alone. Treat it as a first-class resource.
Key Files
| File | What it does |
|---|
clusters.yaml | Defines all clusters, modules, and per-installation config (replicas, log levels, runner limits, etc.) |
justfile | All operations (deploy, lint, test, setup, kubeconfig) |
mise.toml | Tool versions (tofu, kubectl, helm, crane, ruff, shellcheck, etc.) + env vars |
pyproject.toml | Python dependencies + dev deps, managed by uv |
scripts/cluster-config.py | Reads clusters.yaml for justfile/shell consumption |
scripts/bootstrap-state.sh | Creates S3 bucket + DynamoDB for tofu state |
scripts/destroy-cluster.sh | Operator-facing helper for tearing down a cluster's tofu-managed infrastructure |
scripts/cleanup-git-cache.sh | Transitional operator helper — removes orphaned in-cluster git-cache resources left over from #703 code removal; delete once all clusters are cleaned |
scripts/mise-activate.sh | Sourceable helper — adds mise tools to PATH for shebang recipes |
scripts/deploy-log.sh, scripts/deploy-status.py | Deploy logging + status reporting helpers (called by justfile) |
scripts/helm-upgrade.sh, scripts/kubectl-apply.sh, scripts/yaml-diff.py, scripts/kubeconfig-lock.sh | Wrappers used by module deploy.sh scripts |
scripts/python/configure_harbor_projects.py | Configures Harbor proxy cache projects via API |
scripts/python/ (analysis) | analyze_node_utilization.py, simulate_cluster.py, instance_specs.py (two dicts: INSTANCE_SPECS compute/memory/arch keyed by AWS instance type, ENI_MAX_PODS AWS-stock max-pods from awslabs/amazon-eks-ami eni-max-pods.txt, used by simulators and analyzers), pytorch_workload_data.py, daemonset_overhead.py, , — capacity / utilization analysis tooling |
Base Component Notes
Node Taint Remover
base/kubernetes/node-taint-remover/ runs an operator that strips startup taints once their gating DaemonSets are Ready on a node. The taint registry is generated by lib/taint_remover.py and consumed by base DaemonSets (cache-enforcer, registry-mirror-config, node-performance-tuning). When adding a new startup-taint-bearing DaemonSet to base or a module, register the taint here.
Node Compactor
base/node-compactor/ is a node consolidation controller that taints underutilized Karpenter nodes to compact workloads WITHOUT eviction. It relies on Karpenter's WhenEmpty consolidation to clean up empty nodes. See osdc-runners-nodepools skill for configuration details.
Documentation (docs/)
Keep this list in sync. When adding, removing, or significantly editing a document in docs/, update this table (description, filename, or entry presence).
| Document | Description |
|---|
arc-fork-build-deploy.md | Building and deploying the ARC controller fork (image build, push, version pinning) |
architecture.md | Modular platform architecture — base vs modules, deployment layers |
ci-pre-merge.md | Pre-merge CI workflow and gating |
cluster-recreation.md | Operator runbook for destroying and recreating an OSDC cluster (per-cluster cutover, accepted data losses for Harbor S3 + EFS pypi-cache, pre-flight checks) |
current_runner_load_distribution.md | 30-day job counts and peak concurrency analysis by runner type |
loki_query.md | CLI guide for querying Grafana Cloud Loki logs (credentials, LogQL patterns) |
mimir_query.md | CLI guide for querying Grafana Cloud Mimir metrics (credentials, PromQL patterns) |
modules.md | Module contract and deployment order (Terraform -> Kubernetes -> deploy.sh) |
node-utilization-optimization.md | Runner-to-node packing efficiency analysis and instance type recommendations |
node-warmup-and-scheduling-gates.md | Node startup flow — startup taints, gating DaemonSets (hooks, registry mirrors, perf tuning), init container gates, taint summary |
observability.md | Three-Alloy observability architecture (metrics Deployment + logs DaemonSet + events Deployment -> Grafana Cloud) |
observability-estimates.md | Per-unit cost estimates for metrics cardinality and log volume (Grafana Cloud billing) |
operations.md | Operational procedures — bootstrap, deploy, and manage clusters |
prod-cluster-ha-us-west-1.md | Production cluster HA design notes for us-west-1 |
pypi-package-cache.md | PyPI wheel cache architecture and operational guide |
runner-image-autoresolve.md | Deploy-time auto-resolution of the runner image to a digest-pinned reference; ConfigMap shape, rollback paths, failure modes |
