osdc-cli-debugging

# OSDC CLI Debugging (Read-Only) **You are encouraged to run read-only CLI commands to debug, investigate, and understand cluster and infrastructure state.** This is the fastest way to diagnose issues — use it proactively. All tools below are managed by mise via `osdc/mise.toml`. Run commands from the `osdc/` working directory so mise activates the correct tool versions. If running from elsewhere, prefix with `mise exec --`: ```bash mise exec -- kubectl get pods -n arc-runners ``` ## clusters.yaml Location All clusters are defined in `clusters.yaml` in the project root. Run `just list` for full details. Read the file directly for config values. The `<cluster>` argument to every `just` recipe is the **YAML key** (e.g. `meta-prod-aws-ue1`, `meta-staging-aws-ue1`, `arc-cbr-production`), not the `cluster_name` field — they sometimes differ (e.g. `arc-cbr-production` resolves to `cluster_name: pytorch-arc-cbr-production`). ## lf-* Clusters Are NOT Accessible From This Laptop The `lf-prod-aws-ue1` / `lf-prod-aws-ue2` clusters (state buckets `lf-osdc-tfstate-prod-*`, admin role `lf_osdc_admin`) are operated by another org. **From this laptop the operator has no AWS credentials, no kubeconfig access, and no tofu state read access for them.** Any `just kubeconfig lf-prod-*`, `kubectl` against an `lf-*` context, `aws eks describe-cluster --name lf-prod-*`, or `tofu init` against an `lf-osdc-*` state bucket will fail on authentication. Accessible from this laptop (admin role `osdc_gha_prod` / `osdc_gha_staging`): - `meta-staging-aws-ue1` (us-east-1) - `meta-prod-aws-ue1` (us-east-1) - `arc-cbr-production` → `pytorch-arc-cbr-production` (us-east-2) For lf-* investigations: read `clusters.yaml`, read the module code, read the manifests in `base/` and `modules/` — but do not attempt live cluster commands. If the user explicitly asks for lf-* live state, tell them you cannot reach the cluster and ask them to run the command and paste output. ## Kubeconfig: Use `just kubeconfig <cluster>` (MANDATORY) Before running any kubectl or helm commands, configure kubeconfig using the just recipe — **never run `aws eks update-kubeconfig` directly.** ```bash just kubeconfig <cluster> ``` The just recipe reads `clusters.yaml` to resolve the cluster name and region automatically and delegates to `scripts/kubeconfig-lock.sh` (which serializes concurrent kubeconfig writes via a file lock). It aliases the kubectl context to the **cluster_name** (e.g. `pytorch-arc-cbr-production`, `meta-prod-aws-ue1`, `meta-staging-aws-ue1`) — there is no `eks-...` prefix. Switch contexts with `kubectl config use-context <cluster_name>` or scope a single call with `kubectl --context <cluster_name> ...`. Running `aws eks update-kubeconfig` by hand risks wrong cluster name, wrong region, or wrong kubeconfig context. ## Corporate Proxy Bypass for kubectl / aws / EKS (MANDATORY) This sandbox inherits Meta's x2p proxy (`HTTPS_PROXY=http://localhost:10054`), which blocks the EKS API endpoints. **Every ad-hoc `kubectl`, `aws eks`, or `helm` call against the cluster MUST inline the bypass** — `export` does not persist across separate Bash tool invocations, so prefix each command: ```bash NO_PROXY="${NO_PROXY:-},.eks.amazonaws.com" no_proxy="${no_proxy:-},.eks.amazonaws.com" kubectl get pods -A NO_PROXY="${NO_PROXY:-},.eks.amazonaws.com" no_proxy="${no_proxy:-},.eks.amazonaws.com" aws eks describe-cluster --name <cluster-name> --region <region> ``` The shipped recipes (`just kubeconfig`, `just deploy-*`, `scripts/destroy-cluster.sh`, etc.) already inject the bypass. The raw kubectl/aws/helm examples in the sections below omit the prefix only for readability — you still must add it when you actually run them. `aws` calls that do NOT touch the EKS API endpoint (`aws s3 ...`, `aws ec2 describe-*`, `aws autoscaling ...`, `aws ecr ...`, `aws iam ...`, etc.) generally work without the bypass, because Meta's proxy already whitelists the regular AWS service endpoints. Add the bypass when in doubt — it is harmless on non-EKS calls. ## Where Modules Deploy (Namespace Map) | Namespace | What lives there | |-----------|------------------| | `arc-systems` | ARC controller + per-runner-set listener pods | | `arc-runners` | Runner pods, AutoscalingRunnerSets, zombie-cleanup CronJob | | `karpenter` | Karpenter controller | | `harbor-system` | Harbor + harbor-cache-recovery CronJob | | `buildkit` | BuildKit builder pods | | `monitoring` | Alloy metrics + kube-prometheus-stack (default; overridable via `monitoring.namespace` in `clusters.yaml`) | | `logging` | Alloy logging + Alloy events (default; overridable via `logging.namespace` in `clusters.yaml`) | | `pypi-cache` | PyPI wheel cache pods | | `kube-system` | node-compactor, cache-enforcer, NVIDIA device plugin, registry mirror config, algif mitigation, dirtyfrag mitigation, node-performance-tuning, image-cache-janitor, node-local-dns DaemonSet (see `base/kubernetes/` for the full list) | | `osdc-system` | Deploy-audit ConfigMaps (labeled `app.kubernetes.io/managed-by=osdc-deploy-log`) read by `just deploy-status` and `just deploy-history` | ## kubectl (Kubernetes) ```bash kubectl get nodes # List nodes and status kubectl get pods -A # All pods across namespaces kubectl get pods -n arc-runners # Runner pods kubectl get pods -n arc-systems # ARC controller pods kubectl get pods -n karpenter # Karpenter pods kubectl get pods -n harbor-system # Harbor pods kubectl get pdb -n harbor-system # list Harbor PDBs and disruption budgets kubectl describe pdb harbor-core -n harbor-system # see selector matches and current/desired healthy kubectl get pods -n buildkit # BuildKit builder pods kubectl get pods -n monitoring # Alloy metrics + kube-prometheus-stack pods kubectl get pods -n logging # Alloy log collector pods kubectl get pods -n pypi-cache # PyPI wheel cache pods kubectl get ds -n logging # Logging DaemonSet (should match node count) kubectl get nodepools # Karpenter NodePools kubectl get nodeclaims # Karpenter NodeClaims (what Karpenter has provisioned) kubectl get ec2nodeclasses # Karpenter EC2NodeClasses (AMI / IAM / tag config) kubectl get autoscalingrunnersets -n arc-runners # ARC runner scale sets (one per runner type, named arc-<runner_name>) kubectl get pods -n arc-systems -l app.kubernetes.io/component=runner-scale-set-listener # ARC listener pods kubectl describe pod <pod> -n <ns> # Pod details and events kubectl logs <pod> -n <ns> # Pod logs kubectl get events -n <ns> --sort-by=.lastTimestamp # Recent events kubectl top nodes # Node resource usage kubectl top pods -n <ns> # Pod resource usage kubectl get pods -A --field-selector=status.phase=Pending # All pending pods (often Karpenter is missing capacity) kubectl get pods -A --sort-by=.status.containerStatuses[0].restartCount | tail -20 # Top restart counts (crashloop suspects) kubectl get pods -A -o wide | awk '$4 != "Running" && $4 != "Completed" && NR>1' # Anything not Running/Completed ``` ## aws (AWS CLI) ```bash aws eks describe-cluster --name <cluster-name> --region <region> aws eks list-access-entries --cluster-name <cluster-name> --region <region> # Cluster auth entries (who can get in) aws eks describe-access-entry --cluster-name <cluster-name> --principal-arn <arn> --region <region> aws ec2 describe-instances --filters "Name=tag:eks:cluster-name,Values=<cluster-name>" --query 'Reservations[].Instances[].{ID:InstanceId,Type:InstanceType,State:State.Name,AZ:Placement.AvailabilityZone,Launch:LaunchTime}' --output table --region <region> aws ec2 describe-nat-gateways --filter "Name=tag:Cluster,Values=<cluster-name>" --query 'NatGateways[].{ID:NatGatewayId,State:State,Subnet:SubnetId}' --output table --region <region> aws ec2 describe-vpc-endpoints --filters "Name=tag:Cluster,Values=<cluster-name>" --query 'VpcEndpoints[].{ID:VpcEndpointId,Service:ServiceName,State:State}' --output table --region <region> aws ecr describe-repositories --region <region> aws autoscaling describe-auto-scaling-groups --query 'AutoScalingGroups[].{Name:AutoScalingGroupName,Desired:DesiredCapacity,Min:MinSize,Max:MaxSize}' --output table --region <region> aws s3 ls s3://<state_bucket>/ --recursive | head # Peek at tofu state object layout for a cluster ``` Use the access-entries commands when troubleshooting "why can't I get into this cluster" — `clusters.yaml` has an `access_config` block that controls EKS access entries. For Karpenter-provisioned nodes the source of truth is the `NodeClaim` CR (`kubectl get nodeclaims`), not `aws autoscaling` — Karpenter nodes are NOT in an ASG. ASGs only back the **base** node group (provisioned by EKS managed node groups via tofu). `aws ec2 describe-instances` will show both kinds. ## helm ```bash helm list -A # All installed releases helm list -n arc-runners # Per-namespace (arc-runners has many: arc-<runner_name> per runner type) helm status <release> -n <ns> # Release status helm get values <release> -n <ns> # Current values helm history <release> -n <ns> # Release history ``` ## just (read-only recipes) ```bash just list # All clusters with details just show <cluster> # Dry-run: cluster name, region, bucket, modules, tofu vars just plan <cluster> # Tofu plan (no apply, no k8s/helm side effects — safe from CI on PRs) just deploy-status <cluster> [module] # Deployed versions for cluster (reads ConfigMaps in osdc-system, labeled app.kubernetes.io/managed-by=osdc-deploy-log). Optional second arg narrows to one module. just deploy-history <cluster> # Recent deploy history (same ConfigMap source) ``` `just deploy-status` / `just deploy-history` require kubeconfig to be set first (`just kubeconfig <cluster>`) — they read live ConfigMaps from the cluster. ## Deploy-Audit ConfigMaps (osdc-system) Every `just deploy`, `just deploy-base`, and `just deploy-module` records itself as ConfigMaps in the `osdc-system` namespace via `scripts/deploy-log.sh`. These are the source of truth for **what version is running where, who deployed it, when, did it succeed, and is something stuck**. The `osdc-system` namespace exists for nothing else — it is purely the audit log. **Always start here**: `just deploy-status <cluster>` — it parses the ConfigMaps and prints color-coded Current Versions + recent history. Drop to raw kubectl only when you need a field the renderer hides (`branch`, `duration`) or want to diff across clusters. ### What gets written Every recipe writes three ConfigMaps per recorded event, all labelled `app.kubernetes.io/managed-by=osdc-deploy-log`: | Name pattern | When written | Status field | |---|---|---| | `osdc-deploy-<scope>-start-<name>` | Immediately when the recipe begins | `started` | | `osdc-deploy-<scope>-finish-<name>` | When the recipe exits (success or via ERR trap on failure) | `completed` or `failed` | | `osdc-deploy-<scope>-history-<name>` | Both events appended to `.data.entries` (JSONL, trimmed to last 50 lines) | n/a | `<scope>` is `cmd` for the top-level `just` recipe and `module` for the per-module record. `<name>` is the recipe or module name. Examples: - `just deploy meta-prod-aws-ue1` → `osdc-deploy-cmd-{start,finish,history}-deploy` - `just deploy-base meta-prod-aws-ue1` → `osdc-deploy-cmd-{start,finish,history}-deploy-base` - `just deploy-module meta-prod-aws-ue1 karpenter` → writes **two** records: `osdc-deploy-cmd-{...}-deploy-module-karpenter` AND `osdc-deploy-module-{...}-karpenter` A full `just deploy` of an N-module cluster emits one outer `deploy` record, one `deploy-base` record, and 2N module records (cmd-scope + module-scope per module) — roughly 100 ConfigMaps for a 16-module cluster. ### Data schema Common keys on both `start` and `finish`: | Key | Source | Example | |---|---|---| | `commit` | `git rev-parse --short HEAD` on the deploy machine (or `unknown`) | `4fe8f56` | | `branch` | `git branch --show-current` (or `detached` for SHA checkouts; CI is almost always `detached`) | `main` | | `user` | `$USER` on the deploy machine (NOT the GH actor) | `jschmidt` | | `cluster` | The positional arg passed to the `just` recipe | `meta-prod-aws-ue1` | | `timestamp` | `date -u +%Y-%m-%dT%H:%M:%SZ` — ISO-8601 UTC, second precision, lexicographically sortable | `2026-06-30T14:23:45Z` | | `module` or `command` | The recipe/module name | `karpenter` / `deploy-base` | | `status` | `started` (on start CM); `completed` or `failed` (on finish CM) | `completed` | Only on `finish`: `duration` — wall-clock seconds as a string (e.g. `"197"`). History entries are `\n`-separated JSON lines with the same fields plus `event: start|finish`. ### Debugging recipes All commands assume `just kubeconfig <cluster>` has been run and need the EKS proxy bypass — `NO_PROXY=...,.eks.amazonaws.com no_proxy=...,.eks.amazonaws.com` prefix, omitted below for readability. **What version of module X is on this cluster?** ```bash kubectl get cm osdc-deploy-module-finish-<module> -n osdc-system -o yaml ``` Read `.data.commit`, `.data.branch`, `.data.timestamp`, `.data.user`, `.data.status`. **Does prod match staging for module X?** ```bash just kubeconfig meta-prod-aws-ue1 PROD_COMMIT=$(kubectl get cm osdc-deploy-module-finish-karpenter -n osdc-system -o jsonpath='{.data.commit}') just kubeconfig meta-staging-aws-ue1 STAGING_COMMIT=$(kubectl get cm osdc-deploy-module-finish-karpenter -n osdc-system -o jsonpath='{.data.commit}') echo "prod=$PROD_COMMIT staging=$STAGING_COMMIT" git log --oneline "$PROD_COMMIT..$STAGING_COMMIT" -- modules/karpenter/ ``` **Who deployed last on this cluster, and when?** ```bash just deploy-status <cluster> # color-coded; reads all CMs at once ``` Or raw, sorted by actual deploy time (NOT `creationTimestamp`): ```bash kubectl get cm -n osdc-system -l app.kubernetes.io/managed-by=osdc-deploy-log \