Read-only CLI commands for debugging OSDC clusters: kubectl, aws, helm, tofu. Includes command references, safety boundaries, and the deploy-audit ConfigMaps in osdc-system (what version is running where, who deployed it, when, did it succeed, is a deploy stuck). Applies to ~/meta/ci-infra/osdc. Load when investigating cluster state, debugging pods, inspecting infrastructure, comparing deployed versions across clusters, or diagnosing a failed or stuck deploy.
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Read-only CLI commands for debugging OSDC clusters: kubectl, aws, helm, tofu. Includes command references, safety boundaries, and the deploy-audit ConfigMaps in osdc-system (what version is running where, who deployed it, when, did it succeed, is a deploy stuck). Applies to ~/meta/ci-infra/osdc. Load when investigating cluster state, debugging pods, inspecting infrastructure, comparing deployed versions across clusters, or diagnosing a failed or stuck deploy.
OSDC CLI Debugging (Read-Only)
You are encouraged to run read-only CLI commands to debug, investigate, and understand cluster and infrastructure state. This is the fastest way to diagnose issues — use it proactively. All tools below are managed by mise via osdc/mise.toml.
Run commands from the osdc/ working directory so mise activates the correct tool versions. If running from elsewhere, prefix with mise exec --:
mise exec -- kubectl get pods -n arc-runners
clusters.yaml Location
All clusters are defined in clusters.yaml in the project root. Run just list for full details. Read the file directly for config values.
The <cluster> argument to every just recipe is the YAML key (e.g. meta-prod-aws-ue1, meta-staging-aws-uw1, arc-cbr-production, arc-cbr-production-uw1), not the cluster_name field — they sometimes differ (e.g. arc-cbr-production resolves to cluster_name: pytorch-arc-cbr-production).
lf-* Clusters Are NOT Accessible From This Laptop
The lf-prod-aws-ue1 / lf-prod-aws-ue2 clusters (state buckets , admin role ) are operated by another org. Any , against an context, , or against an state bucket will fail on authentication.
lf-osdc-tfstate-prod-*
lf_osdc_admin
From this laptop the operator has no AWS credentials, no kubeconfig access, and no tofu state read access for them.
just kubeconfig lf-prod-*
kubectl
lf-*
aws eks describe-cluster --name lf-prod-*
tofu init
lf-osdc-*
Accessible from this laptop (admin role osdc_gha_prod / osdc_gha_staging):
For lf-* investigations: read clusters.yaml, read the module code, read the manifests in base/ and modules/ — but do not attempt live cluster commands. If the user explicitly asks for lf-* live state, tell them you cannot reach the cluster and ask them to run the command and paste output.
Kubeconfig: Use just kubeconfig <cluster> (MANDATORY)
Before running any kubectl or helm commands, configure kubeconfig using the just recipe — never run aws eks update-kubeconfig directly.
just kubeconfig <cluster>
The just recipe reads clusters.yaml to resolve the cluster name and region automatically and delegates to scripts/kubeconfig-lock.sh (which serializes concurrent kubeconfig writes via a file lock). It aliases the kubectl context to the cluster_name (e.g. pytorch-arc-cbr-production, meta-prod-aws-ue1, meta-staging-aws-uw1) — there is no eks-... prefix. Switch contexts with kubectl config use-context <cluster_name> or scope a single call with kubectl --context <cluster_name> .... Running aws eks update-kubeconfig by hand risks wrong cluster name, wrong region, or wrong kubeconfig context.
Corporate Proxy Bypass for kubectl / aws / EKS (MANDATORY)
This sandbox inherits Meta's x2p proxy (HTTPS_PROXY=http://localhost:10054), which blocks the EKS API endpoints. Every ad-hoc kubectl, aws eks, or helm call against the cluster MUST inline the bypass — export does not persist across separate Bash tool invocations, so prefix each command:
NO_PROXY="${NO_PROXY:-},.eks.amazonaws.com" no_proxy="${no_proxy:-},.eks.amazonaws.com" kubectl get pods -A
NO_PROXY="${NO_PROXY:-},.eks.amazonaws.com" no_proxy="${no_proxy:-},.eks.amazonaws.com" aws eks describe-cluster --name <cluster-name> --region <region>
The shipped recipes (just kubeconfig, just deploy-*, scripts/destroy-cluster.sh, etc.) already inject the bypass. The raw kubectl/aws/helm examples in the sections below omit the prefix only for readability — you still must add it when you actually run them.
aws calls that do NOT touch the EKS API endpoint (aws s3 ..., aws ec2 describe-*, aws autoscaling ..., aws ecr ..., aws iam ..., etc.) generally work without the bypass, because Meta's proxy already whitelists the regular AWS service endpoints. Add the bypass when in doubt — it is harmless on non-EKS calls.
Use the access-entries commands when troubleshooting "why can't I get into this cluster" — clusters.yaml has an access_config block that controls EKS access entries.
For Karpenter-provisioned nodes the source of truth is the NodeClaim CR (kubectl get nodeclaims), not aws autoscaling — Karpenter nodes are NOT in an ASG. ASGs only back the base node group (provisioned by EKS managed node groups via tofu). aws ec2 describe-instances will show both kinds.
helm
helm list -A # All installed releases
helm list -n arc-runners # Per-namespace (arc-runners has many: arc-<runner_name> per runner type)
helm status <release> -n <ns> # Release status
helm get values <release> -n <ns> # Current values
helm history <release> -n <ns> # Release history
just (read-only recipes)
just list # All clusters with details
just show <cluster> # Dry-run: cluster name, region, bucket, modules, tofu vars
just plan <cluster> # Tofu plan (no apply, no k8s/helm side effects — safe from CI on PRs)
just deploy-status <cluster> [module] # Deployed versions for cluster (reads ConfigMaps in osdc-system, labeled app.kubernetes.io/managed-by=osdc-deploy-log). Optional second arg narrows to one module.
just deploy-history <cluster> # Recent deploy history (same ConfigMap source)
just deploy-status / just deploy-history require kubeconfig to be set first (just kubeconfig <cluster>) — they read live ConfigMaps from the cluster.
Deploy-Audit ConfigMaps (osdc-system)
Every just deploy, just deploy-base, and just deploy-module records itself as ConfigMaps in the osdc-system namespace via scripts/deploy-log.sh. These are the source of truth for what version is running where, who deployed it, when, did it succeed, and is something stuck. The osdc-system namespace exists for nothing else — it is purely the audit log.
Always start here: just deploy-status <cluster> — it parses the ConfigMaps and prints color-coded Current Versions + recent history. Drop to raw kubectl only when you need a field the renderer hides (branch, duration) or want to diff across clusters.
What gets written
Every recipe writes three ConfigMaps per recorded event, all labelled app.kubernetes.io/managed-by=osdc-deploy-log:
Name pattern
When written
Status field
osdc-deploy-<scope>-start-<name>
Immediately when the recipe begins
started
osdc-deploy-<scope>-finish-<name>
When the recipe exits (success or via ERR trap on failure)
completed or failed
osdc-deploy-<scope>-history-<name>
Both events appended to .data.entries (JSONL, trimmed to last 50 lines)
n/a
<scope> is cmd for the top-level just recipe and module for the per-module record. <name> is the recipe or module name. Examples:
just deploy meta-prod-aws-ue1 → osdc-deploy-cmd-{start,finish,history}-deploy
just deploy-base meta-prod-aws-ue1 → osdc-deploy-cmd-{start,finish,history}-deploy-base
just deploy-module meta-prod-aws-ue1 karpenter → writes two records: osdc-deploy-cmd-{...}-deploy-module-karpenter AND osdc-deploy-module-{...}-karpenter
A full just deploy of an N-module cluster emits one outer deploy record, one deploy-base record, and 2N module records (cmd-scope + module-scope per module) — roughly 100 ConfigMaps for a 16-module cluster.
Data schema
Common keys on both start and finish:
Key
Source
Example
commit
git rev-parse --short HEAD on the deploy machine (or unknown)
4fe8f56
branch
git branch --show-current (or detached for SHA checkouts; CI is almost always detached)
main
user
$USER on the deploy machine (NOT the GH actor)
jschmidt
cluster
The positional arg passed to the just recipe
meta-prod-aws-ue1
timestamp
date -u +%Y-%m-%dT%H:%M:%SZ — ISO-8601 UTC, second precision, lexicographically sortable
2026-06-30T14:23:45Z
module or command
The recipe/module name
karpenter / deploy-base
status
started (on start CM); completed or failed (on finish CM)
completed
Only on finish: duration — wall-clock seconds as a string (e.g. "197").
History entries are \n-separated JSON lines with the same fields plus event: start|finish.
Debugging recipes
All commands assume just kubeconfig <cluster> has been run and need the EKS proxy bypass — NO_PROXY=...,.eks.amazonaws.com no_proxy=...,.eks.amazonaws.com prefix, omitted below for readability.
What version of module X is on this cluster?
kubectl get cm osdc-deploy-module-finish-<module> -n osdc-system -o yaml
just kubeconfig meta-prod-aws-ue1
PROD_COMMIT=$(kubectl get cm osdc-deploy-module-finish-karpenter -n osdc-system -o jsonpath='{.data.commit}')
just kubeconfig meta-staging-aws-uw1
STAGING_COMMIT=$(kubectl get cm osdc-deploy-module-finish-karpenter -n osdc-system -o jsonpath='{.data.commit}')
echo"prod=$PROD_COMMIT staging=$STAGING_COMMIT"
git log --oneline "$PROD_COMMIT..$STAGING_COMMIT" -- modules/karpenter/
Who deployed last on this cluster, and when?
just deploy-status <cluster> # color-coded; reads all CMs at once
Or raw, sorted by actual deploy time (NOT creationTimestamp):
kubectl get cm -n osdc-system -l app.kubernetes.io/managed-by=osdc-deploy-log \
-o jsonpath='{range .items[*]}{.data.timestamp}{"\t"}{.metadata.name}{"\t"}{.data.user}{"\t"}{.data.status}{"\n"}{end}' \
| sort | tail -20
Why did the last deploy fail?
just deploy-status <cluster> # red "failed" rows identify the failing scope# Or list every failure on the cluster:
kubectl get cm -n osdc-system -l app.kubernetes.io/managed-by=osdc-deploy-log -o json \
| jq -r '.items[] | select(.data.status=="failed") | "\(.data.timestamp)\t\(.metadata.name)\t\(.data.commit)\t\(.data.user)"'
The ConfigMap only records THAT it failed, not WHY — pair this with the actual deploy logs (CI job, terminal scrollback). When deploy-base fails inside just deploy, you typically see TWO failed records (the inner deploy-base and the outer deploy).
Is a deploy in progress right now?
just deploy-status <cluster> | grep -i "in progress"
deploy-status flags "in progress" when the start ConfigMap's timestamp is newer than the finish one's. There is no TTL or liveness check — an orchestrator that was SIGKILL'd hours ago shows as in-progress forever until somebody re-runs the same scope/name. Sanity-check the start timestamp: anything >1h old with no recent module activity is almost certainly a dead orchestrator, not a live deploy.
Timeline of every deploy event today, narrowed to one scope:
just deploy-status <cluster> <name> # name = "deploy" | "deploy-base" | "deploy-module-<MOD>" | "<MOD>"
The optional <name> filter switches the history table to include BOTH start and finish events (richer timeline) and bumps the row cap to 20. Without it, only finish events are shown, capped at 10.
Full JSONL history for one record:
kubectl get cm osdc-deploy-module-history-karpenter -n osdc-system -o jsonpath='{.data.entries}'
Gotchas
.metadata.creationTimestamp is NOT the deploy time — it is when the ConfigMap was first created (possibly weeks ago). just deploy-history sorts by creationTimestamp and is therefore mostly an inventory of which records exist, not a timeline. Always use .data.timestamp for actual deploy time.
"In progress" can be stale. No TTL, no heartbeat. A re-run of the same scope/name is the only thing that clears it.
user is $USER, not the GH actor. CI runs as whatever user the runner pod uses — usually meaningless for attribution. Cross-reference with the CI job URL.
branch=detached is normal in CI (SHA checkouts). Not a bug.
Removed modules leak ConfigMaps. Records for a module dropped from clusters.yaml stay in osdc-system until someone kubectl delete cms them.
Audit writes are non-fatal and silent on failure. If the audit kubectl call itself fails (network blip, RBAC), the deploy still proceeds and the audit record simply doesn't exist — only a Warning: deploy-log failed … on stderr.
Direct ./modules/foo/deploy.sh invocations produce zero audit records. Module scripts don't source deploy-log.sh; only the just recipes do.
History caps at 50 entries combined (start + finish) → effectively the last ~25 deploys per record. High-churn clusters lose old history fast.
tofu (OpenTofu) — read-only
NEVER run terraform — this is OpenTofu-only. Running terraform against the same state directory will corrupt state with no recovery. Use tofu or the just recipes (which call tofu internally) for everything.
Base infra terraform code lives in modules/eks/terraform/ even though state is keyed under ${CLUSTER}/base/terraform.tfstate (base here is the state slot name; the directory base/terraform/ does not exist — the real code is in modules/eks/terraform/).
Each cluster's state lives in its own state_bucket (defined in clusters.yaml, e.g. ciforge-tfstate-meta-staging-aws-uw1, ciforge-tfstate-arc-cbr-prod, ciforge-tfstate-arc-cbr-prod-ue1, ciforge-tfstate-arc-cbr-prod-uw1). Lock table is shared: ciforge-terraform-locks. To inspect state for a specific cluster, first init with the cluster's backend config:
cd modules/eks/terraform
tofu init -reconfigure \
-backend-config="bucket=ciforge-tfstate-meta-staging-aws-uw1" \
-backend-config="key=meta-staging-aws-uw1/base/terraform.tfstate" \
-backend-config="region=us-west-2" \
-backend-config="dynamodb_table=ciforge-terraform-locks"
tofu show # Current state
tofu output # Output values
tofu state list # All managed resources
tofu state show <addr> # Inspect a single resource
For module state (karpenter, arc, monitoring, etc.) the pattern is the same but the code lives in modules/<module>/terraform/ and the key is <cluster>/<module>/terraform.tfstate. The simplest way to plan-only a whole cluster is just plan <cluster> — it handles init + plan for base and every module in order.
The state bucket and lock table always live in us-west-2 regardless of the cluster's own workload region. This is the hardcoded STATE_REGION constant in scripts/state-config.sh (the single source of truth) and is what every justfile recipe and module deploy.sh uses. mise.toml also exports AWS_REGION=us-west-2 by default for the same reason. Don't substitute the cluster's region here, even for clusters running outside us-west-2.
lf- state buckets* (lf-osdc-tfstate-prod-ue1, lf-osdc-tfstate-prod-ue2) live in another AWS account and are not readable from this laptop — tofu init against them will fail on S3 auth.
Boundaries: What NOT to Do with CLI
This skill is READ-ONLY. Everything below is forbidden without explicit user instruction — and even then, prefer surfacing the appropriate just recipe over running the raw command.
NEVER run terraform (any subcommand, including terraform init/plan) — this project is OpenTofu-only and running terraform against the same directory will corrupt state with no recovery. Always use tofu or just recipes.
NEVERkubectl apply/create/delete/edit/patch/replace/scale/cordon/drain/taint/label/annotate — use just recipes for deployments
NEVERkubectl exec into a pod for anything other than running idempotent read-only diagnostic commands (ls, cat, env, ps, netstat, nslookup). No mutating commands, no shell sessions left running.
NEVERhelm install/upgrade/uninstall/rollback — use just deploy-* recipes
NEVERtofu apply/destroy/import/state rm/state mv/taint/untaint/refresh -auto-approve — use just deploy-* recipes; tofu show/output/state list/state show/plan are the only safe verbs
NEVER scale, cordon, drain, or taint nodes directly
NEVER write to the project's git repo (no git commit, git push, git merge, git rebase, git tag, branch deletion). Read-only git commands are fine.
NEVER edit kubeconfig by hand — always go through just kubeconfig <cluster> (which uses scripts/kubeconfig-lock.sh to serialize concurrent writes)
If you need to make a change, find or suggest the appropriate just recipe. Don't experiment with the cluster — even on staging.
Node Compactor Debugging
# Controller logs
kubectl logs -n kube-system deploy/node-compactor -f
# Check taint state on nodes
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key'
NodeLocal DNSCache (NLD) Debugging
NLD is a per-node DaemonSet (hostNetwork: true, listens on fd00::10 and the kube-dns Service ClusterIP). Pods continue resolving via the unchanged kube-dns ClusterIP — NLD intercepts via NOTRACK ip6tables (IPv6-only EKS) on a dummy nodelocaldns interface.
# DaemonSet health snapshot (DESIRED/CURRENT/READY should match node count)
kubectl get ds node-local-dns -n kube-system
# Per-node pod placement (verify one pod per node, identify any not-Running)
kubectl get pods -n kube-system -l k8s-app=node-local-dns -o wide
# Recent logs across all NLD pods
kubectl logs -n kube-system -l k8s-app=node-local-dns --tail=50
# Verify DNS resolution from a debug pod (both paths)
kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never -- \
nslookup kubernetes.default.svc.cluster.local fd00::10
kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never -- \
nslookup kubernetes.default.svc.cluster.local # via kube-dns ClusterIP
Container is a minimal image — kubectl exec ... -- env (or any shell/busybox cmd) does NOT work. For runtime env var inspection use /proc/1/environ if accessible; otherwise rely on functional DNS queries as the authoritative test. Setup errors surface via the coredns_nodecache_setup_errors_total metric (note: NOT nodelocaldns_setup_errors_total).
VPC CNI Custom Networking — ENIConfigs / base node label
Legacy — inert under IPv6 mode. VPC CNI Custom Networking is unsupported in IPv6-only EKS; the ENIConfig CRs and the ipam.osdc.internal/eni-config node labels are kept inert as a rollback fallback to IPv4 + Custom Networking. Under IPv6 the commands below still execute but the labels carry no behavioral meaning.
# List ENIConfig CRs (one per AZ for base nodes)
kubectl get eniconfigs.crd.k8s.amazonaws.com
# Verify base nodes carry the AZ-matched eni-config label
kubectl get nodes -l role=base-infrastructure \
-o custom-columns='NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,ENI:.metadata.labels.ipam\.osdc\.internal/eni-config'
