kubernetes-manifests

name	kubernetes-manifests
description	Construct Kubernetes Jobs, StatefulSets, and Services for GPU-accelerated RL training workloads on EKS

Skill 09: Kubernetes Manifests

Purpose

Construct Kubernetes manifests for deploying RL training workloads on EKS. This includes Job definitions, headless Services for multi-node discovery, MPIJobs for NCCL tests, Secrets, and resource requests for GPUs, EFA, and shared storage.

Source of truth: The actual deployed manifests live in examples/<example>/manifests/. The content below explains the patterns and design decisions. If this skill and the manifests ever diverge, the manifests are authoritative.

Reference manifests: container-test.yaml, training-statefulset.yaml, training-service.yaml in infrastructure/manifests/. Per-example manifests in examples/<example>/manifests/.

Example (RLinf): See examples/<example>/manifests/ for the RLinf-specific manifests and examples/AGENTS.md for RLinf-specific configuration values (CONFIG_NAME, VENV_NAME, MODEL_PATH, etc.).

NCCL test manifest: infrastructure/manifests/nccl-tests-mpijob.yaml -- MPIJob pattern from awslabs/awsome-distributed-ai. Uses pre-built public ECR image.

Architecture: Single-Node vs Multi-Node vs MPIJob

Single-Node (1 pod, 8 GPUs)

+-------------------------------------------+
| Kubernetes Job (completions: 1)           |
|                                           |
|  +-------------------------------------+ |
|  | Pod: training-0                      | |
|  | - 8x nvidia.com/gpu                  | |
|  | - 4x vpc.amazonaws.com/efa           | |
|  | - /fsx mounted                       | |
|  | - Ray head (local)                   | |
|  | - FSDP across 8 GPUs                 | |
|  +-------------------------------------+ |
+-------------------------------------------+

Multi-Node (2 pods, 16 GPUs)

+--------------------------------------------------+
| Headless Service: training-svc                    |
|                                                   |
|  +---------------------+  +---------------------+|
|  | Pod: training-0     |  | Pod: training-1     ||
|  | (Ray head + worker) |  | (Ray worker)        ||
|  | 8x GPU, 4x EFA      |  | 8x GPU, 4x EFA      ||
|  | /fsx mounted        |  | /fsx mounted        ||
|  +---------------------+  +---------------------+|
|         NCCL over EFA (allreduce)                 |
+--------------------------------------------------+

Single-Node Training Job

The single-node Job runs the training script directly -- no init container is needed because the training script handles model preparation internally. The image URI uses ${ECR_URI} which is substituted at deploy time via envsubst.

# training-job.yaml
#
# Before applying, substitute the ECR image URI:
#   export ECR_URI=<account>.dkr.ecr.<region>.amazonaws.com/rlinf-on-eks/rlinf
#   envsubst < manifests/training-job.yaml | kubectl apply -f -
#
# Or manually replace ${ECR_URI} below.
apiVersion: batch/v1
kind: Job
metadata:
  name: rlinf-training  # Replace with your reference name
  labels:
    app: rlinf  # Replace with your reference name
    reference: rlinf
spec:
  backoffLimit: 0
  template:
    metadata:
      labels:
        app: rlinf
    spec:
      restartPolicy: Never
      serviceAccountName: training-sa
      nodeSelector:
        role: gpu-training
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

      containers:
        - name: training
          image: ${ECR_URI}:latest  
          
             
               
          
            
               
               
               
               
            
               
               
               
               
          
                
               
              
               
              
               
              
                 
              
                 
              
                 
              
                 
              
               
              
               
              
               
              
               
              
               
              
               
              
               
              
               
          
              
               
              
               

      
          
          
             
          
          
             
             

Key design decisions (single-node)

Decision	Rationale
No init container	The training script (<launch-script>) handles checkpoint preparation internally, removing the need for a separate pre-step.
${ECR_URI}:latest image	Uses envsubst at deploy time so the same manifest works across accounts/regions without editing.
CONFIG_NAME, VENV_NAME, MODEL_PATH, EXPERIMENT_NAME env vars	Parameterized so the same manifest works for different examples (swap values per experiment).
EFA/NCCL env vars	FI_PROVIDER=efa, FI_EFA_USE_DEVICE_RDMA=1, FI_EFA_FORK_SAFE=1 configure libfabric for RDMA; TORCH_NCCL_AVOID_RECORD_STREAMS=1 reduces CUDA memory fragmentation.

Multi-Node Training (StatefulSet + Headless Service)

Headless Service

The headless Service provides DNS-based pod discovery. It must be applied before the StatefulSet so that DNS records are available when pods start.

# training-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: training-svc
  labels:
    app: rlinf-multi  # Replace with your reference name
spec:
  clusterIP: None  # Headless -- pods discover each other via DNS
  selector:
    app: rlinf-multi
  ports:
    - name: ray-head
      port: 6379
      targetPort: 6379
    - name: ray-dashboard
      port: 8265
      targetPort: 8265
    - name: ray-client
      port: 10001
      targetPort: 10001

StatefulSet

The StatefulSet deploys N pods (one per GPU node). Pod-0 starts a Ray head, waits for all workers to connect, then launches the training script. Pod-1+ are Ray workers that connect to the head and block. The image URI uses ${ECR_URI} substituted via envsubst at deploy time.

# training-statefulset.yaml
#
# Before applying, substitute the ECR image URI:
#   export ECR_URI=<account>.dkr.ecr.<region>.amazonaws.com/rlinf-on-eks/rlinf
#   envsubst < manifests/training-statefulset.yaml | kubectl apply -f -
#
# Or manually replace ${ECR_URI} below.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: training
  labels:
    app: rlinf-multi  # Replace with your reference name
    reference: rlinf
spec:
  serviceName: training-svc
  replicas: 2  # Number of nodes
  podManagementPolicy: Parallel  # Start all pods simultaneously
  selector:
    matchLabels:
      app: rlinf-multi
  template:
    metadata:
      labels:
        app: rlinf-multi
    spec:
      serviceAccountName: training-sa
      nodeSelector:
        role: gpu-training
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
           
       

      
      
      
      
      
          
           
           
          
            
               

      
      
        
          
             
                
                    
                     
                    
                       
               

      
          
             
          
             
             
             
               [    ] 
                 
                    

                 
                
                
                 [         ] 
                   
                    
                   [    ] 
                     
                     
                  
                
                 

                   
              
                 
                   
              
          
              
              
                
                   
              
               
              
               
              
                 
              
                 
              
                 
              
                 
                
               
              
               
              
               
              
               
              
               
              
               
              
               
              
               
              
               
          
            
               
               
               
               
            
               
               
               
               
          
              
               
              
               
          
          
            
              
                 
                 
                 
             
             
             
          
              
               
              
               

      
          
          
             
          
          
             
             

Key design decisions (multi-node)

Decision	Rationale
POD_NAMESPACE via Downward API	The head address FQDN includes the namespace (training-0.training-svc.<ns>.svc.cluster.local). Using fieldRef: metadata.namespace makes the manifest namespace-portable rather than hardcoding default.
topologySpreadConstraints on topology.k8s.aws/network-node-layer-2	Prefers co-locating pods under the same layer-2 switch for lowest NCCL latency. ScheduleAnyway means it is best-effort -- the cluster still works if the label is absent.
podAntiAffinity on kubernetes.io/hostname	Ensures exactly one training pod per physical node. Two pods on the same node would contend for GPUs.
Ray worker detection via python3 -c 'import ray; ...'	Counts connected Ray nodes programmatically with a 300-second timeout. More reliable than parsing ray status output with grep.
No separate init container	The training script (<launch-script>) handles all setup internally via VENV_NAME and CONFIG_NAME env vars.
Liveness probe on ray status	Detects Ray head crashes. The `
${ECR_URI}:latest image	Same envsubst pattern as the single-node Job for account/region portability.

Supporting Resources

Resource Sizing Guide

Instance Type	GPU Request	EFA Request	CPU Request	Memory Request
p4de.24xlarge	8	4	90	1000Gi
p5.48xlarge	8	32	180	1800Gi

/dev/shm Sizing

PyTorch DataLoader workers and NCCL use shared memory. Always mount a large emptyDir with medium: Memory:

volumes:
  - name: dshm
    emptyDir:
      medium: Memory
      sizeLimit: "64Gi"  # At least 64Gi for multi-GPU training

Without this, training will fail with "Bus error" or "Insufficient shared memory".

YAML/Bash Interaction Pitfalls

Bash scripts embedded in YAML | block scalars are a common source of subtle failures in training manifests. YAML parsers can misinterpret:

• Semicolons (;): Can terminate YAML values. if test -f foo; then echo yes; fi may fail.
• Curly braces ({}): Interpreted as YAML flow mappings. ${ } and { echo ...; } can cause parse errors.
• &&/|| chains: Long chains with variable assignments may lose variable scope (subshell execution).
• for loops with ;: for x in a b; do ... done -- the ; before do breaks in many YAML parsers.
• Colons (:): Bare colons (e.g., in image tags $ECR_URI:latest) are parsed as YAML key-value separators.

Prescriptive Approach

1. Keep inline bash simple -- sequential commands, basic if/then/else/fi (with then on the same line using newline, not ;).
2. For complex logic, use a ConfigMap-mounted script:

volumes:
  - name: scripts
    configMap:
      name: training-scripts
      defaultMode: 0755
containers:
  - name: training
    command: ["/scripts/run.sh"]
    volumeMounts:
      - name: scripts
        mountPath: /scripts

3. For test suites, use Python instead of bash -- Python has no YAML interaction issues when mounted via ConfigMap.
4. Always validate: kubectl apply --dry-run=client -f manifest.yaml before deploying.

Image Pull Considerations

Physical AI RL container images are typically 15-20 GB compressed. Image pull time impacts pod startup and job deadlines.

Scenario	Pull Time	Recommendation
First pull (cold)	5-10 min	Account for this in activeDeadlineSeconds
Same image, same node (cached)	0s	Default IfNotPresent policy
Updated :latest tag	5-10 min	Only if imagePullPolicy: Always
Specific version tag (cached)	0s	Best practice for production

Best Practices

• Do NOT set imagePullPolicy: Always for large training images. The default (IfNotPresent) is correct.
• Tag images with specific versions (e.g., :20260401-085549), not just :latest. Update the tag in manifests when the image changes.
• Add 10 minutes to activeDeadlineSeconds to account for cold image pulls. A 1-hour training test should use activeDeadlineSeconds: 4200 (70 min).
• Pre-pull images on GPU nodes using a DaemonSet if predictable startup time is critical.

Deployment

All manifests use ${ECR_URI} as a placeholder for the container image. Substitute it at deploy time with envsubst:

# Set your ECR image URI
export ECR_URI=<account>.dkr.ecr.<region>.amazonaws.com/rlinf-on-eks/rlinf  # Replace with your values

# Single-node
envsubst < manifests/training-job.yaml | kubectl apply -f -

# Multi-node
kubectl apply -f manifests/training-service.yaml
envsubst < manifests/training-statefulset.yaml | kubectl apply -f -

# Monitor
kubectl logs -f job/rlinf-training       # single-node
kubectl logs -f training-0                      # multi-node head
kubectl logs -f training-1                      # multi-node worker

Validation Checklist

• Pod schedules on GPU node with all resources allocated
• nvidia-smi inside pod shows 8 GPUs
• /fsx is mounted and readable/writable
• /dev/shm is mounted with sufficient size
• EFA devices are available (ls /sys/class/infiniband/)
• Training starts and logs appear in MLflow
• For multi-node: Ray workers connect to head, FSDP initializes across all ranks
• For multi-node: ray status on head pod shows all nodes connected
• For multi-node: liveness probe does not restart pods during training

Related Skills

• Skill 07: Container Image Building - Image referenced in manifests
• Skill 08: PyTorch Training Scripts - Scripts launched by these manifests
• Skill 10: Kubernetes Native Features - Scheduling and topology