e2e-test-runner

name	E2E Test Runner
description	Provides the ability to run and iterate on HyperShift e2e tests. Auto-applies when implementing features that require e2e validation, fixing e2e test failures, or working on tasks that need live cluster testing.

HyperShift E2E Test Runner

This skill enables autonomous iteration on e2e tests - running tests, analyzing failures, making fixes, and re-running until tests pass.

When to Use This Skill

This skill automatically applies when:

• Implementing a feature that needs e2e test validation
• Fixing a failing e2e test
• Working on a task where the user wants you to iterate until tests pass
• Debugging test failures in the test/e2e/ directory
• The user mentions running e2e tests or validating changes against a live cluster

Prerequisites

Source the environment file before using this skill:

source dev/claude-env.sh

Environment Configuration

Environment variables from dev/claude-env.sh:

Variable	Description
E2E_PLATFORM	Test platform (AWS, Azure, etc.)
AWS_CREDENTIALS	Path to AWS credentials file
OIDC_BUCKET	S3 bucket for OIDC
BASE_DOMAIN	Base DNS domain
PULL_SECRET	Path to pull secret file
AWS_REGION	AWS region
E2E_ARTIFACT_DIR	Directory for test artifacts
MGMT_KUBECONFIG	Path to management cluster kubeconfig
CPO_IMAGE_REPO	Custom CPO image repository
RUNTIME	Container runtime (podman/docker)

Running E2E Tests

Step 1: Check if Test Binary Needs Rebuilding

CRITICAL: Before running any e2e test, you MUST check if the test binary needs rebuilding:

# Check if binary exists
if [ ! -f ./bin/test-e2e ]; then
  echo "Test binary missing, building..."
  make e2e
fi

# Check if any test files are newer than the binary
NEWEST_TEST=$(find test/e2e -name "*.go" -newer ./bin/test-e2e 2>/dev/null | head -1)
if [ -n "$NEWEST_TEST" ]; then
  echo "Test files changed (e.g., $NEWEST_TEST), rebuilding..."
  make e2e
fi

Step 2: Run the Test

Build and execute the test command:

KUBECONFIG=$MGMT_KUBECONFIG \
./bin/test-e2e -test.v -test.timeout 2h \
  -test.run "TEST_PATTERN" \
  -test.v \
  --e2e.platform $E2E_PLATFORM \
  --e2e.aws-credentials-file $AWS_CREDENTIALS \
  --e2e.aws-oidc-s3-bucket-name $OIDC_BUCKET \
  --e2e.base-domain $BASE_DOMAIN \
  --e2e.pull-secret-file $PULL_SECRET \
  --e2e.aws-region $AWS_REGION \
  --e2e.artifact-dir $E2E_ARTIFACT_DIR

Step 3: Add Custom CPO Image (When Testing Control Plane Changes)

If you've made changes to control-plane-operator code and built a custom image, add:

-e2e.control-plane-operator-image $CPO_IMAGE_REPO:TAG

Iteration Loop

When working autonomously on a task that requires e2e validation:

1. Initial Test Run

Run the test to establish baseline:

KUBECONFIG=$MGMT_KUBECONFIG ./bin/test-e2e -test.v -test.run "TestName" [flags...]

2. On Failure - Analyze

• Read the test output carefully
• Check artifacts in $E2E_ARTIFACT_DIR/ directory for:
  • Pod logs
  • Events
  • Resource states
• Identify the root cause

3. Make Fixes

• Edit the relevant code (test code, operator code, etc.)
• If you modified test/e2e/*.go files, the binary will be rebuilt automatically on next run

4. Rebuild Images (If Needed)

If you modified control-plane-operator code: Use the build-cpo-image skill to build and push a new image.

$RUNTIME build -f Dockerfile.control-plane --platform linux/amd64 -t $CPO_IMAGE_REPO:NEW_TAG .
$RUNTIME push $CPO_IMAGE_REPO:NEW_TAG

5. Re-run Test

Run the test again with updated code/images. Repeat until passing.

Common Test Patterns

Test Pattern	Description
TestNodePool	All NodePool tests
TestNodePool/HostedCluster0/Main/TestSpotTerminationHandler	Specific spot test
TestNodePool.*Karpenter	All Karpenter-related tests
TestCreateCluster	Cluster creation tests
TestUpgrade	Upgrade tests

Analyzing Test Failures

Check Test Output

The test output includes:

• Test name and status
• Assertion failures with expected vs actual
• Timeout information
• Resource creation/deletion logs

Check Artifact Directory

After a test failure, examine:

ls -la $E2E_ARTIFACT_DIR/
# Contains: cluster manifests, pod logs, events, resource dumps

Common Failure Patterns

Pattern	Likely Cause
context deadline exceeded	Resource didn't reach expected state in time
not found	Resource wasn't created or was deleted prematurely
connection refused	Service not ready or network issue
forbidden	RBAC or permission issue

Building Test Binary

When test code changes, rebuild:

make e2e

This compiles ./bin/test-e2e with all tests from test/e2e/.

Notes

• Tests typically take 10-30+ minutes depending on complexity
• Some tests create real AWS resources (costs money, needs cleanup on failure)
• Use -test.timeout to set appropriate timeouts (default: 2h)
• The artifact directory is overwritten on each run
• For long tests, consider running in background and checking periodically