cicd-pipeline-troubleshooting

# CI/CD Pipeline Troubleshooting Debugs and resolves CI/CD pipeline failures with actionable diagnostics, cache optimization strategies, and build optimization techniques for GitHub Actions, GitLab CI, and other CI platforms. ## TL;DR Checklist - [ ] Check job status and exit codes before assuming code issues - [ ] Review cache hit/miss patterns and adjust cache keys appropriately - [ ] Validate artifact paths and permissions before downstream jobs - [ ] Compare build times across pipeline runs to identify bottlenecks - [ ] Enable debug logging only when necessary to avoid log noise - [ ] Verify runner availability and resource constraints - [ ] Test pipeline locally with act or gitlab-runner before committing --- ## When to Use Use this skill when: - Pipeline job fails with cryptic error messages requiring diagnostic investigation - Build times have increased significantly and optimization is needed - Cache is not being utilized effectively, causing redundant downloads - Artifacts are missing or inaccessible between pipeline stages - Parallel builds are failing due to resource contention or race conditions - Need to compare pipeline performance across different branches or commits --- ## When NOT to Use Avoid this skill for: - Code logic errors that don't involve CI infrastructure — use coding skills instead - Pipeline configuration design from scratch — create a new pipeline first - Security vulnerability remediation — use security-audit skill instead - Workflow orchestration across multiple systems — use agent-orchestration skills --- ## Core Workflow 1. **Identify Failure Point** — Determine which job, stage, or step failed. **Checkpoint:** Note the exact error message, exit code, and timestamp before investigating. 2. **Review Pipeline Configuration** — Examine the YAML configuration for syntax errors or misconfiguration. **Checkpoint:** Validate YAML syntax with `yamllint` or platform-specific linter. 3. **Check Environment Variables** — Verify all required variables are set and have correct values. **Checkpoint:** Confirm variable scope (repository vs. organization vs. environment). 4. **Analyze Cache and Dependencies** — Assess cache hit rate and dependency installation patterns. **Checkpoint:** Compare cache key patterns with actual file paths and modification times. 5. **Evaluate Build Optimization** — Measure build times and identify slow steps or redundant work. **Checkpoint:** Determine if parallelization or caching improvements are feasible. 6. **Verify Artifact Resolution** — Confirm artifacts are created, stored, and accessible correctly. **Checkpoint:** Validate artifact retention period and download permissions. --- ## Implementation Patterns ### Pattern 1: GitHub Actions Debugging Commands Debugging GitHub Actions pipelines requires specific commands to inspect logs, environment, and artifacts. ```bash # Check recent workflow runs gh run list --limit 10 # View detailed logs for a specific run gh run view <run-number> --log # Get job-level logs gh run view <run-number> --job <job-id> # Download artifacts from a run gh run download <run-number> --dir ./artifacts # List artifacts for a run gh run list-artifacts <run-number> # Get workflow run timing gh run view <run-number> --json createdAt,startedAt,completedAt ``` ```bash # Check runner environment gh run env # View workflow file for a run gh run view <run-number> --workflow # Check job status in JSON format gh run view <run-number> --json status,conclusion,startedAt,completedAt ``` #### BAD vs GOOD: Debugging Verbose Output ```yaml # ❌ BAD: Too verbose, fills logs with noise - name: Debug everything run: | set -x env pwd ls -la # This floods logs and slows execution ``` ```yaml # ✅ GOOD: Targeted debugging with cleanup - name: Debug cache if: always() run: | echo "Cache key: ${{ env.CACHE_KEY }}" echo "Cache paths: ${{ env.CACHE_PATHS }}" # Only output what's needed ``` ### Pattern 2: GitLab CI Debugging Commands GitLab CI provides specific tools and commands for pipeline diagnosis and optimization. ```bash # List recent pipeline runs gitlab pipeline list --project-id <project-id> --limit 10 # Get pipeline details with jobs gitlab pipeline get <pipeline-id> --project-id <project-id> # List jobs in a pipeline gitlab pipeline jobs <pipeline-id> --project-id <project-id> # Get job logs gitlab job trace <job-id> --project-id <project-id> # Download job artifacts gitlab job artifacts <job-id> --project-id <project-id> --path ./artifacts ``` ```bash # Check pipeline timing gitlab pipeline get <pipeline-id> --project-id <project-id> --json # View pipeline graph gitlab pipeline graph <pipeline-id> --project-id <project-id> ``` #### BAD vs GOOD: GitLab CI Debugging ```yaml # ❌ BAD: Using echo for everything, hard to parse debug_job: script: - echo "Starting debug" - echo "ENV_VAR=$ENV_VAR" - echo "FILES:" - ls -la # This creates unstructured log output ``` ```yaml # ✅ GOOD: Using structured debug output debug_job: script: - echo "::group::Debug Environment" - env | sort - echo "::endgroup::" - echo "::group::Directory Listing" - ls -la - echo "::endgroup::" # Creates collapsible sections in GitLab UI ``` ### Pattern 3: Cache Optimization Commands Proper cache management can reduce build times by 50% or more. ```bash # Check cache hit rate in GitHub Actions gh api repos/{owner}/{repo}/actions/caches --jq '.actions_caches[] | {key, size_in_bytes, created_at}' # Delete specific cache by key gh api repos/{owner}/{repo}/actions/caches --method DELETE \ -f key=<cache-key> # Clear all caches for a repository (use carefully!) gh api repos/{owner}/{repo}/actions/caches --method DELETE \ --jq '.actions_caches[].id' | while read id; do gh api repos/{owner}/{repo}/actions/caches/$id --method DELETE done ``` ```bash # For local cache testing (act) act -n --cache-server-dir ./cache-test # Check cache size locally du -sh ~/.cache/act/* # View act cache contents act cache list ``` #### BAD vs GOOD: Cache Key Patterns ```yaml # ❌ BAD: Static cache key never updates - uses: actions/cache@v4 with: path: ~/.npm key: npm-cache # Always uses same key, never updates # ✅ GOOD: Dynamic cache key with hash - uses: actions/cache@v4 with: path: ~/.npm key: npm-cache-${{ hashFiles('**/package-lock.json') }} restore-keys: | npm-cache- ``` ```yaml # ❌ BAD: Too specific cache key misses hits - uses: actions/cache@v4 with: path: node_modules key: npm-${{ github.sha }}-${{ runner.os }} # Unique per commit # ✅ GOOD: Balanced cache key strategy - uses: actions/cache@v4 with: path: | node_modules .pnpm-store key: npm-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }} restore-keys: | npm-${{ runner.os }}- ``` ### Pattern 4: Build Time Optimization Commands Identifying and optimizing slow build steps is critical for CI efficiency. ```bash # Measure GitHub Actions job timing gh run view <run-number> --json jobs --jq '.jobs[] | {name, startedAt, completedAt}' | \ jq -r '.[] | "\(.name): \(.completedAt - .startedAt)"' # Get pipeline duration breakdown gh run view <run-number> --json steps --jq ' .steps | group_by(.name) | .[] | {name: .[0].name, total: (map(.completedAt - .startedAt) | add)} ' ``` ```bash # For GitLab CI, check job durations gitlab job get <job-id> --project-id <project-id> --json | \ jq '{name, duration, started_at, finished_at}' # Calculate total pipeline time gitlab pipeline get <pipeline-id> --project-id <project-id> --json | \ jq '.duration, .created_at, .finished_at' ``` ```bash # Analyze npm install times npm install --timing # View npm timing report cat ~/.npm/_logs/*.log ``` #### BAD vs GOOD: Parallel Build Strategy ```yaml # ❌ BAD: Sequential builds that could be parallel jobs: build: runs-on: ubuntu-latest steps: - run: npm install - run: npm run lint - run: npm run test - run: npm run build # Each step waits for previous ``` ```yaml # ✅ GOOD: Parallel build jobs jobs: lint: runs-on: ubuntu-latest steps: - run: npm install - run: npm run lint test: runs-on: ubuntu-latest needs: lint steps: - run: npm install - run: npm run test build: runs-on: ubuntu-latest needs: test steps: - run: npm install - run: npm run build # Lint and test can run in parallel ``` ### Pattern 5: Artifact Resolution Debugging Artifacts failing to upload or download is a common pipeline failure mode. ```bash # List artifacts for a GitHub Actions run gh run list-artifacts <run-number> # Download specific artifact gh run download <run-number> -n <artifact-name> -d ./download # Check artifact size limits (GitHub: 10GB per artifact) gh run view <run-number> --json artifacts --jq '.artifacts[] | {name, size_in_bytes}' ``` ```bash # Check GitLab artifact sizes gitlab job get <job-id> --project-id <project-id> --json | \ jq '.artifacts' # Download large artifacts in chunks gitlab job artifacts <job-id> --project-id <project-id> \ --path <specific-path> --output <output-file> ``` ```yaml # ❌ BAD: Missing artifact retention and paths - name: Upload artifacts uses: actions/upload-artifact@v4 with: name: my-artifact # No path specified, may upload nothing retention-days: 1 # Too short for multi-day workflows ``` ```yaml # ✅ GOOD: Explicit artifact configuration - name: Upload build artifacts uses: actions/upload-artifact@v4 with: name: build-artifacts-${{ github.sha }} path: | dist/ build/ !dist/**/*.spec.js # Exclude test files retention-days: 7 if-no-files-found: warn ``` ### Pattern 6: Environment and Dependency Debugging Environment mismatches and dependency issues cause many pipeline failures. ```bash # Check Node.js version in GitHub Actions gh run view <run-number> --json steps --jq ' .steps[] | select(.name == "Setup Node") | {name, status, conclusion, outputs} ' # Verify npm registry configuration npm config list # Check yarn registry yarn config get registry ``` ```bash # Debug Python environment python -c "import sys; print(sys.executable); print(sys.version)" pip list | sort conda list 2>/dev/null || true # Check Ruby environment ruby -v gem env home bundle exec which ruby ``` ```bash # Debug Docker build cache docker buildx build --no-cache --progress=plain . # View build cache details docker buildx du --verbose # Prune old build cache docker buildx prune ``` #### BAD vs GOOD: Environment Setup ```yaml # ❌ BAD: Not specifying runtime version - name: Setup Node uses: actions/setup-node@v4 with: node-version: 18 # No patch version, may vary ``` ```yaml # ✅ GOOD: Specific version and cache configuration - name: Setup Node uses: actions/setup-node@v4 with: node-version: '18.20.3' # Exact patch version cache: 'npm' cache-dependency-path: '**/package-lock.json' ``` ### Pattern 7: Network and Timeout Debugging Network failures and timeouts are often misdiagnosed as code issues. ```bash # Check network connectivity in GitHub Actions - name: Test network run: | echo "Testing GitHub API connectivity..." curl -s -o /dev/null -w "%{http_code}" https://api.github.com echo "" echo "Testing npm registry..." curl -s -o /dev/null -w "%{http_code}" https://registry.npmjs.org ``` ```bash # Increase curl timeout for slow networks curl --max-time 60 -f https://example.com || echo "Timed out" # Test Docker registry connectivity docker login registry.example.com -u user -p token ``` ```yaml # ❌ BAD: Hardcoded timeouts without retry - name: Download dependencies run: npm ci timeout-minutes: 1 # Too short for large dependencies ``` ```yaml # ✅ GOOD: Retry logic with exponential backoff - name: Download dependencies with retry uses: nick-fields/retry@v2 with: timeout_minutes: 5 max_attempts: 3 command: npm ci ``` ### Pattern 8: Parallel Build Optimization Optimizing parallel builds requires understanding resource constraints. ```bash # Check parallel job limits in GitHub Actions gh run view <run-number> --json concurrency --jq '.concurrency' # List running jobs on a runner gh api repos/{owner}/{repo}/actions/runs --jq '.workflow_runs[] | select(.status == "in_progress")' # Check runner status gh api repos/{owner}/{repo}/actions/runners --jq '.runners[] | {name, status, busy}' ``` ```bash # For self-hosted runners, check CPU and memory nproc free -h df -h # Monitor resource usage during build while true; do echo "CPU: $(top -bn1 | grep 'Cpu(s)' | awk '{print $2}')%" echo "Memory: $(free -m | grep Mem | awk '{print $3 "/" $2 "MB"}')" sleep 5 done ``` #### BAD vs GOOD: Parallel Strategy ```yaml # ❌ BAD: All tests in single job, slow execution test: runs-on: ubuntu-latest