cicd-pipeline

name	cicd-pipeline
description	Create, debug, and manage CI/CD pipelines with GitHub Actions. Use when the user needs to set up automated testing, deployment, releases, or workflows. Covers workflow syntax, common patterns, secrets management, caching, matrix builds, and troubleshooting.
metadata	{"clawdbot":{"emoji":"🚀","requires":{"anyBins":["gh","git"]},"os":["linux","darwin","win32"]}}

CI/CD Pipeline (GitHub Actions)

Set up and manage CI/CD pipelines using GitHub Actions. Covers workflow creation, testing, deployment, release automation, and debugging.

When to Use

• Setting up automated testing on push/PR
• Creating deployment pipelines (staging, production)
• Automating releases with changelogs and tags
• Debugging failing CI workflows
• Setting up matrix builds for cross-platform testing
• Managing secrets and environment variables in CI
• Optimizing CI with caching and parallelism

Quick Start: Add CI to a Project

Node.js project

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm test
      - run: npm run lint

Python project

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - run: pytest
      - run: ruff check .

Go project

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version: "1.22"
      - run: go test ./...
      - run: go vet ./...

Rust project

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - uses: Swatinem/rust-cache@v2
      - run: cargo test
      - run: cargo clippy -- -D warnings

Common Patterns

Matrix builds (test across versions/OSes)

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node-version: [18, 20, 22]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm ci
      - run: npm test

Conditional jobs

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm test

  deploy:
    needs: test
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./deploy.sh

Caching dependencies

# Node.js (automatic with setup-node)
- uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: npm  # or yarn, pnpm

# Generic caching
- uses: actions/cache@v4
  with:
    path: |
      ~/.cache/pip
      ~/.cargo/registry
      node_modules
    key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-deps-

Artifacts (save build outputs)

- uses: actions/upload-artifact@v4
  with:
    name: build-output
    path: dist/
    retention-days: 7

# Download in another job
- uses: actions/download-artifact@v4
  with:
    name: build-output
    path: dist/

Run on schedule (cron)

on:
  schedule:
    - cron: "0 6 * * 1"  # Every Monday at 6 AM UTC
  workflow_dispatch:  # Also allow manual trigger

Deployment Workflows

Deploy to production on tag

name: Release

on:
  push:
    tags:
      - "v*"

jobs:
  release:
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm run build
      - run: npm test

      # Create GitHub release
      - uses: softprops/action-gh-release@v2
        with:
          generate_release_notes: true
          files: |
            dist/*.js
            dist/*.css

Deploy to multiple environments

name: Deploy

on:
  push:
    branches: [main, staging]

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ github.ref == 'refs/heads/main' && 'production' || 'staging' }}
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - run: |
          if [ "${{ github.ref }}" = "refs/heads/main" ]; then
            ./deploy.sh production
          else
            ./deploy.sh staging
          fi
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

Docker build and push

name: Docker

on:
  push:
    branches: [main]
    tags: ["v*"]

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: |
            ghcr.io/${{ github.repository }}:latest
            ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

npm publish on release

name: Publish

on:
  release:
    types: [published]

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          registry-url: https://registry.npmjs.org
      - run: npm ci
      - run: npm test
      - run: npm publish --provenance
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

Secrets Management

Set secrets via CLI

# Set a repository secret
gh secret set DEPLOY_TOKEN --body "my-secret-value"

# Set from a file
gh secret set SSH_KEY < ~/.ssh/deploy_key

# Set for a specific environment
gh secret set DB_PASSWORD --env production --body "p@ssw0rd"

# List secrets
gh secret list

# Delete a secret
gh secret delete OLD_SECRET

Use secrets in workflows

env:
  # Available to all steps in this job
  DATABASE_URL: ${{ secrets.DATABASE_URL }}

steps:
  - run: echo "Deploying..."
    env:
      # Available to this step only
      API_KEY: ${{ secrets.API_KEY }}

Environment protection rules

Set up via GitHub UI or API:

• Required reviewers before deployment
• Wait timers
• Branch restrictions
• Custom deployment branch policies

# View environments
gh api repos/{owner}/{repo}/environments | jq '.environments[].name'

Workflow Debugging

Re-run failed jobs

# List recent workflow runs
gh run list --limit 10

# View a specific run
gh run view <run-id>

# View failed job logs
gh run view <run-id> --log-failed

# Re-run failed jobs only
gh run rerun <run-id> --failed

# Re-run entire workflow
gh run rerun <run-id>

Debug with SSH (using tmate)

# Add this step before the failing step
- uses: mxschmitt/action-tmate@v3
  if: failure()
  with:
    limit-access-to-actor: true

Common failures and fixes

"Permission denied" on scripts

- run: chmod +x ./scripts/deploy.sh && ./scripts/deploy.sh

"Node modules not found"

# Make sure npm ci runs before npm test
- run: npm ci     # Install exact lockfile versions
- run: npm test   # Now node_modules exists

"Resource not accessible by integration"

# Add permissions block
permissions:
  contents: write
  packages: write
  pull-requests: write

Cache not restoring

# Check cache key matches - use hashFiles for lockfile
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
# NOT: key: ${{ runner.os }}-node-${{ hashFiles('package.json') }}

Workflow not triggering

• Check: is the workflow file on the default branch?
• Check: does the trigger event match? (push vs pull_request)
• Check: is the branch filter correct?

# Manually trigger a workflow
gh workflow run ci.yml --ref main

Workflow Validation

Validate locally before pushing

# Check YAML syntax
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" && echo "Valid"

# Use actionlint (if installed)
actionlint .github/workflows/ci.yml

# Or via Docker
docker run --rm -v "$(pwd):/repo" -w /repo rhysd/actionlint:latest

View workflow as graph

# List all workflows
gh workflow list

# View workflow definition
gh workflow view ci.yml

# Watch a running workflow
gh run watch

Advanced Patterns

Reusable workflows

# .github/workflows/reusable-test.yml
name: Reusable Test
on:
  workflow_call:
    inputs:
      node-version:
        required: false
        type: string
        default: "20"
    secrets:
      npm-token:
        required: false

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ inputs.node-version }}
      - run: npm ci
      - run: npm test

# .github/workflows/ci.yml - caller
name: CI
on: [push, pull_request]
jobs:
  test:
    uses: ./.github/workflows/reusable-test.yml
    with:
      node-version: "20"

Concurrency (prevent duplicate runs)

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true  # Cancel previous runs for same branch

Path filters (only run for relevant changes)

on:
  push:
    paths:
      - "src/**"
      - "package.json"
      - "package-lock.json"
      - ".github/workflows/ci.yml"
    paths-ignore:
      - "docs/**"
      - "*.md"

Monorepo: only test changed packages

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      api: ${{ steps.filter.outputs.api }}
      web: ${{ steps.filter.outputs.web }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            api:
              - 'packages/api/**'
            web:
              - 'packages/web/**'

  test-api:
    needs: changes
    if: needs.changes.outputs.api == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cd packages/api && npm ci && npm test

  test-web:
    needs: changes
    if: needs.changes.outputs.web == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cd packages/web && npm ci && npm test

Tips

• Use workflow_dispatch on every workflow for manual triggering during debugging
• Pin action versions to SHA for supply chain security: uses: actions/checkout@b4ffde...
• Use continue-on-error: true for non-critical steps (like linting)
• Set timeout-minutes on jobs to prevent runaway builds (default is 360 minutes)
• Use job outputs to pass data between jobs: outputs: result: ${{ steps.step-id.outputs.value }}
• For self-hosted runners: runs-on: self-hosted with labels for targeting specific machines