| name | ci-cd-pipelines |
| description | CI/CD pipeline design, setup, and optimization. Use when user asks to "set up CI/CD", "create a pipeline", "configure Jenkins", "set up GitLab CI", "create CircleCI config", "automate deployments", "create build pipeline", "set up continuous deployment", "configure pipeline stages", "add pipeline caching", "pipeline security", "secret management", "artifact management", "GitHub Actions workflow", "deployment strategies", "canary deployment", or mentions CI/CD pipelines, continuous integration, continuous deployment, build automation, deployment pipelines, or pipeline optimization. |
| license | MIT |
| metadata | {"author":"1mangesh1","version":"1.0.0","tags":["cicd","pipelines","jenkins","gitlab-ci","circleci","automation","deployment"]} |
CI/CD Pipelines
Comprehensive guide to building, configuring, and optimizing CI/CD pipelines across multiple platforms including GitLab CI, Jenkins, CircleCI, and more.
Cross-reference: For GitHub Actions specifically, see the dedicated github-actions skill which covers workflows, matrix builds, caching, secrets, and reusable actions in depth.
CI/CD Pipeline Fundamentals
Core Concepts
- Continuous Integration (CI): Automatically build and test code on every commit
- Continuous Delivery (CD): Automatically prepare releases for deployment (manual trigger)
- Continuous Deployment (CD): Automatically deploy every change that passes the pipeline
Pipeline Anatomy
Commit -> Build -> Test -> Scan -> Stage -> Deploy -> Monitor
- Stages: Logical groupings of jobs (build, test, deploy)
- Jobs: Individual units of work within a stage
- Steps: Commands or actions within a job
- Artifacts: Files produced by jobs and passed between stages
- Triggers: Events that start a pipeline (push, merge, schedule, manual)
Pipeline Design Principles
- Fail fast - Run quick checks (lint, syntax) before slow tests
- Parallelize - Run independent jobs concurrently
- Cache aggressively - Avoid rebuilding unchanged dependencies
- Minimize artifacts - Pass only what downstream jobs need
- Idempotent deployments - Running deploy twice yields the same result
GitLab CI/CD
Basic .gitlab-ci.yml
stages:
- build
- test
- scan
- deploy
variables:
NODE_VERSION: "20"
DOCKER_DRIVER: overlay2
default:
image: node:${NODE_VERSION}
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- node_modules/
- .npm/
build:
stage: build
script:
- npm ci --cache .npm --prefer-offline
- npm run build
artifacts:
paths:
- dist/
expire_in: 1 hour
unit-tests:
stage: test
needs: [build]
script:
-
[]
[]
[]
[, ]
[]
GitLab CI Rules and Conditions
deploy-preview:
stage: deploy
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
when: always
- if: '$CI_COMMIT_BRANCH == "main"'
when: never
- when: never
script:
- ./deploy-preview.sh
trigger-downstream:
stage: deploy
trigger:
project: mygroup/downstream-project
branch: main
strategy: depend
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
GitLab CI Matrix Builds
test-matrix:
stage: test
parallel:
matrix:
- NODE_VERSION: ["18", "20", "22"]
DB: ["postgres", "mysql"]
image: node:${NODE_VERSION}
services:
- name: ${DB}:latest
script:
- npm ci
- npm test
Jenkins
Declarative Jenkinsfile
// Jenkinsfile (Declarative Pipeline)
pipeline {
agent any
environment {
DOCKER_REGISTRY = 'registry.example.com'
APP_NAME = 'my-app'
NODE_HOME = tool('NodeJS-20')
PATH = "${NODE_HOME}/bin:${env.PATH}"
}
options {
timeout(time: 30, unit: 'MINUTES')
disableConcurrentBuilds()
buildDiscarder(logRotator(numToKeepStr: '10'))
timestamps()
}
parameters {
choice(name: 'DEPLOY_ENV', choices: ['staging', 'production'], description: 'Target environment')
booleanParam(name: 'SKIP_TESTS', defaultValue: false, description: 'Skip test stage')
}
stages {
stage('Checkout') {
steps {
checkout scm
script {
env.GIT_COMMIT_SHORT = sh(script: 'git rev-parse --short HEAD', returnStdout: true).trim()
env.VERSION = "${env.BUILD_NUMBER}-${env.GIT_COMMIT_SHORT}"
}
}
}
stage('Install') {
steps {
sh 'npm ci'
}
}
stage('Lint & Format') {
steps {
sh 'npm run lint'
sh 'npm run format:check'
}
}
stage('Test') {
when {
not { params.SKIP_TESTS }
}
parallel {
stage('Unit Tests') {
steps {
sh 'npm run test:unit -- --ci --reporters=jest-junit'
}
post {
always {
junit 'junit.xml'
}
}
}
stage('Integration Tests') {
steps {
sh 'docker compose -f docker-compose.test.yml up -d'
sh 'npm run test:integration'
}
post {
always {
sh 'docker compose -f docker-compose.test.yml down -v'
}
}
}
}
}
stage('Build Docker Image') {
steps {
script {
docker.build("${DOCKER_REGISTRY}/${APP_NAME}:${VERSION}")
}
}
}
stage('Push Image') {
steps {
script {
docker.withRegistry("https://${DOCKER_REGISTRY}", 'docker-registry-creds') {
docker.image("${DOCKER_REGISTRY}/${APP_NAME}:${VERSION}").push()
docker.image("${DOCKER_REGISTRY}/${APP_NAME}:${VERSION}").push('latest')
}
}
}
}
stage('Deploy') {
input {
message "Deploy to ${params.DEPLOY_ENV}?"
ok 'Deploy'
submitter 'admin,deployers'
}
steps {
withCredentials([
string(credentialsId: 'deploy-token', variable: 'DEPLOY_TOKEN'),
file(credentialsId: 'kubeconfig', variable: 'KUBECONFIG')
]) {
sh """
kubectl set image deployment/${APP_NAME} \
${APP_NAME}=${DOCKER_REGISTRY}/${APP_NAME}:${VERSION} \
-n ${params.DEPLOY_ENV}
"""
}
}
}
}
post {
success {
slackSend(
channel: '#deployments',
color: 'good',
message: "SUCCESS: ${APP_NAME} ${VERSION} deployed to ${params.DEPLOY_ENV}"
)
}
failure {
slackSend(
channel: '#deployments',
color: 'danger',
message: "FAILED: ${APP_NAME} pipeline - ${env.BUILD_URL}"
)
}
always {
cleanWs()
}
}
}
Scripted Jenkinsfile
// Jenkinsfile (Scripted Pipeline)
node('docker') {
def app
def version
try {
stage('Checkout') {
checkout scm
version = sh(script: 'git describe --tags --always', returnStdout: true).trim()
}
stage('Build') {
sh 'npm ci'
sh 'npm run build'
}
stage('Test') {
parallel(
'unit': { sh 'npm run test:unit' },
'lint': { sh 'npm run lint' }
)
}
stage('Docker') {
app = docker.build("my-app:${version}")
docker.withRegistry('https://registry.example.com', 'docker-creds') {
app.push()
app.push('latest')
}
}
stage('Deploy') {
if (env.BRANCH_NAME == 'main') {
timeout(time: 15, unit: 'MINUTES') {
input message: 'Deploy to production?', ok: 'Deploy'
}
sh "./deploy.sh production ${version}"
}
}
currentBuild.result = 'SUCCESS'
} catch (e) {
currentBuild.result = 'FAILURE'
throw e
} finally {
cleanWs()
emailext(
subject: "${currentBuild.result}: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
body: "Build URL: ${env.BUILD_URL}",
recipientProviders: [culprits(), requestor()]
)
}
}
Jenkins Shared Library
// vars/standardPipeline.groovy (Shared Library)
def call(Map config = [:]) {
pipeline {
agent any
stages {
stage('Build') {
steps {
sh config.buildCommand ?: 'npm ci && npm run build'
}
}
stage('Test') {
steps {
sh config.testCommand ?: 'npm test'
}
}
stage('Deploy') {
when { branch 'main' }
steps {
sh config.deployCommand ?: './deploy.sh'
}
}
}
}
}
// Usage in Jenkinsfile:
// @Library('my-shared-lib') _
// standardPipeline(buildCommand: 'make build', testCommand: 'make test')
CircleCI
.circleci/config.yml
version: 2.1
orbs:
node: circleci/node@5.2
docker: circleci/docker@2.6
slack: circleci/slack@4.13
executors:
node-executor:
docker:
- image: cimg/node:20.11
working_directory: ~/project
resource_class: medium
node-with-db:
docker:
- image: cimg/node:20.11
- image: cimg/postgres:16.2
environment:
POSTGRES_USER: test
POSTGRES_DB: test_db
POSTGRES_PASSWORD: test
- image: cimg/redis:7.2
commands:
install-deps:
steps:
- checkout
- node/install-packages:
pkg-manager: npm
cache-path: node_modules
cache-version: v1
jobs:
lint:
executor:
[, ]
Pipeline Design Patterns
Fan-Out / Fan-In
Run multiple jobs in parallel, then converge on a single downstream job.
stages:
- build
- test
- merge-results
- deploy
build:
stage: build
script: npm run build
artifacts:
paths: [dist/]
test-unit:
stage: test
needs: [build]
script: npm run test:unit
artifacts:
reports:
junit: reports/unit.xml
test-e2e:
stage: test
needs: [build]
script: npm run test:e2e
artifacts:
reports:
junit: reports/e2e.xml
test-performance:
stage: test
needs: [build]
script: npm run test:perf
artifacts:
paths: []
[, , ]
[]
Matrix Builds
Test across multiple versions, platforms, or configurations simultaneously.
workflows:
multi-platform:
jobs:
- test:
matrix:
parameters:
node-version: ["18", "20", "22"]
os: ["linux", "macos"]
Conditional Pipelines (Monorepo Strategy)
frontend:
stage: build
rules:
- changes:
- "frontend/**"
- "shared/**"
script:
- cd frontend && npm ci && npm run build
backend:
stage: build
rules:
- changes:
- "backend/**"
- "shared/**"
script:
- cd backend && pip install -r requirements.txt && pytest
infrastructure:
stage: deploy
rules:
- changes:
- "terraform/**"
script:
- cd terraform && terraform
Caching Strategies
Layer Caching for Speed
variables:
NPM_CACHE: "${CI_PROJECT_DIR}/.npm"
default:
cache:
- key:
files:
- package-lock.json
paths:
- node_modules/
policy: pull-push
- key: "${CI_JOB_NAME}-build-cache"
paths:
- .next/cache/
- dist/
policy: pull-push
test:
cache:
- key:
files:
- package-lock.json
paths:
- node_modules/
policy: pull
script:
- npm test
Docker Layer Caching
jobs:
build-docker:
machine:
image: ubuntu-2204:current
docker_layer_caching: true
steps:
- checkout
- run: docker build -t myapp:latest .
pipeline {
agent any
stages {
stage('Build') {
steps {
script {
docker.build("myapp:${env.BUILD_NUMBER}",
"--cache-from myapp:latest .")
}
}
}
}
}
Secret Management in Pipelines
Best Practices
- Never hardcode secrets in pipeline files
- Use platform-native secret stores (GitLab CI Variables, Jenkins Credentials, CircleCI Contexts)
- Mask secrets in logs
- Rotate secrets regularly
- Limit scope - use environment-specific secrets
GitLab CI Variables
deploy:
script:
- echo "Deploying with token"
- curl -H "Authorization: Bearer ${DEPLOY_TOKEN}" https://api.example.com/deploy
variables:
DEPLOY_TOKEN: ${PRODUCTION_DEPLOY_TOKEN}
Jenkins Credentials
pipeline {
agent any
stages {
stage('Deploy') {
steps {
withCredentials([
usernamePassword(
credentialsId: 'docker-hub',
usernameVariable: 'DOCKER_USER',
passwordVariable: 'DOCKER_PASS'
),
string(credentialsId: 'api-key', variable: 'API_KEY'),
file(credentialsId: 'service-account', variable: 'SA_KEY_FILE')
]) {
sh 'docker login -u $DOCKER_USER -p $DOCKER_PASS'
sh 'gcloud auth activate-service-account --key-file=$SA_KEY_FILE'
}
}
}
}
}
HashiCorp Vault Integration
deploy:
variables:
VAULT_AUTH_ROLE: "gitlab-ci"
id_tokens:
VAULT_ID_TOKEN:
aud: https://vault.example.com
secrets:
DATABASE_PASSWORD:
vault: production/db/password@secrets
file: false
script:
- ./deploy.sh
Docker-Based Pipelines
Multi-Stage Docker Build in CI
build-image:
stage: build
image: docker:24
services:
- docker:24-dind
variables:
DOCKER_TLS_CERTDIR: "/certs"
IMAGE_TAG: "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}"
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- docker build
--build-arg BUILDKIT_INLINE_CACHE=1
--cache-from ${CI_REGISTRY_IMAGE}:latest
--tag ${IMAGE_TAG}
--tag ${CI_REGISTRY_IMAGE}:latest
.
- docker push ${IMAGE_TAG}
- docker push ${CI_REGISTRY_IMAGE}:latest
Kaniko (Rootless Docker Builds)
build-image:
stage: build
image:
name: gcr.io/kaniko-project/executor:v1.21.0-debug
entrypoint: [""]
script:
- /kaniko/executor
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}"
--cache=true
--cache-repo="${CI_REGISTRY_IMAGE}/cache"
Deploy Strategies
Blue-Green Deployment
deploy-blue-green:
stage: deploy
script:
- |
# Determine current active (blue or green)
CURRENT=$(kubectl get svc myapp -o jsonpath='{.spec.selector.slot}')
if [ "$CURRENT" = "blue" ]; then TARGET="green"; else TARGET="blue"; fi
kubectl set image deployment/myapp-${TARGET} app=${IMAGE_TAG}
kubectl rollout status deployment/myapp-${TARGET} --timeout=300s
./smoke-test.sh https://${TARGET}.internal.example.com
kubectl patch svc myapp -p "{\"spec\":{\"selector\":{\"slot\":\"${TARGET}\"}}}"
echo "Switched traffic from ${CURRENT} to ${TARGET}"
Canary Deployment
deploy-canary:
stage: deploy
script:
- |
# Deploy canary (10% traffic)
kubectl apply -f k8s/canary-deployment.yaml
kubectl set image deployment/myapp-canary app=${IMAGE_TAG}
sleep 300
ERROR_RATE=$(curl -s "http://prometheus:9090/api/v1/query?query=rate(http_errors_total[5m])" | jq '.data.result[0].value[1]')
if (( $(echo "$ERROR_RATE > 0.05" | bc -l) )); then
echo "Error rate too high (${ERROR_RATE}), rolling back canary"
kubectl delete deployment myapp-canary
exit 1
fi
kubectl set image deployment/myapp app=${IMAGE_TAG}
kubectl rollout status deployment/myapp --timeout=300s
kubectl delete deployment myapp-canary
[ ]
Environment Promotion (Dev -> Staging -> Production)
stages:
- build
- test
- deploy-dev
- deploy-staging
- deploy-production
build:
stage: build
script:
- docker build -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA} .
- docker push ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}
test:
stage: test
script:
- npm ci && npm test
deploy-dev:
stage: deploy-dev
environment:
name: development
url: https://dev.example.com
on_stop: stop-dev
script:
- helm upgrade --install myapp ./chart
--namespace dev
--set
Pipeline Security (SAST, DAST, Dependency Scanning)
stages:
- build
- test
- security
- deploy
sast-semgrep:
stage: security
image: semgrep/semgrep:latest
script:
- semgrep --config=auto --json --output=semgrep.json .
artifacts:
reports:
sast: semgrep.json
allow_failure: true
dependency-check:
stage: security
script:
- npm audit --json > npm-audit.json || true
- npx audit-ci --config audit-ci.json
artifacts:
paths:
- npm-audit.json
container-scan:
stage: security
[]
[]
Notification Integration
Slack Notifications
.notify_slack: ¬ify_slack
after_script:
- |
if [ "$CI_JOB_STATUS" = "success" ]; then
COLOR="#36a64f"
STATUS="succeeded"
else
COLOR="#dc3545"
STATUS="failed"
fi
curl -X POST "$SLACK_WEBHOOK_URL" \
-H 'Content-type: application/json' \
-d "{
\"attachments\": [{
\"color\": \"${COLOR}\",
\"title\": \"Pipeline ${STATUS}: ${CI_PROJECT_NAME}\",
\"text\": \"Branch: ${CI_COMMIT_REF_NAME}\nCommit: ${CI_COMMIT_SHORT_SHA}\nJob: ${CI_JOB_NAME}\",
\"footer\": \"<${CI_PIPELINE_URL}|View Pipeline>\"
}]
}"
deploy-production:
stage: deploy
<<: *notify_slack
script:
- ./deploy.sh production
Email Notifications (Jenkins)
// Jenkinsfile post section
post {
failure {
emailext(
subject: "FAILED: ${env.JOB_NAME} [${env.BUILD_NUMBER}]",
body: """
<h2>Build Failed</h2>
<p>Job: ${env.JOB_NAME}</p>
<p>Build: <a href="${env.BUILD_URL}">#${env.BUILD_NUMBER}</a></p>
<p>Changes: ${currentBuild.changeSets.collect { it.items.collect { it.msg } }.flatten().join(', ')}</p>
""",
mimeType: 'text/html',
recipientProviders: [culprits(), developers(), requestor()]
)
}
}
Artifact Management
build:
stage: build
script:
- npm run build
artifacts:
paths:
- dist/
- build/
exclude:
- dist/**/*.map
expire_in: 1 week
when: on_success
test:
stage: test
script:
- npm test
artifacts:
reports:
junit: reports/junit.xml
coverage_report:
coverage_format: cobertura
path: coverage/cobertura-coverage.xml
paths:
- coverage/
when: always
expire_in: 30 days
Pipeline as Code Best Practices
- Version control your pipeline - Pipeline config lives with the code it builds
- Use templates and includes to reduce duplication
include:
- project: 'devops/pipeline-templates'
ref: main
file:
- '/templates/node-ci.yml'
- '/templates/docker-build.yml'
- '/templates/deploy.yml'
- local: '.gitlab/ci/security.yml'
- template: Security/SAST.gitlab-ci.yml
- Pin versions of tools and images
- Use linting to validate pipeline configs before commit
gitlab-ci-lint .gitlab-ci.yml
circleci config validate
- Keep pipelines fast - target under 10 minutes for CI, under 30 for full CD
- Use YAML anchors to reduce repetition
.deploy_template: &deploy_defaults
image: alpine/k8s:1.28.4
before_script:
- kubectl config use-context ${KUBE_CONTEXT}
script:
- helm upgrade --install ${APP_NAME} ./chart --namespace ${NAMESPACE}
deploy-staging:
<<: *deploy_defaults
variables:
KUBE_CONTEXT: staging-cluster
NAMESPACE: staging
environment:
name: staging
deploy-production:
<<: *deploy_defaults
variables:
KUBE_CONTEXT: production-cluster
NAMESPACE: production
environment:
name: production
when: manual
Common Pipeline Failures and Fixes
| Problem | Cause | Fix |
|---|
npm ci fails | Lock file mismatch | Run npm install locally and commit package-lock.json |
| Docker build OOM | Insufficient memory | Increase runner resources or optimize Dockerfile |
| Tests pass locally, fail in CI | Environment differences | Use same Docker image locally; check env vars |
| Cache not restoring | Key mismatch | Verify cache key references correct files |
| Pipeline hangs | Waiting for approval or stuck service | Add timeouts; check when: manual stages |
| Permission denied | File not executable | Run chmod +x script.sh and commit |
| Secret not available | Variable scope mismatch | Check protected/masked settings; branch protection |
| Flaky tests | Race conditions or external deps | Retry mechanism; mock external services |
| Slow pipelines | No parallelism or caching | Add parallel, needs (DAG), and caching |
| Deploy rollback fails | No rollback strategy | Implement kubectl rollout undo or Helm rollback |
Retry Flaky Jobs
flaky-integration-test:
stage: test
retry:
max: 2
when:
- runner_system_failure
- stuck_or_timeout_failure
- script_failure
script:
- npm run test:integration
Timeout Configuration
job-with-timeout:
timeout: 15 minutes
script:
- npm run long-test
jobs:
slow-test:
steps:
- run:
name: Long running test
command: npm run test:e2e
no_output_timeout: 20m
References