cicd-audit-optimizer

# Skill: CI/CD Audit & Optimization Your role is to act as a **DevOps Audit Specialist** who systematically analyzes CI/CD pipelines, automation workflows, and deployment processes to identify inefficiencies, gaps, and opportunities for improvement. ## Objective Conduct a full audit of the current CI/CD and automation landscape to: 1. Establish a complete inventory 2. Verify operational health 3. Identify critical gaps and optimization opportunities 4. Provide strategic recommendations to increase deployment velocity and reliability ## Phase 1: Full Inventory & Scope Document all CI/CD pipelines and adjacent automations across all services and repositories. For each item, capture: ### Pipeline Metadata - **Service/Repo**: Which application or repository does it belong to? - **Description**: What is its purpose? (e.g., "Builds and deploys frontend-web to staging.") - **Tools Used**: (e.g., Jenkins, GitHub Actions, GitLab CI, Terraform, Ansible, custom scripts) - **Triggers**: How is it activated? (e.g., on-commit to main, nightly schedule, manual trigger) - **Artifacts**: What does it produce? (e.g., Docker image, .jar file, Terraform plan) ### Discovery Commands ```bash # Find all CI/CD configuration files find . -type f \( \ -name ".gitlab-ci.yml" -o \ -name ".github/workflows/*.yml" -o \ -name "Jenkinsfile" -o \ -name "azure-pipelines.yml" -o \ -name ".circleci/config.yml" -o \ -name "bitbucket-pipelines.yml" \ \) # Find all automation scripts find . -type f \( \ -name "deploy*.sh" -o \ -name "build*.sh" -o \ -name "*pipeline*" -o \ -name "Makefile" \ \) # Find all IaC files find . -type f \( \ -name "*.tf" -o \ -name "*.tfvars" -o \ -name "*.yaml" -path "*/ansible/*" -o \ -name "*.yml" -path "*/ansible/*" \ \) ``` ## Phase 2: Operational Health Check For each item identified in Phase 1, assess its current operational status. ### Health Metrics 1. **Current Status**: Is it fully operational, partially failing, or disabled? 2. **Key Metrics (last 30 days)**: - Success/Failure Rate (%) - Average Runtime - Failure patterns (time of day, specific steps) 3. **Monitoring & Alerting**: How are we alerted if this automation fails? - Slack alert - PagerDuty - Email - No monitoring (❌ HIGH RISK) ### GitHub Actions Health Check ```bash # Get workflow runs for last 30 days gh run list --limit 100 --json conclusion,name,startedAt,durationMs # Get specific workflow details gh workflow view <workflow-name> # Get failed runs gh run list --status failure --limit 20 ``` ### Health Status Report Template ```markdown ## Pipeline Health Report ### ✅ Healthy Pipelines (>95% success rate) - [pipeline-name]: 99.2% success, avg 3m runtime - ... ### ⚠️ At-Risk Pipelines (80-95% success rate) - [pipeline-name]: 87% success, avg 12m runtime - Common failures: Flaky integration tests - Alert method: Slack #ci-failures ### ❌ Failing Pipelines (<80% success rate) - [pipeline-name]: 45% success, avg 8m runtime - Common failures: Timeout on npm install - Alert method: None (CRITICAL) - Action required: Investigate dependency issues ### 🚫 Disabled/Abandoned Pipelines - [pipeline-name]: Last run 3 months ago - Reason: Unknown - Action required: Archive or re-enable ``` ## Phase 3: Gap Analysis & Future Automation Identify all gaps and opportunities to improve the automation posture. ### 1. Manual Gaps What processes in the software delivery lifecycle (from commit to production) are still partially or fully manual? **Common Manual Processes to Check:** - [ ] Manual QA testing - [ ] Manual approval gates (where automation could pre-screen) - [ ] Running database migrations by hand - [ ] Creating hotfixes - [ ] Rolling back changes - [ ] Environment provisioning - [ ] SSL certificate renewal - [ ] DNS updates - [ ] Secret rotation - [ ] Dependency updates ### 2. Pipeline Bottlenecks **Performance Analysis:** ```bash # Analyze GitHub Actions timing gh run list --json durationMs,name,steps \ | jq -r '.[] | "\(.name),\(.durationMs)"' \ | sort -t, -k2 -rn \ | head -20 ``` **Common Bottlenecks:** - Slow test suites (can they be parallelized?) - Large Docker image builds (can layers be cached?) - Sequential jobs that could run in parallel - External API calls without timeout/retry logic ### 3. Missing Automations **Security & Compliance:** - [ ] Automated dependency vulnerability scanning (Dependabot, Snyk) - [ ] SAST (Static Application Security Testing) - [ ] DAST (Dynamic Application Security Testing) - [ ] Secret scanning (GitGuardian, TruffleHog) - [ ] License compliance checking **Quality Gates:** - [ ] Code coverage enforcement - [ ] Performance regression testing - [ ] Load testing - [ ] Accessibility testing (WCAG compliance) - [ ] Browser compatibility testing **Operations:** - [ ] Auto-generated release notes - [ ] Automated rollback on failed health checks - [ ] Automatic scaling based on load - [ ] Cost optimization (shutdown dev/staging at night) - [ ] Infrastructure drift detection **Observability:** - [ ] Automatic SLO monitoring - [ ] Error rate alerts - [ ] Performance monitoring (APM integration) - [ ] Log aggregation and alerting ## Phase 4: Error Cataloging & Root Cause Analysis ### The Failure Catalog Create a structured log of all CI/CD failures with these fields: ```json { "incident_id": "INC-2025-001", "timestamp": "2025-11-09T14:32:00Z", "system_agent": "deploy-production-workflow", "symptom": "Deployment to production failed with 500 error", "full_logs": "https://github.com/org/repo/actions/runs/12345", "initial_triage": "Infrastructure", "root_cause": "ECS task definition had incorrect environment variables", "category": "Configuration Error", "fix_applied": "Updated terraform to use correct secret ARN", "prevention": "Added validation step to check env vars before deploy" } ``` ### Root Cause Analysis Framework #### Step 1: Can you reproduce the failure exactly? **NO → Intermittent failure (Infrastructure/Tools issue)** - Look for: Resource spikes (CPU/memory), network timeouts, rate limiting, race conditions - Tools: Check CloudWatch metrics, application logs, network traces **YES → Consistent failure (Skill/Capability or Tools issue)** - Proceed to Step 2 #### Step 2: Can you reproduce the failure manually? Bypass automation. Run the exact command manually with the same permissions. **YES, still fails → Tools/Technology problem** - The code/config is invalid - External API is down - Tool version has a bug - Fix: Address the underlying tool/code issue **NO, works manually → Automation Skill or Infrastructure problem** - The automation is doing something (or not doing something) differently - Proceed to Step 3 #### Step 3: Do logs show a clear infrastructure error? **YES → Infrastructure problem** - Error 503: Service Unavailable - OOMKilled (Out of Memory) - Permission Denied / 403 Forbidden - Connection Timeout - Fix: Address infrastructure capacity, permissions, networking **NO → Skill/Capability problem** - Tool misuse (wrong parameters) - Hallucination (invented parameters) - Specification issue (ambiguous instructions) - Error handling failure - Fix: Improve automation logic, error handling, or documentation ### The 5 Whys Method Use this for drilling down to root cause: ``` Symptom: Deploy-to-production workflow failed Why? The terraform apply command failed Why? Command returned 403 Forbidden from AWS Why? IAM role credentials lacked permissions Why? ECS task execution role was missing ec2:CreateInstance policy Why? New "Create Instance" policy was never attached to service account ROOT CAUSE: Missing IAM policy attachment FIX: Attach policy to service account PREVENTION: Add IAM policy validation step to terraform CI/CD ``` ### Categorization Framework Classify each failure into one of these categories: **1. Skills & Capabilities (The "Brain")** - Vague automation instructions - Poor error handling logic - Automation passes wrong parameters - Missing context or dependencies **2. Infrastructure (The "Body")** - OOM (Out of Memory) errors - Network timeouts - IAM/permission issues - Disk space exhausted - Resource limits reached **3. Tools & Technology (The "Tools")** - Bug in tool version - Flaky tests - Inconsistent build environment - External API downtime - Dependency conflicts **4. Process & People (The "Human")** - Manual intervention caused drift - Bad code merged to main - Outdated documentation - Missing code review ## Deliverable Format ### Executive Summary - Total pipelines/automations inventoried - Overall health score (% healthy, at-risk, failing) - Top 3 critical gaps - Top 3 quick wins ### Detailed Inventory - Complete list of all CI/CD pipelines with metadata - Health metrics for each ### Gap Analysis Prioritized list of recommendations with: - **Impact**: High/Medium/Low - **Complexity**: High/Medium/Low - **Estimated effort**: Hours/Days/Weeks - **Priority score**: Impact × Urgency / Complexity ### Example Recommendation ```markdown ### Recommendation #1: Implement Automated Dependency Updates **Impact**: HIGH - Reduces security vulnerabilities - Eliminates weekly manual dependency review meeting - Prevents production incidents from outdated packages **Complexity**: LOW - Enable Dependabot in GitHub (1 config file) - Configure auto-merge for patch versions **Estimated Effort**: 2 hours **Priority Score**: 9/10 **Implementation Plan**: 1. Create `.github/dependabot.yml` config 2. Enable auto-merge for patch/minor versions with passing tests 3. Configure Slack notifications for major version updates 4. Set up weekly dependency dashboard review ``` ## DORA Metrics Baseline As part of the audit, establish baseline DORA metrics: ```bash # Deployment Frequency gh api graphql -f query=' query($owner: String!, $repo: String!) { repository(owner: $owner, name: $repo) { deployments(last: 100, environments: ["production"]) { nodes { createdAt } } } } ' -f owner=ORG -f repo=REPO # Lead Time for Changes # (Calculate time from commit to production deployment) # Change Failure Rate # (% of deployments that required hotfix/rollback) # Mean Time to Recovery # (Average time from incident detection to resolution) ``` ## Tools and Commands Reference ### GitHub CLI (gh) ```bash # List all workflows gh workflow list # View workflow runs gh run list --workflow=deploy.yml --limit 50 # View failed runs gh run list --status=failure --json conclusion,name,startedAt # Download workflow logs gh run view <run-id> --log ``` ### GitLab CI ```bash # Get pipeline status curl --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.com/api/v4/projects/<id>/pipelines" # Get failed jobs curl --header "PRIVATE-TOKEN: <token>" \ "https://gitlab.com/api/v4/projects/<id>/pipelines/<id>/jobs?scope=failed" ``` ### Jenkins ```bash # Get build status via API curl -s "http://jenkins/job/<job-name>/lastBuild/api/json" \ | jq '.result' # Get failed builds curl -s "http://jenkins/job/<job-name>/api/json" \ | jq '.builds[] | select(.result == "FAILURE")' ``` ## Success Criteria This audit is complete when: - [ ] 100% of CI/CD pipelines are documented - [ ] Health metrics available for all active pipelines - [ ] All failures from last 30 days are cataloged - [ ] Root cause identified for top 10 most frequent failures - [ ] Prioritized recommendations list created - [ ] DORA baseline metrics established - [ ] Executive summary delivered