| name | cicd-diagnostics |
| version | 1.0.0 |
| last_updated | 2025-04-12 |
| description | Use this skill to investigate and troubleshoot AWS CI/CD pipeline problems by analyzing CodePipeline executions, CodeBuild projects, CodeDeploy deployments, artifact management, and following structured runbooks. Activate when: pipeline execution failures, source stage errors, approval timeouts, cross-account pipeline issues, build failures, build timeouts, out-of-memory builds, Docker build issues, deployment failures, rollback problems, health check failures, lifecycle hook errors, artifact upload or download failures, encryption issues, S3 permissions, GitHub/CodeCommit source problems, ECR image issues, Lambda deploy failures, pipeline or deployment notifications not working, slow builds, slow deployments, or the user says something is wrong with their CI/CD pipeline without naming specific symptoms.
|
| compatibility | Requires AWS CLI with codepipeline, codebuild, codedeploy, s3, kms, iam, events, sns, ecr, and cloudtrail permissions. Optional: GitHub CLI, Docker CLI.
|
CI/CD Pipeline Diagnostics
When to use
Any AWS CI/CD investigation where the console alone is insufficient — CodePipeline execution failures, CodeBuild build errors, CodeDeploy deployment problems, artifact management, cross-account pipelines, notification issues, or performance troubleshooting.
Investigation workflow
Step 1 — Collect and triage
# Pipeline state — current status of all stages and actions
aws codepipeline get-pipeline-state --name <pipeline-name>
# Pipeline execution history — recent executions and their outcomes
aws codepipeline list-pipeline-executions --pipeline-name <pipeline-name> --max-items 10
# Specific execution details
aws codepipeline get-pipeline-execution --pipeline-name <pipeline-name> --pipeline-execution-id <execution-id>
# CodeBuild build details — phases, logs, environment
aws codebuild batch-get-builds --ids <build-id>
# CodeDeploy deployment details — status, instances, errors
aws codedeploy get-deployment --deployment-id <deployment-id>
# CodeDeploy deployment targets
aws codedeploy list-deployment-targets --deployment-id <deployment-id>
Triage returns:
- Pipeline execution status across all stages (Source, Build, Deploy, Approval)
- Failed action name, error code, and error message
- Build phase details (SUBMITTED, QUEUED, PROVISIONING, DOWNLOAD_SOURCE, INSTALL, PRE_BUILD, BUILD, POST_BUILD, UPLOAD_ARTIFACTS, FINALIZING)
- Deployment status, deployment type (in-place vs blue/green), and instance health
- Artifact locations and encryption configuration
If the pipeline itself is misconfigured (no stages, bad IAM role), that IS the root cause domain. Don't chase downstream build/deploy symptoms.
Step 2 — Domain deep dive (only if needed)
# CodeBuild logs from CloudWatch
aws logs get-log-events --log-group-name /aws/codebuild/<project-name> --log-stream-name <build-id>
# CodeDeploy instance-level deployment status
aws codedeploy get-deployment-target --deployment-id <deployment-id> --target-id <instance-id>
# Pipeline artifact store configuration
aws codepipeline get-pipeline --name <pipeline-name> --query 'pipeline.artifactStore'
# S3 artifact bucket permissions
aws s3api get-bucket-policy --bucket <artifact-bucket>
# KMS key policy for artifact encryption
aws kms describe-key --key-id <key-id>
aws kms get-key-policy --key-id <key-id> --policy-name default
# CloudTrail for pipeline API events
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=codepipeline.amazonaws.com --max-results 20
# EventBridge rules for pipeline triggers
aws events list-rules --name-prefix "codepipeline"
Read references/cicd-guardrails.md before concluding on any CI/CD issue.
Step 3 — Detailed investigation (low-confidence cases only)
# Pipeline definition — full structure including all stages, actions, and configuration
aws codepipeline get-pipeline --name <pipeline-name>
# CodeBuild project configuration — environment, source, artifacts, cache
aws codebuild batch-get-projects --names <project-name>
# CodeDeploy application and deployment group configuration
aws codedeploy get-deployment-group --application-name <app-name> --deployment-group-name <dg-name>
# IAM role policies for pipeline service role
aws iam get-role --role-name <pipeline-role>
aws iam list-attached-role-policies --role-name <pipeline-role>
aws iam list-role-policies --role-name <pipeline-role>
# CodeBuild service role policies
aws iam list-attached-role-policies --role-name <codebuild-role>
# Cross-account trust relationships
aws iam get-role --role-name <cross-account-role> --query 'Role.AssumeRolePolicyDocument'
# SNS topic subscriptions for notifications
aws sns list-subscriptions-by-topic --topic-arn <topic-arn>
# ECR image details
aws ecr describe-images --repository-name <repo> --image-ids imageTag=<tag>
Tool quick reference
| Tool / Command | When to use |
|---|
aws codepipeline get-pipeline-state | Current status of all stages and actions |
aws codepipeline get-pipeline-execution | Details of a specific execution |
aws codepipeline list-pipeline-executions | Recent execution history |
aws codepipeline get-pipeline | Full pipeline definition and configuration |
aws codebuild batch-get-builds | Build details, phases, logs, environment |
aws codebuild batch-get-projects | Project configuration, source, cache, environment |
aws codedeploy get-deployment | Deployment status, type, configuration |
aws codedeploy get-deployment-target | Instance-level deployment status and lifecycle events |
aws codedeploy list-deployment-targets | All targets in a deployment |
aws codedeploy get-deployment-group | Deployment group config, auto-rollback, alarms |
aws s3api get-bucket-policy | Artifact bucket permissions |
aws kms describe-key | Artifact encryption key details |
aws events list-rules | EventBridge rules for pipeline triggers |
aws logs get-log-events | CodeBuild logs from CloudWatch |
| CloudTrail | Pipeline, build, and deployment API events |
Gotchas: CI/CD Pipeline
These are the mistakes commonly made during CI/CD troubleshooting.
- Pipeline execution order matters: if multiple commits are pushed rapidly, CodePipeline processes them sequentially. A new execution supersedes an in-progress one in the same stage, causing the older execution to be SUPERSEDED (not failed). This is expected behavior, not an error.
- CodeBuild timeout defaults to 60 minutes. Maximum is 8 hours (480 minutes). If your build consistently times out, check the
timeoutInMinutes setting in the project configuration. Long-running builds (Docker layer caching, large test suites) often need this increased.
- CodeDeploy has two deployment types: in-place and blue/green. In-place updates instances one at a time (or in batches). Blue/green creates a new set of instances, shifts traffic, then terminates the old set. The troubleshooting path is completely different for each type.
- Artifact encryption: CodePipeline encrypts artifacts in S3 using AWS-managed keys (aws/s3) by default, or a customer-managed KMS key. Cross-account pipelines MUST use a customer-managed KMS key — the default aws/s3 key cannot be shared across accounts.
- Cross-account pipelines require KMS key sharing. The KMS key policy must grant
kms:Decrypt and kms:DescribeKey to the target account's role. The artifact bucket policy must also allow cross-account access. Missing either one causes "Access Denied" on artifact download.
- CodeBuild local caching can dramatically speed up builds. Three modes:
SOURCE_CACHE (caches source), DOCKER_LAYER_CACHE (caches Docker layers), CUSTOM_CACHE (caches specified paths). Docker layer cache requires privileged mode enabled. Cache is per-build host and not guaranteed between builds.
- buildspec.yml phases execute in order: install → pre_build → build → post_build. If a phase fails, subsequent phases are skipped EXCEPT post_build, which always runs (with
CODEBUILD_BUILD_SUCCEEDING=0). Use post_build for cleanup that must happen regardless of build success.
- appspec.yml hooks execute in a specific order for EC2: BeforeInstall → AfterInstall → ApplicationStart → ValidateService. For Lambda: BeforeAllowTraffic → AfterAllowTraffic. For ECS: BeforeInstall → AfterInstall → AfterAllowTestTraffic → BeforeAllowTraffic → AfterAllowTraffic. Hook scripts that exit non-zero fail the deployment.
- CodeDeploy rollback triggers: automatic rollback can be configured on deployment failure OR CloudWatch alarm threshold breach. If an alarm is in INSUFFICIENT_DATA state, it does NOT trigger a rollback — only ALARM state does. Misconfigured alarms cause rollbacks to not fire when expected.
- Pipeline source detection: polling checks for changes every minute (less efficient, adds latency). Webhooks provide near-instant detection. EventBridge (CloudWatch Events) is the recommended approach for CodeCommit and S3 sources. GitHub connections use webhooks via AWS CodeStar Connections.
Anti-hallucination rules
- Always cite specific AWS CLI output, build logs, deployment events, or CloudTrail entries as evidence.
- Never claim CodePipeline can run stages in parallel by default — stages execute sequentially. Only actions within a stage can run in parallel.
- Never suggest modifying CodeDeploy agent configuration files directly on managed instances — use Systems Manager or user data scripts.
- Never claim CodeBuild caches are persistent across all builds — local caches are per-build host and not guaranteed.
- Never assume cross-account pipelines work with the default AWS-managed S3 encryption key — they require a customer-managed KMS key.
- Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.
30 runbooks
Runbooks are organized by failure domain. Use the appropriate runbook based on the symptom category.
| Category | IDs | Covers |
|---|
| A — Pipeline | A1–A4 | Execution failures, source stage errors, approval timeout, cross-account pipeline |
| B — CodeBuild | B1–B4 | Build failures, timeout, OOM, Docker build issues |
| C — CodeDeploy | C1–C4 | Deployment failures, rollback, health check failures, lifecycle hook errors |
| D — Artifacts | D1–D3 | Artifact upload/download, encryption, S3 permissions |
| E — Integration | E1–E3 | GitHub/CodeCommit source, ECR image, Lambda deploy |
| F — Notifications | F1–F2 | Pipeline notifications, deployment notifications |
| G — Performance | G1–G2 | Slow builds, slow deployments |
| Z — Catch-All | Z1 | General CI/CD troubleshooting |
