| name | ci-cd-investigate |
| description | Use when a CI pipeline (GitLab / GitHub Actions / Bitbucket / etc.) failed or is stuck, when prod deploy seems broken, when the user says "пайплайн не прошёл" / "CI красный" / "deploy упал" / "прод лежит", or when investigating any CI/CD or production deployment issue — orchestrates evidence gathering via host MCP, prod SSH, observability stack, and decides between retry / fix-forward / rollback |
CI/CD Investigation
Investigation toolkit for your CI pipeline and production deployment.
Core principle: Get the failed-job log first, classify (transient vs reproducible), then decide retry vs forward-fix vs rollback. Don't push blindly to "see if it works."
This skill is host-agnostic. Substitute <HOST> with gitlab_*, github_actions_*, bitbucket_* or your CI MCP namespace. Substitute <PROJECT_ID> with your project id / slug. Substitute <PROD_HOST> and <PUBLIC_URL> accordingly.
Investigation Flow
digraph ci_flow {
rankdir=TB;
"Pipeline broken / prod issue" [shape=doublecircle];
"Get pipeline state via MCP" [shape=box];
"Identify failed jobs" [shape=box];
"Pull failed job log (last ~200 lines)" [shape=box];
"Classify failure" [shape=diamond];
"Transient (network/DNS/registry)" [shape=box, style=filled, fillcolor="#FFE082"];
"Reproducible (code/config/dep)" [shape=box, style=filled, fillcolor="#EF9A9A"];
"Infra (runner/DinD/disk)" [shape=box, style=filled, fillcolor="#90CAF9"];
"Retry job via MCP" [shape=box];
"Patch + commit + push" [shape=box];
"SSH prod / CI admin" [shape=box];
"Verify prod health" [shape=box];
"Resolved?" [shape=diamond];
"Rollback or escalate" [shape=octagon, style=filled, fillcolor=red, fontcolor=white];
"Done" [shape=doublecircle];
"Pipeline broken / prod issue" -> "Get pipeline state via MCP";
"Get pipeline state via MCP" -> "Identify failed jobs";
"Identify failed jobs" -> "Pull failed job log (last ~200 lines)";
"Pull failed job log (last ~200 lines)" -> "Classify failure";
"Classify failure" -> "Transient (network/DNS/registry)" [label="timeout/refused/registry 503"];
"Classify failure" -> "Reproducible (code/config/dep)" [label="compile/test/migration fail"];
"Classify failure" -> "Infra (runner/DinD/disk)" [label="runner offline/no space/cert"];
"Transient (network/DNS/registry)" -> "Retry job via MCP";
"Reproducible (code/config/dep)" -> "Patch + commit + push";
"Infra (runner/DinD/disk)" -> "SSH prod / CI admin";
"Retry job via MCP" -> "Verify prod health";
"Patch + commit + push" -> "Verify prod health";
"SSH prod / CI admin" -> "Verify prod health";
"Verify prod health" -> "Resolved?";
"Resolved?" -> "Done" [label="yes"];
"Resolved?" -> "Rollback or escalate" [label="no"];
}
Phase 1: Get pipeline state
Use your CI's MCP. Example for GitLab (mcp__gitlab__gitlab_*):
mcp__gitlab__gitlab_list_pipelines(project_id="<PROJECT_ID>", ref="main", per_page=5)
mcp__gitlab__gitlab_get_pipeline(project_id="<PROJECT_ID>", pipeline_id="<N>")
mcp__gitlab__gitlab_list_pipeline_jobs(project_id="<PROJECT_ID>", pipeline_id="<N>", include_retried=false, per_page=50)
Always batch the first three calls in parallel.
If user gave "Pipeline #N", skip list_pipelines. If they said "main is red", start with list_pipelines(ref="main", status="failed").
Phase 2: Read the failed job log
mcp__gitlab__gitlab_get_pipeline_job_output(project_id="<PROJECT_ID>", job_id="<N>")
Returns full trace. Expensive (50k–200k bytes). Read last error window only — search for ERROR:, failed, exit code, Cannot, Unable. The actual cause is almost always within the last 50 lines before the runner cleanup section.
If multiple jobs failed, pull only the FIRST chronological failure — later ones are usually downstream.
Phase 3: Classify
| Symptom in log | Class | Action |
|---|
Connection failed, connection timed out, Could not connect to archive.ubuntu.com | Transient (apt mirror) | retry job |
connection refused to docker registry / dial tcp ... timeout | Transient (registry / DinD) | retry job |
error: Failed to fetch, Hash Sum mismatch | Transient (apt cache) | retry, or pin deps |
error CSxxxx, error MSBxxxx, test name Failed | Reproducible (code) | fix-forward |
relation X does not exist, column X does not exist | Reproducible (migration) | new corrective migration |
npm ERR! 404, package X not found | Reproducible (dep pin) | fix package.json / lockfile |
no space left on device | Infra (runner disk) | SSH runner host, prune |
runner is offline / job stuck pending >10min | Infra (runner) | SSH runner host, restart runner |
403 Forbidden to registry | Infra (registry token) | check token rotation |
DEPLOY VERIFICATION FAILED: not all services healthy | Prod-side | SSH prod, see Phase 5 |
Transient ≠ ignore. If the same flake hits 2+ times in a week, fix-forward — pipeline shouldn't be one-network-blip away from a failed deploy.
Phase 4: Action
Retry a single job
mcp__gitlab__gitlab_retry_pipeline_job(project_id="<PROJECT_ID>", job_id="<N>")
Returns the NEW job (different id). Old failed job stays in history. Pipeline status flips back to running.
Retry whole pipeline
mcp__gitlab__gitlab_retry_pipeline(project_id="<PROJECT_ID>", pipeline_id="<N>")
Only retries failed jobs. Successful ones are not re-run. Use this when multiple jobs failed for the same transient cause.
Cancel a hung pipeline
mcp__gitlab__gitlab_cancel_pipeline(project_id="<PROJECT_ID>", pipeline_id="<N>")
Use when pipeline is stuck or you're about to push a fix that supersedes it.
Fix-forward
git checkout dev && git pull
Pipeline rules often use changes: filters — if a service builds only on changes inside its directory, an untouched service won't rebuild. Touch its Dockerfile to force a rebuild.
Trigger a fresh pipeline manually (rare)
mcp__gitlab__gitlab_create_pipeline(project_id="<PROJECT_ID>", ref="main", inputs={})
Use only if commit is already on main and CI didn't auto-trigger.
Phase 5: Production verification
Pipeline ✅ ≠ prod ✅. After deploy verify:
Public smoke checks
curl -fsS <PUBLIC_URL>/ -o /dev/null
curl -fsS <PUBLIC_URL>/.well-known/openid-configuration | jq .issuer
curl -fsS <PUBLIC_URL>/sitemap.xml -o /dev/null
SSH to prod
ssh root@<PROD_HOST>
cd /opt/<project>
docker compose -f docker-compose.prod.yml ps
docker ps --format 'table {{.Names}}\t{{.Status}}'
cat releases/current.env
Service logs (last 5 min)
ssh root@<PROD_HOST> "docker logs --tail=200 --since=5m <service>"
Or via observability stack (Loki):
{container_name="<service>"} | json | level=~"error|fatal"
Common prod symptoms after a "successful" deploy
| Symptom | Likely cause | Fix |
|---|
Service stays (unhealthy) | health endpoint 500 / migration not run | check migration container logs docker logs <service>-migrations |
Migration container Exit 1 | ORM SQL error / connection string drift | inspect log, add corrective migration |
| 502 from gateway | service crashed silently | docker logs <service> last 100 lines |
| Auth flow broken | appsettings.Production.json mismatch with .env | compare prod .env keys to bootstrap reads |
BrokenCircuitException in handlers | inter-service URL going through gateway returns 404 → Polly opens | use direct http://{service}:{port} |
Phase 6: Rollback (last resort)
If your CI has a manual rollback-production job:
mcp__gitlab__gitlab_play_pipeline_job(project_id="<PROJECT_ID>", job_id="<rollback-job-id>")
Or directly on prod (faster, no pipeline wait) — adapt to your deploy structure:
ssh root@<PROD_HOST> 'cd /opt/<project> && \
cp releases/current.env releases/rolled-back-from.env && \
cp releases/previous.env releases/current.env && \
set -a && . .env && . releases/current.env && set +a && \
docker compose -f docker-compose.prod.yml up -d --force-recreate'
After rollback, immediately open a fix-forward issue with prio::high. Rollback is a band-aid, not a fix.
Quick Reference (substitute MCP namespace for your CI host)
| Need | Tool |
|---|
| Latest pipeline on main | list_pipelines(ref="main", per_page=1) |
| Failed jobs in pipeline N | list_pipeline_jobs(pipeline_id="N", include_retried=false) filter status=failed |
| Why job failed | get_pipeline_job_output(job_id="N") — read last 50 lines |
| Retry one job | retry_pipeline_job(job_id="N") |
| Retry all failed in pipeline | retry_pipeline(pipeline_id="N") |
| Cancel stuck pipeline | cancel_pipeline(pipeline_id="N") |
| Manual job (deploy / rollback) | play_pipeline_job(job_id="N") |
| Open MR / PR with fix | create_merge_request(...) / gh pr create |
| Prod logs | SSH + docker logs, or observability MCP |
| Prod metrics | Prometheus MCP |
| Prod incidents | Incident-tracking MCP |
Common Mistakes
- Pushing speculative fixes without reading the log. Always pull the failed job output first. Half the failures are transient and need a retry, not a commit.
- Reading the whole log. It's 200KB+. Search for
ERROR, failed, exit code only. Last 50 lines is usually enough.
- Forgetting required pagination params (
include_retried, page, per_page). Most CI MCP schemas require them.
- Retrying a reproducible failure. Compile errors, test failures, migration issues do not heal themselves.
- Pushing to main directly. All work goes via integration branch → PR. Direct push to main is reserved for hotfixes after explicit user OK.
- Forgetting prod verification. Pipeline green ≠ prod healthy. Always run smoke checks after deploy.
- Touching unrelated files to "force" a rebuild. Use
retry_pipeline_job instead. If a Dockerfile patch is needed, that's a real change, not a force-rebuild.
- Not linking the issue. Pipeline-related fixes get a tracker issue per
board-tracking.md — type::bug + svc::infra. Commit message ends with (#N).
Anti-patterns
git push -f origin main to "fix" a CI hang. → Use cancel_pipeline then create new commit.- Editing the failed job's container manually via SSH to "complete" deploy. → Investigate and roll a real release.
- Skipping pre-commit hooks (
--no-verify) to land a CI fix. → Hooks exist for a reason; if they block legitimate work, fix the hook.
- Burying the actual error under "I'll just retry once more." → After 1 retry of the same job, classify as reproducible and fix forward.
