| name | debugging-running-instance |
| description | Use when debugging integration-service running on a cluster. Covers pod logs, health probes, metrics, webhooks, network policies, environment variables, snapshot GC, and common failure modes. |
Debugging a Running Instance
Overview
The integration-service runs as a Deployment in the integration-service namespace. It exposes health probes, Prometheus metrics, and optional webhooks. A separate CronJob handles snapshot garbage collection.
When to Use
- Controller not reconciling resources as expected
- Webhooks rejecting or silently ignoring mutations/validations
- Metrics not being scraped
- Status reports not appearing on PRs
- Snapshot GC not cleaning up old snapshots
Quick Reference
| What | How |
|---|
| Controller logs | kubectl logs -n integration-service deploy/controller-manager -f |
| Health check | kubectl get -n integration-service deploy/controller-manager -o jsonpath='{.status.conditions}' |
| Readiness probe | HTTP GET /readyz on port 8081 |
| Liveness probe | HTTP GET /healthz on port 8081 |
| Metrics | Port 8080 (HTTPS, self-signed cert), scraped at /metrics |
| Webhook port | 9443 (when enabled) |
| Leader election | Lease-based, 15s lease duration, 10s renew deadline, 2s retry period |
| Pod label | control-plane: controller-manager |
Environment Variables
Controller
| Variable | Default | Effect if wrong |
|---|
CONSOLE_URL | none | PR status comments show CONSOLE_URL_NOT_AVAILABLE |
CONSOLE_URL_TASKLOG | none | Task log links show CONSOLE_URL_TASKLOG_NOT_AVAILABLE |
CONSOLE_NAME | none | Console display name missing from PR comments |
PIPELINE_TIMEOUT | none | Invalid duration is logged as error and skipped |
TASKS_TIMEOUT | none | Invalid duration is logged as error and skipped |
FINALLY_TIMEOUT | none | Invalid duration is logged as error and skipped |
INTEGRATION_NS | integration-service | Wrong namespace for PAC secret lookup |
PAC_SECRET | pipelines-as-code-secret | Can't authenticate to git providers |
Network Policies
Namespaces must be explicitly labeled for traffic to reach the controller:
| Traffic | Required label |
|---|
| Metrics scraping | metrics: enabled on source namespace |
| Webhook calls | webhook: enabled on source namespace |
Missing labels = silently blocked traffic. Check with: kubectl get ns <name> --show-labels
Webhook Behavior
| Webhook | Type | Failure Policy | Effect |
|---|
| IntegrationTestScenario | Mutating | Ignore | Defaults applied silently on failure |
| IntegrationTestScenario | Validating | Fail | Rejects invalid ITS (bad names, conflicting resolver params) |
| ComponentGroup | Validating | Fail | Blocks invalid ComponentGroups |
| Snapshot | Mutating + Validating | Ignore | Failures silently pass through (by design) |
Debugging Checklist
- Controller not starting? Check env var values and RBAC — timeout parsing errors are logged, not fatal
- Not reconciling? Check leader election lease, RBAC, and that CRDs are installed
- PR status missing? Check
CONSOLE_URL env var and PAC secret availability
- Webhooks not firing? Check namespace labels (
webhook: enabled) and cert-manager status
- Metrics missing? Check namespace labels (
metrics: enabled) and ServiceMonitor exists
- Snapshots piling up? Check
snapshotgc CronJob logs — runs every 6h with separate service account
Common Mistakes
| Problem | Fix |
|---|
CONSOLE_URL_NOT_AVAILABLE in PR comments | Set CONSOLE_URL env var with {{NAMESPACE}} and {{PIPELINE_RUN_NAME}} placeholders |
| Controller crash loop | Check env var values — timeout vars must be valid Go durations (e.g., 2h, 30m) |
| Webhook silently not validating Snapshots | By design — Snapshot webhook has failurePolicy: Ignore |
| Metrics endpoint unreachable | Verify namespace label metrics: enabled and that ServiceMonitor target matches |
