ci-cd-quirks

name	ci-cd-quirks
description	Use when CI checks fail unexpectedly, when preparing code for CI, or when encountering non-obvious build and pipeline behavior. Covers vendoring, hermetic builds, security scans, code generation checks, coverage, and webhook validation gotchas.

CI/CD Quirks and Gotchas

Overview

Integration-service CI runs on Konflux (Tekton-based) for builds and GitHub Actions for linting/testing. Several non-obvious requirements catch developers off guard.

When to Use

• CI failed and you don't understand why
• Preparing a change that touches dependencies, CRDs, or RBAC
• Understanding what the build pipeline actually does
• Debugging webhook validation behavior

Must-Run Commands Before Push

When you changed...	Run this	Why
Any go.mod / go.sum	go mod tidy && go mod vendor	Vendor dir is committed; CI fails on drift
Types in api/v1beta2/	make generate manifests	DeepCopy and CRD/RBAC manifests must be in sync
Any .go file	make fmt	CI fails on uncommitted gofmt changes
+kubebuilder:rbac markers	make manifests	RBAC ClusterRole must reflect markers

CI Checks That Surprise People

Vendoring (most common failure)

Go vendoring prunes non-Go files. After go mod vendor, CRD YAML files from dependencies are gone. make download-crds re-downloads them for tests. CI runs this automatically, but if you updated deps and didn't vendor, CI fails.

RBAC Wildcard Check

CI greps config/ for RBAC wildcards (*) and fails if found. Never use wildcard verbs or resource names in +kubebuilder:rbac markers.

Code Generation Drift

CI runs make generate manifests and diffs the result. Any uncommitted generated files = failure. Always commit generated output.

go mod tidy Drift

CI runs go mod tidy and checks for changes. If your go.mod/go.sum differ after tidy, CI fails.

Build Pipeline Details

PR Pipeline (.tekton/integration-service-pull-request.yaml)

• Injects ENABLE_COVERAGE=true — binary built with -cover -covermode=atomic -tags=coverage
• Hermetic build (network-isolated), deps prefetched via Cachi2
• Multi-platform: x86_64 + arm64
• Image tag: on-pr-{{revision}}
• Expires after 5 days, max 3 kept
• Security scans: Clair, Roxctl, Snyk SAST, ClamAV, ShellCheck, Unicode, RPM signature

Push Pipeline (.tekton/integration-service-push.yaml)

• Regular build (no coverage) + extra build-instrumented-image task (with coverage)
• Image tag: {{revision}}
• Same security scans as PR

Dockerfile Gotchas

• Builds two binaries: manager (controller) and snapshotgc (garbage collector)
• Both ship in the same image, entrypoint is /manager
• ENABLE_COVERAGE build arg switches between instrumented and production builds
• Base image: UBI9 minimal, runs as user 65532 (non-root)

Webhook Validation Gotchas

Rule	What happens if violated
ITS name must be DNS-1035	Webhook rejects with validation error (lowercase alphanumeric + hyphen, <63 chars)
SNAPSHOT param in ITS	Cannot be set manually — auto-injected by the service. Webhook rejects.
Git resolver: url vs repo+org	Must use one or the other, not both. Webhook rejects conflicting params.
Snapshot validator	Has failurePolicy: Ignore — validation failures pass through silently (by design)

Coverage

• Tool: codecov, flag: unit-tests
• Threshold: 3% drop from base commit allowed
• Ignores: **/zz_generated*, e2e-tests/**, vendor
• Patch coverage: informational only (non-blocking)

Common Mistakes

Problem	Root cause
CI fails with "uncommitted changes"	Didn't run make fmt, make generate manifests, or go mod tidy
Build fails on missing deps	Didn't run go mod vendor after dependency change
RBAC check fails	Used wildcard * in kubebuilder RBAC marker
Coverage dropped	New code paths not covered by tests (check cover.out)
Webhook rejects valid-looking ITS	Name contains uppercase, underscore, or is >63 chars