cve-mitigation

# CVE Mitigation Fix and validate CVE findings in the Gitpod monorepo. This repo has ~80 Go modules under `components/`, `dev/`, `install/`, and `test/`, plus TypeScript/Node packages. ## Scanning ### Fast local scan (grype) Install grype if missing, then scan individual modules or the whole repo: ```bash # Install grype (one-time) curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin # Scan a single module (fastest — ~2s per module) grype file:components/common-go/go.mod --only-fixed --output table # Scan a single module directory (slower — ~40s, also picks up binaries/lockfiles) grype dir:components/common-go --only-fixed --output table # JSON output for programmatic use grype file:components/common-go/go.mod --only-fixed --output json 2>/dev/null | \ jq '[.matches[] | {id: .vulnerability.id, severity: .vulnerability.severity, package: .artifact.name, installed: .artifact.version, fixedIn: .vulnerability.fix.versions}]' # Scan all Go modules, show only Critical/High with fixes available find components -name "go.mod" -not -path "*/vendor/*" \ -exec sh -c 'echo "=== {} ===" && grype file:"{}" --only-fixed -q 2>/dev/null' \; ``` Use `--fail-on critical` or `--fail-on high` to get a non-zero exit code. ### CI scan (leeway sbom) CI uses `leeway sbom export` + `leeway sbom scan` against `components:needs-vuln-scan`. Leeway embeds both syft (SBOM generation) and grype (vulnerability matching) internally. The CI scan is slow (requires full build cache) and runs on every push to `main` and on PRs. The daily scheduled run (`build.yml`, cron `0 0 * * *`) checks `vulnerability-stats.json` for critical counts and sends Slack alerts via `notify-scheduled-failure`. Read `references/ci-scanning.md` for the full CI pipeline details. ### Trivy image scanning A separate Trivy-based scan (`scripts/trivy/trivy-scan-images.sh`) scans built Docker images. It uses `scripts/trivy/trivyignore.yaml` for suppressions. This is distinct from the Go dependency scanning. ## Triage When a CVE is reported: 1. **Check severity and CVSS score.** Focus on Critical (CVSS ≥ 9.0) and High (CVSS ≥ 7.0) first. 2. **Check if a fix exists.** Use `--only-fixed` with grype. If no fix exists, document and defer. 3. **Check exploitability in context.** Not all CVEs are exploitable in Gitpod's usage. If a CVE is a false positive or not exploitable, add it to `WORKSPACE.yaml` under `sbom.ignoreVulnerabilities` with a `vulnerability` ID and a `reason` explaining why. 4. **Check for breaking API changes.** Some upgrades (e.g., `runc` v1.1.x → v1.2.x) change exported APIs. These require code migration, not just `go get`. ## BuildKit fork updates The image-builder-bob component depends on Gitpod's BuildKit fork. Follow the [fork maintenance procedure](references/buildkit.md) to update it with upstream while preserving Gitpod's customizations. ## Container image digest updates When updating Dockerfile image pins, prefer the tag's image index digest over an individual platform's image manifest digest, especially when the index also references provenance attestations. Preserve those references when available; use a platform-specific digest only when the task requires it or no index exists. Resolve the tag without selecting a platform (for example, `crane digest <image>:<tag>`) and inspect its manifest to confirm the digest identifies an image index or image manifest. Never use an image configuration or layer digest in a `FROM` reference. Compare index entries before treating a digest change as an update: switching from an index to its existing child image does not provide a newer build. ## Remediation (Go dependencies) ### Standard update flow For each affected module: ```bash cd components/<module> # Update a specific dependency go get google.golang.org/grpc@v1.79.3 # Update transitive dependencies that need bumping go get golang.org/x/net@latest # Clean up go mod tidy # Verify it builds go build ./... ``` ### Multi-module coordination This repo has many Go modules that share dependencies via `replace` directives in `go.mod`. When updating a dependency: 1. **Start with leaf modules** (APIs, protocol packages) and work up to service modules. 2. **Check replace directives.** Many modules use `replace` to point to local paths: ``` replace github.com/gitpod-io/gitpod/common-go => ../common-go // leeway ``` If `common-go` bumps a dependency, all modules that replace-reference it will transitively pick up the new version. 3. **Update all modules that directly depend on the vulnerable package.** Use: ```bash grep -rl "google.golang.org/grpc" components/*/go.mod | sort ``` 4. **Don't forget `dev/` and `test/` modules.** CI builds these too. They often lag behind. 5. **Run `go mod tidy` in every updated module.** Missing this causes CI failures. ### OpenTelemetry replace directives Some modules (notably `image-builder-bob`) use `replace` directives to pin OpenTelemetry versions. When upgrading grpc, the OTel SDK version requirements may change. Update the replace directives to match: ```go replace ( go.opentelemetry.io/otel => go.opentelemetry.io/otel v1.39.0 go.opentelemetry.io/otel/sdk => go.opentelemetry.io/otel/sdk v1.39.0 // ... etc ) ``` ### Deprecated package migration When a package is deprecated and replaced (e.g., `dgrijalva/jwt-go` → `golang-jwt/jwt/v5`): 1. Update `go.mod`: remove old dependency, add new one. 2. Update import paths in `.go` files. 3. Verify API compatibility — most maintained forks keep the same API surface. 4. Run `go build ./...` and `go test ./...` to confirm. ### Packages with breaking changes Some upgrades cannot be done with a simple version bump. Document these in the PR under "Not addressed (requires follow-up)" with the specific breaking change. Example: `opencontainers/runc` v1.1.x → v1.2.x made `libcontainer/cgroups/ebpf` functions unexported. ## Suppressing false positives Two suppression mechanisms exist: ### WORKSPACE.yaml (leeway sbom scan) Add entries under `sbom.ignoreVulnerabilities` in `WORKSPACE.yaml`: ```yaml sbom: enabled: true ignoreVulnerabilities: - vulnerability: GHSA-xxxx-xxxx-xxxx reason: | Explanation of why this CVE is not exploitable in Gitpod's usage. ``` ### Trivy ignore (image scanning) Add entries to `scripts/trivy/trivyignore.yaml`: ```yaml vulnerabilities: - id: CVE-YYYY-NNNNN paths: - "app/node_modules/package/package.json" statement: | Explanation. ``` ## Validation After making changes, validate locally before pushing: ```bash # Quick: scan only the modules you changed grype file:components/<module>/go.mod --only-fixed --output table # Thorough: scan all components find components -name "go.mod" -not -path "*/vendor/*" \ -exec sh -c 'echo "=== {} ===" && grype file:"{}" --only-fixed -q 2>/dev/null' \; \ 2>&1 | grep -E "^===|Critical|High" # Build verification for key modules for mod in common-go supervisor public-api-server ws-proxy ws-manager-mk2 \ registry-facade blobserve content-service image-builder-mk3 \ gitpod-cli local-app ide-metrics ide-service node-labeler \ openvsx-proxy service-waiter ws-daemon image-builder-bob usage; do echo "=== Building components/$mod ===" && \ (cd "components/$mod" && go build ./...) || echo "FAILED: $mod" done ``` ## PR conventions - Title: "Fix critical CVEs from daily vulnerability scan" or similar. - Body: Include a table of updated packages with From/To versions and CVE IDs. - Tag with `team: staff-engineers` and `team: team-enterprise`. - Reference the Linear issue (e.g., "Fixes: CLC-XXXX"). - List any CVEs that could NOT be addressed and why under "Not addressed (requires follow-up)". - Co-author: `Co-authored-by: Ona <no-reply@ona.com>` ## Reference files - `references/ci-scanning.md` — CI pipeline details for vulnerability scanning. - `references/go-modules.md` — List of all Go modules and their scan-relevant groupings.