cve-mitigation

# CVE Mitigation Fix and validate CVE findings in the Gitpod monorepo. This repo has ~80 Go modules under `components/`, `dev/`, `install/`, and `test/`, plus TypeScript/Node packages. ## Scanning ### Fast local scan (grype) Install grype if missing, then scan individual modules or the whole repo: ```bash # Install grype (one-time) curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin # Scan a single module (fastest — ~2s per module) grype file:components/common-go/go.mod --only-fixed --output table # Scan a single module directory (slower — ~40s, also picks up binaries/lockfiles) grype dir:components/common-go --only-fixed --output table # JSON output for programmatic use grype file:components/common-go/go.mod --only-fixed --output json 2>/dev/null | \ jq '[.matches[] | {id: .vulnerability.id, severity: .vulnerability.severity, package: .artifact.name, installed: .artifact.version, fixedIn: .vulnerability.fix.versions}]' # Scan all Go modules, show only Critical/High with fixes available find components -name "go.mod" -not -path "*/vendor/*" \ -exec sh -c 'echo "=== {} ===" && grype file:"{}" --only-fixed -q 2>/dev/null' \; ``` Use `--fail-on critical` or `--fail-on high` to get a non-zero exit code. ### CI scan (leeway sbom) CI uses `leeway sbom export` + `leeway sbom scan` against `components:needs-vuln-scan`. Leeway embeds both syft (SBOM generation) and grype (vulnerability matching) internally. The CI scan is slow (requires full build cache) and runs on every push to `main` and on PRs. The daily scheduled run (`build.yml`, cron `0 0 * * *`) checks `vulnerability-stats.json` for critical counts and sends Slack alerts via `notify-scheduled-failure`. Read `references/ci-scanning.md` for the full CI pipeline details. ### Trivy image scanning A separate Trivy-based scan (`scripts/trivy/trivy-scan-images.sh`) scans built Docker images. It uses `scripts/trivy/trivyignore.yaml` for suppressions. This is distinct from the Go dependency scanning. ## Triage When a CVE is reported: 1. **Check severity and CVSS score.** Focus on Critical (CVSS ≥ 9.0) and High (CVSS ≥ 7.0) first. 2. **Check if a fix exists.** Use `--only-fixed` with grype. If no fix exists, document and defer. 3. **Check exploitability in context.** Not all CVEs are exploitable in Gitpod's usage. If a CVE is a false positive or not exploitable, add it to `WORKSPACE.yaml` under `sbom.ignoreVulnerabilities` with a `vulnerability` ID and a `reason` explaining why. 4. **Check for breaking API changes.** Some upgrades (e.g., `runc` v1.1.x → v1.2.x) change exported APIs. These require code migration, not just `go get`. ## Remediation (Go dependencies) ### Standard update flow For each affected module: ```bash cd components/<module> # Update a specific dependency go get google.golang.org/grpc@v1.79.3 # Update transitive dependencies that need bumping go get golang.org/x/net@latest # Clean up go mod tidy # Verify it builds go build ./... ``` ### Multi-module coordination This repo has many Go modules that share dependencies via `replace` directives in `go.mod`. When updating a dependency: 1. **Start with leaf modules** (APIs, protocol packages) and work up to service modules. 2. **Check replace directives.** Many modules use `replace` to point to local paths: ``` replace github.com/gitpod-io/gitpod/common-go => ../common-go // leeway ``` If `common-go` bumps a dependency, all modules that replace-reference it will transitively pick up the new version. 3. **Update all modules that directly depend on the vulnerable package.** Use: ```bash grep -rl "google.golang.org/grpc" components/*/go.mod | sort ``` 4. **Don't forget `dev/` and `test/` modules.** CI builds these too. They often lag behind. 5. **Run `go mod tidy` in every updated module.** Missing this causes CI failures. ### OpenTelemetry replace directives Some modules (notably `image-builder-bob`) use `replace` directives to pin OpenTelemetry versions. When upgrading grpc, the OTel SDK version requirements may change. Update the replace directives to match: ```go replace ( go.opentelemetry.io/otel => go.opentelemetry.io/otel v1.39.0 go.opentelemetry.io/otel/sdk => go.opentelemetry.io/otel/sdk v1.39.0 // ... etc ) ``` ### Deprecated package migration When a package is deprecated and replaced (e.g., `dgrijalva/jwt-go` → `golang-jwt/jwt/v5`): 1. Update `go.mod`: remove old dependency, add new one. 2. Update import paths in `.go` files. 3. Verify API compatibility — most maintained forks keep the same API surface. 4. Run `go build ./...` and `go test ./...` to confirm. ### Packages with breaking changes Some upgrades cannot be done with a simple version bump. Document these in the PR under "Not addressed (requires follow-up)" with the specific breaking change. Example: `opencontainers/runc` v1.1.x → v1.2.x made `libcontainer/cgroups/ebpf` functions unexported. ## Suppressing false positives Two suppression mechanisms exist: ### WORKSPACE.yaml (leeway sbom scan) Add entries under `sbom.ignoreVulnerabilities` in `WORKSPACE.yaml`: ```yaml sbom: enabled: true ignoreVulnerabilities: - vulnerability: GHSA-xxxx-xxxx-xxxx reason: | Explanation of why this CVE is not exploitable in Gitpod's usage. ``` ### Trivy ignore (image scanning) Add entries to `scripts/trivy/trivyignore.yaml`: ```yaml vulnerabilities: - id: CVE-YYYY-NNNNN paths: - "app/node_modules/package/package.json" statement: | Explanation. ``` ## Validation After making changes, validate locally before pushing: ```bash # Quick: scan only the modules you changed grype file:components/<module>/go.mod --only-fixed --output table # Thorough: scan all components find components -name "go.mod" -not -path "*/vendor/*" \ -exec sh -c 'echo "=== {} ===" && grype file:"{}" --only-fixed -q 2>/dev/null' \; \ 2>&1 | grep -E "^===|Critical|High" # Build verification for key modules for mod in common-go supervisor public-api-server ws-proxy ws-manager-mk2 \ registry-facade blobserve content-service image-builder-mk3 \ gitpod-cli local-app ide-metrics ide-service node-labeler \ openvsx-proxy service-waiter ws-daemon image-builder-bob usage; do echo "=== Building components/$mod ===" && \ (cd "components/$mod" && go build ./...) || echo "FAILED: $mod" done ``` ## PR conventions - Title: "Fix critical CVEs from daily vulnerability scan" or similar. - Body: Include a table of updated packages with From/To versions and CVE IDs. - Tag with `team: staff-engineers` and `team: team-enterprise`. - Reference the Linear issue (e.g., "Fixes: CLC-XXXX"). - List any CVEs that could NOT be addressed and why under "Not addressed (requires follow-up)". - Co-author: `Co-authored-by: Ona <no-reply@ona.com>` ## Reference files - `references/ci-scanning.md` — CI pipeline details for vulnerability scanning. - `references/go-modules.md` — List of all Go modules and their scan-relevant groupings.