cve-mitigation

name

cve-mitigation

description

Scan, triage, and remediate CVEs in the Gitpod monorepo. Use when asked to fix vulnerabilities, update dependencies for security, respond to CVE scan failures, or validate that dependency updates resolve known CVEs. Triggers on "fix CVE", "vulnerability scan", "security update", "dependency CVE", "grype scan", "leeway sbom", "CVE remediation", "update vulnerable dependency", "daily scan failure", "critical vulnerability".

CVE Mitigation

Fix and validate CVE findings in the Gitpod monorepo. This repo has ~80 Go modules under components/, dev/, install/, and test/, plus TypeScript/Node packages.

Scanning

Fast local scan (grype)

Install grype if missing, then scan individual modules or the whole repo:

# Install grype (one-time)
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin

# Scan a single module (fastest — ~2s per module)
grype file:components/common-go/go.mod --only-fixed --output table

# Scan a single module directory (slower — ~40s, also picks up binaries/lockfiles)
grype dir:components/common-go --only-fixed --output table

# JSON output for programmatic use
grype file:components/common-go/go.mod --only-fixed --output json 2>/dev/null | \
  jq '[.matches[] | {id: .vulnerability.id, severity: .vulnerability.severity, package: .artifact.name, installed: .artifact.version, fixedIn: .vulnerability.fix.versions}]'

# Scan all Go modules, show only Critical/High with fixes available
find components -name "go.mod" -not -path "*/vendor/*" \
  -exec sh -c 'echo "=== {} ===" && grype file:"{}" --only-fixed -q 2>/dev/null' \;

Use --fail-on critical or --fail-on high to get a non-zero exit code.

CI scan (leeway sbom)

CI uses leeway sbom export + leeway sbom scan against components:needs-vuln-scan. Leeway embeds both syft (SBOM generation) and grype (vulnerability matching) internally. The CI scan is slow (requires full build cache) and runs on every push to main and on PRs.

The daily scheduled run (build.yml, cron 0 0 * * *) checks vulnerability-stats.json for critical counts and sends Slack alerts via notify-scheduled-failure.

Read references/ci-scanning.md for the full CI pipeline details.

Trivy image scanning

A separate Trivy-based scan (scripts/trivy/trivy-scan-images.sh) scans built Docker images. It uses scripts/trivy/trivyignore.yaml for suppressions. This is distinct from the Go dependency scanning.

Triage

When a CVE is reported:

1. Check severity and CVSS score. Focus on Critical (CVSS ≥ 9.0) and High (CVSS ≥ 7.0) first.
2. Check if a fix exists. Use --only-fixed with grype. If no fix exists, document and defer.
3. Check exploitability in context. Not all CVEs are exploitable in Gitpod's usage. If a CVE is a false positive or not exploitable, add it to WORKSPACE.yaml under sbom.ignoreVulnerabilities with a vulnerability ID and a reason explaining why.
4. Check for breaking API changes. Some upgrades (e.g., runc v1.1.x → v1.2.x) change exported APIs. These require code migration, not just go get.

Remediation (Go dependencies)

Standard update flow

For each affected module:

cd components/<module>

# Update a specific dependency
go get google.golang.org/grpc@v1.79.3

# Update transitive dependencies that need bumping
go get golang.org/x/net@latest

# Clean up
go mod tidy

# Verify it builds
go build ./...

Multi-module coordination

This repo has many Go modules that share dependencies via replace directives in go.mod. When updating a dependency:

1. Start with leaf modules (APIs, protocol packages) and work up to service modules.
2. Check replace directives. Many modules use replace to point to local paths:

   replace github.com/gitpod-io/gitpod/common-go => ../common-go // leeway
   

   If common-go bumps a dependency, all modules that replace-reference it will transitively pick up the new version.
3. Update all modules that directly depend on the vulnerable package. Use:

   grep -rl "google.golang.org/grpc" components/*/go.mod | sort
   

4. Don't forget dev/ and test/ modules. CI builds these too. They often lag behind.
5. Run go mod tidy in every updated module. Missing this causes CI failures.

OpenTelemetry replace directives

Some modules (notably image-builder-bob) use replace directives to pin OpenTelemetry versions. When upgrading grpc, the OTel SDK version requirements may change. Update the replace directives to match:

replace (
    go.opentelemetry.io/otel => go.opentelemetry.io/otel v1.39.0
    go.opentelemetry.io/otel/sdk => go.opentelemetry.io/otel/sdk v1.39.0
    // ... etc
)

Deprecated package migration

When a package is deprecated and replaced (e.g., dgrijalva/jwt-go → golang-jwt/jwt/v5):

1. Update go.mod: remove old dependency, add new one.
2. Update import paths in .go files.
3. Verify API compatibility — most maintained forks keep the same API surface.
4. Run go build ./... and go test ./... to confirm.

Packages with breaking changes

Some upgrades cannot be done with a simple version bump. Document these in the PR under "Not addressed (requires follow-up)" with the specific breaking change. Example: opencontainers/runc v1.1.x → v1.2.x made libcontainer/cgroups/ebpf functions unexported.

Suppressing false positives

Two suppression mechanisms exist:

WORKSPACE.yaml (leeway sbom scan)

Add entries under sbom.ignoreVulnerabilities in WORKSPACE.yaml:

sbom:
  enabled: true
  ignoreVulnerabilities:
    - vulnerability: GHSA-xxxx-xxxx-xxxx
      reason: |
        Explanation of why this CVE is not exploitable in Gitpod's usage.

Trivy ignore (image scanning)

Add entries to scripts/trivy/trivyignore.yaml:

vulnerabilities:
  - id: CVE-YYYY-NNNNN
    paths:
      - "app/node_modules/package/package.json"
    statement: |
      Explanation.

Validation

After making changes, validate locally before pushing:

# Quick: scan only the modules you changed
grype file:components/<module>/go.mod --only-fixed --output table

# Thorough: scan all components
find components -name "go.mod" -not -path "*/vendor/*" \
  -exec sh -c 'echo "=== {} ===" && grype file:"{}" --only-fixed -q 2>/dev/null' \; \
  2>&1 | grep -E "^===|Critical|High"

# Build verification for key modules
for mod in common-go supervisor public-api-server ws-proxy ws-manager-mk2 \
           registry-facade blobserve content-service image-builder-mk3 \
           gitpod-cli local-app ide-metrics ide-service node-labeler \
           openvsx-proxy service-waiter ws-daemon image-builder-bob usage; do
  echo "=== Building components/$mod ===" && \
  (cd "components/$mod" && go build ./...) || echo "FAILED: $mod"
done

PR conventions

• Title: "Fix critical CVEs from daily vulnerability scan" or similar.
• Body: Include a table of updated packages with From/To versions and CVE IDs.
• Tag with team: staff-engineers and team: team-enterprise.
• Reference the Linear issue (e.g., "Fixes: CLC-XXXX").
• List any CVEs that could NOT be addressed and why under "Not addressed (requires follow-up)".
• Co-author: Co-authored-by: Ona <[e-mail]>

Reference files

• references/ci-scanning.md — CI pipeline details for vulnerability scanning.
• references/go-modules.md — List of all Go modules and their scan-relevant groupings.