educates-upgrade-cluster-services

name	educates-upgrade-cluster-services
description	Upgrade a vendored upstream cluster-service Helm chart the Educates operator installs for EducatesClusterConfig — cert-manager, Contour, Kyverno, or external-dns. Invoke when asked to "upgrade cert-manager / contour / kyverno / external-dns", "bump a vendored cluster-service chart", or "update the operator's bundled charts".
argument-hint	[cert-manager|contour|kyverno|external-dns] (omit to check all four)
allowed-tools	Read, Edit, Bash, WebFetch, Glob

Educates Cluster-Service Chart Upgrade Protocol

The operator installs four upstream cluster services for EducatesClusterConfig (Managed mode) from Helm chart tarballs vendored at installer/operator/vendored-charts/: cert-manager, Contour, Kyverno, external-dns. The bytes are checked into the repo and embedded into the operator binary via //go:embed. This skill bumps one (or checks all) to a new upstream version.

installer/operator/vendored-charts/README.md is the canonical reference for this workflow — keep it and this skill in agreement.

If $ARGUMENTS names a service, upgrade only that one. With no argument, check all four for newer upstream versions and report before changing anything.

The same directory also holds the in-repo runtime subcharts (secrets-manager, lookup-service, session-manager, node-ca-injector, remote-access). Those are not in scope here — they are repackaged from source via make package-local-charts, not downloaded.

Where each piece of version state lives

A version is pinned in four canonical places that must stay in lockstep (a consistency test enforces it — see Verification):

1. installer/operator/Makefile → VENDORED_CHARTS list: name=version=url.
2. installer/operator/vendored-charts/SHA256SUMS: <sha256> <name>-<version>.tgz.
3. installer/operator/vendored-charts/embed.go: the //go:embed <name>-<version>.tgz directive, the <X>ChartVersion constant, and (Contour / external-dns / Kyverno) the <X>AppVersion constant. cert-manager has a single CertManagerVersion (its chart version equals its appVersion).
4. The tarball file itself on disk.

Beyond these four, some tests hardcode a version string rather than referencing the embed.go constant (an import cycle prevents the helm package's tests from importing vendoredcharts). The DirectoryConsistent test does not cover these, so they go stale silently and only surface as a CI test failure. The known one is installer/operator/internal/helm/load_test.go (TestLoadArchive_VendoredCertManager), which hardcodes the cert-manager version twice (the .tgz filename and the AppVersion assertion). Always finish a bump with the straggler grep below — do not rely on the four-place list alone.

Upstream sources and version semantics

Service	Upstream releases	Chart URL pattern	App version
cert-manager	https://github.com/cert-manager/cert-manager/releases	https://charts.jetstack.io/charts/cert-manager-v<VER>.tgz	== chart version (v<VER>)
contour	https://github.com/projectcontour/helm-charts/releases	https://github.com/projectcontour/helm-charts/releases/download/contour-<VER>/contour-<VER>.tgz	Contour binary version (e.g. 1.33.5)
external-dns	https://github.com/kubernetes-sigs/external-dns/releases	https://github.com/kubernetes-sigs/external-dns/releases/download/external-dns-helm-chart-<VER>/external-dns-<VER>.tgz	external-dns binary version
kyverno	https://github.com/kyverno/kyverno/releases	https://kyverno.github.io/kyverno/kyverno-<VER>.tgz	Kyverno binary version (e.g. v1.18.1)

Use WebFetch on the releases page to find the latest stable chart version (not a pre-release). The chart version and the bundled app version differ for Contour, external-dns, and Kyverno — read the new app version from the tarball's Chart.yaml (appVersion:) after downloading.

Kyverno is the awkward one: its chart version and binary version are on different numbering tracks and live in different places. The github.com/kyverno/kyverno/releases page lists the binary version (appVersion, e.g. v1.18.1); the chart version (<VER> in the URL above, e.g. 3.8.1) only exists in the Helm repo's index.yaml. They are not aligned and you cannot derive one from the other by inspection — you must resolve the chart version that ships a given Kyverno binary from the repo index:

# What chart version ships Kyverno binary v1.18.1? (stable, non-rc rows only)
curl -sSfL https://kyverno.github.io/kyverno/index.yaml \
  | yq -r '.entries.kyverno[]
      | select(.appVersion | test("rc")|not)
      | .version + "  =>  " + .appVersion' \
  | grep ' v1.18.1$'        # → 3.8.1  =>  v1.18.1

To browse the newest stable charts and the binary each one bundles (the first row is the latest stable chart), drop the grep:

curl -sSfL https://kyverno.github.io/kyverno/index.yaml \
  | yq -r '.entries.kyverno[]
      | select(.appVersion | test("rc")|not)
      | .version + "  =>  " + .appVersion' \
  | head

Equivalently with the Helm CLI (helm repo add kyverno https://kyverno.github.io/kyverno/ && helm repo update kyverno && helm search repo kyverno/kyverno --versions), whose CHART VERSION / APP VERSION columns are the same two fields. Either way, that chart version is what goes in <VER> for the download URL, the Makefile, SHA256SUMS, and the embed.go filename + KyvernoChartVersion; the matching appVersion is KyvernoAppVersion (still confirm it from the downloaded tarball's Chart.yaml per step 5).

Current versions: read them from the Makefile VENDORED_CHARTS line and the embed.go constants.

Update process (per chart)

Worked example: contour 0.6.0 → <VER>. Other charts are identical except cert-manager has no separate AppVersion constant.

1. Compute the new tarball's SHA256 from the upstream URL (the "I am deliberately trusting these bytes" step — make vendor-charts refuses to download anything whose hash isn't already recorded):

   curl -sSfL "https://github.com/projectcontour/helm-charts/releases/download/contour-<VER>/contour-<VER>.tgz" | shasum -a 256
   

2. installer/operator/Makefile — update the chart's VENDORED_CHARTS entry (version and URL).

3. SHA256SUMS — replace the chart's line with the new filename + hash.

4. make -C installer/operator vendor-charts — downloads, verifies against SHA256SUMS, and writes the new tarball into place. Then git rm the old tarball (nothing prunes it automatically).

5. embed.go — update the //go:embed filename, the <X>ChartVersion constant, and the <X>AppVersion constant. Read the new appVersion from the tarball:

   tar -xzOf installer/operator/vendored-charts/contour-<VER>.tgz contour/Chart.yaml | grep -E '^(version|appVersion):'
   

6. Grep the repo for stragglers — hardcoded references to the old version that live outside the four canonical places (notably tests). Run this with the OLD version string before you forget it, and update every hit:

   # e.g. old cert-manager version v1.20.2 → catches internal/helm/load_test.go
   git grep -n "<old-version>" -- installer/operator
   

   For cert-manager this always includes internal/helm/load_test.go (filename + AppVersion assertion). Skip the vendored-charts/ line for the tarball you just git rm'd.

7. Re-verify reconciler assumptions (the load-bearing step — see next section). The per-service reconciler gates readiness on specific workload names / webhook behaviour verified against a chart version.

8. installer/operator/vendored-charts/README.md — update the "Current contents" version table.

9. Kyverno only — also re-vendor the bundled policies. See "Kyverno special case" below.

Re-verify reconciler assumptions

Each cluster service has a reconciler in installer/operator/internal/controller/config/ that gates readiness on workload names and, sometimes, CRD/webhook behaviour. A chart bump can rename a Deployment, change a values key, or move a CRD API version and silently break the gate. Render the new chart and confirm:

# extract + template the new chart with the operator's values shape
helm template <name> <path-to-extracted-chart> [--set ...] | grep -E 'kind: (Deployment|DaemonSet)' -A2

Service	Reconciler	What it asserts (verify still true)
cert-manager	certmanager.go	cert-manager, cert-manager-cainjector, cert-manager-webhook Deployments ready; cert-manager.io API discovery responds; CRDs present (deferred watch)
contour	contour.go	contour-contour Deployment + contour-envoy DaemonSet (names from the chart fullname helper); no admission webhook; envoy.useHostPort / envoy.service.type values still honoured
external-dns	externaldns.go	the external-dns Deployment is ready; provider value keys (Route53 / CloudDNS) unchanged
kyverno	kyverno.go	the four controller Deployments (kyverno-{admission,background,cleanup,reports}-controller) ready; validatingpolicies.policies.kyverno.io CRD still installed; RBAC role names unchanged

If a name or values key changed, update the reconciler constant and its "verified against " comment in the same change.

Kyverno special case

Bumping the Kyverno chart usually means bumping the Kyverno binary (KyvernoAppVersion), and the bundled Kyverno security policies are pinned to a matching kyverno/policies release branch. When KyvernoAppVersion changes (e.g. v1.18.x → v1.19.x):

1. Re-vendor the policies from the matching release-1.NN branch — follow installer/charts/educates-training-platform/charts/session-manager/files/kyverno-policies/README.md.
2. Repackage the session-manager subchart so the operator ships the refreshed policies:

   make -C installer/operator package-local-charts
   

   (then restore any subchart tarball that only changed by gzip timestamp).
3. Re-check that kyverno.go's RBAC / CRD assumptions and the session-manager validatingpolicies ClusterRole still match the new Kyverno.

Verification

make -C installer/operator verify-vendored-charts   # SHA256 of every on-disk tarball
make -C installer/operator test                     # runs the consistency + load tests

make test includes:

• TestVendoredCharts_DirectoryConsistent — ties the Makefile VENDORED_CHARTS list, SHA256SUMS, embed.go constants, and the on-disk tarballs together; a half-finished bump (or a leftover old tarball) fails here.
• Test<Service>_Embedded — asserts each embedded chart's name/version/appVersion match the embed.go constants.

The operator image //go:embeds these tarballs, so to test on a cluster rebuild the operator (make image-operator) and redeploy. For a real upgrade, also deploy EducatesClusterConfig in Managed mode and confirm the service reaches Ready and its reconciler condition flips (CertificatesReady, IngressReady, DNSReady, PolicyEnforcementReady).

Notes

• Never edit a vendored tarball or unpack-and-repack it — ship the upstream chart unmodified. Configuration is applied through the operator's Helm values (the per-service render<Service>Values in the reconcilers), not by patching the bytes.
• Major-version bumps: read the upstream chart's changelog for renamed values, workload names, or CRD API-version moves, and flag breaking changes.

Post-upgrade checklist

• VENDORED_CHARTS (Makefile), SHA256SUMS, and embed.go all updated to the new version
• New tarball downloaded via make vendor-charts (hash-verified); old tarball git rm'd
• embed.go <X>AppVersion read from the new tarball's Chart.yaml
• git grep "<old-version>" run; straggler hardcoded refs updated (cert-manager: internal/helm/load_test.go)
• Reconciler workload-name / values / CRD assumptions re-verified; constants + comments updated if changed
• Fine-grained install RBAC regenerated (make generate-installer-rbac) + CLI copy synced (make embed-installer-chart); an "unmapped Kind" error means adding the Kind to hack/generate-installer-rbac.sh
• Kyverno: bundled policies re-vendored + session-manager subchart repackaged (if appVersion changed)
• README "Current contents" table updated
• make verify-vendored-charts and make test pass
• Major-version bumps: changelog reviewed for breaking changes and flagged