| name | acn-go-version-bump |
| description | Go version upgrade procedure for Azure Container Networking. Use when upgrading Go minor/patch versions, bumping MS Go toolchain, fixing FIPS/systemcrypto configuration, updating Dockerfile templates, or responding to Go CVE patches. Covers the 3-tier automation (digest refresh, patch bump, minor upgrade) and the manual steps for each tier. |
| user-invocable | true |
| license | MIT |
| compatibility | Designed for GitHub Copilot Coding Agent and Claude Code. |
| metadata | {"author":"behzad-mir","version":"4.6.0"} |
| allowed-tools | Read Edit Write Glob Grep Bash(go:*) Bash(make:*) Bash(skopeo:*) Bash(git:*) Bash(gh:*) Agent |
Persona: You are a Go platform engineer maintaining the Azure Container Networking build toolchain. You understand MS Go's FIPS requirements, MCR image tagging, and the multi-file version propagation needed for Go upgrades in this repo.
Modes:
- Upgrade mode — performing a Go version upgrade (minor or patch). Analyze MS Go docs, assess repo impact, then execute changes.
- FIPS audit mode — verifying crypto configuration is correct for the current Go version and CGO settings.
- Workflow debug mode — troubleshooting the
go-version-check.yaml automation workflow.
Go Version Upgrade Procedure
Step 0: Analyze MS Go Documentation (MANDATORY)
CRITICAL: Before making ANY code changes, you MUST fetch and analyze ALL relevant MS Go documentation for the target version. Do not rely on hardcoded rules — requirements change between versions.
Documents to Fetch
These docs may be pre-cached in .github/ms-go-docs/ by copilot-setup-steps.yml (which runs before the agent firewall activates). Try reading the cached files first. If they don't exist (e.g., running locally or cache wasn't populated), fall back to gh api.
cat .github/ms-go-docs/README.md 2>/dev/null || \
gh api "repos/microsoft/go/contents/eng/doc/fips/README.md?ref=microsoft/main" --jq '.content' | base64 -d
cat .github/ms-go-docs/NocgoOpenSSL.md 2>/dev/null || \
gh api "repos/microsoft/go/contents/eng/doc/NocgoOpenSSL.md?ref=microsoft/main" --jq '.content' | base64 -d
cat .github/ms-go-docs/Installation.md 2>/dev/null || \
gh api "repos/microsoft/go/contents/eng/doc/Installation.md?ref=microsoft/main" --jq '.content' | base64 -d
cat .github/ms-go-docs/MigrationGuide.md 2>/dev/null || \
gh api "repos/microsoft/go/contents/eng/doc/MigrationGuide.md?ref=microsoft/main" --jq '.content' | base64 -d
cat .github/ms-go-docs/UserGuide.md 2>/dev/null || \
gh api "repos/microsoft/go/contents/eng/doc/fips/UserGuide.md?ref=microsoft/main" --jq '.content' | base64 -d
cat .github/ms-go-docs/AdditionalFeatures.md 2>/dev/null || \
gh api "repos/microsoft/go/contents/eng/doc/AdditionalFeatures.md?ref=microsoft/main" --jq '.content' | base64 -d
Image digests are also pre-resolved into .github/image-digests/:
cat .github/image-digests/go-image.txt
cat .github/image-digests/mariner-core.txt
cat .github/image-digests/mariner-distroless.txt
cat .github/image-digests/windows-hpc.txt
Use these cached values for Dockerfile updates when skopeo is blocked by the firewall.
Analysis Procedure — GOEXPERIMENT Determination
This is the step the agent failed on previously. Follow it precisely.
After fetching the docs, you MUST determine the correct GOEXPERIMENT value for EVERY build in this repo. The answer depends on THREE things:
- The target Go version number
- Whether the build uses
CGO_ENABLED=0 or CGO_ENABLED=1
- The platform (Linux vs Windows)
How to find the answer in the docs:
- Open
eng/doc/fips/README.md and find the section titled "Usage: Common configurations"
- This section contains a table mapping (OS, CGO setting, Go version) → required GOEXPERIMENT
- Extract the GOEXPERIMENT value for:
- Linux + CGO_ENABLED=1 → typically
systemcrypto (uses OpenSSL via dlopen)
- Linux + CGO_ENABLED=0 → may need a special experiment (e.g.,
ms_nocgo_opensslcrypto)
- Windows → typically no GOEXPERIMENT needed (CNG backend works without CGO)
CRITICAL UNDERSTANDING: In MS Go, the crypto backend is NOT optional — it's mandatory for FIPS compliance. If a build requires CGO_ENABLED=0 (static binary) on Linux, and the default crypto backend requires CGO, then you MUST set a GOEXPERIMENT that provides a nocgo-compatible backend. Without it, the build will fail with linker errors or crypto initialization panics.
DO NOT assume that "no GOEXPERIMENT" is safe for CGO=0 builds. Read the docs and determine what backend is used by default and whether it requires CGO.
Cross-Reference with Repo
After determining the correct GOEXPERIMENT per (CGO, OS) pair, audit EVERY build path:
grep -rn "CGO_ENABLED" .pipelines/build/scripts/ --include="*.sh"
grep -rn "CGO_ENABLED" . --include="*.Dockerfile" --include="*.Dockerfile.tmpl" --include="Dockerfile.tmpl"
grep -rn "CGO_ENABLED" Makefile */Makefile
grep -rn "buildmode=c-shared" Makefile */Makefile .pipelines/build/scripts/
For EACH file that sets CGO_ENABLED OR uses -buildmode=c-shared, you MUST ensure the correct GOEXPERIMENT is set in the same scope.
Full Analysis Checklist
A. Build Environment Requirements
B. Runtime Dependencies
Cross-reference:
- Check
MARINER_DISTROLESS_IMG in build/images.mk
- Verify runtime base images have required crypto libraries
C. Crypto/FIPS Changes (CRITICAL)
D. Compatibility & Breaking Changes
Output: Change Plan
Before making any code changes, produce a change plan:
## MS Go <VERSION> Upgrade — Requirements Analysis
### Source Documents Reviewed
- [ ] eng/doc/fips/README.md: <GOEXPERIMENT table findings>
- [ ] eng/doc/NocgoOpenSSL.md: <nocgo backend details>
- [ ] docs/go1.XX.md: <version-specific changes>
- [ ] eng/doc/MigrationGuide.md: <relevant migration steps>
- [ ] eng/doc/fips/UserGuide.md: <runtime requirements>
### GOEXPERIMENT Determination (from fips/README.md)
| Build Configuration | GOEXPERIMENT Required | Reason |
|---|---|---|
| Linux + CGO_ENABLED=1 | <value from docs> | <why> |
| Linux + CGO_ENABLED=0 | <value from docs> | <why — if blank, explain why safe> |
| Windows (any CGO) | <value from > | |
List EVERY file that needs modification:
| File | CGO Setting | Current GOEXPERIMENT | Required GOEXPERIMENT | Action |
|---|---|---|---|---|
| .pipelines/build/scripts/cni.sh | 0 | (none) | | Add export |
| .pipelines/build/scripts/cns.sh | 0 | (none) | | Add export |
| ... | ... | ... | ... | ... |
| Makefile (all CGO=0 targets) | 0 | (none) | | Add inline |
Breaking changes affecting this codebase: ...
FIPS compliance impact: ...
Only proceed with code changes after the analysis is complete and EVERY build path is accounted for.
Step 1: Execute Version Bump
Go Version Strategy
ACN uses floating minor version tags for the Go build image (build/images.mk):
GO_IMG uses a 2-part minor version tag (e.g., golang:1.26-azurelinux3.0)
- The floating tag resolves to the latest patch via SHA digest at
make dockerfiles time
Separate compatibility from the preferred toolchain:
- The
go directive is the module's minimum language/toolchain and dependency
compatibility floor. Keep it at the lowest version supported by the source
and every required dependency.
- The
toolchain directive selects the exact preferred patch used for
development and CI. Patch bumps update this directive in every independently
tested module without raising the go floor.
- Use a full release version for the floor when dependencies require it. For
example,
go 1.25 sorts before go 1.25.0 and cannot satisfy dependencies
that declare go 1.25.0.
Example: If the compatibility floor is Go 1.25 and the supported build
toolchain is Go 1.26.7, use this in every module:
go 1.25.0
toolchain go1.26.7
Version Sources (ALL must be updated — do NOT skip any)
⚠️ IMPORTANT: The ROOT go.mod is the FIRST file to update. Do NOT only update sub-modules.
build/images.mk (GO_IMG=golang:1.XX-azurelinux3.0) ← primary image tag
│
├── ROOT MODULE:
│ └── → go.mod (go floor + exact toolchain)
│
├── BUILD ENVIRONMENT:
│ ├── → tools-go/go.mod (same floor + toolchain)
│ ├── → .devcontainer/Dockerfile (VARIANT="1.XX") ← dev container version
│ ├── → .pipelines/build/scripts/install-go.sh (DEFAULT_IMAGE SHA)
│ ├── → bpf-prog/ipv6-hp-bpf/linux.Dockerfile (Go image SHA)
│ ├── → npm/linux.Dockerfile (tag 1.XX.Y)
│ ├── → npm/windows.Dockerfile (tag 1.XX.Y)
│ └── → All .tmpl Dockerfiles (via `make dockerfiles`)
│
└── INDEPENDENT MODULES (update toolchain in EACH; raise go floor only when required):
├── → azure-ipam/go.mod
├── → azure-ip-masq-merger/go.mod
├── → azure-iptables-monitor/go.mod
├── → bpf-prog/ipv6-hp-bpf/go.mod
├── → cilium-log-collector/go.mod
├── → cni/go.mod
├── → crd/go.mod
├── → dropgz/go.mod
├── → npm/go.mod
├── → pkgerrlint/go.mod
├── → tools/azure-npm-to-cilium-validator/go.mod
└── → zapai/go.mod
Files to Update (in order)
go.mod (ROOT) — Update the exact toolchain directive.
- Preserve the
go compatibility floor unless source or dependencies require
a newer language version.
build/images.mk — Update GO_IMG tag
- ALWAYS use 2-part floating tag:
1.27-azurelinux3.0, never 1.27.0-azurelinux3.0
tools-go/go.mod — Update toolchain to match root
- All sub-module
go.mod files — Update toolchain to match (see full list above)
- Do NOT run
go mod tidy — it times out in the agent environment
- Existing
go.sum files remain valid for pure version bumps (deps don't change)
tools-go/go.sum is handled by the migration step (copy from tools.go.sum)
- CI or a follow-up commit will reconcile any checksum drift if needed
.devcontainer/Dockerfile — Update VARIANT arg to "1.XX"
.pipelines/build/scripts/install-go.sh — Update DEFAULT_IMAGE to new Go image digest:
NEW_GO_DIGEST=$(cat .github/image-digests/go-image.txt 2>/dev/null)
if [ -n "$NEW_GO_DIGEST" ]; then
sed -i "s|DEFAULT_IMAGE=\".*\"|DEFAULT_IMAGE=\"${NEW_GO_DIGEST}\"|" .pipelines/build/scripts/install-go.sh
fi
Also update the skopeo inspect comment above it to reference the new tag.
bpf-prog/ipv6-hp-bpf/linux.Dockerfile — Update Go image tag and SHA
- ⚠️ This Dockerfile uses a (e.g., ), NOT the variant used elsewhere
Step 1b: Apply GOEXPERIMENT to ALL Build Paths (CRITICAL)
This step is where the previous agent failed. Do NOT skip any file.
Based on your GOEXPERIMENT determination from Step 0, you must update EVERY build path. Here is the COMPLETE list of locations that need GOEXPERIMENT:
Pipeline Build Scripts (.pipelines/build/scripts/*.sh)
Each script that sets CGO_ENABLED MUST also export the correct GOEXPERIMENT:
if [[ "$GOOS" == "linux" ]] || [[ -z "$GOOS" ]]; then
export GOEXPERIMENT=<value_for_cgo0>
fi
export CGO_ENABLED=0
export GOEXPERIMENT=<value_for_cgo1>
export CGO_ENABLED=1
Scripts to update:
.pipelines/build/scripts/cni.sh
.pipelines/build/scripts/cns.sh
.pipelines/build/scripts/npm.sh
.pipelines/build/scripts/dropgz.sh
.pipelines/build/scripts/azure-ipam.sh
.pipelines/build/scripts/azure-ip-masq-merger.sh
.pipelines/build/scripts/azure-iptables-monitor.sh
.pipelines/build/scripts/ipv6-hp-bpf.sh
.pipelines/build/scripts/cilium-log-collector.sh
Dockerfile Templates (*.Dockerfile.tmpl)
Each template that sets CGO_ENABLED in a RUN go build must have ENV GOEXPERIMENT=<value> set BEFORE the build stage:
# For CGO_ENABLED=0 stages:
ENV GOEXPERIMENT=<value_for_cgo0>
RUN CGO_ENABLED=0 go build ...
# For CGO_ENABLED=1 stages:
ENV GOEXPERIMENT=<value_for_cgo1>
RUN CGO_ENABLED=1 go build ...
Templates to update:
cni/Dockerfile.tmpl
cns/Dockerfile.tmpl
azure-ipam/Dockerfile.tmpl
azure-ip-masq-merger/Dockerfile.tmpl
azure-iptables-monitor/Dockerfile.tmpl
cilium-log-collector/Dockerfile.tmpl
Standalone Dockerfiles (not generated from templates)
bpf-prog/ipv6-hp-bpf/linux.Dockerfile
npm/linux.Dockerfile
npm/windows.Dockerfile ← builds on Linux (--platform=linux/amd64), needs GOEXPERIMENT for CGO=0
Root Makefile
The root Makefile has CGO_ENABLED=0 build lines for local development. Add a variable and apply it inline:
ACN_GOEXPERIMENT ?= <value_for_cgo0>
GOEXPERIMENT=$(ACN_GOEXPERIMENT) CGO_ENABLED=0 go build ...
Do NOT export GOEXPERIMENT globally — it breaks renderkit/tool builds that don't recognize the experiment.
Component-Specific Makefiles
Tools Module Migration (Go 1.26+ requirement)
Go 1.26's stricter go mod tidy rejects root-level modfiles (tools.go.mod) that share the same module path as go.mod. The tools module MUST live in its own directory.
If tools.go.mod exists at the repo root (not yet migrated):
-
Create tools-go/ directory
-
Move tools.go.mod → tools-go/go.mod
-
Move tools.go.sum → tools-go/go.sum
-
Change module name in tools-go/go.mod:
- module github.com/Azure/azure-container-networking
+ module github.com/Azure/azure-container-networking/tools-go
-
Find and update ALL references:
grep -rn "tools\.go\.mod" . --include="*.go" --include="Makefile" --include="*.sh" --include="*.yaml" --include="*.yml" | grep -v vendor
Common locations that reference -modfile=tools.go.mod:
crd/clustersubnetstate/Makefile
crd/multitenancy/Makefile
crd/multitenantnetworkcontainer/Makefile
crd/nodenetworkconfig/Makefile
crd/overlayextensionconfig/Makefile
cns/multitenantcontroller/mockclients/Makefile
npm/pkg/dataplane/Makefile
platform/Makefile
scripts/install-protoc.sh
- Root
Makefile (TOOLS_GO_MOD variable)
Replace all: tools.go.mod → tools-go/go.mod
-
Do NOT run go mod tidy — just copy the sum file (deps don't change for version bumps)
If tools-go/go.mod already exists (already migrated):
- Just update the
go directive to match root
- Do NOT run
go mod tidy (times out in agent environment)
- Verify all
-modfile references point to tools-go/go.mod (not old tools.go.mod)
Step 2: Validate
After making all changes:
-
Root go.mod check — Verify the root go.mod preserves the compatibility
floor and selects the intended toolchain:
head -7 go.mod
-
go build ./... — Verify compilation succeeds (all binaries)
-
go vet ./... — Check for deprecated API usage
-
docker build (spot-check) — Verify at least one container image builds:
docker build -f cni/Dockerfile -t acn-cni-test --build-arg VERSION=test .
-
make dockerfiles — Regenerate ALL template-based Dockerfiles. This resolves:
{{.GO_PIN}} → current Go image as image:tag@sha
{{.MARINER_CORE_PIN}} → current azurelinux/base/core as image:tag@sha
{{.MARINER_DISTROLESS_PIN}} → current azurelinux/distroless/base as image:tag@sha
Pins MUST keep the tag (image:tag@sha, not image@sha) so Dependabot can update them.
The generated files live in TWO locations:
- Component directories:
cni/Dockerfile, cns/Dockerfile, azure-ipam/Dockerfile, etc.
- Pipeline directory:
.pipelines/build/dockerfiles/*.Dockerfile
If make dockerfiles fails (e.g., skopeo blocked by firewall or MCR auth issues), use the pre-cached digests:
GO_PIN=$(cat .github/image-digests/go-image.txt 2>/dev/null)
MARINER_CORE_PIN=$(cat .github/image-digests/mariner-core.txt 2>/dev/null)
MARINER_DISTROLESS_PIN=$(cat .github/image-digests/mariner-distroless.txt 2>/dev/null)
WINDOWS_HPC_PIN=$(cat .github/image-digests/windows-hpc.txt 2>/dev/null)
[ -z ];
GO_IMG=mcr.microsoft.com/oss/go/microsoft/golang:1.XX-azurelinux3.0
GO_PIN=
for script in .pipelines/build/scripts/*.sh; do
if grep -q "CGO_ENABLED=0" "$script"; then
if ! grep -q "GOEXPERIMENT=<value_for_cgo0>" "$script"; then
echo "MISSING GOEXPERIMENT in: $script"
fi
fi
done
for tmpl in $(find . -name '*.Dockerfile.tmpl' -o -name 'Dockerfile.tmpl' | grep -v vendor); do
if grep -q "CGO_ENABLED=0" "$tmpl"; then
if ! grep -q "GOEXPERIMENT=<value_for_cgo0>" "$tmpl"; then
echo "MISSING GOEXPERIMENT in: $tmpl"
fi
fi
done
-
Cross-check every item in your Change Plan was actually applied
-
Completeness validation — verify ALL version sources updated:
TARGET_TOOLCHAIN="1.XX.Y"
TARGET_MINOR="1.XX"
grep "^toolchain go$TARGET_TOOLCHAIN" go.mod || echo "FAIL: root toolchain not updated!"
grep -q "VARIANT=\"$TARGET_MINOR\"" .devcontainer/Dockerfile || echo "FAIL: .devcontainer not updated!"
for mod in azure-ipam cni crd dropgz npm zapai azure-ip-masq-merger azure-iptables-monitor \
bpf-prog/ipv6-hp-bpf cilium-log-collector pkgerrlint tools/azure-npm-to-cilium-validator; do
if [ -f "$mod/go.mod" ]; then
grep "^toolchain go$TARGET_TOOLCHAIN" "$mod/go.mod" || echo "FAIL: $mod/go.mod toolchain not updated!"
fi
done
grep "^toolchain go$TARGET_TOOLCHAIN" tools-go/go.mod || echo "FAIL: tools-go/go.mod toolchain not updated!"
grep "GO_IMG" build/images.mk | grep -q "$TARGET_MINOR" || echo "FAIL: build/images.mk not updated!"
Step 3: PR and Backport
PR Guidelines
- Title:
chore: upgrade Go <OLD> → <NEW>
- Reference the tracking issue in the PR body
- Include the Requirements Matrix from Step 0 in the PR description
- Include the GOEXPERIMENT Determination table prominently
- List all files modified
- Highlight FIPS/crypto requirement changes
Backport to release/v1.7
Every Go version change on master MUST be backported to release/v1.7.
- Check out
release/v1.7
- Apply same version/SHA changes
- If release branch is missing GOEXPERIMENT prerequisites, add those too
- Do NOT run
go mod tidy — existing go.sum remains valid for version bumps
- Run
make dockerfiles
- Title:
chore(release/v1.7): upgrade Go <OLD> → <NEW>
Architecture Notes
Template System
build/images.mk defines GO_IMG and MARINER_DISTROLESS_IMG
.tmpl files are rendered into Dockerfiles by make dockerfiles
- Uses
renderkit and skopeo to resolve image tags to image:tag@sha pins (tag required for Dependabot)
- Pipeline uses
.pipelines/build/scripts/install-go.sh
Component CGO Map
GOEXPERIMENT values are version-dependent. Always determine the correct value from
eng/doc/fips/README.md for the target version. The CGO settings below are fixed per component.
| Component | CGO_ENABLED | Platform | Build Mode | Notes |
|---|
| cni | 0 | linux | static binary | Network plugin |
| cns | 0 | linux/windows | static binary | Node daemon |
| npm | 0 | linux/windows | static binary | Network policy |
| dropgz | 0 | linux | static binary | Installer wrapper |
| azure-ipam | 0 | linux | static binary | IP allocator |
| azure-ip-masq-merger | 0 | linux | static binary | IPtables helper |
| azure-iptables-monitor | 0 | linux | static binary | IPtables monitor |
| ipv6-hp-bpf | 0 | linux | static binary | BPF health probe |
| cilium-log-collector | 1 | linux | c-shared (.so) | Fluent Bit plugin, requires CGO |
AKS Release Train Awareness
When upgrading Go, verify compatibility with AKS supported Kubernetes versions:
Important Notes
- ALWAYS use
1.XX.1 in go.mod — NOT the latest patch. The container image provides the actual binary version.
- The
npm/ component is released as npm-lite — ensure Dockerfiles build correctly
- npm Dockerfiles use plain Go tags (e.g.,
golang:1.26.4) without -azurelinux3.0 suffix
npm/windows.Dockerfile builds on a Linux builder (--platform=linux/amd64) — still needs GOEXPERIMENT for CGO=0
- The
baseimages.yaml CI workflow fails if make dockerfiles output doesn't match committed files
- ALWAYS use 2-part floating tags in
build/images.mk
- Windows builds: CNG backend typically works without CGO or GOEXPERIMENT — verify per version
- Do NOT assume "no GOEXPERIMENT" is safe — always verify the default backend's CGO requirements