agentgateway-enterprise

name

agentgateway-enterprise

description

Deploy and configure Solo Enterprise for AgentGateway (v2026.6.x) on Kubernetes clusters. Use this skill whenever the user mentions agentgateway, solo.io gateway, AI gateway, LLM gateway, LLM proxy, MCP gateway, or enterprise gateway. Covers both direct Helm install and ArgoCD GitOps deployment, including Solo UI setup, LLM backend configuration (OpenAI, Anthropic, Azure, Bedrock), HTTPRoutes, policies (tracing, auth, rate limiting), Gateway API resources, and troubleshooting. Also use when upgrading from earlier agentgateway versions or adding new LLM backends to an existing deployment.

Solo Enterprise for AgentGateway (v2026.6.x)

Kubernetes-native API gateway for AI workloads built on the Gateway API. Routes to LLM providers, MCP servers, and HTTP backends with auth, rate limiting, observability, and tracing.

Quick Reference

Component	Chart	Version	OCI Registry
Gateway API CRDs	upstream YAML	v1.5.0	github.com/kubernetes-sigs/gateway-api
AgentGateway CRDs	enterprise-agentgateway-crds	v2026.6.3	us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts
Control Plane	enterprise-agentgateway	v2026.6.3	us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts
Solo UI	management	0.4.6	us-docker.pkg.dev/solo-public/solo-enterprise-helm/charts

Resource	API Group	Purpose
Gateway	gateway.networking.k8s.io/v1	Proxy instance with listeners
HTTPRoute	gateway.networking.k8s.io/v1	Route rules mapping paths to backends
AgentgatewayBackend	agentgateway.dev/v1alpha1	LLM/MCP/HTTP backend definition
EnterpriseAgentgatewayPolicy	enterpriseagentgateway.solo.io/v1alpha1	Policies (tracing, auth, rate limiting)

All resources go in agentgateway-system namespace by default.

Prerequisites

• Kubernetes cluster (Kind, EKS, GKE, AKS, Talos)
• kubectl (within one minor version of cluster)
• helm 3.x
• License key: AGENTGATEWAY_LICENSE_KEY env var

Installation Path A: Direct Helm

1. Gateway API CRDs

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.0/standard-install.yaml

2. AgentGateway CRDs

helm upgrade -i --create-namespace \
  --namespace agentgateway-system \
  --version v2026.6.3 enterprise-agentgateway-crds \
  oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway-crds

3. Control Plane

helm upgrade -i -n agentgateway-system enterprise-agentgateway \
  oci://us-docker.pkg.dev/solo-public/enterprise-agentgateway/charts/enterprise-agentgateway \
  --version v2026.6.3 \
  --set-string licensing.licenseKey=${AGENTGATEWAY_LICENSE_KEY}

4. Solo UI (requires license key too)

helm upgrade -i management \
  oci://us-docker.pkg.dev/solo-public/solo-enterprise-helm/charts/management \
  --namespace agentgateway-system \
  --create-namespace \
  --version 0.4.6 \
  --set cluster="mgmt-cluster" \
  --set products.agentgateway.enabled=true \
  --set-string licensing.licenseKey=${AGENTGATEWAY_LICENSE_KEY}

5. Verify

kubectl get pods -n agentgateway-system
kubectl get gatewayclass enterprise-agentgateway

Installation Path B: ArgoCD GitOps

Uses app-of-apps pattern with sync-wave ordering. See references/argocd-gitops.md for the full repo structure and all ArgoCD Application manifests.

ArgoCD Setup

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.12.3/manifests/install.yaml

Connect Private Repo (use Secret, not CLI — the CLI often has gRPC timeout issues)

GH_TOKEN=$(gh auth token)
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: gitops-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
  type: git
  url: https://github.com/<org>/<repo>.git
  username: <github-user>
  password: ${GH_TOKEN}
EOF

Deploy via App-of-Apps

kubectl apply -f argocd/app-of-apps.yaml
kubectl get applications -n argocd    # Watch sync status

ArgoCD Application Structure (sync-wave ordered)

Wave 1: gateway-api-crds          (Kustomize → upstream CRDs)
Wave 2: agentgateway-crds         (Helm → enterprise-agentgateway-crds v2026.6.3)
Wave 3: agentgateway-control-plane (Helm → enterprise-agentgateway v2026.6.3 + license)
Wave 4: vault                     (Helm → HashiCorp Vault, dev mode)
Wave 5: external-secrets          (Helm → External Secrets Operator)
Wave 6: solo-ui                   (Helm → management v0.4.6 + license)
Wave 6: langfuse                  (Helm → langfuse-k8s, self-hosted observability)
Wave 7: agentgateway-config       (Plain YAML → gateway, backends, routes, policies)

GitOps Repo Structure

agentgateway-gitops/
├── argocd/
│   ├── app-of-apps.yaml
│   └── apps/                    # One ArgoCD Application per wave
├── config/
│   ├── gateway/                 # Gateway resources
│   ├── backends/                # AgentgatewayBackend per provider
│   ├── routes/                  # HTTPRoute per backend
│   ├── policies/                # EnterpriseAgentgatewayPolicy
│   └── secrets/                 # API key secrets (use SealedSecrets in prod)
├── platform/gateway-api-crds/   # Kustomize ref
└── scripts/

Gateway Setup

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agentgateway-proxy
  namespace: agentgateway-system
spec:
  gatewayClassName: enterprise-agentgateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All

LLM Backend Pattern (repeat per provider)

Secret

apiVersion: v1
kind: Secret
metadata:
  name: <provider>-secret
  namespace: agentgateway-system
type: Opaque
stringData:
  Authorization: <api-key>

AgentgatewayBackend

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: <provider>
  namespace: agentgateway-system
spec:
  ai:
    provider:
      openai:           # or: anthropic, azure, bedrock
        model: gpt-4o   # provider-specific model name
  policies:
    auth:
      secretRef:
        name: <provider>-secret

HTTPRoute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: <provider>
  namespace: agentgateway-system
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: agentgateway-system
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /<provider>
      backendRefs:
        - name: <provider>
          namespace: agentgateway-system
          group: agentgateway.dev
          kind: AgentgatewayBackend

Test

kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 8080:80 &
curl "localhost:8080/<provider>/v1/chat/completions" -H content-type:application/json \
  -d '{"model":"","messages":[{"role":"user","content":"Hello!"}]}' | jq

Tracing Policy (requires Solo UI running)

apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayPolicy
metadata:
  name: tracing
  namespace: agentgateway-system
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: agentgateway-proxy
  frontend:
    tracing:
      backendRef:
        name: solo-enterprise-telemetry-collector
        namespace: agentgateway-system
        kind: Service
        port: 4317
      randomSampling: "true"

Solo UI Access

kubectl port-forward svc/solo-enterprise-ui -n agentgateway-system 4000:80
# Open http://localhost:4000/age/

Upgrading

Update chart versions in helm commands or ArgoCD Application targetRevision fields:

# Helm
helm upgrade -i ... --version v<new-version> ...

# ArgoCD GitOps — edit targetRevision in apps/, commit, push

Solo UI (management chart) upgrade gotchas

Bumping the management/solo-ui chart runs a ClickHouse schema migration on ui-backend startup. Two failure modes seen on real upgrades:

• Dirty ClickHouse migration — ui-backend CrashLoops with error applying ClickHouse migrations: ... Dirty database version <N>. Fix and force version. The migration's DDL ran but the startup probe killed the pod before it recorded completion. Verify the migration's objects actually exist (SELECT name, metadata_modification_time FROM system.tables WHERE database='platformdb' ORDER BY metadata_modification_time DESC), then force the marker clean by inserting a dirty=0 row with the highest sequence (equivalent to migrate force <N>):

  kubectl exec -n agentgateway-system <clickhouse-pod> -c clickhouse -- clickhouse-client -q \
    "INSERT INTO platformdb.schema_migrations (version,dirty,sequence) \
     SELECT toInt64(<N>),toUInt8(0),toUInt64(max(sequence)+1) FROM platformdb.schema_migrations"
  

  Only force forward when the DDL is confirmed applied; otherwise force to the prior clean version and let it re-run.
• Rollout wedge — the solo-enterprise-ui Deployment uses strategy: Recreate and ArgoCD selfHeal: true. A kubectl rollout restart adds a pod-template annotation that ArgoCD reverts mid-rollout, so the old RS scales to 0 but no new RS is created (0 pods). Recover with kubectl delete rs <newest-solo-enterprise-ui-rs> — the controller recreates and scales a fresh RS.

Troubleshooting

Symptom	Cause	Fix
Pods stuck ContainerCreating	Stale istio-cni in CNI chain	Deploy privileged DaemonSet with hostNetwork:true to rewrite /etc/cni/net.d/10-flannel.conflist removing istio-cni plugin, then restart pods
Solo UI helm template fails	Missing license key	Add --set-string licensing.licenseKey=... to UI chart too
ArgoCD CLI gRPC timeout	Port-forward instability (common on Talos)	Use kubectl Secret to add repo instead of argocd repo add CLI
HTTPRoute shows OutOfSync in ArgoCD	Gateway controller adds status fields	Normal — resource is healthy, ArgoCD detects server-side diff
ArgoCD pods fail after CNI fix	Old pods have stale network namespace	kubectl rollout restart deploy -n argocd && kubectl rollout restart statefulset -n argocd
GatewayClass not found	CRDs not installed yet	Ensure Gateway API CRDs + AgentGateway CRDs install before control plane
ui-backend CrashLoops after UI upgrade	Dirty ClickHouse migration (DDL applied, completion marker not written)	Force the migration marker clean — see "Solo UI upgrade gotchas" above
solo-ui Deployment stuck at 0 pods after rollout restart	Recreate strategy + ArgoCD selfHeal reverted the restart annotation mid-rollout	kubectl delete rs <newest-solo-enterprise-ui-rs> to force recreate
solo-ui ArgoCD app stuck Progressing on Kind	LoadBalancer Services never get an external IP without a cloud LB	Benign — the workload is Healthy; ignore or use MetalLB/NodePort

Cleanup

# Config
kubectl delete AgentgatewayBackend,HTTPRoute,EnterpriseAgentgatewayPolicy --all -n agentgateway-system
kubectl delete gateway agentgateway-proxy -n agentgateway-system

# Helm charts (reverse order)
helm uninstall management -n agentgateway-system
helm uninstall enterprise-agentgateway -n agentgateway-system
helm uninstall enterprise-agentgateway-crds -n agentgateway-system

# ArgoCD (if used)
kubectl delete -f argocd/app-of-apps.yaml