multi-cluster-api-data-mismatch

name	multi-cluster-api-data-mismatch
description	Debug "API returns data that doesn't exist in database" when multiple Kubernetes clusters exist (production, staging, POC). Use when: (1) REST API returns records that direct database queries can't find, (2) Data counts match approximately but specific records don't overlap, (3) kubectl context points to a different cluster than the one serving the public domain, (4) ClickHouse/Postgres queries return stale or different data than the API, (5) "completely different datasets" despite same table names and schemas. Root cause: querying the wrong cluster's database.
author	Claude Code
version	1.0.0
date	"2026-03-02T00:00:00.000Z"

Multi-Cluster API Data Mismatch

Problem

When investigating why an API returns data that doesn't match direct database queries, the root cause may be that your kubectl context points to a different cluster (e.g., POC) than the one actually serving the public API (e.g., production). Both clusters have the same schemas and similar data volumes, making the mismatch non-obvious.

Context / Trigger Conditions

• API returns records that direct database queries (kubectl exec ... clickhouse-client) can't find
• Data counts are similar (e.g., both have ~20k records) but specific records differ
• You're debugging a data pipeline that "should work" but labels/counts never update
• Multiple kubectl contexts exist (production, staging, POC, test)
• The API response header shows server: nginx but no ingress exists in the current cluster's namespace

Solution

Step 1: Verify which cluster serves the domain

# Check DNS for the public API domain
dig relay.divine.video +short
# → 34.58.27.79

# Check your current cluster's gateway/ingress IP
kubectl get gateway -A  # or kubectl get ingress -A
# → 35.184.10.63  (different IP = different cluster!)

Step 2: Check all available contexts

kubectl config get-contexts
# Look for production vs staging vs POC contexts

Step 3: Switch to the correct cluster

kubectl config use-context <production-context-name>

Step 4: Verify the database connection

# Check what database the API actually connects to
kubectl get secret -n <namespace> <db-credentials> -o jsonpath='{.data.DATABASE_URL}' | base64 -d
# Production might use managed services (ClickHouse Cloud, Cloud SQL)
# POC might use in-cluster pods

Step 5: Also check for ExternalSecrets when updating secrets

If secrets are managed by ExternalSecrets operator:

# Check if ExternalSecrets manages the secret
kubectl get externalsecret -n <namespace>

# If yes, update the SOURCE (e.g., GCP Secret Manager), not the k8s secret
gcloud secrets versions add <secret-name> --project=<project> --data-file=-

# Force sync after updating
kubectl annotate externalsecret <name> -n <namespace> force-sync=$(date +%s) --overwrite

# Verify the k8s secret updated
kubectl get secret <name> -n <namespace> -o jsonpath='{.data.<key>}' | base64 -d

Verification

After switching to the correct cluster:

1. Re-run the same database query — the "missing" records should now appear
2. The latest timestamp in the database should match recent API activity
3. kubectl get pods -n <namespace> should show the same pods the API logs reference

Key Indicators You're On the Wrong Cluster

Symptom	Explanation
API video count ~21k, DB count ~20k	Similar but not identical = different datasets
Latest DB record is days old	API is receiving writes on a different cluster
No ingress/gateway routes match the public domain	Traffic routes elsewhere
Managed DB URL (e.g., ClickHouse Cloud) vs in-cluster pod	Production vs POC architecture
API pod logs show only health checks, no real traffic	Real traffic goes to another cluster

Example

# You query POC ClickHouse and find 20,371 videos, latest from Feb 26
# But the API at relay.divine.video returns videos from today
# The API returns a video with d_tag that doesn't exist in your ClickHouse

# Fix: switch to production
kubectl config use-context connectgateway_dv-platform-prod_...

# Now query production ClickHouse Cloud
CH_URL=$(kubectl get secret -n funnelcake funnelcake-clickhouse-credentials \
  -o jsonpath='{.data.CLICKHOUSE_URL}' | base64 -d)
# → https://z1hzyismdt.us-central1.gcp.clickhouse.cloud:8443
# (managed service, not in-cluster pod)

# Query returns 21,221 videos with latest from minutes ago — matches the API

Notes

• Production often uses managed database services (ClickHouse Cloud, Cloud SQL) while POC/staging use in-cluster pods — same schemas, different data
• The similar-but-not-identical record counts are the most misleading symptom
• Always check max(created_at) or max(indexed_at) — if it's days old on a live service, you're querying the wrong instance
• When updating k8s secrets in production, check for ExternalSecrets first — manual kubectl create secret will be immediately overwritten by the operator syncing from GCP Secret Manager / AWS Secrets Manager / Vault