| name | microservice-api-endpoint-routing-404 |
| description | Debug 404 errors for API endpoints in microservice architectures where code exists but
routing sends requests to wrong service. Use when: (1) API endpoint returns 404 despite
code being deployed, (2) Multiple services share same hostname via HTTPRoute/Ingress,
(3) Service logs show "Health server" but you expected API endpoints, (4) Monorepo
deploys multiple images and endpoint might be in different service than expected.
Covers: verifying which service handles which paths, HTTPRoute path precedence,
testing endpoints directly from inside cluster, matching image tags across services.
|
| author | Claude Code |
| version | 1.0.0 |
| date | "2026-01-29T00:00:00.000Z" |
Microservice API Endpoint Routing 404 Debugging
Problem
API endpoint returns 404 even though the code implementing it has been deployed. This
commonly happens in microservice architectures where multiple services share the same
hostname and HTTPRoutes/Ingress rules determine which service handles which paths.
Context / Trigger Conditions
- New endpoint returns 404 but code is definitely deployed
- Multiple services (e.g., relay + api) share the same external hostname
- Service logs show "Health server listening" or similar health-only messages
- HTTPRoute or Ingress rules split traffic by path prefix
- Monorepo builds multiple Docker images from same commit
Solution
Step 1: Identify which service actually handles the path
Check HTTPRoute/Ingress rules to see routing:
kubectl get httproute <name> -n <namespace> -o yaml | grep -A 50 "rules:"
Look for path matching rules and which backend service they point to:
/api/* might go to api-service:8080/ might go to websocket-service:7777
Step 2: Test the endpoint directly on each service from inside cluster
kubectl run curl-test -n <namespace> --image=curlimages/curl --restart=Never \
-- curl -v "http://<service-name>:<port>/<endpoint>" \
&& sleep 5 \
&& kubectl logs -n <namespace> curl-test \
&& kubectl delete pod -n <namespace> curl-test
This bypasses external routing and tests directly against each service.
Step 3: Check service logs for endpoint registration
kubectl logs -l app=<service-name> -n <namespace> --tail=50 | grep -i -E "(endpoint|route|api|listening)"
Look for messages like:
- "Health server listening on 0.0.0.0:8080" = health only, no API
- "API server listening" or "Registered /api/..." = serves API endpoints
Step 4: For monorepos, check if matching image exists for correct service
If feature was added to service A but endpoint is served by service B, check if
service B has the same commit tag:
gcloud artifacts docker images list <registry>/<service-b> --include-tags --limit=10
Step 5: Deploy the correct service with feature + required env vars
Update the service that actually handles the endpoint path:
- New image tag with the feature
- Any environment variables (API keys, URLs) the feature needs
- Any secrets referenced by the feature
Verification
- Service logs show endpoint registered: "Recommendations API enabled at /api/users/:pubkey/recommendations"
- Direct curl from inside cluster returns 200
- External request through HTTPRoute returns expected data
Example
Architecture:
Hostname: relay.example.com
/api/* → funnelcake-api:8080 (REST API)
/ → funnelcake-relay:7777 (WebSocket)
Problem: /api/users/x/recommendations returns 404
Investigation:
- HTTPRoute shows
/api/* → funnelcake-api (not relay)
- Direct test to relay:8080 returns 404 (health server only)
- Direct test to api:8080 also returns 404 (old image)
- Registry shows both api and relay have image tag
3c0696a
- API was running old tag
487178e
Fix: Deploy funnelcake-api with tag 3c0696a + GORSE_URL + GORSE_API_KEY env vars
Notes
- Services in same namespace can share Kubernetes Secrets
- Health endpoints (/livez, /readyz) are often on separate port from API
- HTTPRoute path matching: more specific paths take precedence
- Always check which service handles which paths before assuming endpoint location
