-
Build a diagnostic probe that reads each surface in its native format, separately, then compares them. Don't trust a single surface's self-report. Example: an MCP tool layer shows conclusions as recent and coherent, but a direct SSH query against the backend Postgres database reveals the processing queue is actually stuck. Compare concrete, comparable signals across both surfaces — e.g., latest-derivation timestamp on each side, or error count vs. throughput — to surface gaps neither surface exposes alone (a silent backlog the API side never reports).
-
When a repo has two competing sources of truth for "what's open" (a formal docs/features → docs/plans → docs/tasks/active framework, and a living docs/tasks/backlog.md), audit both but explicitly choose which one is authoritative for the question being asked. The formal framework fits greenfield initiatives with a defined completion state; the backlog fits living infrastructure repos where work is perpetual and "% complete" is the wrong metric because nearly everything eventually ships. Report status by priority (HIGH/MEDIUM/LOW) and theme, not by feature-completion percentage, when the backlog is authoritative. Call out the 2-3 HIGH items separately as immediate operational risk; group MEDIUM items by concern area (observability, security, cluster, etc.) to aid prioritization.
-
When a service appears offline, separate the host-level health signal from real-time application state before declaring it down. A docker ps "unhealthy" status can be stale (frozen during a resource-starvation wedge) rather than reflecting current reality. Confirm true failure by: tailing the process log for fresh activity (not a frozen healthcheck), confirming the daemon actually restarts, and watching live metrics (memory, load, io.pressure) recover once the starvation condition ends. Log "stale healthcheck frozen during a wedge" explicitly as a known footgun so it's recognized faster next time.
-
When a metrics/coverage gap spans two systems (e.g., an API poller and a log-scraper), prefer deriving the signal from whichever system already collects cleaner, closer-to-source data over standing up new infrastructure. Example: if an exporter already polls coverage every 15 minutes, derive throughput as increase(metric_name[1h]) from that existing series rather than adding a separate log-scraper. Point alert rules and dashboard panels at the chosen surface once decided. Validate idempotency of any backfill/reconciliation job (e.g., a second pass reports changed=0) before calling the work done.
