| name | post-deployment-monitor |
| description | Monitor Kubernetes deployments post-release by analyzing code changes and checking metrics. Use when users ask to "monitor deployment", "check deployment health", "verify deployment", "analyze deployment impact", or provide a deployment name with namespace and cluster. Automates deployment image comparison, code diff analysis, repository skill loading, flow impact analysis, and Grafana metrics validation. Generates comprehensive health reports with risk assessment. |
Post-Deployment Monitoring
Automated workflow for monitoring Kubernetes deployments after updates by analyzing code changes and checking observability metrics.
Purpose: Validate deployment health by correlating code changes with production metrics to detect regressions, errors, and performance degradation.
⚠️ CRITICAL: SEQUENTIAL EXECUTION REQUIRED
ALL phases MUST run one at a time, in strict order. NEVER parallelize across phases.
Phase 1: Fetch deployment info (kubectl via remote-friday-mcp-server)
↓ [COMPLETE before starting Phase 2]
Phase 2: Get code diff + read changed files
↓ [COMPLETE before starting Phase 3]
Phase 3: Load repo skills → identify affected flows + endpoints
↓ [COMPLETE before starting Phase 4]
Phase 4: Build Grafana queries (write them out explicitly)
↓ [COMPLETE before starting Phase 5]
Phase 5: Execute Grafana queries → generate health report
Combining any two phases in a single step is not allowed.
Quick Start
User provides:
- Deployment name (e.g.,
payments-card-live)
- Namespace (e.g.,
payments-card)
- Cluster (e.g.,
cde-green)
Phase 1: Fetch Deployment Info from Kubernetes
Tool: remote-friday-mcp-server::kubectl_execute — ALWAYS use this, never gh or Bash for K8s.
1.1: List ReplicaSets to find current and previous images
Use jsonpath output (custom-columns fails on annotation paths):
command: get
options: rs -n <namespace> -l name=<deployment-name> -o jsonpath='{range .items[*]}{.metadata.annotations.deployment\.kubernetes\.io/revision}{"\t"}{.spec.replicas}{"\t"}{.spec.template.spec.containers[0].image}{"\t"}{.metadata.creationTimestamp}{"\n"}{end}'
context: <cluster>-eks
Extract:
- Current: Highest revision number with
replicas > 0
- Previous: Second-highest revision (regardless of replica count)
- Deployment time:
creationTimestamp of the current ReplicaSet
- Commit SHAs: Parse from image tag — e.g.,
c.rzp.io/razorpay/offers-engine:api_2da394ff... → 2da394ff...
Example Output:
12 0 c.rzp.io/razorpay/offers-engine:api_7e9891499298... 2026-04-05T10:12:00Z
13 3 c.rzp.io/razorpay/offers-engine:api_2da394ff5437... 2026-04-06T18:26:36Z
Result:
- Current SHA:
2da394ff5437... (deployed at 2026-04-06T18:26:36Z)
- Previous SHA:
7e9891499298...
1.2: Fallback if jsonpath fails
command: get
options: rs -n <namespace> -l name=<deployment-name> -o wide
context: <cluster>-eks
Then narrow down with:
command: describe
options: rs <replicaset-name> -n <namespace>
context: <cluster>-eks
⚠️ Common Mistakes:
- ❌ Using
deployment.kubernetes.io/change-cause annotation (often stale)
- ❌ Reading image from the Deployment object (may not reflect live pods)
- ✅ Always use ReplicaSet revision history sorted by revision number
Phase 2: Get Code Diff
Tool: Bash with gh api — use GitHub compare endpoint to get commits and changed files.
gh api repos/razorpay/<repo-name>/compare/<previous-sha>...<current-sha> \
--jq '{
total_commits: .total_commits,
commits: [.commits[] | {
sha: .sha[0:7],
message: (.commit.message | split("\n")[0]),
author: .commit.author.name
}],
files_changed: (.files | length),
files: [.files[] | .filename],
stats: {
additions: ([.files[].additions] | add),
deletions: ([.files[].deletions] | add)
}
}'
Then read each changed file using Read tool (if repo is in workspace) or gh api to get file content for analysis.
Assess risk:
- 1–10 commits → 🟢 Low
- 11–50 commits → 🟡 Medium
- 51–100 commits → 🔴 High
- 100+ commits → 🔴 Very High
⚠️ Analyze ALL commits, not just the latest.
Phase 3: Load Repo Skills → Identify Affected Flows and Endpoints
Complete Phase 2 fully before starting this phase.
3.1: Discover skills
If repo is in the current workspace:
Glob: .claude/skills/**/*.md
If not in workspace, use gh api:
gh api repos/razorpay/<repo-name>/contents/.claude/skills/<service>/modules/technical/observability/metrics.md \
--jq '.content' | base64 -d
3.2: Load in priority order
-
Observability docs — modules/technical/observability/metrics.md
- Extract: metric namespace, metric names, label dimensions, datasource UID
-
Domain flow docs — modules/domain/*/flows.md for each affected entity
- Cross-reference changed files against flows
- Identify which flows are modified, which are downstream
-
API routes — modules/integration/apis/*.md
- Map changed server methods to gRPC/HTTP endpoint names
- Extract the metric action label values used per endpoint
3.3: Output of this phase
Explicitly state before moving to Phase 4:
Affected endpoints:
- <endpoint-1> → metric action label: "<action-value-1>"
- <endpoint-2> → metric action label: "<action-value-2>"
Metric namespace: <oe|cardps|...>
Datasource UID: <uid>
Deployment time (RFC3339): <timestamp>
Deployment time (2h before): <timestamp-minus-2h>
⚠️ Metrics Discovery:
- NEVER assume metric names — always read from observability docs
- Each service has unique naming (e.g.,
oe_* vs cardps_* vs grpc_server_*)
Phase 4: Build Grafana Queries
Complete Phase 3 fully before starting this phase.
Write out ALL queries explicitly before executing any. Do not query Grafana yet.
4.1: Time boundaries
- Before: Use
startTime=<deployment-time-minus-2h> in RFC3339 format (e.g., 2026-04-06T16:26:36Z)
- After: Use
startTime=now
⚠️ Use startTime parameter in the MCP tool, NOT @ timestamp in PromQL. The @ modifier approach often returns empty results with the Grafana MCP tool.
4.2: Query templates
Replace <namespace>, <action_regex>, <metric_prefix> with values from Phase 3.
Error Rate:
# BEFORE
sum(rate(<metric_prefix>_error_response_total{kubernetes_namespace="<namespace>",action=~"<action_regex>"}[10m]))
# AFTER
sum(rate(<metric_prefix>_error_response_total{kubernetes_namespace="<namespace>",action=~"<action_regex>"}[10m]))
Traffic:
# BEFORE
sum(rate(<metric_prefix>_requests_total{kubernetes_namespace="<namespace>",action=~"<action_regex>"}[10m]))
# AFTER
sum(rate(<metric_prefix>_requests_total{kubernetes_namespace="<namespace>",action=~"<action_regex>"}[10m]))
P95 Latency:
# BEFORE
histogram_quantile(0.95, sum(rate(<metric_prefix>_durations_ms_histogram_bucket{kubernetes_namespace="<namespace>",action=~"<action_regex>"}[10m])) by (le))
# AFTER
histogram_quantile(0.95, sum(rate(<metric_prefix>_durations_ms_histogram_bucket{kubernetes_namespace="<namespace>",action=~"<action_regex>"}[10m])) by (le))
P99 Latency:
# BEFORE
histogram_quantile(0.99, sum(rate(<metric_prefix>_durations_ms_histogram_bucket{kubernetes_namespace="<namespace>",action=~"<action_regex>"}[10m])) by (le))
# AFTER
histogram_quantile(0.99, sum(rate(<metric_prefix>_durations_ms_histogram_bucket{kubernetes_namespace="<namespace>",action=~"<action_regex>"}[10m])) by (le))
Per-action error breakdown (always include):
sum by (action, error) (rate(<metric_prefix>_error_response_total{kubernetes_namespace="<namespace>",action=~"<action_regex>"}[10m]))
For gRPC services (Razorpay Foundation framework):
# Error rate
sum(rate(grpc_server_handled_total{kubernetes_namespace="<namespace>", grpc_code!="OK"}[10m]))
# P95 latency
histogram_quantile(0.95, sum(rate(grpc_server_handled_duration_seconds_bucket{kubernetes_namespace="<namespace>"}[10m])) by (le))
4.3: Discover datasource UID (if not in skills docs)
Tool: grafana::list_datasources
params: type=prometheus
Default Victoria Metrics UID: ALvd9Tgnz
Phase 5: Execute Grafana Queries + Generate Report
Complete Phase 4 fully (queries written out) before running any Grafana calls.
Tool: grafana::query_prometheus
Execute in this order:
- Error rate BEFORE (
startTime=<2h-before-RFC3339>)
- Error rate AFTER (
startTime=now)
- Traffic BEFORE
- Traffic AFTER
- P95 latency BEFORE
- P95 latency AFTER
- P99 latency BEFORE
- P99 latency AFTER
- Per-action error breakdown AFTER
Report Template
# Post-Deployment Monitoring Report
**Deployment:** <name>
**Namespace:** <namespace>
**Cluster:** <cluster>
**Deployment Time:** <RFC3339 timestamp>
---
## Deployment Scope
- **Previous SHA:** <sha>
- **Current SHA:** <sha>
- **Total Commits:** <count>
- **Files Changed:** <count>
- **Lines Added / Deleted:** <+X / -Y>
---
## Commit Breakdown
### Commit 1: `<sha>` — <title>
**Author:** <name>
**Risk:** 🟢 Low / 🟡 Medium / 🔴 High
**Affects:** <endpoint(s)> → metric action: `<action-label>`
**Summary:** <what changed and why>
---
## Affected Endpoints
| Endpoint | Action Label | Changed By |
|----------|-------------|------------|
| <endpoint> | `<action>` | Commit <sha> |
---
## Metrics Analysis
> **Time-of-day note (if applicable):** BEFORE snapshot taken at <time IST>. AFTER at <time IST>. Traffic deltas may reflect time-of-day variation.
### 1. Traffic
| Action | Before | After | Change | Status |
|--------|--------|-------|--------|--------|
| `<action>` | X.XX/s | X.XX/s | X% | ✅/⚠️/❌ |
| Combined | X.XX/s | X.XX/s | X% | ✅/⚠️/❌ |
### 2. Error Rates
| Action | Error | Before | After | Status |
|--------|-------|--------|-------|--------|
| `<action>` | `<error-code>` | X.XX/s | X.XX/s | ✅/⚠️/❌ |
### 3. Latency
| Metric | Before | After | Change | Status |
|--------|--------|-------|--------|--------|
| P95 | Xms | Xms | +X% | ✅/⚠️/❌ |
| P99 | Xms | Xms | +X% | ✅/⚠️/❌ |
---
## Risk Assessment
**Overall Status:** ✅ HEALTHY / ⚠️ WARNINGS / ❌ ISSUES
### Findings
1. **`<endpoint>`** — ✅/⚠️/❌ <summary>
- <specific metric observation>
- <root cause hypothesis if error>
---
## Conclusion
**Status:** ✅ No regressions / ⚠️ Minor issues / ❌ Rollback recommended
**Report Generated:** <timestamp>
**Data Sources:** Victoria Metrics (Grafana), Kubernetes (remote-friday-mcp-server), GitHub
Alert Thresholds
Error Rate
- 🟢 Green: <10% increase
- 🟡 Warning: 10–20% increase
- 🔴 Critical: >20% increase
4XX Errors
- 🟢 Green: <15% increase
- 🟡 Warning: 15–30% increase
- 🔴 Critical: >30% increase OR new 4xx codes appearing
Latency P95
- 🟢 Green: <20% increase
- 🟡 Warning: 20–30% increase
- 🔴 Critical: >30% increase
Latency P99
- 🟢 Green: <30% increase
- 🟡 Warning: 30–50% increase
- 🔴 Critical: >50% increase
Traffic Volume
- 🟢 Green: <10% change (or explainable by time-of-day)
- 🟡 Warning: 10–50% drop (unexplained)
- 🔴 Critical: >50% drop OR zero traffic on active endpoints
Deployment Size Risk
- 🟢 Low: 1–10 commits
- 🟡 Medium: 11–50 commits
- 🔴 High: 51–100 commits
- 🔴 Very High: 100+ commits
Troubleshooting
jsonpath query returns empty
Use -o wide first to see available fields, then describe for the specific ReplicaSet.
@ timestamp PromQL returns empty in Grafana MCP
Use startTime=<RFC3339> parameter in the tool call instead. Pass the evaluation time as the tool's startTime, not as an in-query modifier.
Metric names return no data
Use grafana::list_prometheus_metric_names with a regex prefix (e.g., ^oe_server) to discover actual metric names before querying.
Label values differ from docs
Use grafana::list_prometheus_label_values with the metric name filter to find actual label values in use.
No previous ReplicaSet
First deployment — report as "Initial deployment, no baseline available."
Multiple ReplicaSets with replicas > 0
Rolling update still in progress. Use highest revision as current; wait for rollout to stabilize.
Tool Reference
| Task | Tool |
|---|
| Get ReplicaSet images | remote-friday-mcp-server::kubectl_execute |
| Get commit diff | Bash gh api repos/razorpay/<repo>/compare/<prev>...<curr> |
| Read repo skill files | Read tool (if in workspace) or gh api for file content |
| Discover metric names | grafana::list_prometheus_metric_names |
| Discover label values | grafana::list_prometheus_label_values |
| Query metrics | grafana::query_prometheus |
Default Victoria Metrics datasource UID: ALvd9Tgnz
Skill Version: 3.0.0
Last Updated: 2026-04-07
