ops-inspector

## Sibling skills (local only) Sibling CloudBase skills ship beside this skill. Use local relative paths such as `../auth-tool-cloudbase/SKILL.md`. If a referenced sibling skill file is missing from this environment, ask the user to install the full CloudBase plugin (or the missing skill). Do **not** HTTP-fetch remote skill or protocol markdown into the agent context. ## Activation Contract ### Use this first when - The user wants to check the health or status of CloudBase resources (cloud functions, CloudRun, databases, storage, etc.). - The user reports errors, failures, or abnormal behavior and wants a quick diagnosis. - The user asks for an "inspection", "health check", "巡检", "诊断", or "troubleshooting" of their CloudBase environment. - The user wants to review recent error logs across services. - The user asks **告警解读** questions: whether a **CPU 告警** is normal, what **峰值 QPS** was, or whether throttle/error metrics look healthy. - The symptom matches a fault playbook: **429 / 限频**, **AI 模型 429 / Token 额度**, **云函数超时或轮询超时（`-404005`/433）**, **云函数 404**, **ACCESS_TOKEN_INVALID**, or **调用量为 0**. - The user wants a **patrol** rather than a one-off check ("每天帮我看看有没有变差", "定时巡检", "值守"), or asks to put an inspection on a schedule. ### Read before writing code if - The inspection reveals code-level issues in cloud functions or CloudRun services — then read the relevant implementation skill before suggesting fixes. - The user wants to fix a problem found during inspection rather than just diagnose it. ### Then also read - Alarm interpretation baselines -> `references/alarm-interpretation.md` - Fault playbooks (429 / AI model 429 / function timeout / 404 / token / zero calls) -> `references/fault-playbooks.md` - Cloud function issues -> `../cloud-functions/SKILL.md` - CloudRun issues -> `../cloudrun-development/SKILL.md` - Database issues -> `../postgresql-development-cloudbase/SKILL.md` for CloudBase PG / PostgreSQL, `../relational-database-mcp-cloudbase/SKILL.md` for MySQL, or `../cloudbase-document-database-web-sdk/SKILL.md` for NoSQL - Auth readiness (token failures) -> `../auth-tool-cloudbase/SKILL.md` - Environment not visible, binding fails, or only one environment shows up -> `../cloudbase-cli/references/core.md` (*Where credentials are read from*) - Platform overview -> `../cloudbase-platform/SKILL.md` ### Do NOT use for - Deploying new resources or writing application code. This skill is read-only and diagnostic. - Replacing the platform's alert rules (SMS / 告警策略). A patrol catches drift and slow degradation between alerts; alert rules catch hard thresholds in real time. Do not disable alerting because a patrol exists. - Setting up the alert rules themselves. For log-level threshold alerts (fire when `statusCode` is 502), route the user to CLS log alerting — deliver environment logs to CLS, confirm the index, then create an alert policy: https://docs.cloudbase.net/logger/alert/log-alert-guide (metric-based alerting is console-native and needs no setup guide). - Writing to the environment on a scheduled run. See *Scheduled Patrol* — every patrol step is read-only. - Directly fixing problems — it diagnoses and recommends; actual fixes should use the appropriate implementation skill. - Fetching metrics by guessing cloud API Actions. For any metric `queryEnv` covers (functions, CloudRun, gateway QPS, MySQL, package usage), always use `queryEnv(action="metrics")` — never `callCloudApi`. The one exception is PostgreSQL: `queryEnv` has no PG metrics, so pull them via `callCloudApi` → `monitor` / `2018-07-24` / `GetMonitorData`, namespace `QCE/POSTGRES`, with the exact Action and dimensions pinned in the `cloud-api-operations` skill — never from memory. ### Common mistakes / gotchas - Running a full inspection without first confirming the environment is bound (`auth` tool must show logged-in and env-bound state). If binding fails or only one environment appears, credentials are being resolved **per directory** — see `../cloudbase-cli/references/core.md` (*Where credentials are read from*); it is usually neither a permissions nor a platform problem. - Ignoring CLS log service status — if CLS is not enabled, `queryLogs` will fail; always check first with `queryLogs(action="checkLogService")`. - Searching logs without a time range — this can return excessive or irrelevant results. Always scope searches to a relevant time window. - Assuming one `queryFunctions(action="listFunctionLogs")` call covered a multi-day window. It is capped at **1 day and 100 rows** per call. For anything longer, loop over day-sized windows or switch to `queryLogs(action="searchLogs")`. - Reading an empty log result as "no errors". Empty has three common causes and only one of them is good news: wrong time window, missing colon in a field comparison (`status_code>200` silently matches nothing), or CLS not capturing that log source. Say which check you ran before concluding a resource is healthy. - Treating a single error log as the root cause without correlating across resources. A function error may stem from a database or config issue. - Answering "峰值 QPS" / "CPU 告警是否正常" from screenshots or memory instead of `queryEnv(action="metrics")`. - Calling `callCloudApi` with invented `GetMonitorData` / `DescribeCurveData` parameters — the metrics branch already wraps Manager SDK. - Re-reporting the same finding on every patrol run. Without state between runs the report repeats itself, gets muted, and stops being read — see *Scheduled Patrol*. - Scheduling a patrol with write-capable tools approved. A scheduled run cannot stop to ask, so anything it can call, it will call unattended. ### Minimal checklist - [ ] Environment is bound and accessible (`queryEnv(action="info")`) — if it fails, or only one environment shows up, confirm the current identity and credential slot first (`../cloudbase-cli/references/core.md`) - [ ] Metrics pulled with `queryEnv(action="metrics")` when the question involves QPS / CPU / throttle / invocation volume - [ ] CLS log service is enabled (`queryLogs(action="checkLogService")`) when log diagnosis is needed - [ ] Matching fault playbook selected when symptoms match 429 / AI model 429 / function timeout / function 404 / ACCESS_TOKEN_INVALID / 调用量为 0 - [ ] Time range is specified for any log or metrics searches - [ ] Findings are summarized with severity levels, **告警解读**, and actionable recommendations --- ## How to use this skill (for a coding agent) ### What this skill covers - **定时巡检** — four patrol types with default cadences, a state file so a run reports only what is new or worse, read-only rules for unattended runs, and the offer to schedule the inspection itself. - **告警解读** — pull metrics, compare to baselines in `references/alarm-interpretation.md`, answer CPU-alert / peak-QPS style questions in plain language. - **故障剧本** — when symptoms match, follow `references/fault-playbooks.md` instead of ad-hoc tool fishing. ### Inspection Modes | Mode | When to use | Scope | |------|-------------|-------| | **Full inspection** | User asks for a general health check / 巡检 / 全面检查 | All resource types + core metrics | | **Targeted inspection** | User reports a specific error or asks about a specific resource | One resource type or playbook | | **Alarm interpretation** | User asks CPU 告警是否正常 / 峰值 QPS / 是否限流 | Metrics-first, then logs | | **Fault playbook** | 429 / AI model 429 / function timeout / function 404 / ACCESS_TOKEN_INVALID / 调用量为 0 | Playbook steps only | ### Full Inspection Workflow Follow these steps in order for a comprehensive environment health check: **Step 1 — Environment Check** ``` queryEnv(action="info") ``` Confirm the environment is accessible. Record the `envId` for console link generation. **Step 2 — Metrics snapshot** ``` queryEnv(action="metrics", envId="<EnvId>", metricName="GatewayTraceEnvQPS") queryEnv(action="metrics", envId="<EnvId>", metricName="FunctionInvocation") queryEnv(action="metrics", envId="<EnvId>", metricName="MysqlCpuUsageRate") ``` Use returned `Summary.max` / `avg` / `allZero` / `peakTimestamp`. Add `FunctionError`, `FunctionThrottle`, or CloudRun `Tke*` metrics when those resources exist. Read `references/alarm-interpretation.md` before concluding. **Step 3 — Log Service Status** ``` queryLogs(action="checkLogService") ``` If CLS is not enabled, note this as a **warning** — log-based diagnosis will be unavailable. Recommend enabling CLS in the console: `https://tcb.cloud.tencent.com/dev?envId=${envId}#/devops/log` **Step 4 — Cloud Functions Inspection** ``` queryFunctions(action="listFunctions") ``` For each function, check: - **Status**: Is the function in an active/deployed state? - **Recent errors**: `queryFunctions(action="listFunctionLogs", functionName="<name>", startTime="<recent>")` - **Common issues**: - Timeout errors (execution exceeded limit) - Memory limit exceeded - Runtime errors (unhandled exceptions) - Cold start frequency - Zero invocations while traffic is expected → Playbook 4 **Step 5 — CloudRun Services Inspection** ``` queryCloudRun(action="list") ``` For each service, check: - **Status**: Is the service running? - **Detail**: `queryCloudRun(action="detail", detailServerName="<name>")` - **Metrics**: `queryEnv(action="metrics", metricName="TkeQPSService", resourceID="<serviceName>")` (resourceID required) - **Common issues**: - Service not running (scaled to zero or crashed) - Image pull failures - OOMKilled events - Health check failures **Step 6 — Error Log Aggregation** (if CLS is enabled) ``` queryLogs(action="searchLogs", queryString="ERROR", service="tcb", startTime="<24h-ago>", limit=50) queryLogs(action="searchLogs", queryString="ERROR", service="tcbr", startTime="<24h-ago>", limit=50) ``` Look for patterns: - Repeated error messages (same error many times) - Cascading failures (errors in multiple services around the same time) - Timeout / 429 / 404 / ACCESS_TOKEN_INVALID patterns → jump to the matching playbook; a **429 from an AI model call** goes to Playbook 5, not Playbook 1 **Step 7 — Summary Report** Generate a structured report: ```markdown # CloudBase Resource Inspection Report **Environment**: ${envId} **Inspection Time**: ${timestamp} ## Overall Health: ✅ Healthy / ⚠️ Warnings Found / ❌ Issues Found ## 告警解读 | 问题 | 指标 | 窗口峰值 | 基线 | 结论 | |------|------|----------|------|------| | 峰值 QPS | GatewayTraceEnvQPS | ... | package default 500 unless known | ... | | CPU 告警是否正常 | MysqlCpuUsageRate | ... | warn≥80 / crit≥90 | ... | ### Cloud Functions | Function | Status | Recent Errors | Invocations | Severity | |----------|--------|---------------|-------------|----------| | ... | ... | ... | ... | ... | ### CloudRun Services | Service | Status | Issues | Severity | |---------|--------|--------|----------| | ... | ... | ... | ... | ### Error Log Summary - Total errors in last 24h: N - Top error patterns: ... ## Recommendations 1. ... 2. ... ## Console Links - Cloud Functions: https://tcb.cloud.tencent.com/dev?envId=${envId}#/scf - CloudRun: https://tcb.cloud.tencent.com/dev?envId=${envId}#/platform-run - Logs: https://tcb.cloud.tencent.com/dev?envId=${envId}#/devops/log - Monitor: https://tcb.cloud.tencent.com/dev?envId=${envId}#/devops ``` ### Targeted Inspection Workflow When the user specifies a resource type or a specific resource: 1. **Cloud function errors**: `queryFunctions(action="listFunctionLogs", functionName="<name>")` then `queryLogs(action="searchLogs", queryString="function_name:\"<name>\" AND status_code:>202", ...)` 2. **CloudRun errors**: `queryCloudRun(action="detail", detailServerName="<name>")` then `queryLogs(action="searchLogs", queryString="ERROR", service="tcbr", ...)` - If logs show DB / Redis connection failures (`ECONNREFUSED`, timeout, "could not connect"): check whether `VpcConf` is set and matches the database VPC. See `cloudrun-development/references/vpc-and-database.md`. 3. **Database issues**: Check `queryPgDatabase(action="context"|"metadata"|"objects")` for CloudBase PG, `queryMysqlDatabase` for MySQL, or `readNoSqlDatabaseStructure` for NoSQL depending on type; for CPU/disk alerts pull `MysqlCpuUsageRate` / `MysqlStorageUsage` — MySQL only. `queryEnv` has **no PostgreSQL metrics**. The platform monitoring set the console shows (namespace `QCE/POSTGRES` — dedicated instances: Cpu / Storage / Memory / MemoryRate / StorageRate / Qps / Connections / ReadWriteCalls / SqlRuntime* / SlowQueryCnt / ActiveConns / IdleConns / LongQuery / LongXact / Deadlocks / LongWaiting / Tps; shared clusters: Dbcpuusage / Dbmemorybytes / Dbsizebytes / Activeconnections / Idleconnections / Totalconnections) is still reachable from MCP: `callCloudApi` → `monitor` / `2018-07-24` / `GetMonitorData` with namespace `QCE/POSTGRES` (dedicated dimensions use `resourceId`, shared uses `DBName` + `AppId` — see the `cloud-api-operations` skill). For a quick check, `queryPgDatabase(action="sql")` on `pg_stat_activity` / `pg_database_size()` covers connections and size without an extra credential scope. 4. **General error search**: `queryLogs(action="searchLogs", queryString="<error-keyword>", ...)` 5. **Alarm / QPS questions**: follow `references/alarm-interpretation.md` 6. **429 / AI model 429 / function timeout / function 404 / ACCESS_TOKEN_INVALID / 调用量为 0**: follow `references/fault-playbooks.md` ### Scheduled Patrol (定时巡检) An inspection answers "what is broken right now". A patrol answers "did anything get worse than last time". Same tools, two extra rules: a cadence, and memory of what was already reported. #### Four patrols and their cadence | Patrol | Checks | Default cadence | |--------|--------|-----------------|