temporal-ops

# Skill: temporal-ops ## Overview This skill operates and diagnoses Temporal environments. It has two modes: - **Operations:** the user wants to do something — create a namespace, rotate a key, check capacity, find unhealthy workflows, cancel a batch, set up export. The skill executes the right commands and interprets the output. - **Diagnosis:** the user arrives with a symptom — a stuck workflow, a cert error, a connection timeout, a non-determinism panic. The skill routes the investigation through a layered, bottom-up diagnosis until a root cause is identified with a confidence score. It does not teach how to write workflows or activities (use `skill-temporal-developer` for that), and it does not reproduce exhaustive CLI flag tables — run `temporal <cmd> --help` for those, and see [cli-conventions.md](references/ops/cli-conventions.md) for cross-command CLI conventions. The boundary is: if the user needs to administer or troubleshoot a running Temporal environment, this skill applies. ## Out of scope - **Writing workflows, activities, or SDK code** → `skill-temporal-developer`. - **Exhaustive CLI flags / command reference** → run `temporal <cmd> --help`; **cross-command CLI conventions** → [cli-conventions.md](references/ops/cli-conventions.md). - **Worker performance tuning, sizing, capacity planning** → `skill-temporal-workertuning`. - **Helm, Kubernetes, database admin, monitoring stack config** for self-hosted — beyond the CLI surface. If the conversation drifts into one of these areas, hand off to the relevant sibling skill rather than improvising. ## Philosophy ### Operator discipline When the user wants to perform an operational task: 1. **Identify the intent and backend.** Is this a Cloud operation (`tcld`) or a self-hosted operation (`temporal operator`)? Data-plane operations (`temporal workflow`, `temporal batch`, etc.) work on both. **If the backend is ambiguous, ask before proceeding — do not assume Cloud or self-hosted and do not output environment-specific commands until you know.** 2. **Execute commands and interpret output.** Run the documented command, read the result, and report what it means — or act on it if the user asked for an action. Read-only commands (`get`, `list`, `describe`, `count`, `show`) run freely. Anything listed under [Destructive operations](#destructive-operations) is proposed to the user first. 3. **Verify the result.** After a mutating operation, confirm the new state matches the user's intent. ### Destructive operations An operation belongs to this tier if it is **irreversible** (`tcld namespace delete`), **revokes access for a live identity** (`tcld apikey delete`), **moves production traffic** (`tcld namespace failover`), or **fans out to every match** (any `--query` form). Apply the test to the operation in front of you — this is a rule, not a list, and a command's absence from any list in this skill does not place it outside the tier. For anything in the tier: gather the evidence and **propose**. Do not run it on your own initiative, and do not run one to find out what it would do. The reference file for each command states its specific blast radius; read that before proposing, not after. 1. **Blast radius as a number, not a description.** For any `--query` form, run `temporal workflow count --query '<query>'` with the byte-identical query first and carry the result into the proposal. A filter with no narrowing predicate beyond `ExecutionStatus="Running"` matches every open Execution in the Namespace. 2. **Name the target.** State the exact command, the target, and the Namespace it resolves to. Connection settings can come from `TEMPORAL_*` env vars or a config-file profile, so the target is frequently not visible in the command text. If the backend or Namespace was inferred from context rather than stated by the user, say so — a destructive command aimed at the wrong Namespace is the most common way this goes wrong. 3. **Ask explicitly, then run it so it completes.** Put the command, the target, and — for any `--query` form — the count from step 1 to the user as a direct question, and wait for an answer. Once they approve, run it with `--yes` on the `--query` batch forms; that flag is what lets an approved batch finish, since the interactive prompt needs a terminal and without one the command reports `user denied confirmation` and does nothing. `--yes` belongs in a command the user approved, never in a retry of one that failed its prompt. Do not substitute a loop over single-target `workflow terminate --workflow-id`. Approval covers one command against one target; it does not carry to the next command, a widened query, or a second Namespace. 4. **Verify, and know the abort path.** Re-run the corresponding `get`, `describe`, or `count`. A batch job drains asynchronously: `temporal batch describe --job-id <id>` shows how far it has gotten and `temporal batch terminate --job-id <id>` stops it before it reaches the rest of its matches. When a reversible sibling reaches the same goal, propose it alongside: `cancel` lets Workflow cleanup code run where `terminate` does not; `apikey disable` is reversible where `delete` is not; `accepted-client-ca add` appends where `set` replaces. Assume nothing in the environment will stop a destructive command on your behalf. Credential scope, command denylists, and confirmation prompts may or may not be configured, and their possible presence is not a reason to skip any step above — you are the safeguard the user is relying on. ### Diagnostic discipline When the user arrives with a failure or anomaly: 1. **Bottom-up diagnosis.** Verify the lower layer before blaming the upper one. The layers, from bottom to top: 1. DNS / network path 2. TCP / port reachability 3. TLS handshake 4. Authentication (API key or mTLS client cert) 5. gRPC health and Temporal frontend reachability 6. Temporal namespace, task queues, workers 7. Workflow code (determinism, signals, timers, child workflows) The full ladder lives in [diagnostic-ladder.md](references/triage/diagnostic-ladder.md). 2. **Always verify the next layer up** rather than prescribing a speculative fix. If TLS works, prove auth works before blaming the workflow. If pollers are present, prove the workflow's last event before blaming the worker. 3. **Attach a confidence score** (1-10) to every proposed diagnosis: - 9-10: symptoms, operation, and confirming signals line up cleanly. - 6-8: evidence is good but at least one alternative remains plausible. - 1-5: the issue is still ambiguous; the "fix" is the next discriminating check, not a root cause. 4. **Name ambiguity explicitly.** Errors like `context deadline exceeded` are not self-describing, and a single code such as `RESOURCE_EXHAUSTED` can mean more than one condition (account-limit throttling vs. per-Workflow lock contention). Surface that, gather more context, and scope the next step narrowly. These are skill conventions, not docs-derived facts. ## Intent routing ### Operations Find the row that matches the user's intent. The reference file contains the commands and procedures. | Intent | Category | Reference | |---|---|---| | Create, get, list, delete a Cloud namespace | Cloud namespace admin | [cloud-namespace-admin.md](references/ops/cloud-namespace-admin.md) | | Add/remove region, failover, HA config | Cloud namespace admin | [cloud-namespace-admin.md](references/ops/cloud-namespace-admin.md) | | Set retention, tags, codec-server, connectivity rules | Cloud namespace admin | [cloud-namespace-admin.md](references/ops/cloud-namespace-admin.md) | | Add or rename Cloud search attributes | Cloud namespace admin | [cloud-namespace-admin.md](references/ops/cloud-namespace-admin.md) | | Check current APS / capacity mode | Cloud capacity | [cloud-capacity.md](references/ops/cloud-capacity.md) | | Switch On-Demand ↔ Provisioned, set TRUs | Cloud capacity | [cloud-capacity.md](references/ops/cloud-capacity.md) | | Understand APS / RPS / OPS limits and throttling | Cloud capacity | [cloud-capacity.md](references/ops/cloud-capacity.md) | | Create, disable, enable, delete an API key | Cloud IAM | [cloud-iam.md](references/ops/cloud-iam.md) | | Invite, list, remove users; set roles/permissions | Cloud IAM | [cloud-iam.md](references/ops/cloud-iam.md) | | Manage user groups and service accounts | Cloud IAM | [cloud-iam.md](references/ops/cloud-iam.md) | | Generate mTLS certs, upload CA, set cert filters | Cloud certs | [cloud-certs.md](references/ops/cloud-certs.md) | | Rotate mTLS certificates | Cloud certs | [cloud-certs.md](references/ops/cloud-certs.md) | | Set up Workflow History Export (S3 / GCS) | Cloud namespace admin | [cloud-namespace-admin.md](references/ops/cloud-namespace-admin.md) | | Set up PrivateLink / PSC, manage connectivity rules | Cloud connectivity | [cloud-connectivity.md](references/ops/cloud-connectivity.md) | | Self-hosted cluster health, describe, namespace CRUD | Self-hosted admin | [self-hosted-admin.md](references/ops/self-hosted-admin.md) | | Self-hosted search attributes, Nexus endpoints | Self-hosted admin | [self-hosted-admin.md](references/ops/self-hosted-admin.md) | | Check or manage a Cloud Nexus Endpoint's caller-Namespace allowlist; the 1,000-caller default | Cloud namespace admin | [cloud-namespace-admin.md#tcld-nexus-endpoint-allowed-namespace](references/ops/cloud-namespace-admin.md#tcld-nexus-endpoint-allowed-namespace) | | Find stuck/hung/unhealthy workflows via list queries | Workflow health | [workflow-health.md](references/ops/workflow-health.md) | | Task queue poller status, workflow counts | Workflow health | [workflow-health.md](references/ops/workflow-health.md) | | Cancel, terminate, or reset workflows | Workflow recovery | [workflow-stuck.md#recovery-commands](references/triage/workflow-stuck.md#recovery-commands) | | Bulk / batch operations on workflows (`--query`) | CLI conventions | [cli-conventions.md](references/ops/cli-conventions.md#the---query--batch-job-bridge) | | Schedule CRUD, time-spec, and operations | CLI conventions | [cli-conventions.md](references/ops/cli-conventions.md#schedule-time-spec-forms) | | Complete or fail an activity externally | CLI conventions | [cli-conventions.md](references/ops/cli-conventions.md#operation--command-index) | | Cloud Ops API access, rate limits, Go SDK | Cloud Ops API | [cloud-ops-api.md](references/ops/cloud-ops-api.md) | | View billing, generate billing report, cost attribution | Cloud billing | [cloud-billing.md](references/ops/cloud-billing.md) | | Audit Logs: view, query via API, configure sink (AWS/GCP) | Cloud audit logs | [cloud-audit-logs.md](references/ops/cloud-audit-logs.md) | | Terraform provider: Namespace/User/SA/API Key/Nexus CRUD | Cloud Terraform | [cloud-terraform.md](references/ops/cloud-terraform.md) | | Expiry alerts (cert, API key, credit), status page | Cloud notifications | [cloud-notifications.md](references/ops/cloud-notifications.md) | | SAML SSO, SCIM provisioning, IdP integration | Cloud SAML/SCIM | [cloud-saml-scim.md](references/ops/cloud-saml-scim.md) | | Migrate self-hosted to Cloud (automated or manual), migrate between Cloud regions | Cloud migration | [cloud-migration.md](references/ops/cloud-migration.md) | | End-to-end ops playbook (setup, rotation, audit, billing, Terraform) | Ops recipes | [ops/recipes.md](references/ops/recipes.md) | ### Diagnosis Find the row that matches the user's symptom. Start the investigation at the first check, then read the linked reference. | Symptom | Category | First check | Reference | |---|---|---|---| | `connection refused`, cannot reach frontend | Connectivity | `nc -zvw10 <host> 7233` | [connectivity.md#connection-refused](references/triage/connectivity.md#connection-refused) | | `no such host`, DNS resolution fails | Connectivity | `dig +short <host>` or `nslookup <host>` | [connectivity.md#dns](references/triage/connectivity.md#dns) | | `tls: handshake failure`, server rejects handshake | Certificates | `openssl s_client -connect <host>:7233 -servername <host> </dev/null` | [certificates.md#handshake-failure](references/triage/certificates.md#handshake-failure) | | `x509: certificate has expired` or `not yet valid` | Certificates | `openssl x509 -enddate -noout -in cert.pem` | [certificates.md#expired-or-not-yet-valid](references/triage/certificates.md#expired-or-not-yet-valid) | | `x509: certificate signed by unknown authority` | Certificates | `openssl verify -CAfile ca.pem client.pem` | [certificates.md#unknown-authority](references/triage/certificates.md#unknown-authority) | | `tcld` session / auth fails, Cloud role unclear | Authentication | `tcld account get` | [authentication.md#cloud-role-and-permission-model](references/triage/authentication.md#cloud-role-and-permission-model) | | `UNAUTHENTICATED`, API key rejected | Authentication | `env \| grep -i TEMPORAL_API_KEY`, then `tcld apikey get --id <apikey_id>` | [authentication.md#things-to-check-when-unauthenticated-is-returned-with-an-api-key](references/triage/authentication.md#things-to-check-when-unauthenticated-is-returned-with-an-api-key) | | `namespace not found` / wrong namespace string with an API key | Authentication | Confirm Regional Endpoint form `<region>.<cloud_provider>.api.temporal.io:7233` | [authentication.md#address-form-for-api-key-connections](references/triage/authentication.md#address-form-for-api-key-connections) | | `RESOURCE_EXHAUSTED` gRPC status | Rate limits | Identify which limit fired: throttle metrics on Cloud v1, the `resource_exhausted_cause` label on v0 / self-hosted | [rate-limits.md#identifying-which-limit-was-hit](references/triage/rate-limits.md#identifying-which-limit-was-hit) | | Task queue shows no pollers | Worker health | `temporal task-queue describe --task-queue <q>` | [worker-health.md#what-no-pollers-looks-like](references/triage/worker-health.md#what-no-pollers-looks-like) | | Workflow stuck on a pending activity / timer / child / signal | Workflow stuck | `temporal workflow describe --workflow-id <id>` | [workflow-stuck.md#the-primary-inspection-command-temporal-workflow-describe](references/triage/workflow-stuck.md#the-primary-inspection-command-temporal-workflow-describe) | | `NondeterminismError`, repeating `WorkflowTaskFailed` | Non-determinism | Identify the last `WorkflowTaskFailed` cause in the Event History | [non-determinism.md#the-wft-failure-signature-of-non-determinism](references/triage/non-determinism.md#the-wft-failure-signature-of-non-determinism) | | Replay fails locally but prod workflow was running | Non-determinism | Fetch the history and run the SDK replayer in one test | [replay.md#step-2--run-the-sdk-replayer-all-supported-sdks](references/triage/replay.md#step-2--run-the-sdk-replayer-all-supported-sdks) | | HA failover did not route traffic to failover region | HA failover | `tcld namespace get --namespace <ns>.<acct>` vs. DNS CNAME | [ha-failover.md#start-here-establish-ground-truth](references/triage/ha-failover.md#start-here-establish-ground-truth) | | Serverless Worker (AWS Lambda) stopped processing work after a Namespace failover | HA failover | Confirm a `FailoverNamespace` audit event, then compare the new active region against the Lambda ARN on the Worker Deployment Version | [ha-failover.md#symptom-serverless-workers-kept-running-in-the-old-region-after-failover](references/triage/ha-failover.md#symptom-serverless-workers-kept-running-in-the-old-region-after-failover) | | `context deadline exceeded` (unknown layer) | Runtime errors | Identify which operation and SDK emitted it | [runtime-errors.md#deadline-exceeded](references/triage/runtime-errors.md#deadline-exceeded) | | `Workflow is busy` / `ResourceExhausted` on signal/update/query to one Workflow (BusyWorkflow) | Runtime errors | Rule out account-limit throttling, then split `temporal_cloud_v1_resource_exhausted_error_count` by `operation` | [runtime-errors.md#workflow-lock-contention-busyworkflow](references/triage/runtime-errors.md#workflow-lock-contention-busyworkflow) | | `PAYLOADS_TOO_LARGE`, `exceeds size limit`, payload/gRPC blob size error | Blob size limits | Check whether the issue is payload (2 MB) or gRPC message (4 MB) | [blob-size-limits.md](references/triage/blob-size-limits.md) | | Workflow stuck in invisible retry loop (gRPC message too large) | Blob size limits | Check Worker logs for `ResourceExhausted`, reduce batch size | [blob-size-limits.md](references/triage/blob-size-limits.md) | | High schedule-to-start latency, task slot depletion, slow Workflow Tasks | Performance bottlenecks | Check `temporal_workflow_task_schedule_to_start_latency` P95 | [performance-bottlenecks.md](references/triage/performance-bottlenecks.md) |