new-diagnostics-invariant

# Adding a Workflow Diagnostics Invariant Workflow Diagnostics (`service/worker/diagnostics`) reads one workflow execution's history and reports issues, root causes, and runbook links. Each check is an "invariant". Existing ones: `timeout`, `failure`, `retry`, `timeoutrisk`, `antipatterns`. Read the one closest to yours before writing anything. ## The interface Defined in `service/worker/diagnostics/invariant/interface.go`: - `Check(ctx, input)` — scan the history, return issues. The full event list is `input.WorkflowExecutionHistory.GetHistory().GetEvents()`. - `RootCause(ctx, input)` — explain the issues. Can be a no-op. - Each issue is an `InvariantCheckResult{IssueID, InvariantType, Reason, Metadata}`. `IssueID` starts at 0 and counts up within one `Check` call — it is not globally unique; `InvariantType` is what tells issues apart. `Metadata` is JSON, built with `invariant.MarshalData(...)`. Two templates to copy from: - **Pure history scan** → copy `invariant/retry/`: empty struct, `NewInvariant()` with no arguments, `RootCause` returns an empty non-nil slice with a comment saying the issues explain themselves. - **Needs live lookups** (e.g. `DescribeTaskList`) → copy `invariant/timeout/`: client passed in via `NewInvariant(Params{Client: ...})`, real `RootCause`. New root-cause categories are `RootCause` consts in `interface.go`, shared by all invariants. ## The new package: `service/worker/diagnostics/invariant/<name>/` Package name is one lowercase word (`timeout`, `retry`, `timeoutrisk`). **types.go** - `<Name>Type` string type + consts + `String()` — these become the `InvariantType` values. - `IssueType` string type + consts + `String()` — these become the `Reason` values. Each reason is a plain sentence ending with a period. Never build reasons with `fmt.Sprintf` — numbers belong in the metadata. - Thresholds are package consts, each with a comment saying what it means. - One `<Name>IssuesMetadata` struct per invariant, and `Check` marshals it directly. Fields shared by every check are top-level, and every issue sets a top-level `EventID` (the event the issue anchors on). Check-specific fields go in a per-check struct held by a pointer field named after the check, tagged `json:",omitempty"`, with exactly one set per issue (see `timeoutrisk.TimeoutRiskIssuesMetadata`). If all checks share one shape, skip the sub-structs (see `retry.RetryMetadata`). Never nest event IDs: cadence-web links only top-level `EventID`, `ActivityScheduledID`, `ActivityStartedID` keys and renders nested objects as one raw-JSON row. **<name>.go** - `type <Name> invariant.Invariant`, an unexported struct, `NewInvariant()`. - In `Check`: find the started event by scanning for a non-nil `GetWorkflowExecutionStartedEventAttributes()`, and handle it being missing. One flat loop over events. One `issueID` counter, +1 per issue. - Don't deduplicate by ActivityID — repeated scheduled events are separate invocations, and server-side retries don't write new scheduled events. **<name>_test.go** - Plain table-driven `Test__Check`. No testify/suite, no testify mocks — use uber-go/mock if a client is needed (see `timeout_test.go`). - Build fixtures by hand: `*types.GetWorkflowExecutionHistoryResponse{History: &types.History{Events: ...}}` with explicit event IDs. - Expected metadata is `json.Marshal` of the struct; expected reasons come from the `IssueType` consts. - Cover: each check firing alone, values right at each threshold, several issues at once (IssueIDs count up), a healthy workflow, and a history with no started event. ## Gotcha: history shows what the server saved, not what the client asked for The server rewrites decision attributes before saving them (`validateActivityScheduleAttributes` in `service/history/decision/checker.go`, plus siblings for child workflows and timers). For activities: - All four timeouts get capped at the workflow's total execution timeout. A "timeout > workflow timeout" check can never fire; a silently capped value shows up as exactly equal to the workflow timeout. - With a retry policy, ScheduleToStart and ScheduleToClose get raised up to that cap. Equality there is normal, not a mistake. Before writing any check on configured values, read the validator to see what the server rewrites. Also check the existing invariants so you don't repeat a check (`retry` already flags heartbeat >= StartToClose and expiration < initial interval). ## Wiring checklist (all required) 1. `service/worker/diagnostics/workflow.go`: - Add `<Name>s *<name>Diagnostics` to `DiagnosticsWorkflowResult`. - Add `<name>Diagnostics{Issues, Runbook}` and `<name>IssuesResult{IssueID, InvariantType, Reason, Metadata}` structs. Add a `RootCause` field only if `RootCause` does real work. - Add `retrieve<Name>Issues(checkResult)`: check membership in your `InvariantType` consts and unmarshal the metadata into `<Name>IssuesMetadata` (copy `retrieveTimeoutRiskIssues`). - In `DiagnosticsWorkflow`, add an `if len(issues) > 0 { ... Runbook: linkTo<Name>Runbook }` block and a field in the final result. 2. `service/worker/diagnostics/activities.go`: add `linkTo<Name>Runbook` (`https://cadenceworkflow.io/docs/workflow-troubleshooting/<topic>/`). The 10-issues-per-invariant cap applies on its own. 3. `service/worker/diagnostics/parent_workflow.go`: add an `issueType<Name>` const and a branch in `getIssueType` (same `-` joining as the others). 4. `cmd/server/cadence/server.go` (~line 306): add `<name>.NewInvariant(...)` to `params.DiagnosticsInvariants`, plus the import. 5. Existing tests: add the invariant to the lists in `workflow_test.go` `SetupTest` and `activities_test.go` `testDiagnosticWorkflow()`. Check first that existing fixtures don't set off your checks; update expectations if they do. Add `Test__retrieve<Name>Issues` to `workflow_test.go`, copying `Test__retrieveTimeoutRiskIssues`. ## Runbook page (cadence-docs repo) The `linkTo<Name>Runbook` URL has to exist eventually. Write the page in your local cadence-docs checkout as `docs/08-workflow-troubleshooting/<NN>-<topic>.md`, taking the next free number. `01-timeouts.md` on origin/master shows the style (the local checkout can lag — check against origin/master): - Frontmatter: `layout: default`, `title`, a one-sentence `description`, `keywords`, and `permalink: /docs/workflow-troubleshooting/<topic>` — must match the `linkTo<Name>Runbook` path. - No H1 in the body (the title comes from frontmatter). Start with a 2-3 sentence intro. - One `##` section per issue. The heading describes the situation ("Many activities scheduled in quick succession"), not a command. Then a paragraph on the cause, then the fixes: a single fix is a paragraph starting with `Mitigation:`; multiple fixes go under a `Mitigations:` line as a bulleted list, best fix first. - Internal doc links are relative paths like `/docs/concepts/activities#timeouts` — no domain, no trailing slash. Only external links (GitHub, pkg.go.dev) are absolute. - Point to built-in solutions first (`workflow.NewBatchFuture` for big fan-outs, Schedules instead of cron) before manual workarounds. - Describe thresholds loosely ("50 or more within a few seconds") so the page doesn't go stale when the consts are tuned. The page is a separate PR to cadence-workflow/cadence-docs. Until it merges, say in the server PR that the page doesn't exist yet. ## Verify ```bash go build ./service/worker/... ./cmd/server/... go test -race -count=1 ./service/worker/diagnostics/... make lint # no codegen needed — Invariant has no generated mock ``` Before the PR: `make pr GEN_DIR=service/worker/diagnostics`. PR title: `feat(diagnostics): ...`.