| name | scheduler-change |
| description | Safely implement, diagnose, or review changes to gocron scheduling and task execution. Use for cron registration, leader election, task dispatch, concurrency queues, manual runs, retries, timeouts, cancellation, task logs, agent execution, failover, or shared scheduler state. |
Change the gocron scheduler
Assume scheduler bugs can duplicate, lose, or overlap production jobs. Make the
execution semantics explicit before editing.
Compatibility invariants
- When wire behavior changes, test
N gocron with N-1 gocron-node and the
reverse. Preserve execution, output, cancellation, and terminal status.
- Evolve protobuf/gRPC additively: never renumber or reuse field numbers,
remove a supported field, or change an existing RPC's semantics. Reserve
removed identifiers.
- Keep the previous RPC/execution path during the compatibility window. Check
capability and use a deterministic fallback before invoking new behavior.
- Never retry or reroute after an ambiguous remote acceptance; that can run a
user's job twice. Preserve retry counts and status transitions on fallback.
- When distributed ownership or state changes, verify temporary
N/N-1 HA
mixtures do not duplicate dispatch, stick in Running, or falsely cancel.
Performance invariants
- Establish a baseline for materially affected hot paths with representative
task/node counts and measure applicable scheduler metrics.
- Bound goroutines, queues, retries, log buffers, streams, timers, and retained
execution state. Define backpressure and overload behavior explicitly.
- Do not hold locks during database/network/filesystem/command/notification I/O
or serialize unrelated tasks behind one global lock.
- Batch or coalesce high-frequency status/log writes when semantics permit.
Verify SQLite concurrent read/write and busy/lock behavior separately.
- Add a benchmark or deterministic load regression test for material hot-path
changes. Compare before/after under the same workload and report the result.
Define invariants
Write down which behavior the change must preserve:
- whether execution is at-most-once, at-least-once, or best-effort;
- what happens during leader loss, restart, network timeout, or agent retry;
- how scheduled and manual runs interact;
- whether the same task may overlap and how queue limits behave;
- when task counts, instance maps, logs, and final status are updated;
- who owns cancellation and how resources are released.
If the intended behavior is ambiguous and different choices change user-visible
execution, ask before implementing.
Inspect the whole execution path
Trace from route or cron callback through internal/service/task.go, scheduler
state, models, RPC/agent code, task logs, and notifications. Search every read
and write of changed global state. Identify goroutine ownership, locks, channel
capacity, blocking sends, callbacks, timers, and cleanup paths.
Do not:
- hold a mutex while performing database, network, command, or notification I/O;
- start an unbounded goroutine per event;
- close a channel from a receiver or from multiple owners;
- use check-then-act state without the same lock or atomic operation;
- treat a timeout as proof that remote execution did not start;
- silently drop queued work or overwrite the terminal task status.
Keep refactors incremental. Extracting a component is useful when it creates a
test seam; moving globals without defining ownership is not.
Test adversarial sequences
Add deterministic tests for the changed invariant, including the relevant
subset of:
- simultaneous triggers for one task;
- queue full, cancellation, timeout, and shutdown;
- disable/remove while queued or running;
- repeated callback or retry;
- leader handoff and stale leader behavior;
- remote execution accepted but response lost;
N-1 protocol doubles in both client/server directions and fallback paths;- panic/error cleanup and counter consistency.
Avoid sleep-based assertions where a channel, fake clock, barrier, or polling
condition can make the test deterministic.
Run focused tests first, then:
go test -race ./internal/service/... ./internal/modules/... ./internal/rpc/...
go test -race -count=10 <changed-package>
Adapt package paths if the changed execution path differs. Invoke $verify
before committing. Report the promised execution semantics, race-sensitive
state touched, failure cases tested, performance evidence when relevant, and
any behavior that remains best-effort.
