isc-async-scheduling

# Asynchronous calls in BIND 9 Three primitives schedule a callback. They are not interchangeable, and picking the wrong one is either a data race or a stalled event loop. | Need | Use | Runs on | |---|---|---| | Defer within the current loop (break lock re-entrancy, unwind the stack, re-arm) | `isc_job_run(loop, &obj->job, cb, arg)` | same loop, next iteration | | Hand a callback to *another* loop, or schedule from a non-loop thread | `isc_async_run(loop, cb, arg)` / `isc_async_current(cb, arg)` | target loop | | Blocking or CPU-heavy work (disk I/O, zone load/dump, crypto) | `isc_work_enqueue(loop, lane, cb, done_cb, arg)` | worker thread; `done_cb` back on the origin loop | `isc_async_run()` is **the only thread-safe one.** Everything else — the loop, timers, netmgr sockets, `isc_job_run()` — must be touched from the owning loop thread only. ## isc_job_run — cheapest, same loop only - Caller owns the `isc_job_t` storage: embed it in the object (`ISC_JOB_INITIALIZER`), don't heap-allocate one. No allocation, no atomics — an intrusive list plus a `uv_idle_t`. - **Not thread-safe, and not checked.** There is no `REQUIRE(VALID_LOOP)` and no tid assertion in `isc_job_run()`; passing a foreign loop corrupts that loop's list silently. It trusts you. - **One in-flight arming per `isc_job_t`.** `isc_job_run()` does `ISC_LINK_INIT()` right before appending, which defeats `ISC_LIST_APPEND()`'s `!ISC_LINK_LINKED` assertion — re-arming a job that is still queued corrupts `run_jobs` with no diagnostic. If one object can have two of these in flight, it needs two `isc_job_t`s. - **Re-arming from inside the job's own callback is fine and is the intended pattern.** `isc__job_cb()` copies out `cb`/`cbarg` and unlinks the job *before* invoking it, so the callback may re-arm the job — or free the object that contains it. - A job scheduled from within a running job does **not** run in the same drain; it runs on the loop's next iteration. And while any job is armed the idle handle keeps uv from blocking in poll — a perpetually self-re-arming job chain spins the loop at 100% CPU. ## isc_async_run — the thread-safe one - Any thread → any loop, including your own (`isc_async_current()`). Lock-free enqueue (urcu `cds_wfcq`) plus `uv_async_send()`; the send is only issued on the empty→non-empty transition, and `uv_async_send()` coalesces anyway. - **Always deferred, never inline** — even when the target is the current loop. That is exactly why netmgr routes callbacks through it: the caller may be holding a lock the callback wants to take. If you catch yourself invoking a user callback directly from a netmgr code path, this is the fix. - **It takes no reference on the loop.** The caller must guarantee the loop outlives the callback — `isc_work` does this explicitly with `isc_loop_ref()`. Forgetting it is the easy use-after-free here. - Allocates the `isc_job_t` from the *target* loop's `mctx` (fine — `isc_mem` is thread-safe) and frees it after the callback returns. - Do not rely on any ordering between `isc_job_run()` and `isc_async_run()` callbacks. Two `isc_async_run()` calls to the same loop keep their order. ## isc_work_enqueue — get off the loop - Each loop owns one worker thread **per lane**: `ISC_WORKLANE_FAST` (short bounded tasks, e.g. crypto — `dns_message` sig checks, validator) and `ISC_WORKLANE_SLOW` (blocking/long — master file load/dump, xfrin apply, catz/rpz updates). Keeping slow work on its own lane stops it from queueing behind—and delaying—fast work. - `REQUIRE(loop == isc_loop())`: you may only enqueue onto your *own* loop's worker. From another thread, `isc_async_run()` to the target loop first, then enqueue there. - `done_cb` always runs back on the origin loop, with whatever `cb` returned — that is where you touch loop-owned state again. The work `cb` itself runs on the worker thread and must not touch the loop. - **A worker thread has no loop and no tid.** `workthread_thread()` never sets the thread-locals, so inside a work `cb` `isc_loop()` is NULL and `isc_tid()` is `ISC_TID_UNKNOWN` (asserted in `tests/isc/work_test.c`). Anything sharded by tid — see [[per-loop-affinity]] — is therefore unusable from a work callback; capture what you need in `cbarg` before enqueueing, and do the loop-side work in `done_cb`. - **Cancellation is a tombstone, not a removal.** `isc_work_cancel()` CASes `QUEUED → CANCELED` and returns true only if it won that race (`uv_cancel` semantics). Nothing is freed, the node stays in the queue, and **`done_cb` still runs**, with `ISC_R_CANCELED`. The handle is valid only until `done_cb` has run — never afterwards. (No caller and no test exercises this yet; verify against `work.c` before leaning on it.) - **At shutdown the work callback can run on the loop thread.** Once the worker's SHUTDOWN bit is set, `isc_work_enqueue()` stops queueing and routes `work_run` through `isc_async_run()` instead — so a "blocking" callback executes on the event loop. Work callbacks must tolerate that. ## Cross-cutting rules - **Never hold `rcu_read_lock()` across a callback return.** The loop's `uv_prepare` (`quiescent_cb`) reaches a quiescent state and goes RCU-offline every iteration; non-QSBR builds assert `!rcu_read_ongoing()` there. Read-side sections live and die inside one callback. - **Exclusive mode is genuinely exclusive.** `isc_loopmgr_pause()` parks every other loop *and* every loop's worker threads before returning, so no work callback is running concurrently either. Paused loops and workers go RCU-offline, so `synchronize_rcu()` under pause won't hang. - **Shutdown and destroy are two different phases; only the second one is a trap.** At *shutdown* (`shutdown_cb`), teardown jobs are spliced onto the async queue and run as ordinary async jobs on a still-live loop — they may freely schedule more work, and normally do (that is where you drop references). Only when the last loop reference goes away does *destroy* (`destroy_cb`) close the handles; `isc__job_close()` and `isc__async_close()` then drain the job list and the async queue exactly **once** each, and `loop_close()` INSISTs both are empty. Anything scheduled from within that final close-phase drain is never run and trips the assertion. - Per-loop state that these callbacks touch is owned by its loop; see [[per-loop-affinity]] for the sharding discipline and [[rcu-mutation]] for publishing changes readers see concurrently. ## Where to look `lib/isc/async.c`, `lib/isc/job.c`, `lib/isc/work.c`, wired together in `lib/isc/loop.c` (`loop_init`, `shutdown_cb`, `destroy_cb`, `pause_loop`). Design overview in `doc/dev/loopmgr.md`. Behavioural tests, which are the fastest way to check a claim, are in `tests/isc/{async,job,work,loop}_test.c`.