ros2-executors

# ROS 2 Executors, Callback Groups, Thread Safety ## Mental model 1. An **executor** schedules ready callbacks onto threads. 2. A **callback group** constrains *which* of those callbacks may run together. 3. A **mutex** protects *data* when different groups (or non-ROS threads) can touch it concurrently. `MultiThreadedExecutor` **without** extra callback groups is a no-op for concurrency: the node’s **default group is MutuallyExclusive**, so everything still runs one-at-a-time. If you chose MT, you **must** assign groups (or you are lying to yourself). ## Decision tree — when to create a callback group | Situation | What to do | |-----------|------------| | Simple node, no waits in callbacks | Default group + `SingleThreadedExecutor` / `spin(node)`. Done. | | Sync service/action wait **inside** a sub/timer/service callback | **Deadlock** on default/ST. Put **caller** and **client** in different groups (any types) **or** one **Reentrant** group, **and** use `MultiThreadedExecutor`. Prefer **async** instead. | | Two sensors that may process in parallel; each must not re-enter itself | Two **different MutuallyExclusive** groups (not one Reentrant). | | Action/service must handle overlapping instances of the *same* callback | That entity’s group = **Reentrant**. | | Timer reads buffers filled by subscriptions | Groups alone do **not** serialize across groups — use a **mutex** (or put timer+subs in one MutuallyExclusive group and accept no overlap). | | Heavy work (vision, planning) | Do **not** do it in the callback. Bound queue → worker thread / task system; callback only enqueues. | | Component in `component_container` (ST) | No parallel callbacks regardless of groups. Need `component_container_mt` + groups. | | Soft realtime control loop (Nav2-style controller) | Dedicated high-rate timer in its own MutuallyExclusive group; keep it short; optional `SCHED_FIFO` on that thread — never put DDS waits in that loop. | ## Group types (exact semantics) | Type | Same callback overlapping itself? | Different callbacks in group overlapping? | Overlap with *other* groups? | |------|-----------------------------------|---------------------------------------------|------------------------------| | **MutuallyExclusive** | No | No | **Yes always** | | **Reentrant** | **Yes** | **Yes** | **Yes always** | Critical: **different groups never protect each other**. Cross-group shared state ⇒ **you** lock. ## Sync wait rules (official) If any callback does a **synchronous** wait (`spin_until_future_complete`, sync service call, blocking action result): - Caller callback and the **client** must be in **different groups**, **or** both in one **Reentrant** group. - Executor must be able to run the response path (`MultiThreadedExecutor` with enough threads). - If that conflicts with thread-safety needs → **use async only** (always safe w.r.t. this deadlock). ## Patterns (Nav2 / MoveIt / production stacks) **A — Sensor fan-in + control timer (very common)** ``` sub_scan → MutuallyExclusive group A sub_odom → MutuallyExclusive group B control_timer → MutuallyExclusive group C shared buffers protected by std::mutex (or one mutex per buffer) executor: MultiThreadedExecutor(≥3) ``` Nav2-style: costmaps / controllers take sensor data under lock, run a bounded control tick; never call services from the tick sync. **B — Client used from a callback (Nav2 BT / planner services)** - Prefer `async_send_request` + done callback / future polled from BT tick. - If sync is unavoidable: client in group `client_cbg`, BT/timer in `bt_cbg`, MT executor. **C — Action server (MoveIt execute / Nav2 navigate)** - Goal/cancel callbacks: short, MutuallyExclusive (or default). - Execute may be long: often runs as a **thread** or work queue kicked from execute; cancel must be able to run while execute is active → cancel path not blocked by execute’s group (Reentrant execute group **or** execute off-executor + atomic cancel flag). - Reject or preempt a second goal; do not silently pile up. **D — Composition (Nav2 default bringup)** - Whole server in one composable node; container_mt if internal concurrency needed. - Intra-process does not remove the need for groups/mutexes. ## Mutex / data rules - Lock **data**, not the entire ROS API (never call `publish` / wait while holding a lock another callback needs). - Prefer: copy under lock → process unlocked → publish unlocked. - `std::mutex` in callbacks that share state across groups; `recursive_mutex` only if you know you re-enter (usually a design smell). - Atomics for flags (cancel, “new data”) when that is all you share. - Non-ROS threads (drivers, GPU): same rules — treat them like another callback group. ## Executor choice | Executor | Use | |----------|-----| | `SingleThreadedExecutor` | Default; no blocking waits in callbacks | | `MultiThreadedExecutor` | Concurrent groups; set `number_of_threads` ≥ distinct ready groups you care about | | `StaticSingleThreadedExecutor` | Fixed entity set; slightly cheaper | | `EventsExecutor` | Many entities / event-driven (confirm distro) | `rclcpp::spin(node)` ⇒ single-threaded. Production: construct the executor explicitly. ## Anti-patterns - MT executor + only default group (“I enabled multi-thread”). - Sync service call in a subscription on ST executor. - One giant Reentrant group “so nothing deadlocks” — races everywhere. - Holding a mutex across `spin_until_future_complete`. - Doing planning/vision inside a 10–50 Hz control callback. - `spin_once` from inside a callback already being spun. ## Python (`rclpy`) Same group rules. Wall-timer overrun + Reentrant can stack callbacks — avoid. Web/async: dedicated spin thread; never mix `asyncio` event loop and `spin` on one thread without a bridge. ## Debug checklist 1. Is the wait sync and in a callback? → groups/async. 2. Is MT used without extra groups? → add groups or drop MT. 3. Shared buffers across groups? → mutex/copy. 4. Container ST vs MT? 5. Log thread ids in suspect callbacks to see actual overlap.