miniextendr-worker

name	miniextendr-worker
description	Use when the user asks about the miniextendr worker thread, run_on_worker, with_r_thread, Sendable<T>, the worker-thread cargo feature, why Rust code runs on a separate thread from R, how panics on the worker thread become R errors, SEXP Send constraints, the R main thread invariant, miniextendr_runtime_init, or the 16MB R thread stack limit.

miniextendr Worker Thread

R's C API is single-threaded. All calls to Rf_* functions, SEXP manipulation, and the garbage collector must happen on the thread that loaded the package (the R main thread). miniextendr provides an opt-in worker thread pattern that lets Rust computations run off the main thread while safely calling back into R when needed.

When to use this skill

• "Why does miniextendr run Rust code on a separate thread?"
• "How do I use run_on_worker?"
• "What is with_r_thread?"
• "What is Sendable<T> and when do I need it?"
• "I'm getting a Send bound error involving SEXP."
• "What is the worker-thread feature?"
• "How does miniextendr_runtime_init work?"
• "How do panics on the worker thread propagate to R?"
• "Can I call R API from the worker thread?"

Key concepts

R's threading model

R's runtime has no thread safety mechanism for its GC, symbol table, or C API. Every R API call must happen on the main thread. miniextendr records the main thread ID during package initialization and uses it for debug assertions throughout the FFI layer.

miniextendr_runtime_init

Called from the R_init_<pkg> entry point generated by miniextendr_init!(pkg). It:

1. Records R_MAIN_THREAD_ID via OnceLock — idempotent.
2. With the worker-thread feature: spawns the worker thread and establishes the bidirectional channel pair.
3. Registers the process-level panic hook via crate::backtrace::miniextendr_panic_hook_install.

Must be called from R's main thread before any R API calls.

The corresponding shutdown is miniextendr_runtime_shutdown, called from R_unload_<pkg>. With worker-thread, it sends Shutdown to the worker and blocks on JoinHandle::join() — fully synchronous so library.dynam.unload cannot unmap the DLL while the worker is still executing. See miniextendr-api/src/worker.rs for the WorkerMsg::Shutdown protocol.

run_on_worker — dispatching to the worker thread

#[doc(hidden)]
pub fn run_on_worker<F, T>(f: F) -> Result<T, String>
where
    F: FnOnce() -> T + Send + 'static,
    T: Send + 'static,

Dispatches a closure to the worker thread. Returns Ok(T) on success or Err(String) on panic (the panic message as a string). The Err path also fires PanicSource::Worker telemetry.

The main thread blocks inside a recv loop while the worker runs. During that loop, with_r_thread calls from the worker are served by executing work on the main thread and sending the result back via channel.

Without the worker-thread feature:

• run_on_worker(f) expands to Ok(f()) — the closure runs inline on the current thread.

with_r_thread — calling R API from the worker

From the worker thread (inside a run_on_worker closure), use with_r_thread to execute R API calls on the main thread:

run_on_worker(|| {
    // This runs on the worker thread
    let n: i32 = heavy_computation();

    // This runs on the main thread
    let sexp = with_r_thread(|| {
        unsafe { Rf_ScalarInteger_unchecked(n) }
    });

    sexp
});

with_r_thread is bidirectional: if called from the main thread, it runs the closure inline (re-entrant). If called from the worker, it sends the closure to the main thread and blocks until the result arrives.

Without worker-thread, with_r_thread(f) runs f() directly and panics if called from a non-main thread.

Re-entry guard: calling run_on_worker from inside a run_on_worker closure (nested dispatch) panics immediately with a clear "re-entry" message. The single worker thread cannot pick up a new job while executing the current one. Use with_r_thread for any R API calls needed from within a worker job.

Sendable<T> — bridging non-Send types

R's SEXP is not Send (and cannot safely cross thread boundaries on its own). Sendable<T> is a #[repr(transparent)] newtype that implements Send unconditionally:

#[repr(transparent)]
#[derive(Clone, Copy)]
pub struct Sendable<T>(pub T);

unsafe impl<T> Send for Sendable<T> {}

Use Sendable only when you have established that the value will be used exclusively on the receiving thread. The classic case: wrap a SEXP in Sendable to carry it from the main thread into a worker closure, then use it inside with_r_thread (which runs back on the main thread):

let input = Sendable(input_sexp);  // wraps a SEXP the main thread owns

run_on_worker(move || {
    // input.0 must only be accessed on the main thread
    with_r_thread(move || {
        process(input.0)  // safe: running on main thread
    })
});

SEXP is not Send — automatic main-thread dispatch

Functions with #[miniextendr] that take SEXP arguments do not need unsafe(main_thread) annotations. Because SEXP is not Send, the generated C wrapper body cannot be dispatched to the worker — the code compiles only when it runs on the main thread. This is by design: TryFromSexp conversions happen before the closure boundary, so the worker receives already-converted Rust types.

ALTREP callbacks on main thread

ALTREP callbacks (length, elt, dataptr, serialized_state, extract_subset, etc.) are dispatched by R's own runtime, always on the main thread. They cannot be re-routed to the worker. Inside ALTREP callbacks:

• Call R API directly (with appropriate guard mode set via #[altrep(r_unwind)]).
• Do not call run_on_worker — the single worker thread cannot pick up a new job while blocking on the main thread's recv loop.
• with_r_thread is a no-op (already on main thread) and is safe but unnecessary.

See miniextendr-altrep for the full ALTREP model.

16 MB R thread stack limit

R's main thread has a 16 MB stack. Proptest strategies that recurse deeply or allocate large vecs inside with_r_thread closures will stackoverflow on R's thread. If you use proptest inside with_r_thread, use TestRunner with fork: false and keep strategies simple (e.g., Vec<Option<T>> with at most ~10 cases).

How it works

The worker thread is a blocking recv() loop (worker_loop in worker.rs). The main thread holds a SyncSender<WorkerMsg> with capacity 0 (rendezvous channel). When run_on_worker(f) is called:

1. Main thread sends WorkerMsg::Job(boxed_closure) and enters its own recv loop.
2. Worker receives the job, sets up thread-local channels (WORKER_TO_MAIN_TX, MAIN_RESPONSE_RX), runs the closure inside catch_unwind.
3. If the closure calls with_r_thread(g): worker sends WorkerMessage::WorkRequest(g) to main. Main receives it, executes g inside R_UnwindProtect, sends the result or error back.
4. When the closure finishes, worker sends WorkerMessage::Done(result). Main receives it, returns from run_on_worker.

The rendezvous channel (capacity 0) means at most one job is in flight. The response_tx / response_rx pair (capacity 1) handles the worker-to-main callback flow for with_r_thread calls.

Decision trees

Worker thread or main thread for my Rust code?

• My code calls R API (allocates SEXPs, reads R objects): must run on main thread — use with_r_thread.
• My code is pure Rust computation (no R API): can run on worker thread — use run_on_worker.
• My code mixes both: put the computation in run_on_worker, and the R API parts inside with_r_thread from within that closure.

Do I need Sendable?

• I am crossing a thread boundary with a SEXP or other non-Send type: yes, wrap in Sendable.
• I am only passing Rust types that are already Send (i32, Vec<f64>, etc.): no, just use them directly in the closure.
• I am wrapping a type that is logically main-thread-only but not structurally Send: yes, wrap in Sendable and use it only inside with_r_thread.

I'm hitting Send compile errors — what to do?

• Error: "closure is not Send" in run_on_worker:
  • Inspect which captured variable is not Send. It is probably a SEXP, a Rust type containing SEXP, or a type with Rc.
  • If it is a SEXP: wrap in Sendable and access it only inside with_r_thread.
  • If it is a type with Rc: switch to Arc or restructure to avoid capturing it.
• Error: "cannot be sent between threads safely":
  • Same diagnosis. The closure captures something non-Send.
• The code was compiling before and broke: check if you recently added a SEXP-containing type to the closure capture list.

worker-thread feature: on or off?

Off (default): all code runs inline on the main thread. run_on_worker(f) returns Ok(f()). Suitable for packages where all work is I/O bound or the computation is fast enough not to block R's event loop.

On: a background thread is spawned at package load. Use for CPU-intensive work that would freeze R's UI for more than a second or two. The worker-default feature implies worker-thread.

Key files

• miniextendr-api/src/worker.rs — run_on_worker, with_r_thread, Sendable<T>, miniextendr_runtime_init, miniextendr_runtime_shutdown, worker_loop, dispatch_to_worker, route_to_main_thread, WorkerMsg, WorkerState.
• miniextendr-api/src/unwind_protect.rs — with_r_unwind_protect used inside the main-thread work loop.
• miniextendr-api/src/sys.rs — #[r_ffi_checked] thread assertions; is_r_main_thread() helper.
• miniextendr-api/src/panic_telemetry.rs — PanicSource::Worker telemetry fired on worker panics.

Common pitfalls

• Re-entry deadlock: calling run_on_worker from inside a run_on_worker closure deadlocks because the single worker thread is already busy. The re-entry guard panics immediately with a clear message. Use with_r_thread instead for R API calls from worker code.

• Not initializing runtime: with_r_thread panics with "miniextendr_runtime_init() must be called" if package_init() has not been invoked. This happens when embedding miniextendr in tests without going through R_init_<pkg>. Call miniextendr_runtime_init() directly in test setup.

• SEXP across thread boundary without Sendable: capturing a raw SEXP in a run_on_worker closure is a compile error (SEXP is not Send). Wrap in Sendable, carry across the boundary, and use only inside with_r_thread.

• Accessing Sendable on the worker thread: Sendable only declares Send; it does not make the value thread-safe. Accessing Sendable(sexp).0 directly on the worker thread (instead of inside with_r_thread) is unsound — the GC may run on the main thread concurrently.

• Large stack frames in with_r_thread closures: the R main thread has a 16 MB stack. Deep recursion or large stack allocations inside with_r_thread overflow it. Move large allocations to the heap (Box, Vec) before passing into the closure.

• atexit not registered: miniextendr does not register an atexit handler for the worker shutdown. The normal shutdown path is R_unload_<pkg> → miniextendr_runtime_shutdown. Abnormal paths (process exit without unload) let the OS reap the worker thread. Registering atexit would cause a use-after-unmap crash if the package is unloaded before process exit (issue #277).

Related skills

• miniextendr-ffi — #[r_ffi_checked], _unchecked variants, with_r_unwind_protect, GC protection.
• miniextendr-altrep — why ALTREP callbacks stay on the main thread and cannot use run_on_worker.
• miniextendr-getting-started — overall dev loop, where miniextendr_runtime_init fits in package init.