nanobrain-executors

name

nanobrain-executors

description

Choosing and configuring an Executor (Local, Thread, Process, Parsl, GlobusCompute). Covers the async contract (`process()` must be async), executor-specific deadlock risks, the Parsl/HPC code path (Aurora PBS, ProxyStore, dynamic scaling), the Globus Compute remote-endpoint path (dispatch a step to Aurora via Globus Compute + Globus Auth), per-step `executor_config:` binding, and verbatim error messages. Read whenever you set or change a step's `executor:` field, especially before declaring distributed execution "working".

nanobrain-executors

Distributed silent-failure on missing auto_transfer: When a Parsl worker step writes its output and the downstream DirectLink lacks auto_transfer: true, the worker completes successfully, the executor reports "step done", and no data ever transfers across the process boundary. The trigger cascade silently no-ops. Cross-reference nanobrain-data-units-triggers-links warning. Check every link that crosses an executor boundary.

AcademyManagerWrapper singleton: Per workspace CLAUDE.md, the AcademyManagerWrapper enters a process-wide context on first use. Tests that touch Academy MUST call shutdown_academy_manager() in teardown, otherwise singleton state bleeds across tests.

Parsl worker_init MUST point at .venv/bin/python, not system Python. Workers that load with the wrong interpreter fail with the "missing nanobrain" symptom (see nanobrain-testing-debugging § Wrong Python interpreter).

Why this skill exists

The wrong executor for a workload is one of the most common nanobrain performance bugs:

• LocalExecutor for a CPU-bound step → starves the event loop.
• ThreadExecutor for an async def process → returns a coroutine the thread can't await; deadlock or RuntimeError.
• ProcessExecutor for a step holding non-pickleable state → silent submission failure.
• ParslExecutor outside an HPC environment → silent local fallback (or hang on Parsl initialization).

Beyond that, the framework's distributed code path is partially stubbed. Several "distributed" features are mocks. Verify before claiming they work.

File:line ground truth

Concern	File	Approx. line
ExecutorConfig	nanobrain/nanobrain/core/executor.py	30
LocalExecutor	nanobrain/nanobrain/core/executor.py	512–615
ThreadExecutor	nanobrain/nanobrain/core/executor.py	624–711
ProcessExecutor	nanobrain/nanobrain/core/executor.py	722–810
ParslExecutor	nanobrain/nanobrain/core/executor.py	936–1459
Parsl execution context detection	nanobrain/nanobrain/core/executor.py	1003–1127
Parsl app for step execution	nanobrain/nanobrain/core/executor.py	822–905
build_executor_from_config (shared dispatch)	nanobrain/nanobrain/core/executor.py	1978+
GlobusComputeExecutor + GlobusComputeConfig	nanobrain/nanobrain/core/distributed/globus_compute_executor.py	full file
build_globus_app (shared Globus Auth helper, 3-tier creds)	nanobrain/nanobrain/core/distributed/globus_auth.py	full file
globus_credentials (keyring secure store + insecure-backend guard)	nanobrain/nanobrain/core/distributed/globus_credentials.py	full file
apecx-globus-setup CLI (store / status / test / endpoint-config / clear)	apecx-mcp-integration/src/apecx_integration/cli/globus_setup.py	full file
GlobusTransferStep (data staging)	nanobrain/nanobrain/library/steps/globus_transfer_step.py	full file
Per-step executor_config: binding	nanobrain/nanobrain/core/step.py	resolve_dependencies ~820
DynamicExecutorConfigGenerator	nanobrain/nanobrain/core/dynamic_executor_config.py	30–248
PBSResourceManager	nanobrain/nanobrain/core/pbs_resource_manager.py	66–340
DistributedResourceRegistry	nanobrain/nanobrain/core/distributed_resource_registry.py	full file
WorkerStepPool	nanobrain/nanobrain/core/worker_step_pool.py	34–250
ResourceMonitor	nanobrain/nanobrain/core/resource_monitor.py	78–348
Distributed workflow stub	nanobrain/nanobrain/core/distributed/workflow_execution.py	1–103

Decision matrix

You have...	Use
Single-threaded async work, dev/testing, integration tests	LocalExecutor
I/O-bound work (API calls, disk, network), < 10 workers	ThreadExecutor
CPU-bound, pickleable state, no shared memory	ProcessExecutor
HPC cluster (Aurora, PBS) with parallel nodes, you control the Parsl config	ParslExecutor (with caveats below)
Dispatch a step to a remote Globus Compute endpoint (e.g. a managed endpoint on Aurora), no SSH, Globus Auth	GlobusComputeExecutor (see below)
Anything that includes mock:// URLs or claims to mock distributed exec	Stop and verify

ExecutorConfig common fields

executor_type: local                          # local | thread | process | parsl | globus_compute
name: my_executor
max_workers: 4
timeout: 60                                   # seconds
parsl_config: { ... }                         # Parsl only
globus_compute: { ... }                       # GlobusCompute only (see below)
default_resource_specification: { ... }       # Parsl only

LocalExecutor

class: "nanobrain.core.executor.LocalExecutor"
config:
  executor_type: local
  name: local_dev
  max_workers: 1
  timeout: 30

Behavior: runs in the current event loop. Async functions are wrapped in asyncio.create_task(). Sync functions are called directly. If a sync function returns a coroutine, the executor awaits it.

Best for: tests, smoke verification, single-machine work.

ThreadExecutor

class: "nanobrain.core.executor.ThreadExecutor"
config:
  executor_type: thread
  name: io_pool
  max_workers: 8
  timeout: 60

Behavior: submits to ThreadPoolExecutor via loop.run_in_executor. Threads cannot await coroutines. If your process is async def, the thread runs a coroutine object — undefined behavior, almost certainly a bug.

Best for: I/O-bound sync code (legacy libraries, blocking SDKs).

If your step is async def process, do NOT use ThreadExecutor for the step itself. Use LocalExecutor and await asyncio.to_thread(...) inside process for the blocking parts.

ProcessExecutor

class: "nanobrain.core.executor.ProcessExecutor"
config:
  executor_type: process
  name: cpu_pool
  max_workers: 4
  timeout: 300

Behavior: submits to ProcessPoolExecutor. Tasks must be pickleable — no closures, no lambdas, no references to non-pickleable objects. Failures at submission can be silent.

Best for: CPU-bound pure functions with serializable state.

ParslExecutor

class: "nanobrain.core.executor.ParslExecutor"
config:
  executor_type: parsl
  name: aurora_pssm
  max_workers: 12
  timeout: 3600
  parsl_config:
    strategy: null
    app_cache: true
    checkpoint_mode: task_exit
    retries: 1
    executors:
      - label: aurora_pssm_htex
        class: parsl.executors.HighThroughputExecutor
        max_workers_per_node: 4
        cores_per_worker: 1
        worker_debug: true
        heartbeat_period: 30
        provider_config:
          class: parsl.providers.PBSProProvider
          queue: debug
          account: FoundEpidem
          nodes_per_block: 1
          cpus_per_node: 4
          walltime: "00:25:00"
          worker_init: |
            #!/bin/bash
            module load frameworks
            source /home/onarykov/miniconda3/etc/profile.d/conda.sh
            conda activate nanobrain-upd
            export PYTHONPATH="/home/onarykov/nanobrain:$PYTHONPATH"
          parallelism: 1.0
  fallback:
    enable_fallback: false
    executor_type: local
    max_workers: 4

For local development, use a LocalProvider instead of PBSProProvider:

parsl_config:
  executors:
    - label: htex_local_fast
      class: parsl.executors.HighThroughputExecutor
      max_workers_per_node: 2
      provider_config:
        class: parsl.providers.LocalProvider
        min_blocks: 1
        init_blocks: 1
        max_blocks: 1

Parsl execution context detection

ParslExecutor detects whether it's running inside an existing PBS allocation (via PBS_JOBID and PARSL_WORKER_RANK env vars). Three modes:

• Top-level (no PBS env): uses your specified executor (HTEX, WorkQueue).
• Subworkflow (inside Parsl worker): forces WorkQueueExecutor + LocalProvider to reuse the existing allocation. You don't get to override.
• Local (no PBS): uses LocalProvider.

Async contract on Parsl

Parsl detects asyncio.iscoroutinefunction(step.process); if true, the worker runs an event loop and uses asyncio.run_until_complete(). So async def process works on Parsl.

Required environment for Aurora HPC

• PBS_JOBID, PBS_NCPUS, PBS_NODEFILE — set by PBS.
• PYTHONPATH — must include nanobrain source root.
• conda activate nanobrain-upd (or your env) in worker_init.
• proxystore installed (for cross-node resource sharing).
• Shared filesystem (e.g., /lus/flare) for ProxyStore file connector.

GlobusComputeExecutor

Dispatches a step's process() to a remote Globus Compute endpoint (e.g. a managed endpoint on Aurora). Unlike ParslExecutor — which needs you to author a Parsl config and, for PBS, to be on a login node — the endpoint owns the Parsl/PBS config; the client only needs the endpoint UUID + Globus Auth. Works from anywhere (no SSH).

class: "nanobrain.core.executor.ExecutorConfig"     # the config class
config:
  executor_type: globus_compute
  globus_compute:
    endpoint_id: "${AURORA_GC_ENDPOINT_ID}"          # YOUR endpoint UUID
    auth_mode: client_credentials                    # client_credentials | native
    # client_credentials reads $GLOBUS_COMPUTE_CLIENT_ID / _SECRET
    # (or pass client_id / client_secret here explicitly)
    task_timeout_seconds: 3600                        # HPC queue waits are slow
    resource_specification: { ... }                   # optional, endpoint-dependent
    user_endpoint_config: { ... }                     # optional, multi-user endpoints

Dispatch contract (approach B — same as ParslExecutor): the executor introspects the step's execute_wrapper closure, extracts (step_config_path, step_class_name, input_data), and ships ONLY those — a module-level _run_step_on_endpoint function on the worker does importlib → step_class.from_config(path) → process(). No live nanobrain object is pickled across the wire. Therefore the remote endpoint MUST have: the same nanobrain (pin the commit — version skew is a silent-failure risk), the step's package + deps, and the step's config YAML resolvable on its filesystem. The step must be loaded from a YAML file (it needs _config_path or config.source_path) — not an inline dict.

Globus Auth is via the shared build_globus_app helper (core/distributed/globus_auth.py). client_credentials (a confidential client) is the default and the headless/automation mode; native is an interactive browser login. A confidential client is never silently downgraded to interactive — missing client_id/client_secret FAIL-LOUDs.

Credential resolution is 3-tier: explicit config args → environment variables ($GLOBUS_COMPUTE_CLIENT_ID / _SECRET) → the OS keychain (via core/distributed/globus_credentials.py, keyring-backed). Store once with the apecx-globus-setup store CLI and you set nothing else; CI uses env vars; an explicit arg always wins. globus_credentials.py has an insecure-backend guard — store_credentials FAIL-LOUDs if the active keyring backend is fail.Keyring or a plaintext backend rather than silently writing the secret in the clear. The apecx-globus-setup CLI (apecx_integration/cli/globus_setup.py) also has status, test [--endpoint-id] [--round-trip] (the real auth + endpoint + dispatch verification), endpoint-config, and clear.

FAIL-LOUD, no silent fallback: missing globus_compute_sdk, missing endpoint_id, auth failure, a remote exception on the endpoint (re-raised locally with the endpoint's traceback), and a timeout all raise. A "successful" dispatch that produced no usable result raises.

Route 2 (also valid): parsl 2026.5.4 ships parsl.executors.GlobusComputeExecutor. A parsl_config whose executor class is that makes the existing ParslExecutor dispatch through Globus Compute with zero new code — but auth + endpoint config get buried in the Parsl config dict. Prefer GlobusComputeExecutor (Route 1) for explicit auth + FAIL-LOUD control.

Setup runbook for an Aurora endpoint: apecx-mcp-integration/docs/globus_compute_aurora_runbook.md. Companion data-staging step: GlobusTransferStep (below).

GlobusTransferStep — data staging

GlobusComputeExecutor moves code execution; it does not move files. For a step whose inputs/outputs are large files on a remote filesystem, stage them with GlobusTransferStep (nanobrain.library.steps.globus_transfer_step.GlobusTransferStep) before / after the compute step — it wraps globus_sdk.TransferClient, polls the transfer task to completion, and FAIL-LOUDs on a failed/timed-out transfer. It shares the same build_globus_app auth helper, so one confidential client authorizes both Compute and Transfer.

Per-step executor_config: binding

A step YAML can bind itself to a non-local executor via executor_config: — this is how you put one step of a workflow on a remote executor while the others run locally. Because ExecutorConfig is a ConfigBase (no inline-dict construction), use the class: + config: indirection:

# my_step.yml
class: "my_project.steps.MyStep"
name: my_step
# ... the step's own data units / triggers ...
executor_config:
  class: "nanobrain.core.executor.ExecutorConfig"
  config: "my_step_executor.yml"      # a separate ExecutorConfig YAML

# my_step_executor.yml
executor_type: globus_compute
globus_compute:
  endpoint_id: "${AURORA_GC_ENDPOINT_ID:-unset}"
  auth_mode: client_credentials

Executor precedence in BaseStep.resolve_dependencies (highest first):

1. a pre-built executor object passed programmatically;
2. the step's own executor_config: (this path — built via the shared build_executor_from_config dispatch);
3. the workflow-level executor (inherited);
4. default LocalExecutor.

An unknown executor_type, or an invalid globus_compute block, FAIL-LOUDs at step-build time — it never silently falls back to Local.

Resource management subsystems

Component	Role
PBSResourceManager	Discovers PBS nodes via pbsnodes -a -F json; allocates cores per workflow
DynamicExecutorConfigGenerator	Generates per-workflow Parsl config so concurrent workflows don't oversubscribe
DistributedResourceRegistry	ProxyStore-backed registry for cross-node resource discovery
WorkerStepPool	Per-worker step instances for parallel processing
ResourceMonitor	Watches disk space; pauses workflow when critical
ProgressiveScalingMixin	Optional mixin for steps to start small and scale up

Stubbed / mock paths to be aware of

The framework's own CLAUDE.md states "Mock implementations for many distributed features." Concrete mock sites:

1. distributed/workflow_execution.py:1–103 — execute_workflow_distributed Parsl app is stubbed; not integrated end-to-end. Distributed workflow execution may silently fall back to local.
2. mcp_support.py:343–451 — when aiohttp missing, MCP client is mocked. mock:// URLs trigger mock mode.
3. distributed_resource_registry.py:68 — ProxyStore optional; without it, cross-node registry doesn't work but no error is raised.
4. pbs_resource_manager.py:135, 153 — defaults to "8 cores per node" if pbsnodes data is absent. May undersubscribe non-standard configurations.
5. Parsl resource_specification (executor.py:1478–1491) — collected from YAML and passed through, but the Parsl worker app does nothing with it.
6. WorkerStepPool shutdown — incomplete in the version analyzed; check for resource leaks in long-running pools.

Per workspace policy: do not call distributed execution "tested" until you have run it against real Parsl on real nodes with real (small) data and recorded the outcome.

Verbatim error messages

ImportError: Parsl not available. Install with: pip install parsl

Install Parsl, or change the executor to non-Parsl.

RuntimeError: ParslExecutor not initialized

Awaited an execute call before await executor.initialize().

ValueError: Step instance missing _config_path for Parsl execution

The Parsl app needs to recreate the step on the worker via step_class.from_config(step_config_path). Step instances created from inline dicts lack _config_path. Use a YAML file.

RuntimeError: Parsl execution failed: {error}\n{traceback}

Read the worker-side traceback. Common: missing module on worker (PYTHONPATH), import error, missing data file (shared FS path mismatch).

RuntimeError: Both PARSL and fallback execution failed.
PARSL error: {original}, Fallback error: {fallback}

Both paths broken. Inspect both errors; usually the original Parsl error is the one to fix.

FAIL-FAST: GlobusComputeExecutor requires the 'globus_compute_sdk' package

pip install globus-compute-sdk in the client-side venv. The dependency is only needed when a workflow actually selects executor_type: globus_compute.

FAIL-FAST: build_globus_app auth_mode='client_credentials' requires
confidential-client credentials. Missing: ...

Export $GLOBUS_COMPUTE_CLIENT_ID / $GLOBUS_COMPUTE_CLIENT_SECRET (or pass client_id / client_secret in the config). A confidential client is never silently downgraded to interactive login.

GlobusComputeExecutor: step execution FAILED on endpoint ... Remote traceback: ...

The endpoint worker raised. Common: ModuleNotFoundError: nanobrain (the endpoint's worker_init did not activate an env with nanobrain), or FileNotFoundError on the step config path (stage it to the endpoint's filesystem — see the Aurora runbook + GlobusTransferStep).

ValueError: Invalid config type: {type}. Expected str or ExecutorConfig

You passed something other than a path or ExecutorConfig to from_config.

ImportError: ProxyStore not installed.
Install with: pip install 'nanobrain[academy]' or pip install proxystore

Install ProxyStore for distributed resource registry.

Pitfalls

1. ThreadExecutor for an async-def process. Subtle and often hangs.
2. ProcessExecutor for a step that captures unpickleable state. Silent submission failure or pickle error.
3. Parsl with no PBS_JOBID claiming "distributed". It's actually local.
4. Parsl worker_init script that doesn't activate the right conda env. Worker can't import nanobrain; cryptic ImportError.
5. Step uses inline dict config, then runs on Parsl. Worker can't recreate the step (no _config_path).
6. Two workflows sharing the same Parsl config. Oversubscription on PBS; use DynamicExecutorConfigGenerator.
7. Disk fills during a long Parsl run; ResourceMonitor not enabled. Workflow crashes mid-execution.
8. Trusting mock:// URLs as real MCP. Always verify with aiohttp import and a real server URL.

Checklist

• Executor matches the workload class (Local for async dev, Thread for blocking I/O, Process for CPU-bound pickleable, Parsl for HPC).
• If process is async def, executor is Local or Parsl (not Thread/Process directly).
• Parsl config explicitly sets worker_init, walltime, cpus_per_node.
• On Parsl, every step is loaded from a YAML file (not inline dict).
• PYTHONPATH includes the project root in worker_init.
• aiohttp and proxystore installed if you depend on MCP / distributed registry.
• If using enable_fallback: true, you've tested both the Parsl path AND the fallback path.
• Distributed execution tested against real cluster on small real data, recorded.

Tool-backend adapters (2026-05-09 -> 2026-05-11)

Distinct from step executor: (which controls how the Step itself runs — local / thread / process / Parsl), tool-backend adapters control how individual TOOL CALLS dispatch. Three concrete adapters ship in-tree:

• rhea — Rhea fork's RheaMCPDispatcher (remote MCP)
• local_parsl — G11-completion LocalParslAdapter (local Python callables via Parsl). Default preset is ThreadPoolExecutor (P0++b).
• http — G38 HTTPBackendAdapter (generic HTTP)

Adapters live at nanobrain.library.tools.* and register via ToolBackendRegistry.register(adapter). ToolExecutionStep consults the registry by BACKEND_NAME derived from the UTD's descriptor_id prefix. See nanobrain-agents-tools skill for picker.