subsystem-summary-of-process

# Process Subsystem — Technical Summary ## Overview The process subsystem provides asynchronous subprocess management for stellar-core, wrapping platform-specific process spawning (POSIX `posix_spawnp` and Windows `CreateProcess`) behind a unified asio-integrated interface. It enables running external commands (e.g., history archival tools like `gzip`, `gunzip`, `curl`) asynchronously, with configurable concurrency limits, output file capture, graceful shutdown, and process lifecycle tracking. No facilities exist for reading/writing subprocess I/O ports — this is strictly for "run a command, wait to see if it worked." ## Key Files - **ProcessManager.h** — Abstract interface `ProcessManager` and the `ProcessExitEvent` class for async process completion notification. - **ProcessManagerImpl.h / ProcessManagerImpl.cpp** — Concrete implementation of `ProcessManager`; contains all lifecycle management, platform-specific spawning, signal handling, and shutdown logic. - **PosixSpawnFileActions.h / PosixSpawnFileActions.cpp** — POSIX-only RAII wrapper around `posix_spawn_file_actions_t` for redirecting subprocess stdout to a file. --- ## Key Classes and Data Structures ### `ProcessManager` (abstract, inherits `std::enable_shared_from_this<ProcessManager>`, `NonMovableOrCopyable`) The public interface for subprocess management. One `ProcessManager` exists per `Application` instance, created via the static factory `ProcessManager::create(Application&)`. **Key virtual methods:** - `runProcess(cmdLine, outputFile)` → `std::weak_ptr<ProcessExitEvent>` — Queues or immediately launches a subprocess. If `outputFile` is non-empty, stdout is captured to a temp file and atomically renamed on success. - `getNumRunningProcesses()` — Count of active (non-shutting-down) child processes. - `getNumRunningOrShuttingDownProcesses()` — Count of all tracked child processes including those being terminated. - `tryProcessShutdown(pe)` — Synchronously cancels a `ProcessExitEvent` and attempts to terminate the associated process (SIGTERM on POSIX, `GenerateConsoleCtrlEvent` on Windows). Returns `true` if the termination signal was sent successfully. - `shutdown()` — Marks the manager as shut down, cancels all pending processes, and attempts polite shutdown of all running processes. - `isShutdown()` — Returns whether shutdown has been initiated. ### `ProcessExitEvent` An asio-compatible event object that clients use to await subprocess completion. It simulates an event notifier using a `RealTimer` set to maximum duration. When the subprocess exits (or is cancelled), the timer is cancelled with an appropriate error code. **Members:** - `mTimer` (`std::shared_ptr<RealTimer>`) — The underlying asio timer used for async waiting. - `mImpl` (`std::shared_ptr<Impl>`) — Platform-specific implementation details (command line, output file, process handle, lifecycle state). - `mEc` (`std::shared_ptr<asio::error_code>`) — Shared error code for communicating the real exit status past asio's timer cancellation (which always delivers `operation_aborted`). **Key methods:** - `async_wait(handler)` — Registers a callback that fires when the process exits. The handler receives an `asio::error_code` where a zero value means success, and a non-zero value encodes the process exit status. ### `ProcessExitEvent::Impl` (internal, inherits `std::enable_shared_from_this`) Holds all per-process state and manages the platform-specific spawning logic. **Members:** - `mOuterTimer` / `mOuterEc` — Shared pointers back to the owning `ProcessExitEvent`'s timer and error code. - `mCmdLine` (`std::string const`) — The command line to execute. - `mOutFile` (`std::string const`) — The desired final output file path (may be empty if no capture). - `mTempFile` (`std::string const`) — Temporary file path for capturing stdout before atomic rename. - `mLifecycle` (`ProcessLifecycle`) — Tracks the process through its states. - `mProcessId` (`int`) — PID of the spawned process (-1 before launch). - `mProcessHandle` (Windows only, `asio::windows::object_handle`) — Waitable handle for the Windows process object. - `mProcManagerImpl` (`std::weak_ptr<ProcessManagerImpl>`) — Weak back-reference to the owning manager. **Key methods:** - `run()` — Platform-specific process spawning. On POSIX: splits the command line, sets up `PosixSpawnFileActions` for output redirection, sets `FD_CLOEXEC` on all open file descriptors ≥ 3, then calls `posix_spawnp()`. On Windows: sets up `STARTUPINFOEX` with inheritable handles, calls `CreateProcess()`, and registers an async wait on the process handle. - `finish()` — Called on termination; renames the temp file to the final output file (atomic move). Returns `false` if the rename fails. - `cancel(ec)` — Sets the outer error code and cancels the outer timer, firing all registered `async_wait` handlers. ### `ProcessManagerImpl` (inherits `ProcessManager`) The concrete implementation that manages the full lifecycle of subprocesses. **Members:** - `mProcessesMutex` (`std::recursive_mutex`) — Guards `mProcesses` and `mPending` since subprocess exits arrive asynchronously. - `mProcesses` (`std::map<int, std::shared_ptr<ProcessExitEvent>>`) — Maps PID to running or shutting-down processes. - `mPending` (`std::deque<std::shared_ptr<ProcessExitEvent>>`) — Queue of processes waiting to be launched (when concurrency limit is reached). - `mIsShutdown` (`bool`) — Set once `shutdown()` is called. - `mMaxProcesses` (`size_t const`) — Maximum concurrent subprocesses, from `Config::MAX_CONCURRENT_SUBPROCESSES`. - `mIOContext` (`asio::io_context&`) — The application's I/O context for async operations. - `mSigChild` (`asio::signal_set`) — On POSIX, listens for `SIGCHLD` to detect child process exits. Unused on Windows. - `mTmpDir` (`std::unique_ptr<TmpDir>`) — Temporary directory for capturing subprocess output files. - `mTempFileCount` (`uint64_t`) — Monotonic counter for generating unique temp file names. ### `ProcessLifecycle` (enum, anonymous namespace) Tracks the state of a subprocess through its lifetime: - `PENDING` (0) — Queued, not yet spawned. - `RUNNING` (1) — Spawned, waiting for exit. - `TRIED_POLITE_SHUTDOWN` (2) — SIGTERM (POSIX) or CTRL_C_EVENT (Windows) sent. - `TRIED_FORCED_SHUTDOWN` (3) — SIGKILL (POSIX) or TerminateProcess (Windows) sent. - `TERMINATED` (5) — Exit detected and handled. ### `PosixSpawnFileActions` (POSIX only) RAII wrapper around `posix_spawn_file_actions_t`. Lazily initializes the actions object on first `addOpen()` call. Provides an implicit conversion to `posix_spawn_file_actions_t*` (returns `nullptr` if never initialized, meaning no file actions). **Key methods:** - `addOpen(fildes, fileName, oflag, mode)` — Registers a file-open action for the child process (used to redirect fd 1 / stdout to a temp file). - `initialize()` — Calls `posix_spawn_file_actions_init()`; idempotent. - Destructor calls `posix_spawn_file_actions_destroy()` if initialized. --- ## Key Control Flows ### Process Launch Flow 1. Client calls `ProcessManagerImpl::runProcess(cmdLine, outFile)`. 2. A new `ProcessExitEvent` is created with a `RealTimer` set to max duration. 3. A `ProcessExitEvent::Impl` is created holding the command line, output file, a generated temp file path, and a weak reference to the manager. 4. The event is pushed onto `mPending`. 5. `maybeRunPendingProcesses()` is called, which pops events from `mPending` while `getNumRunningOrShuttingDownProcesses() < mMaxProcesses`. 6. For each dequeued event, `Impl::run()` is called: - **POSIX:** Command line is split on whitespace into argv. `PosixSpawnFileActions` is set up if output capture is needed. All file descriptors ≥ 3 are marked `FD_CLOEXEC`. `posix_spawnp()` is called. Lifecycle transitions to `RUNNING`. - **Windows:** `STARTUPINFOEX` and handle inheritance are configured. `CreateProcess()` is called with `CREATE_NEW_PROCESS_GROUP`. An async wait is registered on the process handle. Lifecycle transitions to `RUNNING`. 7. The PID is recorded in `mProcesses`. 8. A `weak_ptr<ProcessExitEvent>` is returned to the caller, who calls `async_wait()` to register a completion handler. ### Process Exit Handling (POSIX) 1. `SIGCHLD` arrives, handled by `asio::signal_set` → `handleSignalChild()`. 2. `handleSignalChild()` re-registers the signal handler (via `startWaitingForSignalChild()`), then calls `reapChildren()`. 3. `reapChildren()` iterates all tracked PIDs, calling `waitpid(pid, &status, WNOHANG)` for each. 4. For each successfully reaped child, `handleProcessTermination(pid, status)` is called. 5. `handleProcessTermination()` maps the exit status to an `asio::error_code` (via `mapExitStatusToErrorCode`), calls `Impl::finish()` to rename the temp output file, removes the process from `mProcesses`, calls `maybeRunPendingProcesses()` to launch queued processes, then fires the callback via `Impl::cancel(ec)`. ### Process Exit Handling (Windows) 1. The `asio::windows::object_handle::async_wait` fires when the process handle becomes signaled. 2. The callback calls `GetExitCodeProcess()` and passes the result to `handleProcessTermination()`. 3. The rest follows the same flow as POSIX. ### Shutdown Flow 1. `shutdown()` sets `mIsShutdown = true`. 2. All pending (not yet launched) processes are cancelled with `ABORT_ERROR_CODE`. 3. `tryProcessShutdownAll()` iterates all running processes and calls `tryProcessShutdown()` on each. 4. `tryProcessShutdown()` uses a two-phase approach based on lifecycle state: - If `RUNNING`: calls `politeShutdown()` (SIGTERM / CTRL_C_EVENT), advances to `TRIED_POLITE_SHUTDOWN`. - If `TRIED_POLITE_SHUTDOWN`: calls `forcedShutdown()` (SIGKILL / TerminateProcess), advances to `TRIED_FORCED_SHUTDOWN`. 5. The destructor (`~ProcessManagerImpl()`) ensures cleanup: it cancels the `SIGCHLD` handler, calls `shutdown()`, then loops up to 3 times sleeping 10ms, reaping children, and re-triggering progressively more forceful shutdown. ### Concurrency Control - The maximum number of concurrent subprocesses is controlled by `mMaxProcesses` (from `Config::MAX_CONCURRENT_SUBPROCESSES`). - When the limit is reached, new processes are queued in `mPending` (a FIFO deque). - After each process exit (`handleProcessTermination`), `maybeRunPendingProcesses()` is called to launch queued processes up to the limit. - All access to `mProcesses` and `mPending` is guarded by `mProcessesMutex` (a recursive mutex), since process exits arrive asynchronously from signal handlers or async I/O callbacks. --- ## Ownership Relationships ``` Application └── ProcessManagerImpl (shared_ptr, via ProcessManager::create) ├── mProcesses: map<pid, shared_ptr<ProcessExitEvent>> │ └── ProcessExitEvent │ ├── mTimer: shared_ptr<RealTimer> │ ├── mEc: shared_ptr<asio::error_code> │ └── mImpl: shared_ptr<ProcessExitEvent::Impl> │ └── mProcManagerImpl: weak_ptr<ProcessManagerImpl> (back-ref) ├── mPending: deque<shared_ptr<ProcessExitEvent>> ├── mSigChild: asio::signal_set (POSIX only) └── mTmpDir: unique_ptr<TmpDir> ``` - `ProcessManagerImpl` owns all `ProcessExitEvent` objects (via `mProcesses` and `mPending`). - `ProcessExitEvent::Impl` holds a `weak_ptr` back to `ProcessManagerImpl` to avoid circular ownership. - Callers receive a `weak_ptr<ProcessExitEvent>` from `runProcess()`, so the manager controls the event's lifetime. - The `Impl::run()` method on Windows captures a `shared_from_this()` to keep `Impl` alive through the async wait callback. --- ## Key Data Flows 1. **Command → Process:** `runProcess(cmdLine, outFile)` → queued in `mPending` → dequeued by `maybeRunPendingProcesses()` → `Impl::run()` spawns the OS process. 2. **Process Exit → Callback:** OS signal (SIGCHLD) or handle wait → `reapChildren()` / handle callback → `handleProcessTermination()` → `Impl::finish()` (rename temp file) → `Impl::cancel(ec)` → timer cancelled → `async_wait` handler fires with exit code. 3. **Output Capture:** Subprocess stdout is redirected to a temp file in `mTmpDir`. On successful termination, `Impl::finish()` atomically renames the temp file to the requested output file path. If the output file already exists, the rename fails and an error is propagated. 4. **Exit Code Mapping:** `mapExitStatusToErrorCode()` translates OS-level exit status (including POSIX `WIFEXITED`/`WEXITSTATUS` macros) into `asio::error_code`. Special handling for exit code 127 on Linux (likely missing command). On Windows, the exit code is used directly. 5. **Shutdown Signal Flow:** `shutdown()` → cancel pending → polite shutdown (SIGTERM) → forced shutdown (SIGKILL) → destructor reaps remaining children with retry loop. --- ## Platform Differences | Aspect | POSIX | Windows | |--------|-------|---------| | Process spawning | `posix_spawnp()` | `CreateProcess()` with `EXTENDED_STARTUPINFO_PRESENT` | | Exit detection | `SIGCHLD` via `asio::signal_set` + `waitpid(WNOHANG)` | `asio::windows::object_handle::async_wait` | | Output redirection | `posix_spawn_file_actions_addopen()` on fd 1 | `CreateFile()` + `STARTF_USESTDHANDLES` | | Polite shutdown | `kill(pid, SIGTERM)` | `GenerateConsoleCtrlEvent(CTRL_C_EVENT, pid)` | | Forced shutdown | `kill(pid, SIGKILL)` | `TerminateProcess()` | | FD cleanup | `FD_CLOEXEC` on fds 3..SC_OPEN_MAX (with gap heuristic) | Handle inheritance via `PROC_THREAD_ATTRIBUTE_HANDLE_LIST` | --- ## Error Handling Notes - If `posix_spawnp()` or `CreateProcess()` fails, the `ProcessExitEvent` is cancelled with `std::errc::io_error` and the error is logged. - On Linux, exit code 127 triggers a prominent warning about a likely missing command (since `posix_spawnp` does not fault on file-not-found in the parent). - The timer-based `async_wait` pattern works around asio always delivering `operation_aborted` on timer cancel: the real error code is stored in a shared `mEc` variable and the handler reads from that instead of using the asio-provided code. - `checkInvariants()` validates consistency: pending processes must be in `PENDING` state, running processes must not be `PENDING`, and PIDs must match map keys. Called at key state transitions.