matlab-set-up-worker-state

name	matlab-set-up-worker-state
description	Set up worker environment and per-worker state for parallel pools. Use when code needs paths, environment variables, database connections, loaded libraries, or expensive objects available on workers before parfor/parfeval runs. Teaches parallel.pool.Constant, parfevalOnAll, and parpool name-value pairs. Also use when refactoring existing code that uses spmd for side-effect setup (an anti-pattern). Triggers: worker setup, pool constant, per-worker state, non-serializable, loadlibrary on workers, database connection parfor, addpath workers, spmd before parfor, worker environment, reduce parfor overhead, parfor setup, resource creation in parallel loop, cannot serialize error, undefined function or variable on workers error, load data per worker, reduce data transfer, parallelize setup, improve parallel code.
license	https://www.mathworks.com/content/dam/mathworks/license/pmrl/license.md
metadata	{"author":"MathWorks","version":"2.0"}

Set Up Worker State for Parallel Pools

By default, process workers in a parallel pool inherit MATLAB path state from their controlling client, but they do not inherit loaded libraries, open connections, or expensive pre-computed objects. Code that relies on any of these needs explicit setup. This skill teaches the correct APIs for each scenario. Most patterns target process-based pools; thread pool applicability is noted where relevant.

When to Use

• Code needs a non-serializable resource on workers (database connections, COM objects, loaded shared libraries, file handles)
• Code needs expensive one-time setup per worker (large object construction, data loading) that should not repeat every parfor iteration
• Code needs paths or environment variables set on workers
• User has existing code using spmd for side-effect setup before parfor (anti-pattern — help them modernise)
• User sees errors about objects not being serializable when passed to parfor

When NOT to Use

• Data already in client memory used in a single parfor loop (MATLAB broadcasts it automatically — Constant adds complexity for no benefit in this case)
• Choosing between process and thread pools (out of scope — this skill assumes a pool type is already chosen)

Decision Framework

Scenario	Correct API	Why
Non-serializable resource needing cleanup (connections, libraries)	parallel.pool.Constant(@buildFcn, @cleanupFcn)	Constructs on each worker; automatic cleanup on delete
Data from a file needed on workers	parallel.pool.Constant(@() load(file).var)	Default for file-based data. Each worker loads from disk; client never holds the dataset. Always prefer this when the source is a file.
Data already in client memory, used in multiple parfor loops	parallel.pool.Constant(data)	Transfers once for pool lifetime; without Constant, broadcast re-sends for every parfor loop. Not needed for a single parfor — let MATLAB broadcast.
One-shot side effect, no return value needed	parfevalOnAll(pool, @fcn, 0)	Runs once on all workers; use fetchOutputs to surface errors
Paths needed on workers — parpool call is in your code	parpool(..., AdditionalPaths=paths)	Cleanest option when you can modify the parpool call; client path entries are inherited automatically
Paths needed on workers — pool opened elsewhere (can't modify call)	parfevalOnAll(pool, @addpath, 0, p)	Never delete and recreate a pool just to add paths — use parfevalOnAll on the existing pool
Environment variables on workers — parpool call is in your code	parpool(..., EnvironmentVariables=vars)	Forwards named env vars from client to workers at startup; set values with setenv on the client before this call
Environment variables on workers — pool opened elsewhere (can't modify call)	parfevalOnAll(pool, @setenv, 0, k, v)	Never delete and recreate a pool just to set env vars — use parfevalOnAll on the existing pool

Thread pool notes: parallel.pool.Constant and parfevalOnAll also work on thread pools. However, thread workers share the client's process, so path and environment variable changes on the client are visible to threads automatically — AdditionalPaths and EnvironmentVariables do not apply. To modify a thread worker's environment, alter the client environment before the parfor.

The spmd Anti-Pattern

This applies equally to process pools and thread pools — prefer parallel.pool.Constant over spmd for worker state setup regardless of pool type.

What it looks like

% ANTI-PATTERN: Do not do this
spmd
    loadlibrary("mylib", "mylib.h");
end
parfor i = 1:100
    result(i) = calllib("mylib", "compute", data(i));
end
spmd
    unloadlibrary("mylib");
end

Why it's fragile

1. No cleanup guarantee — if the parfor errors, the second spmd never runs
2. Makes the pool fragile to worker disconnection — once a pool has run an spmd block, a single worker losing connection tears down the entire pool. Pools that never use spmd continue operating with fewer workers if one disconnects (no replacement occurs — the pool simply shrinks).
3. Composite variables cannot be used inside parfor — values created in spmd are Composite objects that require indexing on the client; they cannot be referenced directly from within a parfor body

How to modernise

Replace with parallel.pool.Constant:

C = parallel.pool.Constant( ...
    @() loadAndReturn(), ...
    @(~) unloadlibrary("mylib")); %#ok<NASGU> — kept alive for cleanup

result = zeros(1, 100);
parfor i = 1:100
    result(i) = calllib("mylib", "compute", data(i));
end

% Cleanup happens automatically when C goes out of scope

Or with parfevalOnAll for simple side effects:

f = parfevalOnAll(pool, @setupWorker, 0);
fetchOutputs(f);

result = zeros(1, 100);
parfor i = 1:100
    result(i) = doWork(data(i));
end
% No automatic cleanup

Prefer parallel.pool.Constant when cleanup is needed. Use parfevalOnAll only for operations where cleanup is not required.

Patterns

Pattern 1: Loading data from a file for use in parfor

Always use the build-function form when the source is a file. This avoids loading the data into client memory entirely — each worker loads directly from disk. This is critical for large files but is also the correct default for any file-based data because it scales without code changes as file size grows.

c = parallel.pool.Constant(@() load("costSurface.mat").costSurface);

results = zeros(1, 10000);
parfor i = 1:10000
    results(i) = processWithLookup(input(i), c.Value);
end

Each worker calls load() once. The client never holds the full dataset.

Do NOT do this — loading on the client then wrapping defeats the purpose:

% WRONG: loads entire file into client memory, then copies to each worker
data = load("costSurface.mat").costSurface;
c = parallel.pool.Constant(data);  % client already paid the memory cost

The only time parallel.pool.Constant(data) (without a build function) is appropriate is when the data is already in client memory and used across multiple parfor loops (avoids re-broadcasting each loop). For a single parfor loop, just let MATLAB broadcast — Constant adds no benefit.

Pattern 2: Shared library (non-serializable, needs cleanup)

c = parallel.pool.Constant( ...
    @() loadLibraryAndReturn("mylib", "mylib.h"), ...
    @unloadlibrary);

results = zeros(1, 100);
parfor i = 1:100
    results(i) = calllib(c.Value, "compute", data(i));
end
% Cleanup runs automatically when c goes out of scope

function libName = loadLibraryAndReturn(libName, headerFile)
    loadlibrary(libName, headerFile);
end

The build function returns the library name so that c.Value is naturally referenced in the parfor body — this triggers lazy initialisation. The cleanup function receives the same value and calls unloadlibrary on it.

Pattern 3: One-shot setup with parfevalOnAll (no cleanup needed)

pool = gcp;
f = parfevalOnAll(pool, @setupWorker, 0);
fetchOutputs(f);

Pattern 4: Database connection (use createConnectionForPool)

Database connections are a common specific case that requires Database Toolbox. Prefer the purpose-built helper createConnectionForPool over a hand-rolled parallel.pool.Constant. It returns a parallel.pool.Constant whose value is a per-worker connection to the configured data source, and it handles worker-side initialisation that a naive database() call inside a Constant build function does not.

pool = gcp;
c = createConnectionForPool(pool, "MyDataSource", "", "");

results = cell(1, 1000);
parfor i = 1:1000
    conn = c.Value;
    results{i} = fetch(conn, sprintf("SELECT val FROM t WHERE id=%d", i));
end

The returned Constant is the standard parallel.pool.Constant type, so it participates in the normal lifetime model: connections are torn down when the Constant goes out of scope or the pool shuts down.

Recognising When pool.Constant Is Needed

Look for these signals in user code:

Signal	Indicates
Object created and destroyed every parfor iteration	Per-worker resource needed
"Cannot serialize" or "not valid on workers" errors	Non-serializable resource
loadlibrary/unloadlibrary inside parfor	Library should persist across iterations
database()/close() inside parfor	Connection should persist
fopen/fclose inside parfor (same file)	File handle should persist
Large variable used read-only in multiple parfor loops (>100 MB)	Send-once with Constant
spmd block doing setup before parfor	Modernise to Constant or parfevalOnAll

Common Mistakes

What the agent does wrong	Why it's wrong	Correct approach
Uses spmd before parfor for setup	Fragile, no cleanup, pool tears down if any worker disconnects	parallel.pool.Constant with build + cleanup functions
Creates resource every iteration	Massive overhead (N connections for N iterations)	parallel.pool.Constant — one per worker
Uses persistent variables on workers	No cleanup, hard to reason about lifecycle	parallel.pool.Constant — explicit lifecycle
Tries to send non-serializable object from client	Will error — can't serialize connections, libraries	Use build function form: parallel.pool.Constant(@buildFcn)
Build function calls a function not on the worker path	Workers can't find it — "Unable to create parallel.pool.Constant on the workers"	Ensure build function helpers are on the path (or use AdditionalPaths). Local functions in the same file work — MATLAB captures them with the handle.
Uses addpath inside parfor on process pool	Runs every iteration unnecessarily	parpool(..., AdditionalPaths=...) or parfevalOnAll
Deletes and recreates pool to change paths or env vars	Extremely expensive — pool startup can take 30+ seconds and may queue on a scheduler	Use parfevalOnAll on the existing pool instead

Key Functions

Function	Product	Available From	Purpose
parallel.pool.Constant	Parallel Computing Toolbox	R2015b	Per-worker state with lifecycle management
parfevalOnAll	Parallel Computing Toolbox	R2013b	Run a function once on all workers
parpool(..., AdditionalPaths=)	Parallel Computing Toolbox	R2025a	Set worker paths at pool creation
parpool(..., EnvironmentVariables=)	Parallel Computing Toolbox	R2017b	Forward env vars to workers at pool creation
createConnectionForPool	Database Toolbox	R2019a	Per-worker database connections (optional)

Conventions

• Prefer parallel.pool.Constant over spmd for any worker state that persists across parfor iterations
• The build-function form (parallel.pool.Constant(@buildFcn)) is lazily initialised — the build function does not run until c.Value is first accessed on a worker. Always reference c.Value inside the parfor body to ensure initialisation occurs.
• Always provide a cleanup function when the resource needs explicit teardown
• Use parfevalOnAll only when no handle is needed back and no cleanup is required
• Use parpool name-value pairs when the setup is needed for the entire pool lifetime (paths, env vars)
• When modernising spmd-based setup code, preserve the original's intent (what resource, what cleanup) while switching to Constant

---

Copyright 2026 The MathWorks, Inc.

---