| name | hosting |
| description | Load, run, and test VST3 / AU / CLAP / LV2 plugins from Pulp code. Use
when working on `core/host/` (scanner, plugin_slot, signal_graph), when
adding a new format backend, when wiring a plugin into a SignalGraph, or
when writing an integration test that needs a real plug-in binary.
|
hosting
When this skill applies
- Adding or modifying a format backend under
core/host/src/plugin_slot_<format>.cpp.
- Extending
PluginSlot::load() to handle a new format in core/host/src/plugin_slot.cpp.
- Building or routing nodes in
SignalGraph.
- Writing tests that need to load a real plug-in binary.
Mental model
PluginSlot is the uniform interface. Each format backend is a single
free function โ load_<format>_plugin(info) โ that returns a
std::unique_ptr<PluginSlot> or nullptr. PluginSlot::load() in
plugin_slot.cpp is a small compile-time dispatcher. There is no dynamic
registry and no plug-in-per-file hooks; adding a format means:
- Write
core/host/src/plugin_slot_<fmt>.cpp that defines
std::unique_ptr<PluginSlot> load_<fmt>_plugin(const PluginInfo&).
- Add the file to
core/host/CMakeLists.txt under a if(PULP_HAS_<FMT>)
guard. Link the format's SDK. Define PULP_HOST_HAS_<FMT>=1.
- Forward-declare the loader and add a
case PluginFormat::<FMT>: to
the dispatcher in plugin_slot.cpp, guarded by the same macro.
Everything else โ tests, scanner, graph wiring โ is format-agnostic.
CLAP reference backend
plugin_slot_clap.cpp is the simplest backend to study for dlopen,
factory lifetime, parameter metadata, automation, MIDI, and state patterns.
VST3 / AU / LV2 also have real loaders, so treat CLAP as a reference for
its ABI shape rather than as the only implemented backend. Patterns to mirror:
dlopen(RTLD_LAZY | RTLD_LOCAL); on macOS resolve
<bundle>.clap/Contents/MacOS/<name> before dlopen.
dlsym("clap_entry"); call entry->init(path) exactly once before
entry->get_factory(...), and entry->deinit() + dlclose() in the
slot's destructor.
- Pick factory descriptor by
info.unique_id when set, else first
available. Then fill the returned PluginInfo with any missing
name / vendor / version / id fields from the descriptor.
- The slot must own the
clap_host_t it exposes to the plug-in; the
plug-in stores the pointer and will deref it later.
- After a successful
CLAP_EXT_STATE restore, clear any cached host
parameter edits in the slot. Otherwise get_parameter() can report a
stale host-side value even though the plug-in restored its own state.
VST3: re-sync a separated edit controller on state restore
VST3 splits a plug-in into an IComponent (processor) and an
IEditController. When they are separate objects, restoring only the
component state (IComponent::setState) leaves the controller โ and the
vendor UI โ showing stale values. The host contract is: after
component->setState, push the same processor state into the controller with
IEditController::setComponentState, and separately save/restore the
controller's own getState/setState. Combined plug-ins (one object
implementing both interfaces) skip all of this โ see the identity trap below
for how to detect the combined case.
A separated controller also needs setComponentState at load, not only on
restore: a factory-created controller comes up on its own defaults, so its
parameter cache โ and the editor about to open on it โ shows values the
processor will not render. vst3_push_component_state (in vst3_state_sync.hpp)
is that push and Vst3Slot's constructor calls it.
VST3: combined-vs-separated is an FUnknown QUERY, never a pointer cast
static_cast<FUnknown*>(component) == static_cast<FUnknown*>(controller) looks
like the identity test and is wrong. A combined plug-in inherits IComponent
and IEditController separately, so each interface carries its own FUnknown
base subobject at a different address โ the cast-and-compare therefore calls
every combined plug-in "separated". Verified against the SDK's own
SingleComponentEffect. COM identity is defined by the query: ask both sides
for FUnknown::iid and compare the returned pointers (vst3_same_object /
vst3_is_separated in vst3_connection.hpp; release both +1s).
Getting it backwards is not cosmetic โ a combined plug-in misread as separated
gets IPluginBase::terminate() called twice on unload, has its state stored
twice, and would be connected to itself.
VST3: connect the separated halves, and give them a real IHostApplication
A separated plug-in's two halves only reach each other through
IConnectionPoint โ the host queries both, connects each to the other, and
disconnects before either terminates (vst3_connection.hpp). Everything a
plug-in cannot express as a parameter (preset banks, meter feeds, editor
handshakes) travels as IMessage over that link, so an unconnected plug-in
opens an editor that can never talk to its own processor.
Those IMessage / IAttributeList objects are allocated by the host, through
IHostApplication::createInstance. A host-application stand-in that returns
kNotImplemented there silently makes the connection useless, so Vst3Slot's
HostApp derives from the SDK's Vst::HostApplication (which also supplies
IPlugInterfaceSupport) and only overrides getName plus the refcount, because
the instance is a process-wide singleton a plug-in must not be able to delete.
That pulls hostclasses.cpp + pluginterfacesupport.cpp + stringconvert.cpp
commonstringconvert.cpp into the vst3-sdk CMake target โ the last two are
transitive and only show up as a link error in an unrelated tool.
Vst3Slot lives in an anonymous namespace, so this logic lives in a
free-function seam, pulp::host::detail::vst3_state_sync.hpp
(vst3_serialize_state / vst3_restore_state) โ that is also the only place a
host-slot test can reach it. It writes a versioned PV3S container (component
plus an optional controller section); a blob without the magic is treated as
legacy raw-component state, so sessions saved before the container existed
still load. Length fields are bounds-checked as len > remaining (never
remaining - len) so a malformed blob cannot underflow into an out-of-bounds
read.
VST3: embed the editor like CLAP โ a parent-consuming IPlugView
IEditController::createView("editor") hands back an IPlugView that, like
CLAP set_parent, CONSUMES a parent: IPlugView::attached(container, "NSView")
inserts the plug-in's view into a host-owned container rather than returning a
view. So the VST3 editor path reuses the same container as CLAP
(create_editor_container), and the ordering matters โ query
IPlugView::getSize first to size the container, create the container (already
in the parent window), THEN attached(). Tear down in reverse: removed()
before release(), and close the editor before terminating the controller it
came from (createView's view must not outlive its controller).
The host MUST install an IPlugFrame before attached(). IPlugView
documents IPlugFrame::resizeView() as callable from inside attached() โ
it is how a plug-in reports the size it really wants โ so a view with no frame
either mis-sizes itself or refuses to attach outright, and a refused attached()
is exactly the "editor returns null, nothing embeds" symptom. setFrame goes on
immediately after createView, before isPlatformTypeSupported; every release
path clears it first (vst3_release_editor_view does the setFrame(nullptr))
so a plug-in can never call into a destroyed frame. Because the plug-in may
resize during attached(), the slot publishes editor_view_/editor_container_
before the attach call and reports the post-attach size, not the size queried
before the view existed.
The AppKit part is only the container; the VST3 negotiation (createView,
setFrame, isPlatformTypeSupported, getSize, attached,
onSize/checkSizeConstraint, removed) is a pure-interface seam,
pulp::host::detail::vst3_editor.hpp, which
is where a headless test drives it with a fake IEditController / IPlugView
(inherit the SDK EditController / CPluginView bases). The full container
attach is only reachable with a real native window, so it is proven by the
real-DAW smoke, not the unit test.
AU: the Cocoa UI RETURNS a view โ adopt it, don't offer a parent
An AUv2's editor is a Cocoa-view factory, the mirror image of CLAP/VST3: query
kAudioUnitProperty_CocoaUI for an AudioUnitCocoaViewInfo (a bundle URL + a
class conforming to AUCocoaUIBase), load the bundle, and call
uiViewForAudioUnit:withSize: to get an NSView the plug-in already laid out. So
the host does NOT hand the plug-in an empty container to fill โ it creates the
container and ADOPTS the returned view into it (editor_container_adopt_view,
which sets an autoresize mask so the view tracks the container). Two traps: the
property hands back +1 CF references (the bundle URL and the class name) โ
CFRelease both on every exit path; and destroying the container already
releases the adopted view (it is a subview), so the slot keeps no separate view
handle. The Cocoa-UI negotiation needs a real AU with a view, so it is proven by
the real-DAW smoke; only the container adoption is unit-tested.
Defensive boundary for entry / factory calls
scanner_clap.cpp wraps entry->init() and entry->get_factory() in
try/catch. Throws across the dlopen boundary abort the whole scan
otherwise โ observed in production with bundles whose static-init throws
C++ exceptions during dlopen. The fallback
emits a synthesized PluginInfo (filename-derived name, no metadata)
so the scan still surfaces the bundle. Static-init throws that fire
before dlsym returns can't be caught at this layer; that's the
case pulp scan --no-load exists for. When adding new entry-point
calls, wrap them too โ the goal is "one bad bundle never crashes a
scan."
dlerror() must be cached
Never call dlerror() more than once per failure log line. POSIX
clears dlerror's internal buffer after every call, so a ternary like:
runtime::log_warn("dlopen failed: {}",
dlerror() ? dlerror() : "unknown");
calls dlerror() twice and the SECOND call returns nullptr.
std::format's string_view(char const*) ctor then runs
strlen(nullptr) and ASan flags a SEGV in libsystem_platform's
_platform_strlen. The bug is invisible on non-ASan builds โ
release builds happen to survive the near-null strlen because the
zero page is read-protected one access deep. The fixed behavior caches
dlerror() before formatting so the null-returning second call cannot
reach std::format.
Correct idiom:
const char* err = dlerror();
runtime::log_warn("dlopen failed: {}", err ? err : "unknown");
The other format backends (plugin_slot_vst3.cpp, plugin_slot_lv2.cpp,
plugin_slot_clap.cpp, core/runtime/src/dynamic_library.cpp) all
already cache via a local. When adding a new format backend that calls
dlopen, mirror the cache-into-local pattern.
Testing against a real plug-in
For repeatable black-box measurements of an installed Audio Unit instrument,
build or install pulp-au-instrument-probe. It renders offline without opening
an audio device, requires an explicit --name, lists vendor parameter IDs, can
apply plain-domain parameter values and timestamped MIDI hits, and writes a
local float WAV. By default an accidentally silent render is a failure;
--allow-silent is reserved for experiments where silence is itself expected.
pulp-au-instrument-probe --name "Reference Instrument" --list-params \
--note 60 --seconds 2 --hits "0:100,250:80" --out /tmp/reference.wav
The probe is a bench oracle, not an implementation oracle: keep commercial
renders out of version control, record the recipe and numeric measurements,
and derive DSP from published specifications or independently authored models.
Reference-specific names, parameter maps, and corpora belong in the private
validation project, not in the SDK tool.
For unattended, scriptable interrogation prefer the isolated CLI/MCP surfaces:
pulp audio plugin-inspect --plugin /path/to/plugin.component --format au
pulp audio render --plugin /path/to/plugin.component --format au \
--input-signal noise:7 --duration-ms 1000 --warmup-ms 1000 \
--initial-param 12=0.5 --settle-ms 250 --wav-format float32 --out /tmp/out.wav
plugin-inspect reports the complete host-visible parameter API. Both commands
instantiate vendor code in disposable child processes with timeouts. That is
crash/hang containment, not a security sandbox. The rich pulp audio compare
step is an optional Audio Quality Lab add-on; inspection, rendering, metrics, and
pulp audio validate compare are stock Pulp.
Integration tests gate on a compile-time path macro:
if(PULP_BUILD_TESTS AND NOT ANDROID AND TARGET PulpGain_CLAP)
foreach(_pulp_clap_host_test IN ITEMS pulp-test-host pulp-test-host-regression)
if(TARGET ${_pulp_clap_host_test})
target_compile_definitions(${_pulp_clap_host_test} PRIVATE
PULP_TEST_CLAP_PATH="${CMAKE_BINARY_DIR}/CLAP/PulpGain.clap")
add_dependencies(${_pulp_clap_host_test} PulpGain_CLAP)
endif()
endforeach()
endif()
Keep this wiring after add_subdirectory(examples): the top-level build
registers test/ before examples/, so test/CMakeLists.txt cannot
reliably see PulpGain_CLAP at configure time.
Tests check fs::exists(PULP_TEST_CLAP_PATH) and WARN + return if the
plug-in isn't built, so the suite still passes on configurations that
skip the plug-in builds (Android, CI without GPU examples, etc.).
Pattern for a process test: load, prepare(48000, 256), fill an input
buffer with non-zero samples, call process, assert the output buffer
has non-zero energy. A gain plug-in is the cheapest target โ one param,
predictable output, no MIDI.
For a quick manual load/inspect of one installed plug-in, examples/plugin-host-demo
doubles as a format-neutral analyzer. --path <bundle> loads a CLAP / VST3 / LV2
by inferring the format from the bundle extension; an Audio Unit has no bundle
path, so select it with --id TYPE:SUBT:MANU (the OSType triplet printed by
--list). --path deliberately skips the full installed-plugin scan โ a bulk
scan runs third-party discovery code across the machine, which is unrelated to a
trusted, user-named probe. Two extension gotchas when inferring format from a
path: shell tab-completion appends a trailing / to a bundle directory, which
empties std::filesystem::path::extension() (step up via parent_path() when
filename() is empty), and a plain dlopen of a relative bundle path triggers
the @rpath search dance โ pass an absolute path.
--editor embeds the loaded plug-in's own editor in a window via the
hosted-editor path (create_hosted_editor โ EditorAttachment), auto-closing
after --editor-ms (default 3000). It is the manual smoke for the whole
host-side editor chain (CLAP / VST3 / AU): the negotiation seams are unit-tested
headlessly, but the editor actually rendering needs a display and a real plug-in
GUI, so run --editor (or load a Pulp-hosted plug-in in a DAW) to confirm it by
eye. Heads-up: opening an editor can make the plug-in active and audible โ the
bounded duration keeps that contained.
Headless Audio Unit event-loop servicing
Licensed Audio Units may complete initialization asynchronously via XPC,
timers, or dispatch-main callbacks. GUI hosts service these naturally; an
offline analyzer may otherwise produce plausible but incorrect audio or ignore
early parameter writes without reporting an error.
Use pulp::events::MessageLoopIntegration::pump_main_loop_for() from the
process main/control thread to service bounded slices before parameter access
and between offline render blocks. Never call it from the audio callback. The
result reports event-loop progress only, not license or plug-in readiness, so a
tool must own a configurable warm-up and post-write/render-settle policy. Put
the entire instantiate/process probe in a child process; isolated scanning alone
does not contain crashes in deeper plug-in code.
Signal graph gotchas
SignalGraph dispatches plugin nodes through the additive
PluginSlot::process(format::ProcessBuffers&, ...) overload. The default
implementation projects the active main input/output bus back to the legacy
process(output, input, ...) callback, and still calls legacy processing with
empty audio views for MIDI-only slots. Override the ProcessBuffers overload
when a hosted format or fixture needs direct bus metadata for sidechains,
auxes, surround, or multi-output products.
- Canonical-executor routing (DEFAULT ON;
set_canonical_executor_routing_enabled
toggles it). The routed executor is the primary inter-node backend for every
eligible graph; it is bit-identical to the legacy walk for that subset AND reports
the same per-node node_loads() telemetry (the executor times each node's work via
a per-binding AudioProcessLoadMeasurer wired from the host's persistent node-load
map), so the default-ON flip is behaviour-preserving where it takes effect. Force it
OFF to render the walk โ the routed-vs-walk parity oracles (run_legacy,
signal_graph_block) do that so the walk stays an independent reference. EVERY
node kind SignalGraph produces is now eligible โ the only remaining walk triggers
are an unprepared graph or a routed snapshot/pool BUILD failure (e.g. a topology past
GraphRuntimeLimits); the walk is the deliberate reference/fallback for those, the
independent parity oracle, and is NOT slated for deletion. routed_walk_fallbacks()
counts blocks where a routed path was ELIGIBLE but its dispatch returned failure so
process() silently fell back to the walk โ that degradation is invisible to the parity
test (the walk is both oracle and fallback), so this counter (plus a once-per-graph
debug warning) is the only signal an eligible graph stopped routing. It stays 0 in
healthy operation; a normal fallback (routing disabled / ineligible / walk-by-choice)
does NOT increment it. An eligible graph โ
nodes AudioInput / AudioOutput / Gain / Plugin (a Plugin with NO live slot routes as
pass-through-or-zero via custom_binding(nullptr), exactly matching the walk's
missing-plugin behavior) / MidiInput / MidiOutput / Custom (CustomNodeType,
stateless process or stateful process_instance; routed via custom_binding,
an unresolved/shape-mismatch custom node pass-through-or-zeros exactly as the
walk does; custom output regions are pinned persistent_output like plugins so
a partial writer keeps its stale tail), connections audio (feedforward,
one-block feedback, or sidechain โ a sidechain edge routes as plain audio into
a higher input port of the destination plugin), MIDI (connect_midi event
edges), or parameter automation โ sparse (connect_automation, two control
points) and dense (connect_audio_rate_modulation, per-sample) โ can be driven
through the canonical GraphRuntimeExecutor instead of the legacy walk via
set_canonical_executor_routing_enabled(true). Output is bit-identical to the
legacy walk (signal_graph_executor_routing.{hpp,cpp} translates the graph;
test_signal_graph_executor_parity is the guard). Plugin output slots are
pinned persistent in the buffer assignment (the persistent_output spec
flag), so a plugin that does not fully overwrite its output carries the same
stale tail across blocks that SignalGraph's per-node buffer does โ the reason a
Plugin node needs a live slot to be eligible (a null-slot placeholder would
take the legacy pass-through-or-zero branch, which the executor does not
reproduce). A latency-reporting plugin IS eligible: the routed gather applies
the same per-connection plug-in delay compensation as the legacy walk
(per-node latency is propagated through the topology in the buffer assignment,
and each feedforward connection that needs it gets a delay ring sized in the
GraphRuntimeBufferPool), so fan-in paths of differing latency time-align
identically. Each pool ring's mutable samples + write cursor live in a
shareable state object while the RT lookup still returns raw pointers; this
is the seam prepare_swap uses off-RT to adopt identity-matched PDC history
without reading or copying live ring contents. MIDI edges route through
per-node MIDI scratch buffers owned by
the executor (GraphRuntimeMidiScratch); SignalGraph bridges its MIDI
mailboxes (inject_midi / extract_midi) around the routed call. External
per-block parameter events cross the same boundary through a per-node
inject_parameter_events mailbox. The routed call appends that publication
after executor-generated automation and commits its sequence only after the
whole dispatch succeeds, matching serial fallback and anticipation behavior.
Parameter automation routes through a GraphRuntimeAutomationScratch
(per-node parameter event queue + per-connection slew state + per-node dense
buffers): sparse edges sample the source at the block edges, map/slew/mix per
the connection's resolved bounds, and emit two control points; dense
audio-rate edges map every sample (through the same per-connection PDC delay
ring as audio), mix into a per-node buffer, and emit one event per sample โ
both bit-identical to the walk and built into the same per-node event queue.
A node exceeding
kMaxParamsPerNode (64) distinct sparse OR dense params is kept on the legacy
walk.
- Where the walk lives. The legacy serial reference walk is no longer
inline in
process_impl; it lives in
core/host/src/signal_graph_reference_walk.cpp and is entered via
SignalGraph::run_reference_walk_ when no routed path takes a block. It is
kept deliberately INDEPENDENT of signal_graph_executor_routing.{hpp,cpp}
โ do not share or merge its gather / PDC / feedback / MIDI / automation
execution with the executor. The MIDI-block helpers shared by both the
routed dispatch and the walk (clear_midi_block, midi_block_has_drops,
copy_midi_block) live in the shared header
core/host/src/signal_graph_internal.hpp. The dual-maintenance rule still
applies: any audio-output-affecting edit to the walk must be mirrored in the
executor (and vice versa), guarded by test_graph_routing_differential_parity,
test_signal_graph_executor_parity, and test_signal_graph_offline_parity.
- Where the live-swap engine lives. The no-silence topology-edit machinery โ
the swap policy and scanned-plugin catalog, the staged-replacement pipeline,
the
begin_swap_edit / prepare_swap / abort_swap_edit transaction with its
crossfade publish and rollback, snapshot_is_plugin_reinit_free_locked_, and
the load-admission gate โ lives in core/host/src/signal_graph_live_swap.cpp,
so signal_graph.cpp keeps the topology/compile/process spine. The file
arrangement mirrors signal_graph_reference_walk.cpp, but the reason does
NOT: these are still ordinary SignalGraph members that name its private
nested types, only their definitions moved. The reference walk's
independence rule does not transfer here โ there is no second
implementation to stay bit-exact against, and factoring shared code out of
live-swap into signal_graph.cpp (or signal_graph_internal.hpp, where
prepare_midi_block_storage already single-sources every graph MIDI block's
real-time capacities for both the compile path and the swap warm-up probe) is
a fix, not a violation. Everything in the live-swap TU runs on the CONTROL
thread.
- Gap-free PDC carry is identity-based and conservative.
connections_
has a private parallel vector of monotonic identities; every insertion and
erasure must update both vectors. CompiledGraph snapshots those identities
beside connections. During prepare_swap, the old and candidate delayed
edge sets must form an identity-keyed bijection with equal delay sizes and
equal total graph latency. The candidate then shares, never copies, each old
domain's audio-thread-owned ring state: legacy ConnectionDelay,
routed.serial.pool, and routed.parallel.pool. Disconnect+reconnect mints a
new identity and is refused even when the public Connection values compare
equal. Because those domains keep independent histories, a PDC-active
CompiledGraph pins the execution domain chosen during prepare(); relaxed
routing toggles remain dynamic only for zero-PDC snapshots, and a live swap
that would change the pinned domain is refused. Feedback graphs,
routed-validity changes, and latency/delay-structure changes are also refused.
Tests: test_signal_graph_pdc_swap_continuity.cpp
uses D=97 with 64-frame blocks across all three execution domains and includes
the reconnect negative plus a concurrent swap hammer.
_locked_ is a contract, and it is now asserted. A SignalGraph helper
suffixed _locked_ requires the caller to already hold
graph_mutation_mutex_; the convention now spans signal_graph.cpp and
signal_graph_live_swap.cpp, so it is easier to violate from the far side than
it was when one TU held every caller. Helpers whose call graph is entirely
internal open with assert_graph_mutation_locked_(), which reads a debug-only
owner record โ so take the mutex through GraphMutationLock, never a bare
std::lock_guard/unique_lock. A bare guard locks correctly but leaves the
owner record empty, and the next _locked_ helper you call asserts as though
you forgot the lock entirely (a debug-only false failure that reads like a real
one). Only prepare_swap calls GraphMutationLock::unlock() early, to drop
the lock before invoking user callbacks. Two _locked_ helpers cannot assert โ
has_path_locked_ (reached via would_create_cycle) and
total_declared_ports_locked_ (via validate_generated_graph /
estimate_generated_graph_work_units) โ because those public entry points do
not lock; their suffix documents the internal contract only. (node_load_mu_
is a different mutex and is correctly taken with a plain lock_guard.)
CompiledGraph::routed groups what is only ever valid together. Each
RoutedPath (routed.serial, routed.parallel) owns its own snapshot,
pool, plugin_ctx, custom_ctx, and valid flag โ driving one path's
snapshot against the other path's pool is the bug the grouping exists to make
hard, since the parallel path's assignment is reuse-free and the serial path's
is compact. The MIDI scratch, automation scratch, and MidiInput/MidiOutput node
lists sit on routed itself and are deliberately SHARED by both paths: the
plans are identical and only ONE path runs per block. That sharing is load-
bearing โ anything that makes those structs carry per-path state (or that runs
both paths in a block) breaks it silently, because the MIDI mailbox bridge and
the automation queues would then interleave across paths.
build_executor_snapshot prefers the ExecutorSnapshotBinders struct. The
positional overload still exists as a forwarder for unmigrated call sites, and
it is exactly where a resolver can go wrong quietly: several of its resolvers
are same-shaped std::function<T*(NodeId)>, so swapping two by argument
position compiles and mis-binds. Every binder field is optional and has a
documented fallback (plugin_latency_for / plugin_params_for empty means
"fall back to the live slot", which is fine for baked/anticipation callers but
NOT on the swap path, where the point of the cached accessors is that a
swap-time build makes no live PluginSlot metadata call).
- Multi-path routing has an arithmetic guard.
test/test_signal_graph_audio_parity.cpp (target
pulp-test-signal-graph-audio-parity) renders a fan-out/fan-in topology whose
paths carry distinct non-commutative transfer functions, and checks the output
bit-exactly against expectations derived by hand from the stimulus โ on the
walk, the routed serial path, and the routed parallel path. Nothing in it is
captured from a build, so a routing change cannot move the bar with it: a
dropped, swapped, or reordered path fails. It also asserts
routed_walk_fallbacks() / routing_executor_stats(), since a routed case
that quietly degraded into the walk would otherwise pass vacuously (the walk is
both oracle and fallback).
- Connection lane CLASSIFICATION is single-sourced (distinct from the
execution-independence rule above). Which lane a host
Connection carries โ
audio / event(MIDI) / automation, plus the orthogonal feedback flag and the
dense-vs-sparse audio-rate split โ is decided in ONE place: classify() in
signal_graph_executor_routing.{hpp,cpp}, returning a ConnectionClass. The
runtime structs carry this as a typed graph::GraphRuntimeConnectionKind
discriminator (Audio/Event/Automation, default Audio) instead of the
old independent event/is_automation bools; read it via the
pulp::graph::is_event / is_automation_conn / carries_audio accessors.
BOTH classification surfaces route through classify(): the executor-routing
gather (building GraphRuntimeConnectionSpecs) AND the compile-time
reference-walk edge bucketer in SignalGraph::compile_ (audio / MIDI /
sparse-automation / dense-audio-rate / feedback buckets). The PDC/latency
passes share the matching connection_affects_latency() predicate. A
sidechain edge deliberately classifies as Audio (it is plain audio into a
higher dest port). This is single-sourced CLASSIFICATION only โ the gather
math, PDC delay rings, and MIDI/automation evaluation stay independent and
dual-maintained per the rule above. New lane mappings are pinned by
test_connection_classify.
- Transport-aware
process(). Alongside the no-transport
process(out, in, n) there is an additive
process(out, in, n, const format::ProcessContext& transport) overload. Both
delegate to one private process_impl(..., const format::ProcessContext*); the
3-arg form passes nullptr and is bit-identical to its prior behaviour. When a
transport is supplied it populates the routed ProcessBlock
(block.transport = &ctx, block.mode = ctx.process_mode) so nodes that consume
it (e.g. a ProcessorNode, via context_for_block) see the host playhead, mode,
and render-speed hint. block.render_speed stays the numeric 1.0: the
render-speed hint is categorical and travels through *block.transport, never the
multiplier. Nodes that ignore block.transport are bit-identical to the
no-transport path, so routed-vs-walk parity is unaffected. Under active
anticipation the transport stays LIVE: transport-sensitive nodes are excluded
from the ahead-rendered interior (see the per-node opt-in below and the
anticipation gotchas), so every ahead-rendered node is transport-insensitive by
construction and the forwarding is inert for it.
- Per-node transport opt-in (
PluginSlot::wants_transport() + transport
process() overload / a transport-aware custom callback). A routed plugin or
custom node OPTS INTO the host transport: a PluginSlot overrides
wants_transport() to return true and overrides the appended
process(ProcessBuffers&, midi_in, midi_out, param_events, n, const format::ProcessContext&) overload; a custom node's type sets
process_transport (stateless) or process_instance_transport (stateful).
compile_ resolves the capability ONCE into the cached, prepare-stable
GraphNode::transport_sensitive (Plugin: from slot->wants_transport();
Custom: from a non-empty transport callback registration), resolved BEFORE the
anticipation eligibility analysis. That ONE cached bit is read by BOTH the
routed binding (PluginBindingContext::wants_transport /
CustomBindingContext::process_transport, which forward the live transport when
the block carries one) AND the anticipation analyzer (which seeds
AnticipationExclusion::TransportSensitive), so the partition and the bindings
can never disagree. INVARIANT: never call a live slot->wants_transport() per
block on the audio thread โ the bit is cached at compile and a capability change
requires a re-prepare. A node that does not opt in is byte-for-byte unchanged.
- Parallel-executor routing (opt-in, default OFF, independent of the serial
opt-in).
set_parallel_routing_enabled(true) drives the SAME eligible subset
through GraphRuntimeExecutor::process_parallel โ a levelized fork-join over a
persistent GraphRuntimeWorkerPool (the audio thread is participant 0). Output
is bit-identical to the serial executor and the legacy walk; the per-node body
(run_routed_node) is shared. Dispatch order in process(): parallel (if
enabled + valid + pool running + fits) โ serial executor (if its toggle on) โ
legacy walk. The two routed branches share one dispatch_routed bridge, and
every executor zeroes the output bus + the MIDI ingress is idempotent (consumed
mailbox sequences aren't committed until run() succeeds), so a failed parallel
attempt re-renders the block on a lower tier with no doubled output or
double-consumed MIDI. SignalGraph::set_parallel_min_work_units(n) forwards
to the executor's channel-sample break-even gate; default 0 preserves the
original "parallelize every eligible level" behavior, while a positive value
keeps low-cost levels serial to avoid fork/join overhead on small graphs. Use
routing_executor_stats() to verify the live path when testing the threshold.
GOTCHAS: (1) the parallel snapshot uses a REUSE-FREE
buffer assignment (parallel_safe=true) โ concurrent same-level nodes must not
alias a recycled scratch slot; process_parallel refuses a non-parallel-safe
snapshot. (2) Levels containing an AudioOutput node run SERIALLY in topo order
(AudioOutput += accumulates into the shared output bus; float add is
non-associative, so order is load-bearing for โฅ3 sinks). (3) WORKER-POOL
LIFECYCLE is load-bearing: the pool is started ONCE (size = clamped hardware
concurrency, guarded by worker_count() == 0) and NEVER stopped/resized on a
re-prepare โ start()/stop() join threads + reset the epoch/completion
counters, a UAF if run against an in-flight audio run(). The only legal stop
is ~GraphRuntimeWorkerPool at SignalGraph destruction. Don't make the thread
count runtime-variable without a drain handshake. The pool's completion barrier
counts PARTICIPANTS finished (not tasks): an empty-range participant must still
register done, or it can race the next batch's published state.
(4) WORKGROUP CHANGES are generation-published, not applied from the caller:
SignalGraph implements format::AudioWorkgroupClient, and each persistent
worker leaves/joins on its own thread. run() executes inline while any worker
still advertises an older generation, so an AU renderContextObserver change
cannot dispatch a deadline into the previous workgroup. A failed non-null join
does not acknowledge the generation: the worker retries and run() stays
inline. For an explicitly owned device, publish null and call the
off-render-thread acknowledgment barrier before switching or closing it; only
then may the borrowed OS handle be invalidated. Re-query and publish the
replacement before rendering resumes. Do not cache a device-owned handle past
that drain point. Close must first disable new device-change notifications and
serialize with any switch already in flight, then publish null and drain again
under that serialization boundary; an external null publication alone can race
a switch which rebinds immediately before close. AU render-context teardown
remains publication-only.
- Anticipative-rendering eligibility (
anticipation_eligibility.{hpp,cpp}).
analyze_anticipation_eligibility(nodes, connections) is the static SAFETY
contract for rendering a latent subgraph ahead of the audio deadline: it
classifies each node None (passed) or a hard-exclusion reason โ seeds live
AudioInput/MidiInput nodes, both endpoints of every feedback edge, any node with
a sidechain inbound edge, and any node with GraphNode::transport_sensitive set
(the per-node host-transport opt-in), then propagates each exclusion forward
along feedforward (non-feedback) edges to a fixpoint so anything downstream of an
excluded node is excluded too. It's deliberately conservative: a false exclusion
only forfeits a speed-up, but a false inclusion would render an unsafe node
ahead. Host-clock sensitivity is handled by the TransportSensitive seed: a
host-clock-dependent node opts in via wants_transport() / a transport-aware
custom callback (resolved into transport_sensitive at compile), so it โ and
its downstream cone โ is kept out of the ahead-rendered interior and runs live.
passes_static_exclusions(i) true therefore IS sufficient for the partition to
treat node i as ahead-renderable. The SignalGraph anticipative splice gates on
this analysis when set_anticipation_enabled(true) is prepared.
- Anticipation partition (
anticipation_partition.{hpp,cpp}).
build_anticipation_partition(nodes, connections, eligibility) carves the
renderable eligible INTERIOR (eligible nodes minus the live AudioOutput/MidiOutput
sinks, which are consumed at the real deadline and must never be written ahead)
and the BOUNDARY edges (interior-source -> outside-the-interior), which are the
splice points the renderer pre-computes and the live graph reads. cost_weight
(the same coarse max(in,out) proxy the parallel cost gate uses) +
worth_anticipating() gate out trivial/no-boundary partitions. Still pure static
analysis โ no rendering, no RT path. Builds on the 6a eligibility result and is
rejected (ok=false) if that result isn't ok or doesn't match the node span.
- Anticipation sub-graph (
anticipation_subgraph.{hpp,cpp}).
build_anticipation_subgraph(nodes, connections, partition) turns a partition
into a standalone renderable graph: it copies the interior nodes verbatim (plugin
slots/gain/ports preserved) and the internal edges, and synthesizes ONE
AudioOutput sink whose input ports correspond to the DISTINCT boundary output
ports (fresh id above every existing node id, so no collision), fed so boundary
output i lands on sink input/output-bus channel i โ so the sub-graph renders
through the ordinary build_executor_snapshot + process_routed and its output
bus carries exactly the boundary signals without summing them together.
outputs[] maps each output-bus channel back to the (source_node, source_port)
it captures. GOTCHA: the interior plugin
GraphNodes are copied by value, so the SAME plugin instances render here โ which
means a live splice (a later slice) must NOT also process those instances, or
their state double-advances. This slice does extraction only; it neither renders
nor changes any RT path.
- Anticipation lane (
anticipation_lane.{hpp,cpp}). AnticipationLane renders
an eligible sub-graph AHEAD of the deadline into a PlanarAudioRingBuffer:
prepare() (off-RT, quiescent) builds the executor snapshot + sizes the ring for
a FIXED block size; render_ahead() (single background producer) advances the
interior's plugin state and pushes whole blocks; consume() (audio thread,
RT-safe, no-alloc) pops a pre-rendered block or reports underrun so the caller
falls back to a synchronous render. The block size is PINNED at prepare (before
any thread exists) so producer/consumer stay in lockstep and there's no
cross-thread block-size field โ the consumed sequence is bit-identical to a
block-by-block synchronous render. GOTCHAS: (1) the interior plugins are advanced
ONLY by render_ahead โ a live splice that uses a lane must not also process those
nodes or their state double-advances; (2) render_ahead is SINGLE-producer (all
calls, including priming, must be serialized โ they share unsynchronized
executor/pool/scratch); only the ring mediates against the consumer.
- Anticipation splice (
set_anticipation_enabled, default OFF; runs on the
canonical executor path). When enabled + the routed snapshot is eligible + the
graph has an eligible latent interior, compile_ builds an AnticipationLane +
a skip_mask over the routed plan (the interior nodes) + a prefill map (each
lane output channel โ the interior boundary-source's exec_pool output slot).
The host drives pump_anticipation() from ONE background thread (the producer);
process() consumes a pre-rendered block, copies it into the prefill slots (or
zeros them on underrun / block-size mismatch), and runs process_routed with the
interior masked โ bit-identical to the canonical interior-live render. GOTCHAS:
(1) the branch is TERMINAL once entered โ on a routed failure it zeros the output
and returns rather than falling through to a path that would re-run (double-
advance) the producer-owned interior. (2) pump_anticipation pins the live
snapshot (RCU object-lifetime only) and is single-producer-guarded, but the host
MUST stop/join the pump before any prepare()/mutation โ prepare reinitializes
the SAME plugin instances the pump renders (a data race otherwise; same rule as
"no process() during prepare"). (3) Host-clock-sensitive nodes opt in via the
per-node transport capability (wants_transport() / a transport-aware custom
callback โ cached GraphNode::transport_sensitive). The eligibility pass seeds
AnticipationExclusion::TransportSensitive on that bit, so a transport-sensitive
node โ and its downstream cone โ is EXCLUDED from the ahead-rendered interior and
always runs live/exterior, where it receives the live transport. The former
blanket transport suppression under anticipation is therefore RETIRED: the
transport stays populated on every block (inert for the transport-insensitive
interior). transport_suppressed_for_anticipation() is repurposed to count the
transport-sensitive nodes anticipation forced exterior (resolved at compile),
not per-block drops.
(A masked node must not be an AudioOutput or a feedback endpoint; the
partition guarantees this and process_routed debug-asserts it.) (4) The lane
uses a FIXED block size (the prepared max). A block of a different size โ or a
ring underrun โ silences the interior for that block (the interior is never
re-rendered live, so bit-identical-to-canonical holds only for fixed-size,
kept-up blocks); and an interior param/gain edit takes effect at render-ahead
time, a lead earlier than a live render. The anticipation branch is
STRUCTURALLY terminal once anticipation_valid โ it never falls through to the
parallel/legacy paths (which would run the producer-owned interior live), even
if the pool can't fit the block (then: silence).
connect() returns false on cycle โ always check. would_create_cycle
lets you preview without mutating.
processing_order() is recomputed each call; cache it in the audio
thread, don't recompute per block.
- Removing a node invalidates its
NodeId. Connections referencing a
removed node are pruned automatically.
- For fail-closed structural publication while audio remains live, use
begin_prepared_topology_edit() instead of mutating the owner eagerly. Build
the complete candidate through its PreparedTopologyEdit, call
prepare(sample_rate, max_block_size), verify
routed_execution_ready(max_block_size) when the caller holds a routed-only
lease, then commit(). A failed mutation poisons the one-shot edit, and
destroying it before commit rolls back topology, private connection IDs,
next IDs, the custom registry, routing flags, and the compiled snapshot.
Existing PluginSlots are never re-prepared off-side; dimension changes are
accepted only without plugin nodes and when every retained custom type has
neither prepare nor release. New edit-owned custom instances may be
prepared. Unchanged PDC rings carry by private connection identity plus equal
shape in the same execution domain; removed rings retire, while new,
reshaped, or reconnected rings start fresh. Preserve connection identity for
unchanged owned routes and prune unused generated custom types in the same
edit so registry churn stays bounded. MidiInput ingress may carry through
its shared sequence mailbox, but MidiOutput egress is snapshot-local so an
old snapshot can expose its pending output exactly once. A prepared edit
returns MidiOutputSnapshotLocalRequired before callbacks whenever either
the live or candidate graph contains a MidiOutput; do not adopt or share
that output mailbox. Baseline plugin removal, and baseline custom removal
when its registered type has release, are also explicit fail-closed
results: ordinary graph release remains the only lifecycle-callback owner.
Commit's exception boundary is before authoring mutation: preflight the
generic live runtime::Slot retirement capacity and reserve destination
node_load_ buckets first. After every failure gate, transfer load measurers
with C++17 unordered-map node handles (matching integral hash/equality and
allocator), move the authoring containers, and call the Slot's noexcept
prepared publication. Never put an allocating emplace or ordinary
Slot::publish after that boundary.
set_live_dsp_telemetry_enabled() is a control-thread operation serialized
by the graph mutation lock: a prepared commit re-seeds its snapshot from the
owner's authoritative desired toggle immediately before publication. Do not
move the toggle outside that lock or restore the candidate's creation-time
value; either change can update the retired snapshot while publishing stale
telemetry state.
A caller that has stopped audio processing and anticipation may instead use
prepare_quiesced() for a dimension change involving external plugins or
retained custom instances. Candidate preparation can touch those shared
lifecycle objects, so every non-commit exit โ candidate failure, routed
rejection, commit rejection, exception, or simple edit destruction โ must
restore each retained object whose candidate prepare callback was entered
before the old snapshot resumes. Track entry immediately before invoking user
code: candidate preflight can fail before every callback, and plugin/custom
preparation can stop midway. On an unprepared base, releasing an untouched
retained object is an unbalanced lifecycle call just as surely as failing to
release a touched one. A successful candidate prepare is not ownership
transfer; only a successful commit cancels that rollback obligation. If
restoration fails, the graph deliberately unpublishes and reports
QuiescedRollbackFailed; a coupled binding must unpublish too. Never resume a
partially restored graph. New custom instances created before a later prepare
failure still require their control-thread release callback.
- Per-node CPU load:
process() wraps each node's work in a persistent
per-node audio::AudioProcessLoadMeasurer (keyed by NodeId in
node_load_), read via node_loads(). The measurers live on the
SignalGraph (not the snapshot) and compile_() only ever ADDS to the map โ
never erase while a snapshot may be live, or the audio thread's raw
NodeRuntime::load pointer dangles. begin()/end() are relaxed-atomic and
RT-safe (proven under the no-alloc trap in test_signal_graph_rt_safety).
- Per-node live-DSP telemetry (
audio::LiveDspTelemetryStore): richer than the
load measurer โ fixed-slot p50/p95/p99 + jitter + over-budget attribution.
Disabled by default (set_live_dsp_telemetry_enabled(); the audio path is one
predicted-not-taken branch when off); drain + read a snapshot copy via
poll_live_dsp_telemetry() (single non-RT poller). Unlike node_load_, the
store is PER-CompiledGraph (not a SignalGraph member): it rides the RCU
snapshot lifetime, so telemetry resets on a topology recompile (a new topology
is a new timing baseline) and no re-prepare races the audio thread. Recording
is PATH-AGNOSTIC and lives at ONE site: a guard at the top of process_impl
destructs after the block (reverse-order vs the graph-load end guard) and reads
the values BOTH the routed serial executor and the legacy walk already stamped
into each node's persistent AudioProcessLoadMeasurer + graph_load_, then
pushes one fixed-slot record via inject_block() over a pre-sized scratch
(external_record_scratch()). This is why there is NO per-node hook in the
executor or the walk โ both already time per node, so the store just harvests
those measurements once per block. canonical_executor_routing_enabled_
defaults TRUE, so the serial executor is the common path; harvesting the
measurers (not hooking the walk) is what makes telemetry work by default.
Per-node slot == the node's index in ordered_runtime, matching the store
metadata built at compile.
.pulpgraph schema changes must go through the graph serializer migration
path. Bump the graph format version, add a deterministic migrator for older
fixtures, and keep future-version loads fail-closed instead of silently
accepting fields the current reader does not understand.
- Use
connect_automation() for sparse two-point-per-block control events.
Use connect_audio_rate_modulation() only for continuous, automatable
HostParamInfo::rate == AudioRate params; do not route dense CV into
stepped/read-only/control-rate parameters.
- MIDI graph edges carry one block with three parallel payloads: short MIDI
events, SysEx, and optional UMP sidecars. When copying or clearing graph MIDI
scratch, handle all three together. If a
MidiBuffer attaches a UmpBuffer
owned by NodeRuntime, attach it only after the runtime object is in its
final CompiledGraph storage; attaching before a move leaves a stale sidecar
pointer.
SignalGraph::inject_midi() and extract_midi() cross the
control/audio-thread boundary through per-node mailboxes, not by mutating
audio-thread scratch directly. After prepare(), injection is noexcept,
fixed-capacity, lock-free, and allocation-free, so it may run on the audio
callback immediately before process(). Each MidiInput has exactly one
writer โ audio or control, never concurrent. A false return means the live
node is unavailable or the source was truncated; any retained prefix is
still published. Publications are latest-wins and one-shot, and sequence
wrap skips zero. A gap-free prepare_swap() shares the ingress mailbox and
consumed sequence for a stable MidiInput NodeId, preserving unconsumed MIDI
across the swap. MidiOutput egress remains snapshot-local but uses an ordered,
fixed four-block SPSC queue: empty blocks cannot overwrite pending output;
overflow retains the earliest blocks and makes extraction incomplete. When
the destination lacks short-event, SysEx, or attached UMP-sidecar capacity,
extraction returns false and retains the undelivered suffix. Provide storage
and call extract_midi() again; it resumes without replaying the delivered
prefix. When
prepare_swap() returns NeedsEagerPrepare, the old live snapshot remains
valid, so drain it with extract_midi() before eager prepare() replaces it.
SignalGraph::inject_parameter_events() uses a separate prepared per-node
mailbox with one control-side writer. Publications are latest-wins and
one-shot: the next successful serial, routed, parallel, or anticipation
block consumes the newest sequence once. Append injected events after graph
automation before the stable sample-offset sort; this preserves graph
automation when the fixed queue is full and lets injected events win a
same-offset tie. Reuse the mailbox across gap-free snapshots for the same
plugin node so an edit cannot discard a publication made before the swap.
- Keep plugin automation scratch preallocated by
SignalGraph::prepare().
The audio-thread process() path must not create per-block containers for
input pointer casts, sparse automation accumulation, or dense audio-rate
modulation accumulation.
- Custom graph nodes are registered per
SignalGraph with CustomNodeType
(type_id, version, port counts, default name, optional process
callback), then instantiated with add_custom_node(type_id) or
add_custom_node(type_id, version). GraphSerializer resolves exact
(type_id, version) matches with the saved port shape, preserves unresolved
custom identities as placeholder NodeType::Custom nodes, and reports them in
LoadResult::missing_custom_node_types, so do not coerce unknown node
strings to a built-in type. Runtime callbacks are attached only when the
registered version and shape match the node.
- Stateful custom nodes.
CustomNodeType has an
optional lifecycle: set create and the graph owns one opaque instance per
node (RAII via destroy); process_instance runs instead of the stateless
process, and prepare/release/reset/save_state/load_state operate on
it. Empty callbacks = today's stateless node (no instance, no serialized
state). The instance is created/prepared on the UI thread inside
SignalGraph::prepare() (mirroring PluginSlot) and captured into each
CompiledGraph snapshot by shared_ptr โ never allocate or create instances
on the audio thread, and never store a raw GraphNode pointer in the snapshot.
process_instance must be RT-safe; call save_state/load_state only on the
control path (graph not live, or after invalidate + re-prepare). Opaque state
is std::vector<uint8_t> via SignalGraph::custom_node_state /
set_custom_node_state; GraphSerializer persists it as state_b64 and keeps
the blob even for unresolved nodes (save โ load-missing-type โ save keeps
state). Do not pull the pulp_native_state_* C ABI into CustomNodeType;
that belongs to the pulp_node_v1 ABI.
- Signed node-pack loader (
core/host/node_pack.{hpp,cpp}).
load_node_pack(dir, manifest, trust) loads a precompiled pulp_node_v1 node
pack (a .dylib/.so/.dll exporting pulp_node_v1_entry + a JSON manifest).
It verifies trust BEFORE any dlopen: the signer key must be in the
NodePackTrust set, the Ed25519 signature over node_pack_signed_message()
(pack_id + abi_major + binary SHA-256 + declared nodes/resources/requirements)
must be authentic, the on-disk binary's SHA-256 must match the signed hash,
and the entry's abi_major must match โ any failure returns a
NodePackError and loads nothing. Revocation = drop a key from the trust set.
Desktop + Android only; pulp-host (and this loader) is compiled out on iOS,
where native components are static-bundled + signed with the app. The crypto
comes from pulp::runtime (ed25519_verify, sha256_hex); OS
codesign/notarization is a separate, additional distribution step on top of
the manifest signature. Registry/package discovery metadata still needs its
own signed canonical manifest; do not treat screenshots, validation reports,
licenses, or provenance as covered by the node-pack loader signature.
- Routing a
SignalGraph through the canonical executor
(core/host/signal_graph_executor_routing.{hpp,cpp}). The eligible subset is
described above under "Canonical-executor routing" and enforced by
signal_graph_topology_executor_eligible() /
signal_graph_executor_eligible(). The builder fails closed for unsupported
Custom nodes, placeholder Plugin nodes, and per-node automation counts above
the fixed scratch caps. build_signal_graph_executor_routing() translates an
eligible prepared graph into a format::GraphRuntimeSnapshot + pre-sized
GraphRuntimeBufferPool; the live process() path embeds that snapshot and
its scratch pool per CompiledGraph, so a re-prepare rebuilds fresh routing
state without resizing buffers an in-flight audio reader holds. The routing
keeps the live compiled snapshot alive, reads live gain atomics, and invokes
the snapshot's live PluginSlots, so rebuild routing after any re-prepare
and keep this section aligned with test_signal_graph_executor_parity.
Offline graph rendering (OfflineSignalGraphHost)
core/host/offline_signal_graph_host.{hpp,cpp} renders a prepared SignalGraph
offline by stepping a fixed block size across a frame range through the public
SignalGraph::process() โ no live audio device, deterministic, allocation-free per
block (staging + output buffers are sized in prepare()). It is a control-thread
host, not a routing path: it adds no walk of its own and never touches graph
internals, so it stays clear of the in-flight routing/anticipation churn.
Gotchas:
- Block-size silence clamp.
SignalGraph::process() zero-fills any block larger
than the prepared max_block_size (prepared_max_block_size()). prepare() refuses
if the configured block_frames exceeds the graph's prepared max โ otherwise an
offline "one big block" render would silently drop to silence. To render one big
block, re-prepare() the graph at that block size first.
- What "offline equals online" actually means here. A
SignalGraph carries no
ProcessMode/transport into its nodes, so an offline render is NOT distinguishable
from an online one by render mode โ the only variable is the block partitioning. For
deterministic nodes, output is therefore block-size invariant: same input at any
block size โ bit-exact for pure gain/sum, within ~1e-6 across re-partitioning. A node
whose output legitimately depends on block size (the exempt path) is declared
EXEMPT as harness-side metadata today (no per-node ProcessMode opt-out exists yet);
the equivalence harness flags and excludes it rather than failing.
- Keep the executor/parallel/anticipation opt-ins OFF for partition-invariance
fixtures โ anticipation in particular is intentionally not block-size invariant.
Baking a graph to a Processor (BakedGraphProcessor)
core/host/baked_graph_processor.{hpp,cpp} โ bake(const SignalGraph&) freezes a
prepared, fully-lowerable graph into one pulp::format::Processor that runs a frozen
GraphRuntimeSnapshot through the SAME GraphRuntimeExecutor::process_routed() the
live graph uses, so baked output is bit-identical to the live graph for the lowerable
subset. The artifact is a serialized fused plan (data), not generated code โ it
reuses the one backend, so the baked Processor only CALLS process_routed, never
defines a routing entry point.
Gotchas:
- Lowerable subset is narrow by design. Today:
AudioInput/AudioOutput/Gain,
plus a Custom node whose registered type opts in (lowerable = true, shape
match, transport-independent). bake() REFUSES loudly (null processor + a
LowerRejectReason) for an unprepared or executor-ineligible graph, a hosted
Plugin node (opaque external state โ not self-contained), or a Custom node that
does not meet the opt-in bar. The node-kind refusals are checked BEFORE the
eligibility predicate so a Plugin/Custom graph reports its specific reason instead
of a generic NotExecutorEligible.
- The baked Processor owns its Gain values.
bake() copies each Gain's value into
the Processor; prepare() seeds one heap-stable atomic<float> per Gain (a
unique_ptr vector, never a value vector) and resolves the routed Gain bindings to
those owned atomics โ so the baked Processor is independent of the source graph's
live snapshot lifetime. A second prepare() clears the old snapshot/pool/atomics
before rebuilding, so binding pointers never dangle.
- Sizing mirrors live routing.
prepare() builds the snapshot via the same
build_executor_snapshot() the live routing uses and sizes the pool from
buffer_slot_count() ร max_buffer_size plus the per-connection PDC rings, so
process() is allocation-free.
- bake() captures topology + gain values, not hot runtime state. The baked
Processor builds fresh feedback/delay/scratch in
prepare() and starts from zero;
a source graph that has already processed blocks does not transfer its feedback
history. The parity proof covers both directions โ baked output is bit-exact to the
live graph's legacy WALK and to its routed executor (the test asserts the walk case
explicitly by forcing routing OFF, since canonical-executor routing is now ON by
default).
- Signed
.pulpbake files carry authored Custom state, never sampled live
state. bake_to_plan() copies each node's staged
GraphNode::custom_state_blob (set through set_custom_node_state()); it must
never call a live instance's save_state() because bake can run while DSP is
processing and that callback has no concurrent-process contract. Temporal DSP
history is intentionally excluded. Stage the intended publish state before
prepare().
- Disk load verifies first and restores fail-closed.
load_baked() verifies the
Ed25519 signature before bounded parsing, rejects duplicate plan node ids and
duplicate host registry identities, then resolves every Custom record by exact
type/version/shape. A stateful record requires a valid create +
load_state lifecycle (an empty byte span may still be meaningful); null instance
creation or rejected state names the exact offending node and aborts the whole
load. The authenticated blob is restored after the Custom lifecycle's
prepare/reset on every baked prepare(), so those hooks cannot silently erase
authored state. The in-memory bake() path remains a fresh-state stream and does
not perform this disk restore.
- In-place hosts alias input over output โ never let the executor's output-zero
destroy the input.
process_routed() zeroes the main output bus BEFORE its
AudioInput gather reads the input bus (AudioOutput nodes accumulate, so N sinks
mix). Logic-style hosts (AUv2, some AUv3) hand process() input and output views
over the SAME memory, so that zero used to wipe the input โ total silence.
BakedGraphProcessor::process() now detects any input-channel/output-channel
overlap and reads the input from a scratch copy sized in prepare() (audio thread
does pointer compares + copy_n only โ no allocation). Any OTHER direct caller of
process_routed() that bridges host buffers must apply the same guard; the
executor itself deliberately keeps the zero-then-gather order.
- Stateful Custom instances need their lifecycle re-run at baked
prepare().
The captured CustomNodeProcessFn is an opaque closure over the instance; the
instance's prepare/reset hooks are NOT inside it. bake() therefore also
captures per-node CustomNodeLifecycle closures (type prepare + reset bound to
the instance shared_ptr) and BakedGraphProcessor::prepare() runs them โ re-prepare
at the HOST's real rate/block (load_baked only prepares at a nominal 48k/512), then
reset so stale DSP state (a delay line's contents) never survives a re-prepare.
Note the instance is SHARED with the source graph โ the baked path does not clone
it โ so a baked re-prepare also resets that node in a still-live source graph.
Bake-layer parameter injection (control-thread writes into a baked node)
A baked custom node's parameters can be changed at runtime โ sample-accurately,
RT-safely, without re-baking โ via the bake-layer injection primitive:
- Declaring: a
CustomNodeType opts in by filling baked_params (id + range +
default per param) and providing process_instance_baked_param, a param-aware
process callback that reads values through a BakedParamView
(value_at(id, sample_offset) โ offsets must be non-decreasing within a block).
That baked-param DSP runs ONLY in the baked Processor, never on the live graph.
- Injecting:
BakedGraphProcessor::claim_param_injection(node) hands back a
move-only ParamInjector โ an EXCLUSIVE per-node claim (a second claim fails until
the first is released; the handle survives re-prepare). inject() publishes into a
single-writer per-node mailbox the baked process() drains next block; events are
pulp::state::ParameterEvents (immediate or ramped), and a ramp longer than one
block carries across blocks to completion.
- One-param-per-block-or-batch contract:
inject(ParameterEventQueue) is the
batch path โ the whole queue lands as ONE sample-accurate batch, and the latest
published queue REPLACES a still-pending one. inject(ParameterEvent) (single)
ACCUMULATES: it merges into the still-unconsumed pending batch, superseding only
that param's pending entries, so N single injects to different params between
blocks all land. (Pre-fix this was latest-snapshot-wins โ two single injects with
no intervening process() collapsed to the last one, silently dropping a param.
If you need many events for one param in one block, use the queue path; single
inject returns PartialOverflow only when the pending batch is already full of
other params' events.)
test/test_baked_graph_param_injection.cpp is the executable spec (claims, ramps,
sample accuracy, RT-allocation-free drain, the accumulate regression).
Not every knob belongs in baked_params
baked_params is for values a node can accept at ANY sample. A value that
changes the node's TOPOLOGY โ which stages exist, how a buffer is laid out, or
anything whose setter designs a filter โ must be a registration/construction
choice with its own type_id instead, following the per-mode "svf" pattern.
Two reasons, both learned the hard way:
- The artifact's identity. A baked build authored as one thing must stay
that thing for the artifact's life; a control-thread write should not be able
to turn a tape delay into a BBD mid-render.
- The setter runs on the AUDIO thread. Anything reachable from
process_instance_baked_param inherits the RT contract transitively. In
forge_character_delay_catalog.hpp the tape TIER and tape SPEED are
construction config precisely because changing the speed redesigns a bank of
FIRs; the age macro next to them is a baked param only because its filter
banks are pre-designed at prepare() and the audio thread merely interpolates
between two of them.
A related ordering trap: a node's prepare() typically configures construction
options and only THEN calls the DSP's set_sample_rate, so every config setter
runs while the instance is still unsized. Those setters must tolerate being
called before allocation โ store the value and let prepare() design against it
โ or they walk buffers that do not exist yet. That is an out-of-bounds write
that only fires for nodes constructed away from their defaults, so it survives
casual testing; test_character_delay.cpp has an explicit regression case for
it ("configuring the tape speed before the sample rate is safe").
Per-sample params on a block-oriented DSP
When the wrapped block processes buffers rather than single samples, the
faithful wrapping is to call it one sample at a time and apply every param each
sample. That is only cheap if the DSP's setters are stores rather than work โ
smoothing and coefficient recomputation belong INSIDE the block, on its own
control-rate cadence. A wrapper that instead calls setters which recompute
filters is doing a filter design per sample per param. Check what a setter costs
before you put it in the per-sample loop.
Catalog nodes with an internal control cadence read params per CHUNK
pulp/host/forge_fdn_reverb_catalog.hpp (the multirate FDN reverb) is the
pattern to copy for any wrapped engine that runs its own control rate. Its
process_instance_baked_param walks the block in 32-sample chunks and re-reads
the BakedParamView at each one, rather than sampling once per block.
Two things follow from that, and both bite if you copy only half of it:
- Read at the engine's cadence, not the block's. Reading once per block
makes a knob sweep step audibly at large buffer sizes; reading per sample
would be discarded by an engine that re-derives on a 32-sample tick anyway.
Match the engine.
- A param that reconfigures the engine must land on a CHUNK BOUNDARY. The
reverb's
tank_rate re-derives every delay length, filter coefficient and
resampler ratio. Applying that between the two halves of a resampler โ after
the input leg produced its samples, before the output leg consumed them โ
desynchronizes them permanently: a switch UP in rate silenced the wet output
for good, and it never recovered, because the deficit was re-created every
block. The engine now applies a pending rate change before either leg runs.
If you wrap something with a similar "reconfigure everything" param, apply it
at a boundary and add a test that switches in BOTH directions and compares
against a cold render โ a one-directional test passes over this bug.
Node shape matters to the host too: this node is true stereo (2 in / 2 out
as one logical wire, not two mono halves) and wet only, so a graph that
wants dry needs a make_drywet_node after it. test/test_fdn_reverb_catalog.cpp
covers the injection path, the true-stereo claim and the RT probe across a live
rate change.
Common tripwires
- Instruments have no input bus โ never address input element 0 blind.
An AU instrument (
aumu) and a generator (augn) expose zero input
elements; so does a MIDI processor (aumi, which despite the name is
kAudioUnitType_MIDIProcessor, not an instrument โ and it may expose no
output element either). A MIDI effect (aumf) does have audio input and
is unaffected. Setting a per-element input property on an input-less AU โ
kAudioUnitProperty_StreamFormat, kAudioUnitProperty_SetRenderCallback โ
returns kAudioUnitErr_InvalidElement (-10877), not a format error.
Treating that as fatal rejects every instrument on the system while every
effect keeps working, so the failure is invisible to effect-only tests. Ask
kAudioUnitProperty_ElementCount on the scope first and skip the input-side
setup when it is 0 (scope_element_count() in
core/host/src/plugin_slot_au.mm). When the AU does not answer the query,
assume an input bus exists: a non-answering effect then behaves as it
always did, and a non-answering instrument fails loudly at the property set
with a named scope+status. Assuming none would skip input setup on that
effect and leave every AudioUnitRender failing while the caller's buffer
keeps stale contents โ silent wrong audio, the worst of the four outcomes.
The general rule for any backend: derive the bus layout from what the plug-in
reports, never from the assumption that an input side exists. The other
slots already do this and are the pattern to copy โ VST3 loops
component_->getBusCount(kAudio, kInput) and ignores setBusArrangements'
status ("missing buses degrade gracefully"); CLAP and VST3 both size
ProcessData from the caller's view. AU was the outlier.
- The LV2 slot cannot safely host an instrument โ and it fails as UB, not
cleanly. Port discovery keeps only
lv2:AudioPort/lv2:ControlPort stanzas
(plugin_slot_lv2.cpp, if (!is_audio && !is_control) continue;), so atom /
event / CV ports are never seen and never connect_port'd โ yet run() is
called anyway. The LV2 spec requires every port be connected before
run() unless it is lv2:connectionOptional; running with unconnected ports
is undefined behavior and commonly segfaults. Every LV2 instrument has an atom
MIDI input port, and many effects carry atom ports for transport. There is
also no MIDI delivery path for LV2 at all. Unlike the AU trap above this does
not fail loudly โ so do not claim "Pulp hosts instruments" unqualified:
AU yes, VST3/CLAP plausibly, LV2 no. Fixing it means an atom-sequence input
buffer + MIDI mapping; the minimum stopgap is to detect non-audio/non-control
input ports at discovery and refuse prepare() loudly unless
connectionOptional.
- Test hosts against a real instrument, not just a real effect. The
effect-only integration test in
test/test_plugin_slot_au.mm passed happily
through the bug above. Apple's bundled DLSMusicDevice
(kAudioUnitType_MusicDevice + kAudioUnitManufacturer_Apple) ships on every
Mac, so an instrument fixture costs nothing โ
first_apple_instrument_unique_id() is there for this. Any host change that
touches bus/format negotiation needs both shapes, or half the plug-in universe
goes untested. (These tests WARN-and-return when no system AU is registered;
a headless VM may not surface Apple's AUs, so treat a green run in CI as
"not disproven" rather than "covered" โ a skip is never a pass.)
- Building
pulp-host without adding a new .cpp to target_sources โ
the file sits on disk but isn't compiled; link errors fire only in the
dispatcher's case. Always update core/host/CMakeLists.txt
alongside adding a backend.
- Missing
PULP_HOST_HAS_<FMT> define โ dispatcher silently returns
nullptr. Verify grep PULP_HOST_HAS_ build/CMakeCache.txt after
configure.
- CLAP bundles on macOS: don't
dlopen the .clap directory; resolve to
the executable inside Contents/MacOS/ first.
- LV2 manifest URI extraction must only use subject-position
<URI> tokens.
A manifest stanza like <plugin> rdfs:seeAlso <plugin.ttl> ; a lv2:Plugin
should identify <plugin>, not the seeAlso object. Keep parser coverage
in test/test_plugin_info_metadata.cpp or test/test_lv2_host_discovery.cpp
when changing core/host/src/scanner.cpp.
- LV2 invalid-bundle tests deliberately use placeholder
.so / .dylib files.
Keep the loader's magic-byte preflight before dlopen / LoadLibrary so
invalid modules fail quickly and consistently on Windows instead of waiting
on the platform loader.
- A slot's per-block scratch must be reserved in
prepare(), not grown in
process(). The CLAP slot fills in_ptrs_/out_ptrs_ and emplaces into
in_event_storage_ each block; on a default-constructed vector the first
resize/emplace_back allocates on the audio thread. Reserve the channel
vectors from PluginInfo::num_inputs/num_outputs (the graph sizes node
buffers from these, floored at stereo) and the event scratch for
params_.size() + ParameterEventQueue::kCapacity + the realtime MIDI cap.
Guard with a PULP_DBG_ASSERT(capacity >= needed) tripwire (debug-only).
This holds for the graph-driven path; a direct caller passing more channels
or an un-capacity-limited MidiBuffer is outside the contract. No-alloc
coverage lives in test_host.cpp ("ClapSlot::process is allocation-free
after prepare() reserves"), gated on PULP_TEST_CLAP_PATH.
- The AU slot (
plugin_slot_au.mm) has the same rule for its output
AudioBufferList: AuSlot::process builds an ABL pointing at the caller's
channels every block. Size the backing abl_storage_ once in prepare()
(num_channels_) via au_internal::reserve_audio_buffer_list and only
refill it per block (fill_output_audio_buffer_list) โ never allocate a
fresh std::vector in process(). The ABL build lives in
plugin_slot_au_internal.hpp so its no-alloc invariant is unit-tested
(pointer-stable across thousands of refills) without a live AU;
test_plugin_slot_au.mm additionally drives a real system Apple effect AU
through process() (skips honestly when none is registered โ headless CI
may surface no AUs). Do NOT assert allocs==0 over AudioUnitRender itself
(Apple allocates internally); assert the reuse invariant on our buffer.
- The VST3 slot has the same channel-vector issue plus extra per-block
allocation inside the Steinberg helper containers it builds each block
(
Vst::ParameterChanges / EventList from public.sdk/.../hosting), so
reserving in_ptrs_/out_ptrs_ alone does NOT make Vst3Slot::process
allocation-free โ a PULP_TEST_VST3_PATH no-alloc test against
PulpGain.vst3 still trips. Making the VST3 slot RT-safe needs those SDK
containers pre-sized too; tracked as a follow-up, not yet done.
- Fixture wiring (
PULP_TEST_CLAP_PATH, future PULP_TEST_VST3_PATH) lives in
the ROOT CMakeLists.txt block after add_subdirectory(examples) โ NOT in
test/CMakeLists.txt, which is registered before examples/ so it cannot
see the PulpGain_* targets at configure time. A guard placed in test/
silently never runs (its define just appears stale in an incremental build).
Audio-thread snapshot contracts
The host exposes a reader-pinned audio-thread snapshot, not direct member
reads. Anything you write that touches the audio thread (a
graph editor, an MCP bridge, a preset loader) must account for these
rules:
- The snapshot lives in
runtime::Slot<CompiledGraph> (live_slot_), the
shared reader-pinned RCU primitive in core/runtime/include/pulp/runtime/slot.hpp.
It owns the atomic pointer the audio thread loads, the seq_cst reader count,
and the retire list. SignalGraph no longer hand-rolls any of that; the old
live_ / live_raw_ / retired_snapshots_ / active_process_readers_ /
ProcessReadGuard / retire_snapshot_ / prune_retired_snapshots_ /
wait_for_retired_snapshots_ are gone. Don't reintroduce them.
- A pin guarantees LIFETIME, not constness. This is the single most
misread part of the contract.
CompiledGraph is not immutable โ the audio
thread writes every node's scratch buffer through the pin on every block,
inject_midi writes mailboxes, drain() consumes telemetry, set_node_gain
writes a gain. What is immutable is the topology. Slot::ReadGuard::get()
therefore hands back a mutable T*; a genuinely read-only publication says so
in the type (Slot<const T>).
- Pin the exact committed generation when publications are coupled.
ExecutionSnapshot is a strong handle to one specific compiled graph, and its
MIDI, parameter-event, and process() methods never redirect to a newer live
graph. Generic inject_parameter_events writes a node's live mailbox; timeline
device automation instead uses inject_exact_parameter_events (passkey-gated)
into a separate owner-claimed exact-generation mailbox, so a claimed node's
timeline stream and ordinary live injection never share one mailbox. A ramp
event is delivered at its start offset with its ramp duration preserved, so a
hosted adapter that consumes ParameterEvent::ramp_duration_sample_frames
glides across the block instead of stepping at the endpoint.
TimelineGraphBinding publishes that handle together with its immutable
playback program and bound track renderers as one runtime::Slot generation.
Topology and content adoption must replace that one generation; independently
latching the program store or looking up the graph's current live snapshot can
produce a mixed old/new audio block. As with ordinary graph processing, only
one audio thread may process these mutable execution snapshots at a time.
- Read it, don't reach for it. Anything that dereferences the snapshot off
the prepare/release thread must hold a pin for the whole dereference:
auto pin = live_slot_.read(); if (auto* cg = pin.get()) { ... }. That
includes control-thread readers (inject_midi, extract_midi,