| name | modify-distributed-cluster |
| description | Distributed cluster patterns for the CheddarFlow project. TRIGGER when: writing or modifying code involving cross-node communication, CFXRpc, libcluster, Phoenix.PubSub broadcasting, distributed Erlang, or node routing. Also trigger when working with SharedUtils.Cluster, CFXPubSub, or any code that needs to handle dev/test vs prod cluster differences. DO NOT TRIGGER when: working with single-node code that doesn't involve distribution. |
Distributed Cluster Architecture
CheddarFlow runs as a distributed Erlang cluster with 11+ nodes communicating via Phoenix.PubSub and :erpc.
Cluster Formation
- Production: Auto-discover via
libcluster using Cluster.Strategy.EC2Tag with tag Group: "CFX Backend"
- Dev/Test: Clustering disabled —
cfx_rpc config call_directly?: true makes RPC calls local
- Each release app joining the cluster starts
SharedUtils.Cluster.toplogy_supervisor/1
CFXRpc — Cross-Node RPC
Wraps :erpc.call/5 and :erpc.cast/4 with metrics, error handling, and routing:
CFXRpc.call_on_random_node("options_feed", Module, :function, [args])
CFXRpc.route_to_node("options_feed", partition_key, Module, :function, [args])
CFXRpc.call(node, Module, :function, [args], timeout: 10_000)
CFXRpc.cast(node, Module, :function, [args])
Key functions:
call_on_random_node/4,5 — Finds nodes matching name filter, picks random. Retries after 5s if none found.route_to_node/5,6 — Consistent-hash routing (partition term → specific node).call/4,5 — Direct :erpc.call with default 10s timeout.cast/4 — Fire-and-forget.
Error handling:
:badarg → ErrorMessage.bad_request/2:noconnection → ErrorMessage.service_unavailable/2:timeout → ErrorMessage.request_timeout/2
Metrics: All calls go through CFXMetrics.RpcMetrics.span_call/4 / span_cast/4.
CFXRpc.Config
CFXRpc.Config.call_directly?() # true in dev/test, false in prod
CFXRpc.Config.distributed?() # inverse
Pattern for conditional distributed features:
if CFXRpc.Config.call_directly?() or SharedUtils.Cluster.node_equal?(node_type) do
# Run locally
else
# Route via RPC
end
PubSub Topology
Three separate PubSub instances in cfx_pub_sub:
| PubSub Name | Pool Size | Used By |
|---|
CFXWeb.PubSub | pool: 4, registry: 6 | Web subscriptions, feature flag cache busting |
OptionsEventsProcessor.PubSub | pool: 2 | Options trade events |
DarkPoolEventsProcessor.PubSub | default | Dark pool/lit events |
Topic conventions via CFXPubSub:
"options_flow_events" — Bulk options flow"contract_activity" / "contract_activity:<symbol>" — Per-symbol activity"new_contract" / "new_contract:<symbol>" — New contract alerts"price_updates" / "price_updates:<symbol>" — Price changes
Use CFXPubSub.options_events_subscribe/1 and CFXPubSub.options_events_broadcast/2 instead of Phoenix.PubSub directly.
Node Naming
Production: <release_name>@<ip> (e.g., options_feed@10.0.1.55).
SharedUtils.Cluster.filter_nodes/1 matches on node name string.
Design Decisions
- Separate PubSub instances prevent event flooding across subsystems
- ETS for feed state avoids GenServer call bottlenecks
- Consistent hashing for feed routing enables per-query GenServer state reuse
- Retry with sleep in RPC handles transient cluster changes during deploys
