| name | concurrency-control |
| description | Use when a model emits multiple tool calls in a single turn and the harness must execute them in parallel safely, or when subagents/forks run concurrently and share state. Apply when designing the concurrency model for tool dispatch. Covers parallel tool execution, ordering guarantees, cancellation propagation, race detection on shared resources (filesystem, db), partial-failure semantics, and result aggregation. |
Concurrency Control
Pain Signals — You Need This Pattern When:
- Model emits multiple tool calls per turn; serial execution multiplies latency
- Subagents run concurrently and may touch the same workspace
- Tool calls of mixed cost (fast read, slow API call) serialized waste time
- Two tool calls write to the same file and the second silently overwrites the first
- Cancellation needs to stop a fan-out cleanly without zombie operations
Concurrency is latency leverage. Done right: 5× speedup on independent work. Done wrong: race conditions, partial state, debugging nightmares.
Core Principle
Three orthogonal questions per turn:
- Independence. Can these tool calls run without affecting each other?
- Ordering. Does the model expect results in a particular order?
- Failure semantics. What happens when one of N fails — proceed, abort all, retry?
The harness answers these per turn, not globally. Some turns are pure-parallel; some are strictly sequential; many are mixed.
Independence Detection
The model often emits multiple tool-use blocks in one response. They are not always independent.
| Independent | Conflicting |
|---|
Read(a.py), Read(b.py) | Edit(a.py), Edit(a.py) |
Grep(...), Bash(git status) | Bash(cd foo), Bash(ls) (cwd state) |
WebFetch(url1), WebFetch(url2) | Write(out), Read(out) |
| Two reads to same DB | Read-after-write to same row |
Heuristics:
- Reads are always parallelizable (within rate limits)
- Writes to disjoint paths/keys are parallelizable
- Writes to overlapping resources serialize
- Stateful tools (shell with cwd, REPL session) serialize within their state
Build a per-tool concurrency descriptor:
class ToolConcurrency:
safe_parallel: bool
resource_keys: Callable
serializes_with: list[str]
The dispatcher computes a conflict graph from the turn's tool calls and schedules accordingly.
Parallel Dispatch
async def dispatch_turn(tool_calls: list[ToolCall]) -> list[ToolResult]:
groups = topological_groups(tool_calls)
results: dict[str, ToolResult] = {}
for group in groups:
group_results = await asyncio.gather(
*(execute(tc) for tc in group),
return_exceptions=True,
)
for tc, r in zip(group, group_results):
results[tc.id] = normalize(r)
return [results[tc.id] for tc in tool_calls]
Key properties:
- Results returned in the original order the model emitted (so the model sees a predictable sequence)
- Failures wrapped, not raised (one failure doesn't crash the gather)
- Group boundaries enforce serialization where conflicts exist
Ordering vs Parallelism
The model's output order is the intent; execution order is harness's choice as long as causal dependencies are preserved.
- Result feed order = intent order. Always feed
tool_result blocks back in the same order the model emitted tool_use blocks. Out-of-order feeds confuse the model.
- Execution order ≠ feed order. Parallel execution finishes in completion order; collect and re-sort to intent order before feeding.
This is non-negotiable. Without it, the model believes the second tool returned what was actually the first's result.
Race Detection
Even with independence analysis, races slip in. Defenses:
- Optimistic + verify. Tool reads include a hash/mtime; subsequent edits assert it; on mismatch, fail with "external mutation detected." Pairs with workspace-state skill.
- Locks per resource key. Cheap shared map of
key → asyncio.Lock for resources without first-class locking.
- Idempotency keys. Especially for external API writes — same key = same effect.
- Two-phase write. Stage to temp; commit-rename. Handles partial-write crash.
async def edit_with_check(path: str, expected_hash: str, edit_fn):
async with workspace.lock(path):
actual = await workspace.hash(path)
if actual != expected_hash:
raise StaleReadError(path)
await edit_fn(path)
return await workspace.hash(path)
Partial Failure
Three policies; choose per turn or per tool class:
| Policy | Behavior |
|---|
| All-or-nothing | One failure → cancel siblings, return partial results to model with error context |
| Best-effort | Failures recorded; siblings complete; model decides next action |
| Quorum | Need M of N successful; below threshold = turn failure |
Best-effort is the right default for read-heavy parallel turns. All-or-nothing for write fan-outs where partial state is incoherent. Quorum for redundant-source patterns (search 3 indexes, need 2).
The model should always see the full result map — successes and structured failures. Truncating to "first error wins" hides information the model needs to recover.
Cancellation Propagation
Cancellation must reach every in-flight task.
async def dispatch_with_cancel(tool_calls, cancel_event):
tasks = [asyncio.create_task(execute(tc)) for tc in tool_calls]
done, pending = await asyncio.wait(
tasks, return_when=asyncio.FIRST_COMPLETED
)
if cancel_event.is_set():
for t in pending:
t.cancel()
await asyncio.gather(*pending, return_exceptions=True)
...
Cancellation must:
- Reach the tool's underlying I/O (close HTTP connection, kill subprocess)
- Mark partial results as cancelled in transcript
- Not leave orphan resources (locks, temp files)
Tools should accept a cancellation token / context and check it at await points.
Subagent Concurrency
Concurrent subagents inherit all the above plus:
- Workspace isolation per subagent. Different worktrees, sandbox dirs — concurrent writes safe.
- Token budget partitioned. Don't let one subagent exhaust the session budget while siblings run.
- Shared resource access policed. If two subagents query the same paid API, rate limit applies across them.
Most subagent concurrency bugs trace to "subagents both wrote to the same file and the second won." Solve at workspace-isolation layer, not at runtime detection.
Bounded Concurrency
Unbounded parallelism causes new problems: rate limit hits, fd exhaustion, memory pressure.
class BoundedDispatcher:
def __init__(self, max_concurrent: int = 8):
self._sem = asyncio.Semaphore(max_concurrent)
async def execute(self, tc):
async with self._sem:
return await self._tool.run(tc)
Cap based on what the harness's slowest dependency tolerates (model API rate limit, downstream service capacity). 8–16 is a reasonable starting point; tune from telemetry.
Common Pitfalls
| Pitfall | Symptom | Fix |
|---|
| Serial dispatch of independent tools | Latency multiplied | Parallel dispatch with conflict graph |
| Out-of-order result feed | Model misinterprets results | Sort to intent order before feeding |
| Two writes to same file | Last writer wins silently | Per-resource lock + stale-read check |
| Cancellation stops dispatcher, not tools | Zombie operations continue | Propagate to tool execution |
| First failure cancels all | Useful results discarded | Best-effort default; explicit policy |
| Unbounded parallelism | Rate limit storm, fd exhaustion | Semaphore-bounded dispatcher |
| Subagent shares parent workspace | Crossed writes | Worktree isolation per subagent |
| Stateful tool (cd, REPL) parallelized | Nondeterministic state | Mark serializable; run sequentially within state |
| Race in idempotency-key generation | Duplicate ops on retry | Deterministic key from (turn, tool, args) |
When NOT to Use (Concurrency)
- Truly sequential workflows (tool B uses tool A's output) — parallelism gives nothing
- Single-tool turns — overhead exceeds benefit
- Heavy state machines where reasoning about concurrent state is harder than the latency saved
- Strict ordering business requirements (financial, audit) — make it explicit and serial
Design Checklist
