trade-path-audit

name

trade-path-audit

description

Retroactively audit an existing trading executor against every gate, error class, and adversarial check defined in /build-executor. Use when the user asks to "audit my executor", "check for trade-path bugs", "find violations of the new gates", "retrofit my executor with safety checks", "review live/paper code for known bug classes", or when an executor predates the current gate system and needs to catch up. Finds real bugs in already-written code — not a forward-looking build guide. Produces a severity-ranked findings report with file:line citations and fix recommendations.

Trade Path Audit

Audit an existing executor against every safety gate the build-executor skill enforces. This skill exists because:

1. Many executors were built before the current gates existed.
2. Build-executor is forward-looking (build X, verify X). This skill is backward-looking (what's already there, what violates the rules, how do we fix it).
3. The same error classes, adversarial personas, and pre-ship items apply — so this skill heavily reuses /build-executor's references instead of duplicating them.

Before you start

Ask the user which executor to audit. The skill operates on one executor at a time:

executors/<name>/

If there are multiple executors, audit them one at a time — each produces its own findings report.

Load these references (from /build-executor)

Do not duplicate these docs here. Load them from build-executor:

• .agents/skills/build-executor/references/error-classes.md — 21 structural failure patterns with gates
• .agents/skills/build-executor/references/adversarial-trading.md — 3 trading personas
• .agents/skills/build-executor/references/parity-rules.md — live/dry-run parity audit
• .agents/skills/build-executor/references/pre-ship-checklist.md — machine-verifiable checks
• .agents/skills/build-executor/references/agent-learnings.md — cross-cutting gotchas
• .agents/skills/build-executor/references/handicaps.md — the six statistical attestations (slippage, fees, CI, samples, regime, stress)
• .agents/skills/build-executor/references/operational-resilience.md — rate limits, websocket reconnect, clock drift, audit log

The audit procedure below maps each check to these documents. Do not re-explain the rules in your findings — cite the source.

Audit methodology: capability presence vs outcome verification

A recurring failure mode in this audit is marking a gate PASS because the code path EXISTS — "it calls cancelOrder," "it queries fill history," "it reconciles timeouts" — without checking whether the outcome is actually trusted. Both §15 and §16 were found in executors that passed every prior error-class grep: the code called cancel, queried fill history, and reconciled. What the audit missed was that the code then trusted an empty or ambiguous response as "safe" rather than as "unknown."

When you walk the error classes, for every venue-interacting code path, ask both questions:

1. Capability presence (easy): does the code call the API / reach the branch?
2. Outcome verification (hard): when the API call fails, returns a body with missing fields, or returns an empty result during the venue's own propagation lag, does the code halt with a fatal error, or does it mirror optimistic local state and continue?

If the capability exists but the outcome isn't verified, mark the finding CRITICAL. The gate is not "is there a cancel call" — it is "does an un-cancelled order ever survive as locally-mirrored-cancelled state." That second question is what keeps money safe.

This methodology applies most forcefully to §15 and §16, but it's a lens worth applying to §2, §5, §9, §11, §14, §17, §18, §19, §20 as well — any class where "the right code ran" is not the same as "the right outcome was verified."

Audit methodology: promoted numbers vs raw numbers (§17, §18)

A second, orthogonal gap. When auditing a PROMOTED strategy, do not stop at "does the scorer run?" Ask: "is the number the operator sees the honest number?" The hypepipe-class of surprise — "replay said +12%, live said -3%" — almost always traces to one of six unattested handicaps. When auditing a promoted strategy, walk every handicap in references/handicaps.md and verify the tear sheet displays it. Any missing handicap is a CRITICAL promotion bug, even if every code path works — promoting an unattested number is itself the bug.

Procedure

Step 1: Inventory

Before looking for bugs, understand what's there:

ls executors/<name>/          # top-level structure
ls executors/<name>/src/      # source layout
cat executors/<name>/experiment.md 2>/dev/null  # status, thesis, findings
cat executors/<name>/package.json | head -30

Write a short "Executor inventory" section in your findings covering:

• Venue(s) it trades
• Collection/scoring/live stages present
• What packages it uses
• Whether it has an experiment.md and current status
• Code size (rough LOC in src/)

Step 2: Run the machine-verifiable checks first

These are cheap and often conclusive. Both scripts below live in .agents/skills/build-executor/scripts/ — if either is missing from your repo, you have a partial skill install. Copy both before running the audit:

ls .agents/skills/build-executor/scripts/gate-check.sh .agents/skills/build-executor/scripts/parity-audit.sh

If either is missing, run pnpm skills:update (or re-copy from the template) before continuing.

Use the gate-check script for each stage (the promote and live stages tolerate missing research/evidence/ and missing src/ with graceful warnings rather than crashing):

bash .agents/skills/build-executor/scripts/gate-check.sh collect executors/<name>
bash .agents/skills/build-executor/scripts/gate-check.sh score executors/<name>
bash .agents/skills/build-executor/scripts/gate-check.sh promote executors/<name>
bash .agents/skills/build-executor/scripts/gate-check.sh live executors/<name>

The live stage's parity audit is invoked automatically by gate-check.sh live. You can also run it standalone:

bash .agents/skills/build-executor/scripts/parity-audit.sh executors/<name>/src

Record every output in the findings section. Failures here are CRITICAL — they're machine-verified violations.

Step 3: Walk the error classes

Open .agents/skills/build-executor/references/error-classes.md. For each of the 21 error classes, grep the executor source for the pattern, then manually verify whether the gate is upheld.

Some concrete checks to run (by class number):

• §2 (Concurrent writer corruption): grep -rn "acquireLock\|ProcessLock\|lockfile\|PID" executors/<name>/src. Must find lockfile usage in the collector entry point. If not, CRITICAL.
• §3 (String-format divergence): grep -rn "===" executors/<name>/src | grep -i "time\|date\|timestamp". Any raw === comparisons on timestamps? CRITICAL.
• §5 (Replay-time information leak): grep -rn "closeTime\|observedAt\|settlement" executors/<name>/src/scorer. Must find temporal ordering guards. If the scorer pre-loads future state, CRITICAL.
• §7 (Derived ask prices): check whether the collector stores a source flag on asks. If derivedAskShare (or equivalent) is untracked, WARNING.
• §9 (Stale credentials): grep -rn "getBalance\|credential\|preflight" executors/<name>/src. Must find startup validation. If not, CRITICAL for live mode.
• §14 (Fire-and-forget state updates): grep -rn "Promise.race\|\.then(\|setTimeout" executors/<name>/src. For each hit in the trading path (not just tests/utilities), verify the caller awaits the state-update chain. Untrusted hits are CRITICAL. Beyond the grep: apply the capability-vs-outcome lens from the methodology section. Any path where control returns to the scan loop while venue-side truth is still unknown — even if syntactically awaited — has the same failure shape as a literal fire-and-forget. Examples: returning recovered_canceled after a best-effort cancel, returning after a single fill-history query inside the venue's propagation lag, returning after a cancel call without inspecting the response. These pass a naive grep but fail the outcome check.
• §15 (Ambiguous acceptance): grep the OMS for the response paths that handle accepted / status: "accepted" — typically grep -rn "accepted\|filledContracts\|recovered_canceled" executors/<name>/src. Three concrete patterns to flag:
  1. Any code path that returns recovered_canceled, recovered, or equivalent "safe, try again" outcomes after a cancel-then-empty-fill-history sequence. These collapse UNRECONCILED into SAFE and are CRITICAL.
  2. Any path that treats a missing filledContracts field as 0 (e.g., const filled = resp.filledContracts ?? 0) without raising an uncertainty error. Venue silence is not a zero; it's "unknown." CRITICAL.
  3. Any branch where return { accepted: false } (or equivalent) is reached after a successful venue submit response — the runtime will treat this as "order never placed" and can immediately retry, racing the venue's own fill path. CRITICAL. Also verify the CLOSE path carries the same rule as the ENTRY path — asymmetry is CRITICAL. §15 sub-patterns (check each independently):
  4. Accepted-terminal distinction: does the OMS preserve venue terminal-status fields (status: "canceled", remainingContracts) through its normalization layer? If it collapses all accepted-zero-fill responses into generic ambiguity, every normal FAK zero-fill falsely halts the runtime. CRITICAL if collapsed; the fix is to classify accepted+terminal as ordinary rejection while keeping accepted-without-proof as fatal.
  5. Settling persistence across restarts: grep -rn "settling\|settlingRecord\|settlingEntry\|persistSettling" executors/<name>/src. If accepted-but-unresolved orders are tracked only in memory, a process restart drops the guard and permits duplicate submits. CRITICAL for any live executor.
  6. Order-scoped fill recovery: trace the timeout-recovery code path that queries fill history. If it uses marketId + since instead of exact orderId, back-to-back orders on the same market can steal each other's fills. CRITICAL.
  7. AbortController-based timeout: does the OMS submit timeout use AbortController on the real HTTP request, or a Promise.race local timer? grep -rn "AbortController\|Promise.race" executors/<name>/src — if Promise.race is used around venue submit without aborting the request, the original submission can still land after the timeout fires. CRITICAL.
  8. Shutdown refuses exit with open positions: trace the SIGINT/SIGTERM handler. If the first signal exits while positions are open (even after logging a warning), CRITICAL — it abandons live exposure.
  9. Inner-catch preservation: grep -rn "catch\|UnreconciledLiveOrderError" executors/<name>/src — verify every catch block in the trading path rethrows UnreconciledLiveOrderError before soft-logging ordinary failures. A swallowed error defeats the fail-closed design. CRITICAL.
• §16 (Unverified partial-fill remainder cancellation): grep the OMS for cancel call sites — typically grep -rn "cancelOrder\|cancel(" executors/<name>/src. For each call, trace the return value:
  1. If the cancel result is discarded (await cancelOrder(...) with no capture, or a capture that isn't inspected before mirroring local state), that's CRITICAL. The cancel may have failed silently.
  2. If the cancel is wrapped in a try/catch whose catch block swallows the error and proceeds to mirror "cancelled" locally, CRITICAL. A caught error is not a successful cancellation.
  3. If the code checks only for HTTP 200 and not for a venue-side status: "cancelled" / equivalent positive acknowledgement, WARNING — venue 200s don't always mean the order was actually cancelled.
  4. IOC / FAK remainder cancellations that assume "the venue took care of it" without explicit verification are CRITICAL. As with §15, verify close-path parity.
• §17 (Unsimulated market impact): grep the scorer for depth-consumption recording — typically grep -rnE "depthConsumed|depthFraction|ladderWeighted|consumedDepth" executors/<name>/src/scorer. If the scorer does not emit depthConsumedFraction per fill, the strategy's promoted edge is untested for capacity; that's a CRITICAL statistical bug even though no runtime code is broken. Three patterns to flag:
  1. Replayed fills priced at BBO unconditionally, with no consideration of trade size vs depth. CRITICAL — live will underperform replay proportionally to p90 depth-consumption.
  2. Missing slippage haircut on the promoted number despite large trade sizes. Look at the tear-sheet generation code — if no term reduces the edge by modeled slippage, CRITICAL.
  3. Queue-position assumption of "top of book" without calibration against observed queue lengths. WARNING — likely overstates fill rate at live latency. Cross-check: if the executor has @recallnet/skunk-paper-trading wired and rejects empty books, that's capability but not enough — the SCORER must also record depth consumed at fill time. Cite file:line for both.
• §18 (Under-attested promoted edge): walk references/handicaps.md and verify all six attestations appear on the tear sheet:
  1. Slippage haircut applied: tear sheet shows a reduced edge if p90 depthConsumedFraction > 0.10. Missing = CRITICAL.
  2. Fee completeness: every venue fee type is in sizing. Enumerate them; a missing one = CRITICAL.
  3. Confidence intervals: win rate, Sharpe, expected return each shown as point [CI_low, CI_high]. Bare point estimates = CRITICAL.
  4. Sample adequacy: tear sheet shows evidence grade (hypothesis / pilot / deployment / deployment-confident) computed from loss count. Absent or miscomputed = CRITICAL.
  5. Regime coverage: dataset spans ≥2 regimes; per-regime breakdown shown. One regime = NOT PROMOTABLE = CRITICAL.
  6. Worst-loss stress: bankroll survives 3× worst observed loss at maximum simultaneous exposure. Not computed, or fails = CRITICAL. For each attestation: trace the code in buildTearSheet() (or the hand-rolled equivalent) and cite file:line. If the executor doesn't use @recallnet/skunk-strategy-report, that's already WARNING — hand-rolled tear sheets systematically miss attestations.
• §19 (Portfolio / exposure accumulation): grep for correlation-cluster definitions — grep -rnE "correlationCluster|correlation_cluster|clusterId" executors/<name>/src. If the executor can hold multiple simultaneous positions and has no cluster identifier, CRITICAL — every multi-position entry accumulates into a hidden larger bet. Also verify:
  1. Per-cluster exposure cap is enforced BEFORE entry (not after). A cap enforced after a fill already overspends. CRITICAL.
  2. Cap formula: ≤ single-position-cap × independence-factor. A flat N × single-position-cap is an overcount bug. CRITICAL.
  3. Pattern-visibility alert: same-signal entries in a sliding window produce an operator alert. Absent = WARNING. Cite the cluster-id registration code and every enforcement site.
• §20 (Operational fragility): walk the four resilience concerns from references/operational-resilience.md:
  1. Rate limit / 429: grep -rnE "429|Retry-After|retryAfter|exponential" executors/<name>/src. Must have a shared retry-aware helper. Raw fetch() without retry handling = CRITICAL.
  2. Idempotency keys: grep -rnE "clientOrderId|client_order_id|idempotency" executors/<name>/src. Every order submit carries a key. Absence = CRITICAL on retries.
  3. Websocket reconnect + sequence gap: grep -rnE "prevSeq|lastAppliedSeq|sequenceGap|onReconnect.*snapshot" executors/<name>/src. Sequence-gap detection and snapshot-on-reconnect must both be present. Either missing = CRITICAL — the book silently diverges.
  4. Clock drift: grep -rnE "getServerTime|assertClockSync|clockDrift|clock_drift" executors/<name>/src. Startup + periodic drift checks must exist. Missing = WARNING for most venues, CRITICAL for TTL-sensitive venues.
  5. Audit log: grep -rnE "auditLog|audit_log|auditTrail" executors/<name>/src. Every outbound venue request and inbound response must be persisted to a durable append-only JSONL, separate from the ledger. Missing or partial coverage = WARNING (not CRITICAL because it's operational, not financial, but post-mortems will be guesswork).

For the full 21-class walkthrough, open error-classes.md and work through them in order. Record every finding with a file:line citation.

Step 4: Run the three adversarial personas

Open .agents/skills/build-executor/references/adversarial-trading.md. Adopt each persona in sequence and ask the persona's questions against the executor source:

1. Data Skeptic — questions about writer isolation, feed integrity, timestamp handling, warmup, quality gates
2. Backtester — questions about data purity, replay correctness, capital model, stale data
3. Live Operator — questions about parity, order safety, restart safety, budget, fire-and-forget state updates

For each persona, record findings with file:line citations. A persona can legitimately return "no issues found — closest risk examined: X, acceptable because Y" if the investigation was thorough.

Step 5: Run /strategy-verification on recent scoring output

If the executor has evidence artifacts in research/evidence/, load /strategy-verification and run its checks against the most recent score output. This catches data quality and statistical issues that code-level review can miss.

Step 5b: Statistical-honesty audit (§17, §18)

Separate from code bugs, this checks whether the promoted number itself is honest.

1. Open the most recent tear sheet in research/evidence/. Identify the promoted metric the operator is acting on (usually "edge" or "expected return").
2. Verify each of the six attestations from references/handicaps.md appears alongside the metric:
   • Slippage haircut (with p50/p90/p99 depthConsumedFraction shown)
   • Fee completeness (every venue fee type accounted for)
   • Confidence interval (point + CI band, not bare point estimate)
   • Sample adequacy (evidence grade matches loss count)
   • Regime coverage (≥2 regimes, per-regime breakdown)
   • Worst-loss stress (survives 3× worst loss at max simultaneous exposure)
3. For any missing attestation, file a CRITICAL finding: "strategy is being promoted as X% but attestation Y is absent — the number operators see is not the honest number."
4. Reference the relevant handicap formulas when proposing fixes — do not invent thresholds; use the ones in handicaps.md.

This step catches the "14-of-14 passed but lost money" failure mode where every error-class code path is fine but the promoted number is unattested.

Step 5c: Operational resilience audit (§20)

Walk the four concerns from references/operational-resilience.md and produce findings:

1. Rate-limit handling: integration test simulates 429 burst and verifies exponential backoff + Retry-After honoring. Missing test = WARNING.
2. Websocket reconnect + sequence gap: integration test tears the WS mid-run with a dropped sequence and verifies book re-snapshot. Missing = CRITICAL for venues where book state drives fill decisions.
3. Clock drift: startup + periodic drift checks. Inspect the interval; if > 60 min between checks, WARNING.
4. Audit log: every outbound request and inbound response appears in a durable JSONL. Verify by tailing the file during a paper run. Missing = WARNING but rising to CRITICAL if §15/§16 are active risks for this venue (you cannot post-mortem without it).

Step 6: Check the experiment.md against reality

Read experiment.md:

• Is the status field accurate? (e.g., claims active-edge but Trust Checklist is empty)
• Are evidence rounds linked to real directories in research/evidence/?
• If status is active-edge, is the Trust Checklist fully attested?
• If status is testing, is there evidence from the last 24 hours?

Findings here are governance violations, not code bugs, but still important.

Step 7: Cross-model audit stamp

This is the final step. The internal audit catches what you can see; /verify brings in reviewers from different model families to catch what your pretraining blind spots hid.

/verify --payload=trade-path-audit --reviewers=2 --gate=audit

/verify picks two reviewers in families different from the auditing agent AND different from each other. It hands them .agents/skills/trade-path-audit/SKILL.md as their checklist, writes findings to research/evidence/prosecution-<strategy-slug>.md under an ## AUDIT section, and enforces the fix-and-re-review loop. The retrofit is not complete until /verify returns SURVIVED or WEAKENED (acceptable, with rationale) for the audit gate.

The retrofit audit's final verdict is the /verify verdict — not the internal audit's verdict alone. An internal CLEAN that turns up LANDED findings cross-model means the executor is BLOCKED until fixed.

After the retrofit passes, the executor re-enters the normal gate schedule at whichever stage it's promoting into (usually PROMOTE → LIVE).

Findings format

Write the audit report to executors/<name>/research/audit-<YYYY-MM-DD>.md:

# Trade Path Audit: <executor-name>

**Audited:** YYYY-MM-DD
**Auditor:** <agent ID or human>
**Overall verdict:** CLEAN / CONCERNS / BLOCKED

## Executor Inventory

[venue(s), stages present, packages used, experiment status, rough LOC]

## Machine-Verified Checks

[paste gate-check.sh outputs per stage]
[paste parity-audit.sh output]

## Error Classes (21)

### §1. Feed pinning on expired markets

Status: PASS | WARN | FAIL
Evidence: [file:line or rationale]

[...one section per class, skip classes that don't apply with "N/A — no [relevant feature]"]

## Statistical Honesty (§17, §18)

| Attestation              | Status             | Evidence                            |
| ------------------------ | ------------------ | ----------------------------------- |
| Slippage haircut applied | PASS / WARN / FAIL | [file:line or tear-sheet reference] |
| Fee completeness         | ...                | ...                                 |
| Confidence intervals     | ...                | ...                                 |
| Sample adequacy          | ...                | ...                                 |
| Regime coverage          | ...                | ...                                 |
| Worst-loss stress        | ...                | ...                                 |

## Operational Resilience (§20)

| Concern                     | Status             | Evidence |
| --------------------------- | ------------------ | -------- |
| Rate-limit / 429 handling   | PASS / WARN / FAIL | ...      |
| Idempotency keys on submits | ...                | ...      |
| WS reconnect + sequence gap | ...                | ...      |
| Clock drift monitoring      | ...                | ...      |
| Audit log persistence       | ...                | ...      |

## Adversarial Personas

### Data Skeptic

| #   | Severity | Finding | File:line                |
| --- | -------- | ------- | ------------------------ |
| 1   | WARNING  | ...     | src/collector/feed.ts:45 |

### Backtester

[table]

### Live Operator

[table]

## Experiment Doc Governance

[status accuracy, trust checklist completeness, evidence freshness]

## Fix Plan

Ranked by severity. Each fix is one commit:

1. [CRITICAL] Fix fire-and-forget position update in oms-adapter.ts:123 — await the position tracker before returning. See error-classes.md §14.
2. [WARNING] ...

Severity rubric

• CRITICAL — Machine-verified gate failure, or a bug that can lose money in live/paper (e.g., fire-and-forget state updates, missing pending-order gate, empty-book fills accepted).
• WARNING — Pattern that increases risk but isn't an immediate bug (e.g., untracked ask source, soft parity bug gated behind an unreached code path).
• NOTE — Cleanliness/documentation issue (e.g., outdated experiment.md status, missing JSDoc on money-path code).

A CRITICAL in any category makes the overall verdict BLOCKED. Any WARNINGs → CONCERNS. All NOTEs or nothing → CLEAN.

Integration with the rest of the workflow

• Before fixes: the audit report itself is the artifact. Commit it to executors/<name>/research/audit-<date>.md so the history shows what was found when.
• During fixes: use /recall-commit for every fix. One fix per commit keeps the history reviewable.
• After fixes: re-run this skill to produce a fresh audit. The previous audit stays as a historical record.
• If fixes require new packages or venue work: escalate to /package-authoring or /venue-integration as needed.
• If fixes change behavior that's actively running: follow the /build-executor parity rules strictly — changes to live code are immediate-execution priority (see AGENTS.md "Trading-Impact Bugs").

What this skill is NOT

• Not a linter — it uses judgment, cites concrete evidence, and flags governance issues code tools can't see
• Not a refactor — findings are reported but not automatically fixed
• Not a substitute for /adversarial-verifier on new code — use this for audits, use build-executor's personas for fresh builds
• Not a one-time task — run it periodically, especially after the gate system evolves (new error classes, new personas, new checklist items)