| name | sherlock |
| description | First-principles Firefox problem solving for Codex. Use for Bugzilla bugs or Firefox failures that need evidence-based diagnosis, revision-pinned source links, proof tests, invariant-driven solution design, independent evaluation, implementation, and durable follow-up documentation.
|
| metadata | {"short-description":"Diagnose and solve Firefox bugs"} |
Sherlock: Root Cause Analysis and Solution Design
Follow the source-permalinks skill for ALL source and documentation references.
Follow references/spec-check.md when verifying web specification compliance.
Follow references/gecko-architecture.md for Gecko architecture lookups.
Follow references/agent-teams.md for the agent-team I/O contracts and prompts.
Follow references/first-principles.md for the Reframe phase's method.
Follow references/impl-skill-discovery.md for the Implement phase's skill discovery.
Phases
Phases are named, not numbered — they do not always run in a straight line,
and the numbering used to lie about that.
| Phase | Question it answers | Ends with |
|---|
| Intake | Where does this run live? | Run dir + pinned revision |
| Diagnose | Why does this bug happen? | Verified root cause + proof tests |
| Diagnosis Review | Is that root cause actually right? | Review #1 (L / T / R) |
| Gate | — | User agrees the root cause |
| Reframe | What must be true so this cannot happen? | Named design principles + invariants |
| Gate | — | User adopts principles |
| Design | What are all the ways to get there? | Option set + roadmaps + comparison |
| Gate | — | User agrees the option set |
| Decide | Which one, given reality? | Ranked recommendation |
| Gate | — | User reviews the evaluation, before red-pen |
| Decide (cont.) | Does an independent reviewer agree? | Review #2 + response |
| Gate | — | User approves the implementation choice |
| Implement | Make it real. | Patches, FAIL→PASS proof, follow-up list |
| Consolidate | What did we learn? | Every doc reconciled, run FINISHED |
Two independent reviews are mandatory and judge different things: Reviewer R
challenges the root cause; the Decide red-pen challenges the solutions.
The gate contract
All five gates behave identically, and every one of them must do all four steps —
a gate that only handles agreement leaves the run stuck in blocked-on-user the
moment the user pushes back, and a crash there re-presents a summary that no longer
matches the artifacts:
- Set the gate's row
blocked-on-user before presenting.
- On agreement → row
completed, proceed.
- On disagreement → record the objection verbatim in plan.md Notes, set the
gate row back to
pending, set the named upstream rows in-progress, and re-run
forward from there. Re-present the gate only once the artifact has been rewritten.
- Never proceed past a gate on inferred consent. Silence is not agreement.
| Gate | Row | On disagreement, reopen |
|---|
| Root cause agreed | 21 | the offending Diagnose rows through 17.5, then 18–20; invalidate every later derived row |
| Principles agreed | 28 | 22 for framing objections, otherwise 27 (and 23–26 if new evidence is needed); invalidate 28 onward |
| Option set agreed | 34 | 31–33 — append an option, never rewrite one; invalidate 34 onward |
| Evaluation reviewed | 37 | 35–36 if criteria or weights change, otherwise 36; invalidate 37, 39–46 (row 38 remains valid only if the solutions revision is unchanged) |
| Implementation approved | 40 | 36–37 and 39–40 — take the next-preferred option; invalidate 42–46 |
Invalidation rule. Reopening a row also resets every row whose artifact depends
on it to pending (or to skipped when its condition no longer applies). A
completed downstream row is never trusted after its input changes. At minimum:
| Changed boundary | Reset |
|---|
| Diagnose evidence, root cause, or proof | affected Diagnose rows, 17–21, then 22–46 |
| Reframe framing or principles | affected 22–27, then 28–46 |
| Design option set | 31–34, 35–40, 42–46; append a fresh 38.x review row |
| Evaluation criteria or scores | 35–37, 39–40, 42–46; keep row 38 only when its solutions revision is unchanged |
| Implementation choice or implementation | 39–46 (and 36–37 when the recommendation changes) |
Record every invalidation in plan.md Notes. Dynamic child rows follow the same
dependency rule as their parent.
The generate/judge split is deliberate. Reframe and Design deliberately ignore
cost — architecture revamps are on the table, reviewer pushback is not a
consideration. Practicality enters only in Decide. Collapsing the two produces
the smallest local patch every time, which is the failure mode this workflow
exists to prevent.
Sherlock runs are persistent and resumable. Every run writes a plan.md
progress table to its run directory; each team and each reviewer writes its
findings to a dedicated file. If a session halts (server unavailable, context
exhausted, power outage, kill), re-invoke with --resume <run-dir> (or just
/sherlock <bug-id> — the bug id locates the run dir) and the skill continues
from the first non-terminal row. See Intake and references/plan-template.md.
Arguments: $0
Parse the arguments:
--resume <run-dir> present → resume mode (skip fresh setup; see Intake).- Otherwise:
- First numeric token = bug ID (mandatory)
- Tokens containing
/ = path arguments. Disambiguate by checking the target:
- If it's an existing directory → output-dir
- If it's an existing file → report-path
- If two path tokens are given, first = output-dir, second = report-path
- If it's a non-existing path ending with
/ or looks like a directory → output-dir
Gotchas
- Every claim needs evidence or
[Assumption] label — do not state hypotheses
as facts. Read the code before making any claim about code behavior. This
applies in every phase, including the first-principles ones.
- ALWAYS use revision-pinned links — follow the
source-permalinks skill.
Never use trunk/tip URLs (firefox-main/source/...) in any document.
- Tests are PROOFS for root cause claims — they must demonstrate the root cause
is correct. They are reusable for TDD later, but their primary purpose here is evidence.
- Debug logs go in separate files — never inline multi-line log output in the
analysis doc. Store under
firefox/debug/ (or <library>/debug/) as
bug-<id>-debug-<desc>.log and reference with relative links.
- Do NOT write solutions in Diagnose — Diagnose is purely diagnostic.
- Do NOT write patches in Reframe — Reframe produces invariants, elimination
candidates and design principles: what must be true, not what code to write.
If you are writing a diff, you are in the wrong phase.
- Cost is not an argument before Decide — "too big", "the reviewer will
object", "we'd never land that" are out of scope in Reframe and Design. A
pre-filtered design space makes the Decide phase useless.
- Private bugs — never log titles, descriptions, components, or root cause details
outside the output directory. History log for private bugs:
date | bug_id | PRIVATE only.
- Check normal/debug build BEFORE requiring ASan/TSan — try reproducing in
current build first, then debug build, then sanitizer builds.
- NEVER read, parse, or print any API key — all API access goes through
sherlock-config or bmo-to-md. Never read TOML config files containing keys.
Never use python3 to parse config files.
- Delegate research, not synthesis — subagents do bounded fact-finding
(bug fetch, code path tracing, git archaeology, invariant enumeration). The
main agent decides what the facts mean. Never let a subagent declare the root
cause, name the design principles, or pick the solution.
- Three hypotheses minimum — single-hypothesis RCAs anchor too early.
Diagnose.5 is mandatory; do not skip it even when one cause feels obvious.
- The reviewers are independent — when
red-pen returns
or , do not argue with the verdict. Either fix the artifact,
escalate to the user (for ), or loop back. This applies to both
Reviewer R (root cause) and the Decide red-pen (solutions).
Subagent delegation policy
The main agent's context is reserved for synthesis — connecting evidence,
forming and pruning hypotheses, deciding what is verified vs assumed,
articulating the root cause, naming the design principles, categorising options,
and judging. Bounded research tasks are delegated to subagents so the main
context stays focused.
Delegate to a subagent when the task is:
- Bounded: clear input, clear output shape, no interactive judgment.
- Voluminous: produces a lot of intermediate text (raw bug comments,
searchfox dumps, git log archaeology, call-site censuses) that the main agent
does not need in full.
- Parallelizable: e.g., trace Firefox + library simultaneously; brainstorm
one approach per principle.
Do NOT delegate:
- Hypothesis selection or pruning (Team H is advisory; the tree is yours).
- "What does this evidence mean for the root cause?" — Synthesis (
Diagnose.12).
- The verdict, the root cause, and the design-relation sentence.
- The Reframe framing answers (Q1/Q2) and the naming of design principles.
- Design categorisation — clustering, merging, roadmap sequencing.
- The entire Decide phase — criteria, weights, scoring, recommendation.
- Gates and user-facing decisions.
- The structural self-check (
Diagnose.16).
- Any step that requires judging two competing claims.
Bounded research is delegated to agent teams — multiple spawn_agent calls
with fork_turns: "none" in a single message that run concurrently. The prompts
must be self-contained and use absolute paths; no team inherits the parent
conversation. Sherlock's workflow is gated, so teams
launch in four waves (full contracts in references/agent-teams.md):
- Wave 1 — intake teams (
Diagnose.1, before any hypothesis exists):
Team B (bug-context digest) and Team H (hypothesis brainstorm,
advisory only).
- (main-agent gate: failure-pattern classification, investigation plan,
hypothesis tree, working branch — none delegated)
- Wave 2 — research teams (
Diagnose.8, after the primary hypothesis is
chosen): Team C (Firefox code-trace), Team L (library code-trace, when
third-party), Team D (design archaeology), Team X
(cross-browser/spec), Team T (test-framework scout + draft).
- (main agent: Synthesis → verdict + root cause + broken invariant)
- Wave 3 — Reframe teams (
Reframe.2): Team P (problem framing),
Team E (elimination scan + call-site census), Team I (invariant
discovery), Team W (widening & unification).
- (main agent: principles doc)
- Wave 4 — Design fleets (
Design.1): Fleet G (guided, one agent per
adopted principle), Fleet F (free-mind, isolated). Then Team M
(comparison matrix) at Design.3.
Each team writes its full findings to a dedicated file under <run-dir>/teams/
and returns only a ≤10-line summary. The main agent reads the files, never the
transcripts. Each invocation must:
- Pass inputs as file paths and explicit values — not as "the bug we're
investigating".
- Specify the output shape and the target output file.
- Include the framing appropriate to its wave: "return the requested artifact
only", plus the wave's prohibition (no root cause for Waves 1–2, no patches
for Wave 3, no ranking for Wave 4).
A team never declares the root cause, the verdict, the hypothesis ranking, or the
chosen solution.
Phase: Intake
Every run lives in a per-run subdirectory <output-dir>/sherlock-bug-<id>/
(the run dir; the sherlock- prefix keeps it from colliding with other
per-bug skills sharing an output root, e.g. /triage, which uses
triage-bug-<id>/). plan.md (the progress table, from references/plan-template.md)
and all artifacts live inside it. The bug id is the run identity — there is no
slug. Re-running the same bug resumes the existing run dir.
Resume branch
Enter this branch if the invocation contains --resume <run-dir>, or if a
fresh invocation's <output-dir>/sherlock-bug-<id>/plan.md already exists (in which case
ask the user directly: "Found an existing analysis for bug <id> at <path>.
Resume, or start fresh?" — "start fresh" archives or overwrites per the user).
- Read
<run-dir>/plan.md. Recover the bug id, the pinned Searchfox
revision, the progress table, and the Notes.
- Restore session variables: set
$SHERLOCK_REV, $SHERLOCK_BASE, and the
recorded Firefox and library ref tips from plan.md — do NOT re-resolve
recorded values; every link and recovery decision on disk depends on them.
Values recorded as not-created remain unset. Re-resolve only
$SHERLOCK_AUTHOR via sherlock-config --get-patch-author (deterministic).
- If row 8 is not
completed, skip branch recovery — the branch does not exist
yet by design. If row 8 is completed, confirm the Firefox working branch exists:
git rev-parse --verify sherlock/bug-<id>. If it is missing, do not silently
recreate it — an empty branch alongside completed rows 16.x/17 means the run
believes proof tests exist that are gone, and the FAIL→PASS evidence the whole
workflow rests on has quietly evaporated. Instead:
- Try to recover the tip by SHA: the Firefox working-branch tip field in plan.md, then
git reflog. If found, git branch sherlock/bug-<id> <sha> and carry on.
- If unrecoverable, reset the row that created the missing ref and every
dependent row using the invalidation table above. This includes applicable
14.x proof rows, row 16 and its children, rows 17–21, and rows 22–46 when
they were already reached. Tell the user which evidence must be rebuilt.
Note
$SHERLOCK_REV is not a valid recreation point — it may be a searchfox
revision that does not exist locally. SHERLOCK_BASE in plan.md is the branch point.
- Apply the same completed-row check to every recorded library test/debug/fix
ref needed by an applicable 14.x or 42.x row. Recover by its recorded SHA or
invalidate its producing row and dependants. Before re-running any
in-progress
Implement row, check for work in flight:
git status --porcelain and git log --oneline "$SHERLOCK_BASE"..HEAD. Present
uncommitted changes to the user rather than re-implementing over them.
- Find the first non-terminal row in table order. Terminal means only
completed or skipped; therefore pending, in-progress,
blocked-on-user, and blocked-external all stop the scan. Announce in ≤2
lines: "Resuming sherlock bug at ; next task:
."
Fresh-run branch
- Check setup:
.codex/skills/sherlock/sherlock-config --check-setup
reports API key availability, bmo-to-md, searchfox-cli, and configured
output dir. If any prerequisite is missing, present options to the user.
- Resolve output dir (priority): CLI arg →
sherlock-config --get-output-dir
→ ask the user directly, then persist with
sherlock-config --set-output-dir <path>.
- Parse bug id and optional report-path (see Arguments above).
- Create the run dir and subdirs:
mkdir -p <output-dir>/sherlock-bug-<id>/teams
mkdir -p <output-dir>/sherlock-bug-<id>/review
mkdir -p <output-dir>/sherlock-bug-<id>/firefox/fix
mkdir -p <output-dir>/sherlock-bug-<id>/firefox/debug
<library>/ subdirs are created later, when Diagnose.9 activates (T1).
Hereafter <run-dir> = <output-dir>/sherlock-bug-<id>/.
- Resolve
$SHERLOCK_REV (pin a searchfox revision for the whole run so all
links are permanent):
- Get local HEAD:
git rev-parse HEAD
- Validate on searchfox:
fetch
https://searchfox.org/firefox-main/rev/<hash>/moz.configure
- 200 → use this hash as
$SHERLOCK_REV
- 404 (not yet indexed) → fetch
https://searchfox.org/firefox-main/source/moz.configure and extract the
latest indexed revision from the page
- For ESR/beta branches, repeat with the appropriate repo id
(e.g.
firefox-esr128).
- Resolve
$SHERLOCK_AUTHOR:
SHERLOCK_AUTHOR=$(.codex/skills/sherlock/sherlock-config --get-patch-author)
(reads patch_author from config, else git config user.name/user.email).
All generated patches use this as the commit author.
- Create
plan.md from references/plan-template.md in ,
substituting , , ,
, , , for the Firefox
base/tip, for Firefox evidence refs, and for every third-party
field. T1 changes applicable library tips from to . Row 1
, rest .
The main report is resolved once at the start of Diagnose.1: use a supplied
report-path, or fetch into bug-<id>-report/. Team B consumes that input and
gathers duplicates/attachments/Treeherder without duplicating the main fetch.
Output Directory Structure
The run dir uses a uniform layout. Firefox patches go in firefox/, third-party
patches go in <library>/. Both use identical fix/ + debug/ substructure.
Patches are numbered so they apply in order.
<output-dir>/sherlock-bug-<id>/ # the run dir (resume key = bug-<id>)
plan.md # Progress table + resume doc
bug-<id>-analysis.md # Diagnose: the root cause
bug-<id>-principles.md # Reframe: invariants + design principles
bug-<id>-solutions.md # Design: option set + roadmaps
bug-<id>-evaluation.md # Decide: criteria + ranking + review response
bug-<id>-review.md # Decide: red-pen on solutions (-N per re-run)
bug-<id>-followups.md # Implement: follow-up + blocking issues
bug-<id>-report/ # Bug report from bmo-to-md (Team B)
bug-<dup_id>-report/ # One per duplicate (Team B)
bug-<id>-attachments/ # Attachments (Team B)
teams/ # Team output files
team-b-bug-context.md # Wave 1
team-h-hypotheses.md # Wave 1
team-c-code-trace-firefox.md # Wave 2
team-l-code-trace-library.md # Wave 2
team-d-design-archaeology.md # Wave 2
team-x-cross-browser.md # Wave 2
team-t-frameworks.md # Wave 2
synthesis.md # main agent
team-p-problem-framing.md # Wave 3
team-e-elimination.md # Wave 3
team-i-invariants.md # Wave 3
team-w-widening.md # Wave 3
design-g-<principle-slug>.md # Wave 4 (one per adopted principle)
design-f<N>-freemind.md # Wave 4 (one per lens)
analysis-for-fleet-f.md # Wave 4 (stripped analysis doc)
team-m-comparison.md # Wave 4
review/ # Diagnosis Review files
L.md T.md R.md # Reviewer verdicts
bug-<id>-rootcause-review.md # Full red-pen review (REVIEW #1)
firefox/
fix/ # Clean patches for Firefox
<NN>-{test|fix}-<desc>.patch # Actual selected landing order
bug-<id>-verify-run.log # Test or alternative verification
debug/ # Firefox debug artifacts
<NN>-test-<proof-id>-<desc>.patch # Proof/evidence patch(es)
<NN>-debug-firefox-<hypothesis>.patch # Instrumentation on top
bug-<id>-mozconfig # Special build config, if needed
bug-<id>-<hypothesis>-test-run.log # One proof log per hypothesis
bug-<id>-debug-<desc>.log # Captured debug output
<library>/ # Only when third-party involved
bug-<id>-upstream-<library>.md # Upstream report (sanitized)
fix/ # For upstream: clean, applicable
<NN>-{test|fix}-<desc>.patch # Actual selected landing order
debug/ # For upstream: reproduction aid
01-test-<desc>.patch # Test (may include injection)
02-debug-lib-instrumentation.patch # Logging on top of test
bug-<id>-debug-lib-build.log # Library build output
bug-<id>-debug-lib-<desc>.log # Library debug output
The <library>/ folder is created only when Diagnose.9 is active (Branch A, B,
or C). For Branch B (Firefox integration bug), <library>/ may contain only the
T3 diagnostic logs — no fix/ patches since the library has no bug.
The <library>/ output structure mirrors three git branches in the local library
repo (see T3 for details):
sherlock/bug-<id>/test — test/evidence commits (base for debug; fix base only
for benign non-security series)sherlock/bug-<id>/debug — instrumentation on top of testsherlock/bug-<id>/fix — fix series (from the test tip for benign
non-security proofs; from the vendored revision for security/injection proofs)
The firefox/ output structure mirrors the Firefox working branch (see Diagnose.6):
sherlock/bug-<id> — final commits in the order selected by Implement.3sherlock/bug-<id>-debug-<proof-id> — one temporary instrumentation branch
per proof (h1, a2, b1, …)
The Firefox fix may differ from the upstream fix (e.g., upstream takes a wider-scope
fix touching many files, while Firefox takes a less-aggressive local patch touching
one or two files).
Patch ordering rules:
Within each fix/ folder:
- If the regression test can be created without code injection (reproducible with
certain inputs in the built-in test framework), then test patches come first, fix
patches on top.
- If the test requires code injection (e.g., custom malloc returning OOM, mocked
syscalls), then only fix patches go in
fix/ — the injection-based tests go in
debug/ instead.
- Security bugs invert the order at implementation time — see
Implement.3.
Within each debug/ folder:
- Always contains test patches (including injection-based ones), with debug
instrumentation patches on top.
- Goal: developers can apply these patches and immediately reproduce + confirm
the issue.
Numbering: Use two-digit prefixes (01-, 02-, ...) for apply order,
allocated independently per destination series (firefox/fix,
firefox/debug, and each library's fix/debug series). Within one series, two
to-test hypotheses must not both produce an 01-test-*.patch, or git am
order becomes ambiguous. Test patches take the low numbers in hypothesis order
(01-test-h1-…, 02-test-h2-…); fix patches continue from the next free number.
Use descriptive names after the prefix (e.g., 01-test-ogg-truncated-stream.patch,
02-fix-bounds-check-vorbis-window.patch).
Patch format: Always use git format-patch so patches can be applied via
git am -3 (with three-way merge) or git apply. This requires committing
changes before generating the patch.
Patch generation pattern (commits stay on the branch; $SHERLOCK_AUTHOR resolved
in Intake):
git add <files>
git commit --author="$SHERLOCK_AUTHOR" -m "<descriptive message>"
git format-patch -1 --stdout > <run-dir>/<path>/<NN>-<desc>.patch
The commit message should describe what the patch does (e.g.,
"Add gtest for OOB read in vorbis window function" or "Add debug
instrumentation for decode path tracing"). Commits are kept on the working
branch — do NOT reset. The branch history IS the ordered patch series.
<library>/ subdirectories are created when Diagnose.9 activates and the library
is identified (T1): mkdir -p <run-dir>/<library>/fix <run-dir>/<library>/debug.
Phase: Diagnose
Focus entirely on WHY the bug occurs. No solutions, no invariant design — those
are later phases with their own gates.
Diagnose.1: Understand the Bug — Wave 1 teams (Team B + Team H)
First ensure the main report exists: use report-path, or run
sherlock-config --fetch-bug <id> -o <run-dir>/bug-<id>-report. Read only enough
to form a condensed identity/STR/expected/actual/stack input for Team H. Then
launch the Wave 1 intake teams in one message (two spawn_agent calls with
fork_turns: "none") so they run concurrently. Full I/O contracts and prompt
templates are in references/agent-teams.md:
- Team B — bug-context digest. Consumes the resolved main report, fetches each
duplicate plus attachments, Treeherder failure-distribution, and branch flags,
and writes the digest to
<run-dir>/teams/team-b-bug-context.md (sections:
Identity, STR, Attachments, Duplicates, Treeherder, Branch status, Pointers).
This single fetch serves Diagnose.2 (duplicates), Diagnose.3 (failure
pattern), Diagnose.13a (attachments) and Decide.1 (uplift pressure) —
read the file, do not re-fetch. If a report-path was provided in Intake, pass
it as a pointer so Team B skips the main fetch.
- Team H — hypothesis brainstorm (advisory). Brainstorms ≥3 candidate
hypotheses (mechanism / confirming / refuting / probe cost) to feed the
main agent's hypothesis tree (
Diagnose.5). It writes
<run-dir>/teams/team-h-hypotheses.md. It does not rank or pick a primary
hypothesis — that is main-agent work (anti-anchoring, Gotcha #12). Seed it
with the condensed main-report facts (pass inline, not conclusions).
Set plan.md rows 2 (Team B) and 3 (Team H) in-progress before launching. As
each returns, verify its output file is non-empty and mark its row completed;
if a team aborts, leave its row in-progress so --resume re-runs it.
Then read <run-dir>/teams/team-b-bug-context.md and extract for the analysis
doc:
- Bug title and description (condensed)
- Component
- Steps to reproduce (STR)
- Expected vs actual behavior
- Attached testcases or reproduction scripts
- Keywords (check for
sec-* keywords: sec-high, sec-moderate, sec-low, sec-critical)
- Related bugs, duplicates, depends/blocks
- Per-branch status flags (carried forward to
Decide.1)
If the bug has any sec-* keyword or is in a security group, the analysis doc
MUST include a Security Rating section.
Diagnose.2: Check Duplicates and Related Bugs
Mark plan.md row 4 in-progress.
Create the analysis doc now, from references/analysis-template.md, with every
section still a placeholder. Rows 4, 5 and 7 each fill one section of it, and from
here on the doc is only ever edited in place — never re-created, which would
truncate the earlier sections. Diagnose.15 completes the remaining sections rather
than authoring the file.
This ordering exists so those three rows have a real on-disk artifact when they are
marked completed. Otherwise a resume trusts them and the duplicate findings, the
failure-pattern classification, and the hypothesis tree are silently gone — and the
hypothesis tree is what Diagnose.12 re-ranks and Diagnose.16 checks.
Team B's digest already includes duplicate fetches and a "Duplicates" section. Read
that section.
Duplicates are valuable when they contain:
- Additional STR or reproduction scripts
- Independent stack traces that confirm or refine the root cause
- Attached testcases (
testcase keyword)
- Commenter analysis that narrows the failure condition
- Different affected versions or platforms
If the digest's duplicate summary is thin and a duplicate seems important,
read the raw report from <run-dir>/bug-<duplicate_id>-report/ directly
— targeted inspection, not full re-ingestion. If a duplicate adds a meaningfully
different perspective, note it under Related Context in the analysis doc.
Mark row 4 completed.
Diagnose.3: Failure Pattern Analysis
Mark plan.md row 5 in-progress. Team B's digest has the Treeherder block. From
it, extract:
- Platforms (e.g., Windows 11 only → OS-specific; Linux+Mac → cross-platform)
- Test suites (e.g.,
mochitest-media-wmfme → Windows Media Foundation Engine)
- Build types (debug/asan/opt)
- Failure count and trees (autoland, mozilla-central, try)
Classify the failure pattern:
| Pattern | Meaning | Approach |
|---|
| Always fails | Code is wrong or missing | Find the broken code path |
| Intermittent | Race, resource, error-handling | Ask: what happens on rare failures? |
For intermittent bugs, reason in this order:
- Test robustness: Does the test properly handle all error paths?
Promise.all
hangs if any promise never resolves/rejects.
- Error propagation gaps: Missing
RejectPromises call turns decode error into silent hang.
- Concurrent vs sequential: Resource contention (hardware decoder session limits).
Test by running items sequentially — if failure disappears, contention is confirmed.
- Platform-specific conditions: Hardware, OS version, driver.
Never assume that because a test occasionally fails, the feature is broken.
If it passes 95% of the time, something rare causes the failure. Write the
Failure Pattern section into the analysis doc and mark row 5 completed.
Diagnose.4: Plan the Investigation
Mark plan.md row 6 in-progress. Create or update the harness investigation
checklist with update_plan before starting the deep investigation, based on what
Diagnose.1–Diagnose.3 (and Team H's candidate hypotheses) revealed.
The plan should cover:
- Hypothesis: Initial theory about the root cause (to be confirmed or refuted)
- Code areas to investigate: Which files/subsystems to trace (from bug report,
stack traces, component, test suite names)
- Third-party involvement: Whether vendored library code is likely involved
(if yes, which library and why)
- Build requirements: Whether a special build (debug/ASan/TSan) is likely needed
- Test strategy: What kind of proof test is likely appropriate
- Open questions: What information is still missing
Present the plan to the user for review. The user may refine the hypothesis,
suggest additional code areas, or redirect the investigation. Update the
checklist after the user approves or provides feedback. Record plan decisions in plan.md
Notes and mark row 6 completed.
Diagnose.5: Build the Hypothesis Tree
Mark plan.md row 7 in-progress. Read <run-dir>/teams/team-h-hypotheses.md for
Team H's candidates, then build the tree yourself — Team H is advisory; you
own selection, ranking, and pruning. Enumerate at least three candidate
root-cause hypotheses — even when one already feels obvious. Single-hypothesis
RCAs anchor too early; the cost of generating two extra candidates is minutes,
the cost of anchoring on the wrong one is hours.
For each hypothesis, fill in:
| Hypothesis | Failure mechanism | Confirming evidence | Refuting evidence | Probe cost |
|---|
| H1 ({short name}) | {how the bug would manifest if this were the cause} | {what we'd see in code/logs/test if true} | {what we'd see if false} | {minutes / hours / build required} |
| H2 ({short name}) | ... | ... | ... | ... |
| H3 ({short name}) | ... | ... | ... | ... |
Pick the hypothesis with the highest confirm/refute ratio per unit of
probe cost as the primary; keep the others alive in reserve. Save
this table to the analysis doc under a ## Hypothesis Tree section so
reviewers can see what was considered and pruned.
When investigation surfaces evidence that revives a pruned hypothesis, do
not silently re-anchor — re-rank the table and update the analysis doc. Mark
row 7 completed.
Diagnose.6: Create Firefox Working Branch
Mark plan.md row 8 in-progress. Create a working branch in the Firefox tree for
all sherlock test and fix commits:
SHERLOCK_BASE=$(git rev-parse HEAD)
git checkout -b sherlock/bug-<id> "$SHERLOCK_BASE"
All test commits go on this branch first, then fix commits on top (Implement):
HEAD
└── sherlock/bug-<id> ← test-1 → test-2 → ... → fix-1 → fix-2 → ...
└── sherlock/bug-<id>-debug-<proof-id> ← instrumentation for one proof
Each sherlock/bug-<id>-debug-<proof-id> branch is created later
(Diagnose.13e or B1.4) when instrumentation is needed. It forks from the
relevant test commit and is used only for debug capture — it is not part of the
final patch series. Injection-only tests use an evidence branch and never enter
the working/landing branch.
Why the hyphen. Git stores branches as files under refs/heads/, so
sherlock/bug-<id> and sherlock/bug-<id>/debug cannot coexist — the second
git checkout -b fails with cannot lock ref … 'sherlock/bug-<id>' exists. Hence
-debug as a sibling, not /debug as a child. The library repo uses
sherlock/bug-<id>/test|debug|fix and that is fine, because there is no bare
sherlock/bug-<id> ref in that repo. Do not "tidy" the Firefox name to match.
Record the branch point. The reorder in Implement.3 and the recovery path in
Intake both need the commit this branch was cut from, and $SHERLOCK_REV will not
serve — it may be an older searchfox-indexed revision, not local HEAD:
Write SHERLOCK_BASE and the new branch's current SHA into plan.md's Firefox
base/tip fields. Mark row 8 completed.
Diagnose.7: Research Code Paths
Mark plan.md row 8.5 in-progress. This is the row loop-backs target when the
investigation has to be reopened (Diagnose.14 on an unexpected PASS, a red-pen
reject, Decide.2 on a moved root cause, Implement.4 when the test fails to
flip) — so it must exist as a row, not just as a step.
Use searchfox-cli for symbol lookups:
searchfox-cli --id <keyword> --cpp -l 50
searchfox-cli --define <ClassName>
searchfox-cli -q blob --path dom/media <search-term>
For architecture questions, follow references/gecko-architecture.md for structured
approaches to understanding Gecko control flow, ownership, and subsystem interactions.
For web-exposed features, consult references/spec-check.md to verify spec compliance.
Identify 3-5 key files. Check recent history:
jj log -r 'file(path/to/file)' -l 10
For third-party libraries, consult references/upstream-libs.md. Decide here
whether Diagnose.9 will activate — do the suspect files match a path in that
reference? Record the yes/no in plan.md Notes; Diagnose.8 needs the answer to know
whether to launch Team L and whether to skip Team D. Mark row 8.5 completed.
Diagnose.8: Wave 2 research teams (C, L, D, X, T)
Once the primary hypothesis and entry symbols are chosen (Diagnose.5/Diagnose.7),
launch the Wave 2 research teams in one message (multiple spawn_agent calls,
all with fork_turns: "none") so they run concurrently. Full I/O contracts and prompt templates are in
references/agent-teams.md. All Wave 2 teams are read-only — none of them
build. Set each applicable team's plan.md row in-progress before launching:
- Team C — Firefox code-trace (row 9). Numbered, revision-pinned trace of the
call path for the primary hypothesis →
teams/team-c-code-trace-firefox.md.
- Team L — library code-trace (row 10). Only when
Diagnose.9 applies; same
contract with upstream permalinks → teams/team-l-code-trace-library.md.
Pass the exact revision source/path from references/upstream-libs.md; do not
assume every library uses media/<lib>/moz.yaml (some use third_party/,
netwerk/, or Cargo.toml).
- Team D — design archaeology (row 11). Git-history archaeology of the suspect
code →
teams/team-d-design-archaeology.md. Skip (row skipped) when
Diagnose.9 is active — design intention is covered in the branch workflow
(A1/B1/C1). Its function contract output is the primary input to the
Reframe phase, so extract it precisely.
- Team X — cross-browser / spec (row 12). Spec citation + cross-engine
behaviour table →
teams/team-x-cross-browser.md. Skip (row skipped) for
internal-only bugs with no web surface.
- Team T — test-framework scout + draft (row 13). Per live hypothesis: pick a
framework, find a neighbour test, and draft the proof-test source →
teams/team-t-frameworks.md. Team T does NOT build (builds serialize — see
Diagnose.13/Diagnose.14).
As each team returns, verify its output file is non-empty and mark its row
completed; if a team aborts, leave its row in-progress for --resume. The
main agent reads the files (not transcripts) during Synthesis and reasons
about which trace steps correspond to the primary hypothesis. Read
the source-permalinks skill for URL patterns; every code reference is
revision-pinned with $SHERLOCK_REV.
Diagnose.9: Third-Party Library Sub-Workflow (Conditional)
If the suspected failing path does not touch vendored third-party code, mark
plan.md rows 14 and 10 (Team L) skipped and continue at Diagnose.10/Diagnose.12.
The test is the suspected path, not the confirmed root cause — the root cause is
not established until Diagnose.12, well after this row, and Diagnose.8 already
needs the answer to decide whether to launch Team L and whether to skip Team D. Make
the call at the end of Diagnose.7 by checking the suspect files against
references/upstream-libs.md, and record the yes/no in plan.md Notes.
If the suspected failing path involves vendored third-party code (file paths matching
references/upstream-libs.md), activate this sub-workflow. It is plan.md row
14 — an internal diagnostic decision point, not one of the five user phase
gates. T1/T2/T3 are sequential and main-agent-driven (NOT teams): T1 needs
user input; T3 is a serialized diagnostic build whose result chooses the branch.
Ordering with the Wave 2 teams: the read-only traces (Team C + Team L) launched
in Diagnose.8 run before T3's build, so the trace informs which path to test.
After T3 resolves scope, write the scope verdict (Branch A / B / C) into plan.md
Notes and append the branch sub-rows under row 14 (see
references/plan-template.md "Dynamic rows"). Recording the verdict in Notes lets
a resume that halts after T3 but before the rows are written reconstruct them.
Mark row 14 blocked-on-user while T1 waits on the user, in-progress once T2/T3
are running, and completed once the scope is recorded and the branch rows are
appended. Getting that first transition right matters: left as in-progress, a
crash during T1 makes resume re-run the whole gate including T3's diagnostic build. Branch downstream work then runs against those
appended rows.
T1: Check for Local Upstream Repo
Ask the user directly:
- "The issue involves {library} ({upstream_url}). Do you have a local clone?"
- If yes: "Where is it?"
- If no: "Should I clone it? Where?" (suggest
~/Work/{lib-name})
Once resolved, fill every Third-party field in plan.md and initialize the three
library ref tips to not-created.
T2: Initial Scope Hypothesis
Form an initial hypothesis about where the bug lives:
- (a) In the library itself — would reproduce with standalone upstream tests
- (b) In Firefox's integration — library works correctly but Firefox's wrapper
code, IPC, threading, or lifecycle management causes the issue
- (c) In Firefox's local patches — Firefox applies patches on top of the vendored
library that introduce or expose the problem. Check the mapped vendored directory for
.patch
files or diffs from the upstream revision.
This is a hypothesis — T3 will confirm or refute it.
T3: Diagnostic — Reproduce in Upstream Library
This step is mandatory regardless of the T2 hypothesis. Even if you suspect a
Firefox integration issue, always attempt to reproduce in the upstream library
first. This eliminates false assumptions about where the bug lives.
Branch structure in the local library repo:
Work in the library repo uses three branches forked from the upstream revision.
This keeps test, debug, and fix work cleanly separated with proper git history:
upstream HEAD (vendored revision)
└── sherlock/bug-<id>/test ← test commits
├── sherlock/bug-<id>/debug ← debug instrumentation on top of test
└── sherlock/bug-<id>/fix ← fix commits on top of test (created in A4/C4)
1. Create the test branch and write the standalone test:
git checkout -b sherlock/bug-<id>/test <vendored-revision>
Create a minimal test case in the library's native test framework (see the
Library Test Frameworks table in references/upstream-libs.md). The test should
exercise the suspected failure condition.
git add <test files>
git commit --author="$SHERLOCK_AUTHOR" -m "Add standalone test for <desc>"
Generate the test patch:
git format-patch -1 --stdout > <run-dir>/<library>/debug/01-test-<desc>.patch
2. Create the debug branch and add instrumentation:
git checkout -b sherlock/bug-<id>/debug
Add targeted logging to confirm the traced code path is hit during test execution.
Common instrumentation patterns for C/C++ libraries:
fprintf(stderr, "SHERLOCK: %s:%d reached\\n", __FILE__, __LINE__);fprintf(stderr, "SHERLOCK: value=%d\\n", variable);- Library-specific debug macros if available (e.g.,
aom_internal_error, dav1d_log)
git add <instrumented files>
git commit --author="$SHERLOCK_AUTHOR" -m "Add debug instrumentation for <desc>"
git format-patch -1 --stdout > <run-dir>/<library>/debug/02-debug-lib-instrumentation.patch
3. Code path trace: Read and trace the suspected code path in the library's own
source files. Produce a numbered trace using permanent upstream links (e.g.,
https://gitlab.xiph.org/xiph/vorbis/-/blob/{hash}/lib/sharedbook.c#L355).
4. Build and run (on the debug branch — has both test + instrumentation):
<build-command> 2>&1 | tee <run-dir>/<library>/debug/bug-<id>-debug-lib-build.log
<test-command> 2>&1 | tee <run-dir>/<library>/debug/bug-<id>-debug-lib-<desc>.log
5. Switch back to the test branch for clean state:
git checkout sherlock/bug-<id>/test
The debug and test branches are preserved for later use. Implement.2b chooses
the fix branch base: the test tip only for a benign non-security series, otherwise
the vendored revision.
Record the library test and debug branch tips in plan.md immediately. Leave the
library fix tip as not-created until Implement.2b creates it. Library SHAs never
replace the Firefox working-branch tip.
The T3 result determines the scope and which branch to follow:
| T3 Result | Scope | Next Step |
|---|
| Bug reproduces in upstream library | (a) Library bug | → Branch A |
| Bug does NOT reproduce upstream | (b) Firefox integration | → Branch B |
| Bug reproduces differently (e.g., different behavior, partial failure, or only under specific threading/config that Firefox uses) | (a+b) Split scope | → Branch C |
Document the T3 result and confirmed scope in the analysis doc. Write the scope
verdict into plan.md Notes and append the exact dynamic rows from
references/plan-template.md: diagnostic A1–A3, B1, or C1–C3 under row 14,
plus deferred A4, B2, or C4 under row 42. Never put fix work under row 14.
Then mark row 14 completed.
Branch A: Library Bug (scope a)
The bug exists in the upstream library. Investigation and fix happen primarily in
the library repo. Use upstream permanent links for all code references.
A1. Complete library investigation:
- Design intention: Study git history in the library repo (
git log, git blame,
commit messages). Understand why the code was written this way, what constraints
the authors faced. This replaces Diagnose.10 for third-party code — write the
findings to teams/team-d-design-archaeology.md using the same section shape Team D
would have produced (introducing commit, original purpose, rationale, constraints,
function contract, related code, drift signals). Wave 3 depends on that path.
- Verify claims: Apply the two-tier rule (Verified vs
[Assumption]) to all
claims about the library code.
A2. Create Firefox-side regression test:
Create a test in the Firefox tree that reproduces the issue through Firefox's
integration layer. This proves Firefox is affected AND will verify that vendoring
the upstream fix resolves the problem in Firefox.
- Use the PoC from the bug report (test.html, attached testcases) as the basis
- Choose the appropriate Firefox test framework:
- gtest — C/C++ internal paths that call library APIs directly
- crashtest — crash-only via web-facing paths (
<audio>, <video>, etc.)
- WPT — web-exposed, spec-defined behavior
- mochitest — web-exposed, Firefox-specific behavior
- The test MUST fail without the fix and pass with it
- Register the test in the appropriate manifest
- Commit a benign test on the Firefox working branch. Put an injection-only
test on
sherlock/bug-<id>-evidence-a2 instead. Export every proof test to
firefox/debug/; copy a benign non-security test to firefox/fix/, but keep
security tests and injection-only tests out of fix/ until the final series
is constructed in Implement.3:
git checkout sherlock/bug-<id>
git add <test files>
git commit --author="$SHERLOCK_AUTHOR" -m "Add regression test for <desc>"
git format-patch -1 --stdout > <run-dir>/firefox/debug/01-test-a2-<desc>.patch
cp <run-dir>/firefox/debug/01-test-a2-<desc>.patch <run-dir>/firefox/fix/01-test-a2-<desc>.patch
The commit stays on the branch — do NOT reset.
- Build and run against the unfixed tree:
./mach build
./mach test <path> --headless 2>&1 | tee <run-dir>/firefox/debug/bug-<id>-a2-test-run.log
A3. Generate upstream report:
Generate a second, concise analysis document for reporting to the upstream library
maintainers. Read references/upstream-report-template.md for the template.
Write to <run-dir>/<library>/bug-<id>-upstream-<library>.md.
Critical rules for the upstream report:
- Use ONLY upstream permanent links — no searchfox, no Firefox paths
- Do NOT mention Firefox, Gecko, or any browser-specific context
- Do NOT include security exploitation details, sec-* ratings, or how the bug
can be triggered via web content
- Do NOT include Bugzilla links or Firefox bug numbers
- Describe the issue purely in terms of the library's API and internal behavior
- Include the T3 standalone test case (or reference it) — but see the security
exception below
- Include the library-side code path trace from A1
- If a fix is verified (A4), include it as a suggested fix
Security exception — this overrides the bullet above. If the bug carries any
sec-* keyword, do not attach the T3 reproducer. For a memory-safety bug the
reproducer is the exploitation detail, and this report is written to be filed
publicly — which would disclose the vulnerability before any fix ships, the exact
thing Implement.3's commit ordering exists to prevent. Instead: describe the
failing condition abstractly, hold the reproducer for the library's security
contact, note in plan.md Notes that the report was filed in reduced form, and
ask the user before filing anything upstream on a security bug.
This report should be suitable for filing as an upstream bug report or attaching
to a pull request / issue tracker entry.
A4. Fix strategy (deferred to Implement.2b):
Do not create a fix branch or patch during Diagnose. Record these two Design
directions and leave row 42.1 pending:
- Upstream fix: submit to upstream, then update vendored copy via
./mach vendor
or manual update. Preferred for long-term health. Larger scope acceptable.
- Local Firefox patch (if urgent): apply as a patch on top of the vendored
library pending upstream acceptance. Smaller scope, faster to land.
Implement.2b owns branch construction, security ordering, vendoring, and
FAIL→PASS verification after the implementation choice is approved.
Branch B: Firefox Integration Bug (scope b)
The library works correctly — the bug is in how Firefox uses it (wrapper code, IPC
actors, threading model, lifecycle management, error handling around library calls).
B1. Pivot investigation to Firefox code:
Resume the standard investigation steps, focused on the integration layer:
-
Code path trace (Diagnose.8): Trace the Firefox integration code using
searchfox revision-pinned links. Include the boundary where Firefox calls into
the library and how results/errors propagate back.
-
Design intention (Diagnose.10): Study the Firefox integration code's git
history. Why was the wrapper written this way? What assumptions does it make
about the library's behavior?
-
Proof test (Diagnose.13): Create a Firefox test that reproduces the
integration bug. Follow A2 steps 1-6 using proof id b1: choose the framework,
register it in the manifest, commit it on the working branch or the required
evidence branch, apply A2's security/injection export rules, then build and run.
No separate library test is needed — the library is correct.
-
Debug instrumentation (Diagnose.13e): Create sherlock/bug-<id>-debug-b1
branch from the test commit. Add instrumentation, commit, generate
firefox/debug/<NN>-debug-firefox-b1.patch.
-
Run with instrumentation (Diagnose.14): Build on the debug branch
(./mach build), run tests, capture debug logs to firefox/debug/. Switch back
to sherlock/bug-<id> (working branch with test commits only).
B2. Fix strategy (deferred to Implement.2):
Do not implement during Diagnose. Leave row 42.1 pending. Candidate directions
for Design include:
- Correct threading assumptions (e.g., library is not thread-safe but Firefox calls
it from multiple threads)
- Fix lifecycle management (e.g., using library object after shutdown)
- Add missing error handling around library calls
- Correct IPC serialization of library types
No upstream submission needed. The T3 result ("library works correctly") should be
documented to prevent future misattribution.
Branch C: Split Scope (scope a+b, or scope c)
The root cause spans both the library and Firefox's integration. Common patterns:
- Library has an undocumented API contract; Firefox violates it
- Library has a threading assumption; Firefox's threading model breaks it
- Firefox's local patches in the mapped vendored directory introduce a bug not in upstream
- Library returns an error that upstream callers handle but Firefox's wrapper doesn't
C1. Investigate both layers:
You need TWO code path traces, TWO design intention studies, and TWO sets of
permanent links:
- Library side: Code path trace with upstream permanent links. Design intention
from library git history. Document what the library expects (API contracts,
threading model, preconditions).
- Firefox side: Code path trace with searchfox links. Design intention from
Firefox git history. Document where Firefox violates the library's expectations
or fails to handle a library-side edge case.
For scope (c) (Firefox local patches): compare the vendored code against the
upstream revision to identify what the patches changed and whether the patch
introduced the bug:
git diff HEAD:media/<lib>/src/file.c <local-lib-repo>/src/file.c
ls media/<lib>/*.patch
C2. Create and run tests for BOTH layers:
- Library test (from T3): Already created and run in T3. The result (PASS or
FAIL) indicates whether the library itself has a bug or just an undocumented
limitation.
- Firefox test (A2 pattern): Create a Firefox-side test that demonstrates the
integration aspect (e.g., the contract violation, the missing error handling).
Follow A2 steps 1-6 using proof id
c2: choose the framework, register it in
the manifest, use the working or evidence branch as A2 requires, apply A2's
security/injection export rules, run against the unfixed tree, and capture
output to firefox/debug/. This test MUST fail without the fix.
C3. Generate upstream report (if library-side fix needed):
If the library has a bug, undocumented limitation, or missing hardening that
contributes to the issue, generate an upstream report following the same rules
as Branch A step A3. Read references/upstream-report-template.md.
Write to <run-dir>/<library>/bug-<id>-upstream-<library>.md.
For split-scope reports, frame the issue from the library's perspective:
- If the library has a bug: report it as a bug
- If the library has an undocumented API contract: frame as a documentation or
hardening request ("library should validate X" or "document that callers must Y")
- Do NOT reveal the Firefox-side contract violation or exploitation path
- Do NOT include Firefox security ratings or Bugzilla links
C4. Fix strategy (deferred to Implement.2b):
Do not create either fix during Diagnose. Present separate Design strategies for
each layer and leave row 42.1 pending:
| Layer | Fix Type | Scope |
|---|
| Library | Harden API, add validation, document contract | Long-term, submit upstream |
| Firefox | Respect API contract, add error handling, fix threading | Smaller scope, land in Firefox |
Both fixes may be needed. The analysis doc should clarify:
- Which fix is necessary (without it the bug persists)
- Which fix is defensive (hardens against the class of bug)
- Landing order: Firefox fix can land immediately; library fix goes upstream then
gets vendored later
For scope (c): evaluate whether to fix the local patch, replace it with
a better patch, or remove it entirely (if upstream now handles the case).
Implement.2b owns branch construction, security ordering, vendoring, and
verification. If the T3 library test PASSes because the issue is an undocumented
limitation, preserve that PASS as diagnostic evidence and use the upstream
report's hardening/documentation variant; do not invent a FAIL→PASS test patch.
Summary: Required Tests by Scope
| Scope | Library Test (T3) | Firefox Test | Notes |
|---|
| (a) Library bug | Yes — must FAIL | Yes (A2) — must FAIL | Both required |
| (b) Firefox integration | Diagnostic only (PASSES) | Yes (B1) — must FAIL | Only Firefox test is proof |
| (a+b/c) Split | Yes — may PASS or FAIL | Yes (C2) — must FAIL | Both required, separate evidence |
Diagnose.10: Study Design Intention (Team D)
Design archaeology is Team D, launched in the Wave 2 wave (Diagnose.8). Its
full contract and prompt are in references/agent-teams.md; it writes
teams/team-d-design-archaeology.md (introducing commit, original purpose, design
rationale, constraints, function contract, related code, drift signals) and does
NOT claim how the root cause relates to the design.
Team D is replaced, not dropped, when Diagnose.9 is active (set row 11
skipped). Design intention is instead recovered inside the branch workflow —
Branch A in A1, Branch B in B1, Branch C in C1 — but that work must still be
written to teams/team-d-design-archaeology.md, in the same section shape.
The filename is a contract, not a byproduct: Reframe.1 reads it unconditionally,
and Team P and Team I both take it as a required input path. A third-party run that
leaves it absent hands Wave 3 three agents pointing at a file that does not exist.
During Synthesis the main agent reads the Team D file and writes the Design
Intention section in the analysis doc, adding the two sentences the team
declined to write: how the current root cause relates to (violates / reveals a
gap in / drifts from) the original design intention, and the broken invariant
(the property the contract implies should always hold, which the root cause shows
does not). Both are reserved for the main agent, and both are load-bearing for the
Reframe phase — solutions that respect the original design intention are more
likely to be correct and maintainable than patches that only address the symptom.
Diagnose.11: Classify Claims Continuously
This is a continuous discipline, not a progress row that can be completed before
the claims exist. Apply it during Synthesis, proof evaluation, and document
authoring. Diagnose.15a performs the final document-wide audit.
Two-tier rule — every statement must be classified:
| Tier | Label | Meaning | Requirement |
|---|
| Verified | (no label) | Confirmed by code, logs, or data | Cite the file:line or log entry |
| Assumption | [Assumption] | Plausible but unconfirmed | Label clearly; state what would confirm/refute |
Before writing each claim, ask yourself:
- Code behavior — "Function X does Y": Did you read that code? If not, read it
or label
[Assumption].
- Causation — "X causes Y": Can you trace the exact call path? If inferring,
label
[Assumption].
- Environment — "Fails on some drivers": Do you have log evidence? If not,
label
[Assumption: needs log analysis].
- Absolutes — "always", "never", "only": Read the code to confirm.
Mandatory checks before writing Root Cause:
Diagnose.12: Synthesis (main agent, not delegated)
Mark plan.md row 15 in-progress. Read the Wave 1 files (B, H) and every
applicable Wave 2 file whose row is completed — C, T, optional L/X, and D or
its required third-party replacement. Read the files, not subagent transcripts, and
write <run-dir>/teams/synthesis.md:
- Merged code-trace + design-intention narrative; note any drift.
- Re-rank the hypothesis tree against the gathered evidence. Revive any pruned
hypothesis the evidence warrants — do not silently re-anchor.
- Classify each hypothesis as
to-test / refuted / assumption-only, citing
the Team C/L/D/X evidence that drove the classification.
- State the verified root cause with two-tier labels (
Diagnose.11), plus the
sentence on how it relates to the design intention (Diagnose.10).
- State the broken invariant in one sentence — the property Team D's function
contract implies should always hold, and that the root cause shows does not.
This is the hand-off into Reframe; without it, Reframe re-derives the contract
from scratch. It goes into the analysis doc's Design Intention section.
- Append one row 16.x to plan.md per
to-test hypothesis, naming the target
proof-test patch and log paths (see references/plan-template.md). If
Diagnose.9 is active, the proof tests are tracked by the Branch rows under
row 14 (A2/T3/C2) instead — append no 16.x rows and mark row 16 skipped.
A team never declares the root cause — this synthesis is the main agent's. Mark
row 15 completed.
Diagnose.13: Evaluate and Create Proof Tests
This step processes the row 16.x entries Synthesis appended — one per
to-test hypothesis. Builds serialize: Team T already scouted the framework
and drafted the test source in parallel (teams/team-t-frameworks.md); the main
agent now writes, builds, and runs each test one at a time. Mark parent row 16
in-progress, then mark each row 16.x in-progress before its build and
completed or skipped after the result is captured. Mark row 16 completed
only after all children are terminal. If there are no testable hypotheses, mark
row 16 skipped with the alternative evidence in Notes.
Note: If Diagnose.9 is active, skip this step — proof tests are already
created within the branch workflow:
- Branch A: T3 (library test) + A2 (Firefox test)
- Branch B: B1 (Firefox proof test)
- Branch C: C2 (both library and Firefox tests)
Read references/test-frameworks.md for framework selection and FuzzingFunctions mapping.
Diagnose.13a: Check Bug Attachments for Existing Testcases
Team B already fetched attachments to <run-dir>/bug-<id>-attachments — read from
there (do not re-fetch). If a testcase exists and uses FuzzingFunctions, apply
the mapping table from references/test-frameworks.md. Auto-convert to the
appropriate framework.
Diagnose.13b: Determine Test Framework
Team T already chose a framework and found a neighbour test per hypothesis
(teams/team-t-frameworks.md). Confirm its choice against the decision tree in
references/test-frameworks.md:
- Crash → crashtest (HTML-triggerable) or gtest (C++ only)
- Web-exposed + spec-defined → WPT (follow
references/spec-check.md first)
- Web-exposed + Firefox-specific → mochitest
- Internal C++/Rust → gtest
Diagnose.13c: Check Build Requirements BEFORE Writing Tests
- Try reproducing in the current build first
- If that fails, try a standard debug build
- Only if needed: ASan/TSan
Signals for sanitizer builds:
- "data race" / "race condition" → TSan
- "heap-use-after-free" / "buffer-overflow" / ASan signature → ASan
- Bug report explicitly mentions sanitizer output
If a special build is needed:
- Read the mozconfig presets from
references/test-frameworks.md
- Auto-generate a mozconfig file:
<run-dir>/firefox/debug/bug-<id>-mozconfig
- Present to the user directly for review before building
- The user can invoke
/mozconfig for full interactive configuration if preferred
Diagnose.13d: Write Proof Test
Start from Team T's drafted test in teams/team-t-frameworks.md (adapt it; do not
build inside Team T). Assign the hypothesis a stable proof id (h1, h2, …).
For a benign test, write it into the tree and commit it on
sherlock/bug-<id>. For a code-injection test, create
sherlock/bug-<id>-evidence-<proof-id> from the current working tip, commit and
run it there, then switch back without cherry-picking it; its SHA and patch live
only in the evidence-ref field and firefox/debug/. Build serialized (one
./mach build at a time; FE-only tests use ./mach build faster). The test must:
- FAIL without fix — proving the bug exists (the root cause claim is correct)
- Be designed to PASS after fix — making it reusable for TDD development later
- Serve as EVIDENCE for the root cause claim
Diagnose.13e: Add Debugging Instrumentation
Create a proof-specific debug branch from the relevant test/evidence commit:
git checkout -b sherlock/bug-<id>-debug-<proof-id>
Add targeted logging to confirm the traced code path is actually hit during test
execution. Commit and generate the patch:
git add <instrumented files>
git commit --author="$SHERLOCK_AUTHOR" -m "Add debug instrumentation for <desc>"
git format-patch -1 --stdout > <run-dir>/firefox/debug/<NN>-debug-firefox-<proof-id>.patch
Common instrumentation patterns for Firefox C++/JS:
- MOZ_LOG:
MOZ_LOG(gMediaDecoderLog, LogLevel::Debug, ("SHERLOCK: %s:%d", __FILE__, __LINE__));
- printf (quick and dirty):
printf("SHERLOCK: reached %s:%d\n", __FILE__, __LINE__);
- Mochitest JS:
info("SHERLOCK: state=" + variable);
- GTest:
GTEST_LOG_(INFO) << "SHERLOCK: value=" << variable;
The debug branch is temporary — it's used for the build+run+capture cycle in
Diagnose.14, then you switch back to the main working branch.
When NOT to Write a Test
Skip the test entirely when:
- Data race with narrow, platform-gated race window (flaky test adds CI noise)
- Code can't be exercised from JS in standard CI configuration
- Detection rate in typical CI run would be well below 50%
Crashtest is acceptable (~50% detection) when:
- Bug causes outright crash in normal CI builds
- Can run on platforms where crash occurs
- Cheap to write (simple HTML page triggering crash path)
If no test: document the rationale and alternative evidence in Test Evidence,
mark the hypothesis child skipped, and use the no-test verification branch in
Implement.4.
Diagnose.14: Run Tests and Capture Debug Logs
Note: If Diagnose.9 is active, skip this step — tests are already run and
debug logs captured within the branch workflow:
- Branch A: T3 ran the library test; A2 ran the Firefox test
- Branch B: B1 ran the Firefox test with debug instrumentation
- Branch C: T3 ran the library test; C2 ran the Firefox test
1. Build and run on the debug branch (has test commits + instrumentation).
Builds serialize — one at a time:
./mach build
./mach test <path> --headless 2>&1 | tee <run-dir>/firefox/debug/bug-<id>-<proof-id>-test-run.log
Additional debug logs go in the firefox/debug/ directory:
<run-dir>/firefox/debug/bug-<id>-debug-<description>.log
2. Switch back to the working branch:
git checkout sherlock/bug-<id>
The debug branch is preserved. The working branch has only test commits (clean,
ready for fix commits in the Implement phase).
3. Generate test patches from the branch history:
git format-patch -1 <commit> --stdout > <run-dir>/firefox/debug/<NN>-test-<proof-id>-<desc>.patch
cp <run-dir>/firefox/debug/<NN>-test-<proof-id>-<desc>.patch <run-dir>/firefox/fix/<NN>-test-<proof-id>-<desc>.patch
For a sec-* bug, or an injection-only test, the proof-test patch goes only to
firefox/debug/ at this
point — not to firefox/fix/. It enters the fix series in Implement.3, after the
fix commits, so a vulnerability-demonstrating test is never sitting first in the
landing order.
4. Evaluate results (mark each row 16.x completed with its result in Notes):
- Test FAILS as expected → confirms root cause, record as evidence
- Test PASSES (contradicts hypothesis) → re-examine root cause, loop back to
Diagnose.7 (row 8.5)
- Test inconclusive → note as
[Assumption], document what would make it conclusive
After every child is terminal, mark parent row 16 completed (or skipped when
all children were skipped).
Diagnose.15: Generate Analysis Documents
Mark plan.md row 17 in-progress.
Primary analysis document (always required):
The analysis doc already exists — Diagnose.2 created it from
references/analysis-template.md, and rows 4/5/7 filled Related Context, Failure
Pattern and the Hypothesis Tree. Edit the remaining sections in place; do not
recreate or overwrite the whole file. Source content from synthesis.md and the
team files.
Requirements:
- Fill ALL sections with actual content (no placeholders)
- Preserve what rows 4/5/7 already wrote
- Re-read the completed file after creation
- Verify all links are revision-pinned (not trunk URLs)
- Ensure the Design Intention section is present and filled, including both the
root-cause-relation sentence and the broken invariant line
Mark row 17 completed.
Diagnose.15a: Final two-tier claim audit
Mark row 17.25 in-progress. Re-read the completed analysis document from top to
bottom. Every factual statement must either cite code/log/data or carry an
[Assumption] label with what would confirm or refute it. Verify every absolute
(always, never, only) and every causal statement against its source. Mark
row 17.25 completed only after no unaudited claim remains.
Upstream report (required for Branch A and Branch C with library-side fix):
If Diagnose.9 produced a Branch A (library bug) or Branch C (split scope with
library-side component), the upstream report should already have been generated
in step A3 or C3. Verify it exists at <run-dir>/<library>/bug-<id>-upstream-<library>.md.
If not yet created, generate it now using references/upstream-report-template.md.
The upstream report must:
- Contain NO Firefox/browser/Bugzilla references
- Contain NO security exploitation details or sec-* ratings
- Use ONLY upstream permanent links
- Be self-contained and suitable for filing with the library's issue tracker
Diagnose.16: Structural self-check
Mark plan.md row 17.5 in-progress. Before launching the review team, the main
agent does a quick structural self-check (not the full audit — that is the review team's job):
If a check fails, fix it before launching the review team. Record the checklist
result in plan.md Notes and mark row 17.5 completed.
Phase: Diagnosis Review (REVIEW #1)
The deep audit is done by an independent review team, not by the main agent
grading its own work. The project caps delegation at depth one, so there are no
review-wrapper agents. Load/read the red-pen skill in the main agent. Set
plan.md rows 18/19/20 in-progress, perform Red Pen's
local pre-flight, then launch Reviewer L, Reviewer T, and the root-cause critic
directly in one message, each with fork_turns: "none" (full contracts in
references/agent-teams.md).
- Reviewer L (links / citations) →
review/L.md. Opens every local source
target or uses the web open mechanism for remote permalinks; confirms the cited
file:line still says what the doc claims; replaces any trunk URL with a
$SHERLOCK_REV link.
- Reviewer T (test re-runner) →
review/T.md. Re-reads every applicable
firefox/debug/bug-<id>-<proof-id>-test-run.log and confirms the proof test
FAILs on the bug, not on setup. When re-execution is warranted, create a
disposable worktree at $SHERLOCK_BASE, apply the exported patches there with
git am -3, run the tests, then remove the worktree. Never apply patches onto
the working branch, and do not use a one-range git range-diff.
- Reviewer R (red-pen on root cause) → the direct Red Pen critic writes the
full review to an explicit output path so it cannot collide with the
Decide.5 review:
spawn_agent(
task_name: "rootcause_review",
fork_turns: "none",
message: "Read <absolute analysis path> and the Red Pen review template; verify the cited source under <absolute repo root>; write the independent review to <absolute run-dir>/review/bug-<id>-rootcause-review.md; return the required four-line verdict summary. No prior conclusions are provided."
)
It challenges the root cause, hypothesis ranking, and assumption labels. After
it returns, the main agent writes review/R.md containing the returned verdict,
headline, iteration, and a pointer to the exact full-review path. This summary
file is bookkeeping; the independent critic owns the substantive review.
Why the explicit path. red-pen derives its output from the analysis doc
basename, so bug-<id>-analysis.md resolves to bug-<id>-review.md — the file
Decide.5 uses for the solutions review. Left implicit, whichever review ran
second would be pushed to bug-<id>-review-1.md, and Decide.6 would then read
the root-cause review instead of the solutions review it is responding to.
Passing an explicit …-review.md path keeps the two apart.
Reviewer R judges the root cause. The Decide.5 red-pen judges the
solutions. Different targets — both fire, and both are required.
As each reviewer returns, verify its file exists and mark its row completed.
Handle failures by looping back (offending row → in-progress, artifact
rewritten); do not argue with the reviewer (Gotcha #13):
- Reviewer L fail →
Diagnose.15 (re-edit doc + relink).
- Reviewer T fail →
Diagnose.13/Diagnose.14 (fix/re-run the test) or
Diagnose.12 (correct the verdict).
- Reviewer R
revise → Diagnose.15. redesign → escalate to the user.
reject / needs-more-info → back to Wave 2 (gather more evidence).
Record the response, not just the verdict. Reviewer L reports "pass/fail plus
fix-up diffs" — a pass with fix-ups is easy to drop on the floor, because the
loop-back rules only fire on fail. Before the gate, fill the analysis doc's
## Review #1 Response section: the three verdicts, and for each concern whether it
was accepted (and what changed) or rejected (and why). Gotcha #13 allows fixing,
escalating, or looping back — but not ignoring, and the record is what makes the
difference visible at Consolidate.
Only when all three reviewers pass (or their concerns are resolved and recorded)
proceed to the gate.
Gate: Root cause agreed
Mark plan.md row 21 blocked-on-user. Present a summary of the diagnosis to the
user:
- Root cause (1-2 sentences)
- The broken invariant (1 sentence)
- Key evidence (code path trace highlights)
- Test results
- Reviewer verdicts
- Path to the analysis doc
Ask: "Does this root cause analysis look correct? Say 'yes' to move on to
Reframe, or tell me what needs more investigation."
This is a hard gate:
- If user disagrees → loop back to the relevant Diagnose step, update the analysis doc
- If user wants changes → make the changes, re-present
- Do NOT proceed to Reframe without explicit user agreement
On agreement, mark row 21 completed.
Phase: Reframe
Read references/first-principles.md before starting — it defines the five
questions, the invariant discipline, and the widening discipline this phase runs on.
This phase does not produce patches. It produces what must be true. If you
find yourself writing a diff, stop and write the invariant the diff would
establish instead. Concrete approaches are the Design phase's job, and keeping
them separate is what stops the run collapsing into the first plausible fix.
Cost is not an argument here. Architecture revamps are on the table. Reviewer
pushback, patch size, and landing risk are all Decide-phase concerns. A design
space that has already been pre-filtered by practicality makes the Decide phase
a formality.
Reframe.1: Frame the problem (main agent, not delegated)
Mark plan.md row 22 in-progress. Read the analysis doc, teams/synthesis.md,
and teams/team-d-design-archaeology.md. Answer, in writing:
- Why is this a problem? Which guarantee is broken — stated as a proposition,
not as a description of the failure. For whom: end user, web content, the spec,
a named internal caller, a security boundary, a future maintainer. What the
concrete harm is, and its class. And honestly: would anyone notice if it were
never fixed?
- Why do we have this problem in the first place? Which decision, constraint,
absence, or drift permitted it. Classify the origin (missing contract / contract
drift / layering violation / representable illegal state / duplicated path /
lapsed purpose / defensive accretion) and name the specific commit, bug, or
absence. "Technical debt" is not an answer.
Team P (next step) supplies the evidence trail; the framing itself is yours.
Persist the answers before marking the row done. Create
<run-dir>/bug-<id>-principles.md from references/principles-template.md now,
with the header and the Q1/Q2 sections filled and the rest left as template
placeholders. Reframe.3 then edits this file rather than creating it.
Without this the answers live only in the transcript while Wave 3 runs — four
concurrent long-running agents, and a likely place to be interrupted. Resume trusts
completed rows and would read an artifact that does not exist, silently losing the
harm class that Decide.1 derives its weights from. Mark row 22 completed once
the file is on disk.
Reframe.2: Wave 3 teams (P, E, I, W)
Launch all four in one message with fork_turns: "none" (rows 23–26
in-progress before launching).
Full contracts in references/agent-teams.md. All are read-only and none may
