ops-preflight

# ops-preflight Onboarding answers *"will the loops run in this repo?"* This answers the other half: **will they do good work?** They are different questions with different failure modes. `ops-install` can report full coverage for a repo the loops will nonetheless waste every run on, because coverage matches capability **names**, not the state of the product. A repo with no single test command, a build that cannot run in an isolated worktree, and a branch model nobody wrote down passes onboarding and fails at everything after it. **It is not a loop** and **not a capability** — it implements no catalog action. Like `ops-install`, a human runs it, and it takes no `-loop` suffix for that reason. **It blocks nothing.** A repo is allowed to onboard with gaps and close them afterwards. What it must not do is let anyone believe there are none. ## What it does, in order | Step | Deterministic? | |---|---| | 1. Work out which check files apply | `scripts/select-profile.sh` | | 2. Read the repo against them | `scripts/inspect.sh` | | 3. **Ask about everything the files could not answer** | the list is `scripts/questions.sh`; the asking is you | | 4. Plan an issue per real gap | `scripts/plan-issues.sh` | | 5. Score the repo, once every check is answered | `scripts/score.sh` | | 6. File the planned issues, on a yes | you, with `github-ops` | ## Three verdicts, never two | Verdict | Set by | Printed as | |---|---|---| | **present** | a STRONG pattern matched — see below | `present` + the evidence that matched | | **present** | a human confirmed it | `present (declared)` — never with evidence | | **gap** | a human said it is not there | `gap`. The only thing that becomes an issue | | **unknown** | detection found nothing and nobody has been asked | `unknown` | | **unknown** | only a WEAK pattern matched — see below | `ASK`, with what was found | | **unknown** | a whole-product check found strong evidence for SOME active stacks, not all (see below) | `ASK`, naming the stack(s) with no strong evidence | **Silence is never a pass.** This is the same rule as *a gate that cannot run reports blocked*: a check nothing could answer reports `unknown`, not a pass. And a **self-reported yes is not evidence** — that is why `present (declared)` prints differently from a detected one, and why `inspect.sh` alone can never produce a `gap`. Only a human can say something is missing. **Never summarise `unknown` away.** "18 unknown" is not "18 missing" and it is not "18 fine". It is the count of questions still open, and reporting it as either of the other two is the one failure mode of this skill. ## A map, not an entry exam The data still carries two severities, `blocking` and `quality`. Nothing renamed them, and no other tool that reads this data changed. What changed is how a human reads them. This is a diagnostic map of a repo, not a pass/fail test, and nobody clears every box. The report says so at the top, and prints each severity as what it means for the loops rather than as a grade: | Severity (data, unchanged) | Printed as | |---|---| | `blocking` | **Needed for the loops to work** | | `quality` | **Makes the loops better** | ## Step 1 — which checks apply ``` scripts/select-profile.sh <repo-root> ``` Three layers, merged by check `id` — later wins, **by field**, so a profile entry may carry nothing but an `id` and the `detect` it is replacing: | Layer | Where | Ships | |---|---|---| | base | `scripts/checks.json` | engine. Product-agnostic wording only | | stack | `scripts/profiles/<stack>.json` | engine. Loads when its own `when` block matches | | repo | `<repo>/.claude/ops-preflight-profile.json` | the consumer | **Profiles are named for a stack, never a product**, and every matching one loads — a repo holding both a solution and a `package.json` genuinely has both stacks and both sets of checks are true of it. A check that is only true of one product belongs in that repo's own override file; layer 3 exists for exactly that, and putting a product fact in the engine breaks the golden rule. ## Step 2 — read the repo ``` scripts/inspect.sh <repo-root> --out findings.json ``` **One run, both outputs.** The report goes to the screen and the JSON to the file, which steps 3, 4 and 5 all need. Running it twice scans the repo twice: on a large repo that was 65 seconds each. **This is the path to use.** It does the looking itself and it is what the tests cover. It is slow on Windows, around half a minute on a real repo, because it starts about 1,400 programs at roughly 26ms each. That is a local annoyance, not a wrong answer. > ### The Glob and Grep path, and why it is not the default > > `--plan` and `--evidence` below let the harness do the searching with its own tools. It is faster > and you can watch it work. **It is also wrong on any repo big enough to care**, and a live run > found both reasons: > > - **The Glob tool stops at 100 results and cannot be asked for more.** `**/*.cs` on a real repo > is 12,157 files. A check that only asks "does anything match" survives that. A check that needs > strong evidence from every stack does not: the one match that proves the .NET side can fall off > the end of a list of 100, and the report then says a stack has nothing behind it when it does. > - **Glob does not know about `prune.json`.** It returns `node_modules/`, `obj/` and > `.claude/worktrees/`. `--evidence` now drops those on the way in, so this one is fixed, but it > was evidence pointing into a throwaway worktree, which is the exact bug the prune list exists > to stop. > > **If you use it, abandon it the moment any lookup says it truncated**, and run the script. Do not > report a result built on a truncated list. A session that hit this worked it out and switched on > its own; the next one might not. ### 2a. Ask what to look for ``` scripts/inspect.sh <repo-root> --plan > plan.json ``` Each entry in `plan.json` is one lookup, and it is dropped straight into a tool call: | Field | What to do with it | |---|---| | `tool` | `glob` means the **Glob** tool; `grep` means the **Grep** tool | | `glob` | Glob's `pattern`, or Grep's `glob` filter. Hand it over **exactly as written** | | `pattern` | Grep only: its `pattern`. Use `output_mode: "files_with_matches"` | | `id` | The key to file the results under | | `match`, `uses` | Not yours. Step 2c uses them | The patterns need no editing because the catalog is written in the same glob language the Glob tool speaks. `**` reaches into folders, a single `*` does not. If you find yourself rewriting a pattern to make it work, the catalog is wrong and the fix belongs there, not here. ### 2b. Run them, in parallel There are usually well over a hundred lookups, and they are independent. **Put many tool calls in one message** so they run at once. A few at a time turns a fast step into a slow one. Write the results to a file as `{"<id>": ["<path>", ...]}`, one key per lookup, an empty list for a lookup that found nothing. Paths are **relative to the repo root**; trim the absolute prefix the tools return. Record every path a tool gives back, and if a tool says it truncated a long list, say so rather than presenting what came back as the whole of it. ### 2c. Turn the results into the report ``` scripts/inspect.sh <repo-root> --evidence evidence.json # the report scripts/inspect.sh <repo-root> --evidence evidence.json --json # keep this; step 4 needs it ``` Every path has to clear two gates before it counts. It must match the pattern that asked for it, and it must not be under a pruned directory. Then the same verdict rules apply as always. > The two paths are asserted to produce identical findings **when the tool returned everything**. > If they ever disagree, the script is right and this step has a bug. **Show the report verbatim.** It is the honest answer, and a summary of it is not — particularly the `unknown` count. **It groups by section, in a fixed order**, not alphabetically and not by which capability breaks: 1. Release management 2. Testing 3. Harness 4. Environment 5. Frontend 6. Backend 7. Best practices 8. Utilities 9. Misc Release management and Testing come first because they are what unlock the merge and release parts of the pipeline, and they are the two sections to do first if someone only has time for one. **`verify-build-command`, `verify-test-command`, `verify-lint-command` and `verify-warnings-clean` live in Harness, not Backend or Frontend.** Each asks about the WHOLE product: "one command builds THIS repo", not "one command builds the dotnet half", and the source checklist's Harness item is "it runs all linting and tests to validate its own work", so the commands the harness runs belong there, next to `verify-self-check`. `dotnet-analyzers` stays in Backend and `node-component-tests` / `node-lockfile` stay in Frontend: those genuinely are stack-specific, so they stay where a stack-specific check belongs. On a repo with real backend AND frontend code, this leaves Backend and Frontend thin, and that is honest, not a bug: those sections were never about "how much backend or frontend code exists", they are about what is stack-specific ENOUGH that a whole-product answer would be the wrong shape for it. **Within a section, severity is a sub-heading, printed once, not repeated per line.** A `blocking` check sorts above a `quality` one, under its own **Needed for the loops to work** or **Makes the loops better** heading: ``` Release management Needed for the loops to work [unknown] Something already knows how to prepare a release (ops-release cut) why: Preparing a release ... Makes the loops better [unknown] After a release, the branches get put back in step (ops-release sync) why: ... ``` Earlier this repeated the severity phrase on every row, comma-joined onto the title — "Needed for the loops to work, Title" — which read as one broken sentence, and got worse the more checks a section held. Printing it once per group instead is what keeps a 26-row report readable. **It never runs your build.** A preflight that compiles the product only works on a machine that can compile the product, which rules out CI, a routine, and anyone looking at a repo they do not work in. It reads files. That is a deliberate ceiling, and it is why Step 3 is not optional. ### Read the evidence on a blocking `present` before you trust it Detection is a **seed, not an authority** — the same stance `ops-install`'s `detect.sh` takes. A glob is a guess, and a wrong `present` is worse than a wrong `unknown`, because it skips the question instead of asking it. So for every **blocking** check reported `present`, look at what matched. If the evidence is thin or plainly the wrong thing — a build check satisfied by an unrelated script that happens to be called `build-something.sh` — **treat it as unknown and ask anyway**. This costs one question and is the only defence against a confident wrong answer. ### Evidence strength lives on the pattern, not on the whole check Every detection rule in the catalog carries a **strength**, `strong` by default: STRONG means the thing matched is named for, or dedicated to, the exact job the check is asking about — a script literally named `build.sh`, a skill directory named `release-management`, a canonical file like `version.json`. A strong match resolves the check straight to `present`, evidence shown, no question. WEAK means the match only proves something *exists*, not that it does *this* job — a `package.json` proves a package exists, not that the published version lives there; an `azure-pipelines.yml` proves CI exists, not that it publishes a release; a `NuGet.config` proves a private feed *might* need a credential, nothing about whether a restore works without one. A weak match prints as `ASK` with the evidence attached, and counts as `unknown`. **Always ask these, and open with what was found:** *"There's a `NuGet.config` here — does a restore need a credential for a private feed?"* That is a much better question than the blind version, which is the entire reason the evidence is kept. One check can hold **both** kinds of pattern at once. `release-version-source` treats `version.json` as strong (it is the dedicated file) and `package.json` as weak (it is a generic manifest that happens to also exist) — a repo with only `package.json` still gets asked; a repo with `version.json` resolves straight to `present`. A check where **no** file could ever prove the positive fact — only ever hint that it might not hold — is still marked `signal: true` at the check level instead of tagging every pattern `weak` individually; `workspace-isolated-build` and `dotnet-private-feed` are the two that stay this way, because no detectable file proves a bare worktree suffices or that a restore needs no credential. Before pattern strength existed, `dotnet-private-feed` found a `NuGet.config` and reported `present` for *"a restore needs no credential"* — a false pass on a **blocking** check, for precisely the repos most likely to fail. Found in a dry run against a synthetic repo, which is why there is one. A second dry run, against a real repo with three genuine release skills, then surfaced the opposite problem: `signal: true` asked about all three in the same tone it would ask about a stray file that merely had "release" in its name — because a whole-check flag cannot tell strong evidence from weak. Pattern strength is the fix: `release-prepare` and `release-cleanup` now resolve straight to `present` when a skill directory is genuinely named for the job, and only ask when the evidence is generic. ### Evidence: strong first, weak second, never the same file twice A row's evidence line shows what actually matched, split by strength rather than mixed together: a reader has to be able to tell which file earned a `present` without guessing. ``` [PRESENT] The version lives in a known file (ops-release cut) found: version.json also seen (weak): Directory.Build.props, package.json ``` `found:` is always the STRONG matches: what earned the verdict, if the row is `present` at all. `also seen (weak):` only appears when there is ALSO weak evidence, so a reader can see the rest of what was found without mistaking it for what earned the pass. A row with no strong evidence at all (an `ASK`) still gets one plain `found:` line, the same as always. A file that happens to match both a strong pattern and a weak one on the same check (a `post-release-cleanup` skill matches both the strong `*post-release*` glob and the weak `*clean*` one) is shown only once, under `found:`, never repeated under `also seen (weak):` as if it were separate evidence. **JSON never hides a match, strong or weak. Text still caps, but only the weak list, never the strong one.** A real dry run found four `.claude/skills/...` entries (alphabetically first, and weak) filling every slot of a 3-item cap and pushing a genuinely strong `azure-pipelines.yml` match out of the report entirely: a false claim that the file was not detected. `evidence_strong` and `evidence_weak` in the JSON report are both complete, always; only the TEXT rendering of `evidence_weak` caps at 3 with a trailing `(+N more)`, because a weak match is supplementary evidence, never what a `present` rests on. A check can legitimately show dozens of strong matches in text (twenty-two integration test files is twenty-two lines of real evidence), and that is the correct trade: never hiding a strong match matters more than a short report. ### Whole-product checks need EVERY active stack, not just one `verify-build-command`, `verify-test-command`, `verify-lint-command` and `verify-warnings-clean` are marked `whole_product: true` in the catalog (see `checks.schema.json`) because their QUESTION is about the whole repo, not one stack. On a repo with only one active stack profile this changes nothing. On a repo with **two or more** (a repo with both a `.sln` and a `package.json`, which loads both the `dotnet` and `node` stack profiles), a strong match tagged to only ONE of them can no longer resolve the check to `present`. This is the OR-union problem `signal: true` originally existed to prevent, coming back through pattern strength: a real `npm test` script is genuinely strong evidence that the front-end half is tested, and proves nothing about the dotnet half sitting right next to it. A repo reporting `present` on `verify-test-command` from front-end evidence alone, while an equivalent dotnet-only repo reports `ASK` for the identical check, was found in a real dry run, and this is the fix for it. ``` [ASK ] Commands run the tests for the whole product (ops-change verify) found: src/StaticAssets/package.json also seen (weak): Tests/FooTests.cs (+383 more) no strong evidence from: dotnet why: A verify that cannot run the tests reports a pass that means nothing. ... ``` `no strong evidence from: <stacks>` only appears on a `whole_product` check that found strong evidence for some active stacks but not all: that is `source: partial` in the JSON, a third `unknown` source alongside `weak` and `null`. A strong match with no stack tag at all (a literal root `build.sh`, `test.sh` or `lint.sh`) counts for every active stack at once, because a real repo-wide command genuinely answers the question regardless of how many stacks the repo has. It does not need to be repeated once per stack to satisfy this rule. **All four ask for commands, plural.** A repo with a front end and a back end normally builds, tests and lints in two steps, and that is a `present`: what the loops need is that every part is covered by a command someone can run, not that the count is one. Take the list and the order the steps go in. A part with no command at all is the gap, however short the rest of the list is. ## Step 3 — ask about the rest **Read what the repo already says about itself, first.** ``` scripts/list-skills.sh <repo> scripts/list-skills.sh <repo> --json ``` A repo's own skills are a short, purpose-written index of what this codebase knows how to do, and they answer preflight questions outright. A live repo had a `repo-setup` skill describing "git hooks, demo site creation, and dependency installation", a `session-hook-config` skill naming "dotnet restore fails with 401 errors", and a `demo-site-management` skill. All three questions were asked anyway, because nothing read them. It also reports engine skills the repo already has. That same repo had four, so it was part-onboarded and preflight never said so. **A description is a claim, never a verdict.** It says what a skill means to do, not that it works. So this never resolves a check. Use it to put the name in front of the person and let them answer. ### Then find the questions the repo has already answered ``` scripts/list-skills.sh <repo> --for findings.json ``` A repo partway through onboarding ships its own capability skills, and those skills hold the answers. Every check records the capability it is about, so a check whose `consumer` is `ops-change` is answered by the repo's own `ops-change`, if it has one. This prints the join: ``` verify-build-command ops-change · verify .claude/skills/ops-change/SKILL.md workspace-isolated-build ops-workspace · prepare .claude/skills/ops-workspace/SKILL.md ``` **Read the file before asking.** Find the part about that action and lead the question with what it says: > Your `ops-workspace` says a plain worktree is full CI parity, because the tests are > self-contained and the pipeline runs them with no SQL and no container, and that `prepare` must > not stand up a demo site. Still true? Not reading it produced two wrong answers on a live repo. Asked cold what single command builds the product, the answer was "no single command"; that repo's `ops-change` names all three. Asked whether a bare worktree is enough, the answer was "it needs a demo site first"; that repo's `ops-workspace` says the opposite and gives its reasoning. Both answers came from memory of building by hand. Listing the skill names was not enough, because neither answer is in a description. On a repo that has not started onboarding this prints nothing, which is the normal case. **When nothing links, read the list anyway.** The join only works on the `ops-*` names, because a check records the capability it is about and matches a skill of that name. A repo that has not onboarded names its skills its own way: one had `umb-release-notes`, `umb-bump-version` and `umb-review`, which speak to the release and review questions and link to nothing. Scan the plain list for anything that sounds like the question you are about to ask, and read it. The worst case is that you read a file that turns out to be irrelevant.