cargo-crev-review

# cargo-crev: reviewing Rust dependencies as an agent This skill tells you how to review a Rust dependency on the user's behalf and help them produce a signed `cargo-crev` package review proof that they can publish to their proof repository. ## When to use this skill Use this skill when the user asks you to: - review one of their Rust dependencies (or a transitive dep), - audit a specific crate version, - produce a `cargo crev` review/proof for a crate, - help them catch up on crates that still need reviewing. Do *not* use this skill for unrelated security audits, non-Rust code, or trust proofs for other crev ids — those are separate workflows. ## Prerequisites (check these first) 1. The user has a crev id configured: ```sh cargo crev id current ``` If this fails, stop and ask the user to run `cargo crev id new --github-username <them>` first. Do **not** auto-generate an id. 2. The user's proof repo has been set up and is reachable. `cargo crev repo dir` prints the local path. ## High-level workflow 1. **Refresh the user's web of trust.** Run `cargo crev update` once at the start of the session to pull the latest proofs from every trusted reviewer's proof repo (it's a shortcut for `cargo crev repo update`). Without this, both candidate discovery (which depends on the `status` column reflecting current trusted reviews) and the later cross-check against existing reviews can be working off stale data, causing you to either review a crate someone else just signed off on or miss a recent advisory. See "Refreshing the web of trust" below. 2. **Pick the crate + version to review.** If the user names a specific crate, skip to the next step. Otherwise, use the candidate discovery step below to produce a ranked list and confirm the pick with the user. 3. **Start a review report file.** See "The review report file" below. Every finding from every subsequent step — positive or negative — gets appended to this file as you go. This is your scratchpad, your audit trail, and the source you distill into the final `comment`. 4. **Locate the crate's local source.** Use `cargo crev crate dir <name> <version>` to print the path to the crate source tree that cargo downloaded from crates.io. Record the path in the report. 5. **External verification.** Check that what crates.io shipped matches the public upstream repository at a specific commit, and that that commit corresponds to the advertised version tag. See "External verification" below. Record the outcome — pass, partial, or fail — in the report. Any unexpected discrepancy is a real finding and must be raised to the user before proceeding. 6. **Read the code.** See "Reviewing the code" below. Append findings to the report as you go. 7. **Cross-check against existing reviews.** Query the local proof database for any existing reviews of this crate (any version), read them, and reconcile with your own findings. See "Cross-checking existing reviews" below. Record the comparison in the report. 8. **Assemble the unsigned proof file.** Fill in the YAML fields (rating, thoroughness, understanding, comment, issues, advisories, `llm-agent`). The `comment` is **the full text of the report file** by default — see "Assembling the unsigned proof" below. 9. **Validate the proof file.** Round-trip the unsigned proof through `cargo crev review --import-unsigned-from ... --no-store --no-edit --print-unsigned`. This step is **mandatory** — fix and re-validate until it passes. See "Round-trip validating the draft" below. 10. **Hand off to the user.** The agent does **not** sign or publish. Give the user two files — the report and the unsigned proof — along with the exact command they should run to interactively review, edit and sign it. See "Handing off to the user" below. ## The review report file Maintain a single markdown file per review, at: ``` target/crev/reviews/<crate>-<version>.md ``` (Create the directories if they don't exist.) Start it as soon as a candidate is picked, **before** any inspection begins. Append to it throughout the review — treat it as an append-only journal. Minimum structure: ```markdown # Review: <crate> <version> - Local source: <path from `crate dir`> - Upstream repository: <URL or "unknown"> - Upstream commit verified: <sha or "n/a"> - Upstream tag for version: <tag or "n/a"> - Verification outcome: pass | partial | fail ## External verification <details of what was checked, what matched, what differed> ## Code review findings <ongoing notes — one bullet per observation, grouped by file or theme> ## Cross-check against existing reviews <how many prior reviews, agreement, disagreement, verdict on whether they change your draft fields> ## Open questions / things skipped <anything you didn't examine and why> ## Draft review fields - rating: ... - thoroughness: ... - understanding: ... ``` Rules: - **Append, don't rewrite.** If you change your mind about something, add a follow-up note; don't quietly edit earlier entries. The user should be able to see how your understanding evolved. - **Every problem, however small, goes in.** Even if you decide it's not worth reporting in the final proof, it belongs in the report so the user can sanity-check your judgement. - **Record what you did *not* look at.** If you skipped the test directory, or only skimmed `build.rs`, say so explicitly under "Open questions / things skipped". This is what backs honest `thoroughness` / `understanding` values later. - **Hand the report to the user** alongside the unsigned proof file at the end of the workflow (see "Handing off to the user"). The report is how the user actually audits *your* work; the unsigned proof is the thing they sign after auditing. ## Refreshing the web of trust Before any candidate discovery or cross-checking, pull the latest proofs from every trusted reviewer: ```sh cargo crev update ``` This is a shortcut for `cargo crev repo update`. It walks every proof repository in the user's WoT and `git fetch`es it, so the local proof database reflects the current state of the world. It can take a few seconds to a minute depending on how many ids the user trusts and network conditions; that's normal. Run this **once per session**, before the first candidate-discovery pass, and again only if the session has been running long enough that the data is plausibly stale or the user explicitly asks. It is not necessary to re-run it between successive reviews in the same session. If the command fails for some repos (network blip, dead remote, auth issue) but succeeds for others, that is usually fine — note the failures in case they matter, but proceed. If it fails entirely (e.g. no trusted ids configured, so there is nothing to fetch), stop and tell the user: candidate discovery and cross-checking will both be meaningless until the WoT is set up. ## Finding candidate crates to review When the user hasn't named a specific crate, first make sure the WoT is fresh (see "Refreshing the web of trust" above), then capture a stats dump of their dependencies. **`cargo crev verify` is slow** — it walks every dependency, downloads metadata from crates.io, and computes trust graphs. On a non-trivial project it can take **30 seconds to several minutes**. Therefore: - **Run it exactly once per session** and save the output to a file. - **Never re-run it** unless the user explicitly asks or the dependency tree has changed (e.g. after `cargo update`). - All candidate discovery, sorting, and filtering works on the saved file — re-read the file instead of re-running the command. ```sh cargo crev verify \ --show-reviews true --show-downloads true \ --show-loc true --show-flags true \ --force-print-header \ > target/crev-verify.txt 2>&1 ``` Notes: - Only request the columns you actually need for candidate selection: `reviews`, `downloads`, `loc`, and `flags`. The default output already includes `status`, `name`, and `version`. - Do **not** use `--show-all` — it enables many extra columns (geiger, owners, issues, lpidx, latest trusted version, …) that produce wide, hard-to-parse output and are not needed for picking candidates. If you need a specific extra column later (e.g. `--show-geiger true` for a particular analysis), add it to a one-off query for that crate, not to the full dump. - `--force-print-header` makes the tool print the column header even when stdout is redirected. **Always** use this when capturing — the header is the only reliable way to know which column is which. - Exit status `255` is expected and does **not** mean the command failed. It signals `VerificationFailed` — i.e. at least one dependency is not yet fully verified, which is obviously the case when you're about to start reviewing. Only treat it as a real error if the stdout file is empty or the stderr contains an actual error message. - Status column values you'll see: - `local` — crate from a local path source (skip; not reviewable via crates.io). - `none` — no existing trusted review; prime candidate. - `pass` — already has sufficient trusted reviews; skip. - `flagged` / `dangerous` — has issues reported against it; these may be worth confirming rather than reviewing fresh. - `N/A` — there are no trusted ids in the user's WoT at all. If you see this, stop and tell the user they should set up trust first (`cargo crev trust <id>`) before candidate discovery is meaningful. Once the file exists, **work from it for all subsequent candidate selection in this session**. If you need to re-examine the data, read `target/crev-verify.txt` — do not re-run `cargo crev verify`. Before narrowing down, confirm the criteria with the user — see the "Picking candidates" section below. ## Picking candidates Picking what to review is inherently a bit arbitrary, but the default goal is: **find the non-passing, non-local, least-reviewed, least-downloaded package** — i.e. the most obscure crate in the user's dependency tree that nobody else is likely to be reviewing any time soon. Unless the user says otherwise, apply these filters/rankings in order: 1. **Filter by `status`.** Keep only `status = none`. Skip `pass` (already covered), `local` (not from crates.io), and `N/A` (configuration problem, handled separately). `flagged` / `dangerous` are a separate workflow (confirming pre-existing issue reports) — don't pick them unless the user asked. 2. **Sort by lowest total review count.** Use the second `reviews` column (total across all versions, not the per-version count). Zero-review crates come first. 3. **Tiebreak by lowest total download count.** Popular crates will eventually get reviewed by someone else; obscure ones won't. Reviewing the long tail is higher marginal value. Use the second `downloads` column (total). 4. **Tiebreak by `CB` flag.** Among otherwise equal candidates, prefer crates with a custom build script — they run arbitrary code at build time and are higher-risk, so the review is more valuable. If the user has additional or different preferences (e.g. "I only care about direct deps", "prioritize high lpidx", "skip `-sys` crates"), they override the defaults above — ask if unsure. **The user's own crates are valid candidates.** If a top candidate turns out to be authored by the user themselves, don't filter it out — a review where the author says "I wrote this and had an LLM agent audit it" is still a useful signal for downstream consumers, and once published it'll flip to `status = pass` and drop out of the filter naturally. Agent provenance is recorded structurally via the `llm-agent:` field on the proof (see "Disclosing agent provenance" below), so no special comment wording is needed for own-crate reviews. **Pick one candidate, not a shortlist.** Filter the captured stats file by the rules above, sort by the priority order (status → review count → download count → CB flag), and propose the single most promising crate to the user. Don't offer a menu — it just slows things down. After that crate is reviewed and published, repeat the selection process on the same captured file (or re-capture if it's stale) to pick the next one. ### Column legend These are the columns present in the recommended `cargo crev verify` output (with `--show-reviews`, `--show-downloads`, `--show-loc`, `--show-flags`). Run `cargo crev verify --help` for the full legend including columns not shown by default. ``` - status - Trust check result: `pass` for trusted, `none` for lacking reviews, `flagged` or `dangerous` for crates with problem reports. `N/A` when crev is not configured yet. - reviews - Number of reviews for the specific version and for all available versions (total). Two sub-columns: version count / total count. - downloads - Download counts from crates.io (version / total). Two sub-columns: version count / total count. - loc - Lines of Rust code - flgs - Flags for specific types of packages - CB - Custom Build (runs arbitrary code at build time) - UM - Unmaintained crate - crate - Crate name - version - Crate version ``` ## External verification Before reading a single line of the crate's code, verify that what crates.io shipped actually matches the public upstream repository. This catches the single highest-leverage class of attack (a publisher whose account was compromised, who can ship a tarball that doesn't match their public git history). It also tells you *which* upstream commit the reviewed snapshot corresponds to, which is valuable information in its own right. Ideal outcome, recorded in the report file: > Package matches public repo `<url>` at commit `<sha>`, which is the > commit tagged as `<tag>` corresponding to version `<version>`. Anything short of that is a partial or failed verification, and **every discrepancy must be recorded in the report and raised to the user**. Don't silently proceed past a verification issue — if the user hasn't told you what to do in that case, stop and ask. ### Procedure 1. **Get the local source path.** ```sh cargo crev crate dir <name> <version> ``` Record it in the report. Call this path `$SRC`. 2. **Look for `.cargo_vcs_info.json`** at `$SRC/.cargo_vcs_info.json`. When present (it usually is), it contains the upstream git sha the tarball was built from: ```json { "git": { "sha1": "abc123…" }, "path_in_vcs": "subcrate" } ``` Treat this as a **hint, not a guarantee**. The file may be missing (older crates, unusual publish flows), or the sha may be slightly off (dirty worktree at publish time). If it's missing, note that in the report and fall back to searching the upstream repo by version tag instead. 3. **Find the upstream URL.** Read `$SRC/Cargo.toml.orig` (the pristine pre-publish manifest) and extract `package.repository`. If absent or dead, also check `package.homepage`, then crates.io metadata. If none of those point at a reachable repository, **search the web** (e.g. `"<crate-name>" site:github.com` or the author name plus the crate name) — crates sometimes move or get renamed and the manifest stays stale. Record every URL you tried and which one resolved. If you still cannot find a public source after a reasonable search, record the failure and ask the user how to proceed. Do not skip external verification silently. 4. **Clone (or update) the upstream cache.** Use a stable cache directory so repeat runs don't re-clone: ```sh mkdir -p target/crev/review-cache UPSTREAM=target/crev/review-cache/<crate> if [ -d "$UPSTREAM" ]; then git -C "$UPSTREAM" fetch --tags origin else git clone --filter=blob:none "$REPO_URL" "$UPSTREAM" fi ``` `--filter=blob:none` keeps the clone fast even for huge histories — blobs are fetched on demand when you actually diff. 5. **Identify the upstream commit to compare against.** Prefer, in order: 1. The sha from `.cargo_vcs_info.json` (if it exists upstream). 2. The commit that the version tag points to — tag naming conventions to try: `v<version>`, `<version>`, `<crate>-v<version>`, `<crate>-<version>`. 3. If neither matches, report the failure and ask the user. Record in the report which method succeeded and which commit/tag was chosen. Ideally both the sha and the version tag agree — if they disagree (e.g. vcs_info points at a commit the tag does not), that is itself a finding and must be reported. 6. **Check out the commit.** ```sh git -C "$UPSTREAM" checkout "$SHA" ``` 7. **Compare the trees.** Compare `$SRC` against `$UPSTREAM/<path_in_vcs>` (or `$UPSTREAM` if `path_in_vcs` is empty or `.`). Use `diff -r --brief` for an initial pass, then drill into any differing file with a full `diff` to see the content. **Files expected to differ or to exist on only one side** — add these to an ignore list and do not flag them: - `Cargo.toml` — always rewritten by `cargo publish`. Compare `$SRC/Cargo.toml.orig` against upstream's `Cargo.toml` instead. Workspace-inherited fields may differ; those differences are expected and benign as long as they're clearly inheritance artefacts. - `Cargo.toml.orig` — exists only in the crate tarball. - `.cargo_vcs_info.json` — exists only in the crate tarball. - `.cargo-ok` — cargo extraction marker. - `Cargo.lock` — sometimes present, sometimes stripped. - `.gitignore`, `.gitattributes` — often excluded via `exclude`. - `target/` — build artifacts, never in the tarball. Any other difference is real and must be recorded in the report with the file path and a short description of what differs. 8. **Cross-check the manifest.** Diff `$SRC/Cargo.toml.orig` against `$UPSTREAM/<path_in_vcs>/Cargo.toml` at the checked-out sha. Expected differences for workspace-published crates: fields like `version`, `authors`, `edition`, `license` being inlined from the workspace root; `[workspace]` stanza absent from the crate copy; path dependencies resolved to version specs. Flag anything else. 9. **Record the outcome in the report** under "## External verification", e.g.: > Package matches public repo > <https://github.com/foo/bar> at commit > `abcdef0123` tagged `v1.2.3`. Manifest differences are limited > to expected workspace-inheritance artefacts (listed above). All > other files bit-for-bit identical. or, on failure: > Package does **not** cleanly match upstream. The file > `src/util.rs` differs from the upstream copy at the commit > recorded in `.cargo_vcs_info.json` (see diff below). Raising > to user before continuing. Any verification failure blocks the rest of the review until the user says what to do. ## Internal verification (reviewing the code) Once external verification has passed (or the user has explicitly told you to proceed past a failure), start the actual code review. This is "internal verification": convincing yourself that the code inside the verified tarball does what it claims to and nothing more. ### Mindset Approach the code **adversarially**. Assume, as a working hypothesis, that everything you are reading could be wrong or hostile, and it is your job to either falsify that hypothesis or document what you found. This is not paranoia; it's the baseline for a review that is worth signing your name to. Concrete rules that follow from this: