| name | judge-readme |
| description | Use when reviewing, grading, or auditing a README — before publishing a repo, after major doc changes, evaluating an OSS dependency, or when a maintainer asks "is this README good enough?". Applies a rubric across first-impression, install path, accuracy, navigation, mental model, progressive disclosure, maintenance signals, and honesty about limitations. |
Judge a README
Overview
A README is the front door of a repository. Most READMEs fail because they document the surface (what the API looks like) instead of the mental model (what the system is, how it behaves, where it breaks). This skill grades a README across 8 axes and surfaces the single best heuristic for the overall verdict.
Use this rubric to produce a written review. For each section: state what works, what's missing, and what to fix. Be specific — quote lines, point at headings, name the bug.
How to use
- Read the README end-to-end once without judging — first-impression matters.
- Walk the 8 sections below in order. Each section has questions; answer each with evidence from the README.
- Apply the single best heuristic at the end as a sanity check on your verdict.
- Scan for the listed anti-patterns — these are auto-deductions.
- Produce a punch list: what to fix, in priority order.
The Rubric
1. First-30-seconds test
Whether a stranger can orient themselves above the fold.
- What is this? One sentence, above the fold, that a stranger can understand.
- Who is it for? The reader can self-select in or out within the first paragraph.
- Does it work? Evidence of life — CI status, weekly-downloads count, last-release date, version badges, "tested on X". A live downloads number is a stronger signal than a release date; a green CI badge is stronger than either. No one trusts an abandoned repo.
2. Can I install it in under 5 minutes?
Whether a reader who pastes every code block in order ends up with a working thing.
- Prerequisites are explicit — runtime versions, peer deps, external services.
- Install is one copyable block. No "see our docs site for installation."
- Quick Start runs end-to-end. No gaps, no "configure X" without showing how.
- Examples use consistent names.
mainKnowledgePool in setup and 'main' in search is the #1 README bug — it silently breaks every copy-paste user.
3. Does it match reality?
Whether the README has been maintained alongside the code.
- Every code sample compiles. Stale type signatures and renamed exports are worse than no examples.
- Version drift is acknowledged. If a video/blog post/section is outdated, say so in place rather than letting readers discover it the hard way.
- Error responses and edge cases are documented, not just the happy path — readers trust docs that admit what breaks.
4. Is it navigable?
Whether someone returning for the third time can find what they need fast.
- Table of contents for anything over ~300 lines.
- Section depth is meaningful. h4 under h3 under h2 should reflect real containment, not just "I ran out of
##". Orphaned subsections are a tell that the doc grew without being restructured.
- Cross-references are links, not prose. "See above" ages badly;
[Metadata Filtering](#metadata-filtering-where) survives edits.
- Duplicate information is dangerous. If the same endpoint is documented twice, one copy will rot. Pick a canonical home.
5. Does it teach the mental model, not just the API?
Whether the reader leaves with intuition, not just syntax.
- Why, not just how. "Bulk runs are never auto-queued — they must be triggered manually" tells me more than a signature dump.
- Lifecycles and invariants are stated. "Only one bulk run per pool at a time", "deletion can't be disabled", "the id correlates X to Y" — these are the things users discover painfully if they aren't written down.
- Failure modes are named. A Troubleshooting section is a proxy for "the maintainer has actually supported this in production."
Docs-site exception. If the project has a canonical reference site linked above the fold (e.g. zod.dev/api, nextjs.org/docs), grade this section against "does the README teach enough that the reader knows whether to dive into the docs site, and what to look for when they get there?" — not "does it teach the full mental model?". Duplicating the full reference in the README would violate Section 4's "duplicate information is dangerous" rule. The README's job in that case is the bridge, not the encyclopedia.
6. Progressive disclosure
Whether each audience can stop reading at the section they need.
- Quick Start gets you working. Reference gets you unstuck. Examples show idiomatic use.
- Advanced features don't block beginners. Bulk API, provider callbacks, custom adapters — all deferred past the minimum path to a working search.
- Config tables with required/optional columns let people scan instead of read.
7. Maintenance signals
Whether the project looks alive and usable inside a company.
- Changelog linked. No changelog = no signal that the project is maintained.
- License stated. Missing license = unusable in most companies.
- Contribution path is visible. Even one line ("open an issue first") beats nothing.
- Tone matches maturity. An alpha project shouldn't read like it's v3; a stable project shouldn't read like a hackathon pitch.
8. Honest about limitations
Whether the maintainer respects the reader's time.
- Adapter parity, platform support, known gaps are surfaced, not hidden in a GitHub issue.
- A reader who finds a limitation in the README feels informed; the same reader finding it after 2 hours of debugging feels lied to.
The single best heuristic
Apply two heuristics, scoped to the README's audience:
User contract: Can a new user open the README cold and ship a small correct integration without reading the source?
Required for every README.
Contributor contract: Can a new contributor open the README cold and, without talking to the maintainer, ship a small correct PR?
Required if the README is the contributor entry point. Optional if a CONTRIBUTING.md is linked from the README — in that case grade the contributor experience against that file, not the README.
The contributor contract requires the README (or CONTRIBUTING.md) to explain the architecture, not just the surface. Most projects fail this test, and it's usually the difference between a project that gets drive-by contributions and one that doesn't.
If the user contract fails, that overrides surface polish on sections 1–7. If only the contributor contract fails and there's no CONTRIBUTING.md linked, that's still a real defect — flag it but don't let it override user-facing strengths.
Anti-patterns (auto-deductions)
Mark these explicitly when found — they signal larger problems.
| Anti-pattern | Why it's bad |
|---|
| Marketing copy where reference docs should be | Readers need facts, not pitch |
| Screenshots of code instead of code blocks | Not copy-pasteable, not searchable, not accessible |
| "See the source" as documentation | Punts the maintainer's job onto every reader |
| Emoji-driven feature lists that don't map to section headings | Decoration without navigation value |
| Badges without a purpose — more than ~6, or any badge that doesn't map to a distinct signal (CI, license, downloads, version, coverage, support channel) | Cargo-cult signal of quality; 5 purposeful badges fine, 10 decorative ones aren't |
| A "Roadmap" that's actually a wishlist with no signal about priority or timeline | Sets expectations the project can't meet |
| Videos/blogposts linked without a "recorded against vX" disclaimer | Readers waste time on stale content |
Placeholder text shipped to readers (your-repo, TODO, lorem ipsum, broken anchor links) | Single biggest trust-killer above the fold — signals nobody proofread before publishing |
Inverted layering — required helpers (e.g. embedQuery) defined after the Quick Start that uses them | Copy-paste of the "minimum path" produces ReferenceError; the example block isn't actually self-contained |
| Quick Start that ends at config without a "verify it works" snippet | Reader has no way to confirm the install did anything; doc covers ingestion but not retrieval |
Output format
When grading a README, produce:
- Verdict — one line. (e.g. "Strong on install, weak on mental model. A new user ships in 10 min; a new contributor can't.")
- Section-by-section findings — for each of the 8 axes, what works and what's missing, with line/heading references.
- Anti-patterns hit — list any from the table above.
- Best-heuristic answer — yes/no on both the user contract and the contributor contract, with reasoning. If
CONTRIBUTING.md is linked, note that the contributor test is scoped to that file.
- Punch list — fixes in priority order (highest impact first).
