wpt-evaluator

name	wpt-evaluator
description	Evaluate the quality of a WPT test file against guidance in the upstream WPT documentation. Use when reviewing a generated or submitted test before merge.

WPT Evaluator (doc-inputs variant)

This skill evaluates a single WPT test file against the upstream web-platform-tests/wpt documentation directly. It is an advisory signal for human reviewers, not a merge gate.

The evaluator is given a curated reading list of upstream markdown docs and is asked to extract guidance and flag violations itself. Findings cite the source doc + line range that prompted the flag.

When to use

• Reviewing a test file just generated by wpt-gen or written manually, before opening a pull request.
• Reviewing a submitted pull request to surface issues a human reviewer might otherwise have to catch by hand.

Do not use this skill as a substitute for running wpt lint, running the test, or human review. It complements those steps; it does not replace them.

Inputs

• The path to a single WPT test file.

Outputs

For each finding:

• Severity: error, warn, info, or nit. Infer from upstream language — "MUST", "must not", "required" → error; "should", "recommended" → warn; "may", "preferred" → info; "nit" prefixed in the source → nit.
• Line reference into the test file under evaluation.
• Short evidence quote from the test file.
• Source citation: the relevant upstream doc path + line range that prompted the finding (e.g., wpt/docs/writing-tests/general-guidelines.md:L82-L87).
• One-sentence summary of what the guidance says, paraphrased.

Prohibited outputs

The following are hard prohibitions, not preferences:

1. No composite score. Do not aggregate findings into a single number, grade, or pass/fail verdict. Report each finding on its own.
2. No proposed fixes. Findings describe what is wrong and why, never how to fix it. The following all count as proposed fixes and must not appear in a report:
   • "Should use X instead." / "The test should..."
   • "Suggested title:" / "Suggested replacement:" / "Suggested rewrite:"
   • "Could be improved by..." / "Consider..."
   • Concrete code rewrites or before/after snippets.
   • A code block that is not present in the test file as written. Evidence quotes (verbatim from the test) are not fixes and are allowed.
3. No invented rule IDs. The source citation (upstream doc path
   • line range) is the identifier.

If you find yourself writing "should be X" or attaching a code suggestion, stop. State the problem and cite the source; the human reviewer decides the remediation.

Procedure

1. Detect test kind from the file path and contents. Possible kinds: testharness, reftest, print-reftest, crashtest, manual, visual, idl, wdspec, css (orthogonal to the above — a test can be both testharness and css).
2. Load the curated reading list for that kind (see below). Always include the "all kinds" baseline plus any kind-specific docs.
3. Read each listed doc in full. Identify normative statements (look for MUST, SHOULD, MAY, required, must not, do not, never, always, etc., plus list items in the reviewer checklist).
4. Evaluate the test file against each statement. Skip rules already enforced by wpt lint (no tabs, no trailing whitespace, no setTimeout, assert_throws/promise_rejects deprecation, filename duplicate / case-collision rules, etc.) to avoid double-flagging.
5. Follow declared dependencies as needed (see below) when reading them would inform a specific finding.
6. Submit findings by calling report_evaluation_complete with findings and input_scope payloads. Each finding object must populate the fields described in the Outputs section above. Before submitting, verify no finding violates the prohibited outputs: no composite score, no proposed fix, no invented rule ID. Findings describe what is wrong and why, never how to fix it. The wpt-gen CLI renders the final Markdown report and writes it to disk; do NOT format or write the report yourself.

Dependency reading (as-needed)

The test file may declare dependencies on other local files. List every declared dependency you detect, but only read a dependency when doing so is necessary to evaluate a specific piece of upstream guidance. The goal is to keep the corpus small unless the contents of a dependency actually matter for a finding.

What counts as a declared dependency

Detect these forms in the test file's source:

• <script src="..."> — JS includes.
• <link rel="match" href="..."> and <link rel="mismatch" href="..."> — reftest reference files.
• <link rel="stylesheet" href="..."> — CSS includes.
• <link rel="help" href="..."> — spec URL (external; do not read).
• <img src="...">, <iframe src="...">, <source src="...">, <video src="...">, <audio src="..."> — embedded resources.
• // META: script=... — JS dependency (testharness JS-only tests).
• importScripts("...") — worker script imports.
• fetch("...") and similar runtime fetches to local paths.

For each dependency, note the path it points to. Classify it as:

• Framework if it begins with /resources/, /common/, /fonts/, or /media/. These are WPT-provided and should be listed but not read.
• External if it's an absolute URL (http(s)://) or starts with //. List only; do not fetch.
• Local otherwise. May be read on demand per the criteria below.

When to read a local dependency

Read a local dependency only if at least one of these applies:

• Reftest reference verification: the test is a reftest and the upstream reftest guidance ("the reference file should use a different technique that won't fail in the same way as the test" and "the reference file ... does not use the technology under test") cannot be evaluated without comparing the two files. Read the reference file.
• Reference existence: a <link rel="match"> or <link rel="mismatch"> points at a path. Verify the file exists (directory listing is sufficient; reading is not required unless the previous bullet applies).
• META script existence: a // META: script=... points at a local path (not under /resources/ or /common/). Verify the file exists.
• Specific upstream guidance requires it: a particular normative statement in a loaded doc can only be evaluated by inspecting the dependency's contents. State which statement, and why the dependency's contents are needed, when emitting the finding.

Do not read a local dependency:

• To explore the broader test ecosystem.
• To look for similar tests or patterns.
• To check for duplicates (lint already covers basename and case-collision duplication).
• Because it "might be relevant" without a specific upstream statement that requires it.

All dependencies you read must appear in the Input scope section with their byte sizes, alongside the test file and the curated reading list.

Curated reading lists by test kind

All kinds (baseline — always read)

• wpt/docs/writing-tests/general-guidelines.md
• wpt/docs/writing-tests/file-names.md
• wpt/docs/writing-tests/assumptions.md
• wpt/docs/writing-tests/server-features.md
• wpt/docs/reviewing-tests/checklist.md (read the All tests section; the kind-specific sections are loaded below)

testharness

• wpt/docs/writing-tests/testharness.md
• wpt/docs/reviewing-tests/checklist.md (Script Tests Only section)
• For testdriver-driven tests: wpt/docs/writing-tests/testdriver.md

reftest / print-reftest

• wpt/docs/writing-tests/reftests.md
• wpt/docs/writing-tests/rendering.md
• wpt/docs/reviewing-tests/checklist.md (Reftests Only section)
• If the file is a print reftest: wpt/docs/writing-tests/print-reftests.md

crashtest

• wpt/docs/writing-tests/crashtest.md

manual

• wpt/docs/writing-tests/manual.md

visual

• wpt/docs/writing-tests/visual.md
• wpt/docs/writing-tests/rendering.md
• wpt/docs/reviewing-tests/checklist.md (Visual Tests Only section)

idl

• wpt/docs/writing-tests/idlharness.md
• wpt/docs/writing-tests/testharness.md (idlharness builds on testharness)

wdspec

• wpt/docs/writing-tests/wdspec.md

css (additive — read on top of whatever else applies)

• wpt/docs/writing-tests/css-metadata.md

Tracking input scope

While running the evaluation, record every file you opened in service of the evaluation, along with its byte size (wc -c <path>). Include the test file itself. Do NOT count files only used for navigation (directory listings, this skill itself, etc.).

For each tracked file, classify it as one of:

• skill — this SKILL.md.
• reading-list — a doc from the curated reading lists above.
• test — the test file under evaluation.
• dependency — a file referenced by the test and read on demand per "Dependency reading (as-needed)" above.

Separately, list any declared dependencies that you detected but did NOT open (framework + external dependencies — /resources/*, /common/*, absolute URLs).

You submit this data as part of the input_scope payload to the report_evaluation_complete tool. The wpt-gen CLI renders the final Input scope table, computes the total byte count, and computes an approximate input-token figure (bytes ÷ 4) for cross-approach comparison. Do NOT compute or format any of this yourself — just report what you opened.