pmstack

Turns a /run-eval result into a single self-contained, shareable HTML report — a verdict hero (pass/conditional/fail), the per-case table with PASS/FAIL color-coding, and the metrics table — that a PM can paste into Slack, a launch doc, LinkedIn, or a PR. The report carries a "run your own eval" backlink to pmstack. Use when the user says "make this eval shareable," "turn my eval run into a report," "export the eval results," "an HTML report of my eval," "share my eval results," or has a /run-eval output directory and wants a presentable artifact. Renders real run data only (from summary.md) — never invents numbers. Runs bin/eval-report.py on a run directory. Output: report.html next to the run's summary.md.

Grades an eval the user already has against Anthropic's eval-design principles and returns an Eval Report Card — a letter grade A–F, the failures that earned it with the offending line, the single highest-leverage fix, and a shareable verdict card. It is the inverse of /eval (which writes a suite); this checks whether a suite is any good. Use when the user says "grade my eval," "is my eval any good," "review my eval suite," "roast my eval," "what's wrong with this eval," "did I write this eval right," pastes an eval YAML or rubric and asks for feedback, or wants to check an eval against the Anthropic framework before trusting it. Catches grader leakage, missing negative cases, no target, vibes-as-metric, and uncalibrated LLM-as-judge. Accepts a file path, a pasted/attached eval, or a plain-English success metric. Output: outputs/eval-grade-<feature>-<date>.md.

Synthesize many raw customer signals (support tickets, interview snippets, churn reasons, sales notes, NPS verbatims, app reviews) into a small set of ranked, PRD-ready problems. Use when a PM pastes or attaches a pile of feedback and asks "what's the real problem / what should we build first / what are the themes", when there are too many signals to read one by one, or when someone needs to turn voice-of-customer data into prioritized problem statements before writing a PRD. This is the front-of-funnel step that precedes pmstack-prd.

An interactive 9-step tutorial that walks a new user through every pmstack capability using a realistic AI-code-review scenario. Includes Anthropic's vibe-test (read raw transcripts before formalizing an eval) and transcript-review (diagnose failed trials as model / grader / task errors) rituals. Works in claude.ai web, desktop, and mobile (no terminal needed). Trigger when the user says "I just installed pmstack," "how do I use this," "tutorial," "walk me through pmstack," "getting started," or "show me what pmstack does."

Walks a PM through Anthropic's Step 6 ritual — reading transcripts from many trials to diagnose every failed eval task as one of three things — model mistake, grader mistake, or task-spec error. Implements the practice Anthropic describes as "critical" — without it, badly-calibrated graders mask real model improvements. Use when the user has a /run-eval result with failures, asks "why did this fail?", says "let's read the transcripts," wants to debug an eval, suspects grader bugs, or mentions Anthropic's transcript-review practice. Accepts a run folder path, pasted summary + cases, or attached files. Output: a markdown memo at outputs/transcript-review-<feature>-<date>.md with verdict counts and per-trial diagnoses.

Walks a PM through the layer-1 ritual of reading raw transcripts of an AI feature in action — what Anthropic calls "manual testing, dogfooding, and intuition" — before formalizing a structured eval. Surfaces failure patterns, drafts task candidates (including balanced negative cases per Anthropic Step 3), and produces a verdict on whether the feature is ready for /eval. Use when the user says "I want to vibe-test this," "I have transcripts of our AI feature," "before we write evals let's read the data," "we have support tickets / bug reports about this AI feature," or asks how to start an eval from scratch. Accepts pasted transcripts, attached files, or a folder path. Output: a markdown memo at outputs/vibe-test-<feature>-<date>.md.

Design a PM-runnable evaluation suite for an AI feature, implementing Anthropic's eval framework (Demystifying Evals for AI Agents). Walks the user through Anthropic's 8-step roadmap — source tasks from real failures, write unambiguous tasks with reference solutions, build balanced problem sets (test where behavior should AND shouldn't occur), choose graders thoughtfully (code / model / human). Use when a PM mentions "eval", "evaluation suite", "test suite for AI", "agent eval", "AI quality bars", "how would we know if X works", "designing tests for our AI feature", or wants to define what "good" looks like before shipping. Output: a YAML test suite that /run-eval can execute end-to-end.

Execute an evaluation YAML produced by pmstack-eval against a real target system, running n_trials per task and reporting both pass@k and pass^k. Implements the running side of Anthropic's eval framework. HARD-STOPS with educative errors if no target is configured, no tasks are present, or required env vars are missing — never simulates results. Use when the user asks to "run", "execute", or "score" an eval, or wants real metrics from an existing eval design. Requires a runtime that can execute Python and the `claude` CLI (Claude Code, sandboxed code-exec on web, or local terminal).

Re-runs an existing eval suite, diffs against the prior baseline, and writes a release-blocker memo if anything regressed. Use weekly on a schedule, or on demand before a release. Trigger when the user mentions "did my AI feature get worse," "eval drift," "model regression," "eval over time," "weekly eval," "release-blocker check," "is this still working," or asks to re-run /eval-self / /run-eval and compare to last time. Designed for repeated runs — produces durable, dated artifacts.

Pre-launch verifier. Aggregates PRD, metrics, eval design, eval run, premortem, eval-drift, and brief evidence into a single GO/NO-GO/CONDITIONAL verdict with each item showing pass/fail/missing plus the file that proves it. Trigger when the user says "are we ready to ship," "launch readiness," "release check," "pre-launch review," "ship it?" or asks to gate a release. Run before any launch.

Walks the pmstack skill graph against the outputs/ directory to find graph gaps (missing downstream artifacts), cross-artifact drift (PRD-metrics-eval misalignment), and stale candidates (superseded files). Each finding includes a one-line "do this" recommendation. Trigger when the user says "lint my workspace," "graph check," "find drift," "tidy up outputs," "what's stale," or asks for a workspace audit. Run weekly on schedule, or before any review.

Run a Klein-style pre-mortem on a draft PRD before launch — simulates 3 plausible failure stories 6 months in the future, names leading indicators and mitigations, then offers to mutate the PRD's Risks section. Trigger when the user says "pre-mortem," "what could go wrong," "risk pass," "imagine this fails," "stress test the spec," or asks to harden a PRD before review. Use after /prd, before launch.

A Monday-morning self-snapshot for PMs. Three sections only — decisions made, open loops aging out, and one required "thing I changed my mind about." Anti-vanity by design. Trigger when the user says "weekly memo," "weekly recap," "Monday status," "what changed this week," "weekly self-check," or asks to summarize last week's PM activity. Run weekly on schedule, or on demand to catch up.

Run the pmstack self-eval suite to score every pmstack skill against canonical scenarios, with regression check vs the locked golden baseline. Use when the user wants to test pmstack itself, check skill quality after a model upgrade or prompt change, or verify pmstack still works before recommending it to others.

Full PM sprint orchestrator — chains pmstack-prd, pmstack-metrics, pmstack-eval, and pmstack-brief in sequence with explicit user confirmation between each step. Use when the user has a customer signal or feature ask and wants to go end-to-end (problem → spec → metrics → eval design → stakeholder comms) in one guided pass with checkpoints.

Draft a stakeholder brief on a topic, tailored to executive, engineering, customer, or board audience. Use when a PM asks for a "brief", "update", "readout", "exec summary", "eng comms", "customer announcement", or "board update", or when they need to communicate a status / decision / change to a specific audience and need help adapting the framing.

Feature-by-feature comparison of two or more products with built-in eval design and runnable execution. Use when the PM wants to compare specific features across products (e.g., "compare GitHub Copilot vs Cursor", "how does our pricing stack up against X and Y", "evaluate these two AI coding tools head-to-head"). Use pmstack-competitive instead for market-positioning analysis (audience, value prop, white space).

Generate a structured competitive landscape analysis for a market or product category. Use when a PM mentions "competitive analysis", "market positioning", "white space", "where do we wedge in", asks who else is in a space, or wants to size up rivals before strategy/launch. Focus is on market positioning (audience, value prop, white space) — for feature-by-feature parity, use pmstack-compare instead.

Design a measurement framework for an AI product or feature with North Star, supporting metrics, counter-metrics, and AI-specific quality / latency / cost metrics. Use when a PM asks "how do we measure this", "what's the North Star for X", "how would we know X is winning", or mentions instrumentation, OKRs, KPIs, or success criteria for an AI feature.

Turn a raw customer signal (quote, support ticket, feature request, exec ask) into a structured Product Requirements Document. Use when a PM mentions a customer quote and wants a spec, when "let's write a PRD" comes up, when a vague feature request needs scoping, or when the user asks for a problem statement, MoSCoW prioritization, success metrics, or PRD template.