Run any Skill in Manus with one click

pmstack-eval-drift

Stars8

Forks1

UpdatedMay 5, 2026 at 18:11

Re-runs an existing eval suite, diffs against the prior baseline, and writes a release-blocker memo if anything regressed. Use weekly on a schedule, or on demand before a release. Trigger when the user mentions "did my AI feature get worse," "eval drift," "model regression," "eval over time," "weekly eval," "release-blocker check," "is this still working," or asks to re-run /eval-self / /run-eval and compare to last time. Designed for repeated runs — produces durable, dated artifacts.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

RyanAlberts

RyanAlberts/pmstack

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software Quality Assurance Analysts and TestersComputer and Mathematical Occupations·SOC 15-1253

SKILL.md

readonly

name

pmstack-eval-drift

description

eval-drift — scheduled drift watch over an AI feature

A scheduled drift-watch over LLM-substrate behavior. Mechanical diff only — no causal hypotheses.

Required structure for the artifact

# eval-drift — <YYYY-MM-DD>
RELEASE_BLOCKED: true|false
SCOPE: self|user|all
RUNS_SINCE_BASELINE: N

## Per-suite delta
| suite | prev | now | Δ | P0 pass-rate Δ |
...

## Top regressions (max 3)
- <suite> / <case-id>: expected X, got Y
...

## Trailing-4-run trajectory
<short table; only if 4+ prior runs exist>

Hard rules

No causal hypotheses. Never write "this regression is likely due to X." Mechanical diff only — speculation under unattended schedule is hallucinated theater.
No suppression. If P0 regressed by any amount, RELEASE_BLOCKED is true.
First-run bootstrap. When no prior baseline exists, replace the body with BASELINE: this is run 1; drift detection begins next run. Still write the artifact.
Conditional brief. On RELEASE_BLOCKED=true, run the stakeholder-brief skill (audience: engineering) using this artifact as input.

Where to write

With filesystem: outputs/eval-drift-<YYYY-MM-DD>.md and (on regression) outputs/brief-eval-drift-<YYYY-MM-DD>.md.
Inline (web/mobile): emit the artifact as markdown with the suggested filename at the top. The user pastes it into their tracker.

Decision-log entry

Append one line: - <date> — eval-drift: RELEASE_BLOCKED=<bool> — outputs/eval-drift-<date>.md

Tone

Mechanical. Direct. No softening of regressions, no editorializing on causes.

Cold-start behavior

First invocation against an empty baseline still produces a useful artifact (the baseline marker). Do not skip the artifact.

More from this repository

same repository

pmstack-eval-report

RyanAlberts/pmstack

Turns a /run-eval result into a single self-contained, shareable HTML report — a verdict hero (pass/conditional/fail), the per-case table with PASS/FAIL color-coding, and the metrics table — that a PM can paste into Slack, a launch doc, LinkedIn, or a PR. The report carries a "run your own eval" backlink to pmstack. Use when the user says "make this eval shareable," "turn my eval run into a report," "export the eval results," "an HTML report of my eval," "share my eval results," or has a /run-eval output directory and wants a presentable artifact. Renders real run data only (from summary.md) — never invents numbers. Runs bin/eval-report.py on a run directory. Output: report.html next to the run's summary.md.

2026-06-248

pmstack-eval-grade

RyanAlberts/pmstack

Grades an eval the user already has against Anthropic's eval-design principles and returns an Eval Report Card — a letter grade A–F, the failures that earned it with the offending line, the single highest-leverage fix, and a shareable verdict card. It is the inverse of /eval (which writes a suite); this checks whether a suite is any good. Use when the user says "grade my eval," "is my eval any good," "review my eval suite," "roast my eval," "what's wrong with this eval," "did I write this eval right," pastes an eval YAML or rubric and asks for feedback, or wants to check an eval against the Anthropic framework before trusting it. Catches grader leakage, missing negative cases, no target, vibes-as-metric, and uncalibrated LLM-as-judge. Accepts a file path, a pasted/attached eval, or a plain-English success metric. Output: outputs/eval-grade-<feature>-<date>.md.

2026-06-208

pmstack-voc

RyanAlberts/pmstack

Synthesize many raw customer signals (support tickets, interview snippets, churn reasons, sales notes, NPS verbatims, app reviews) into a small set of ranked, PRD-ready problems. Use when a PM pastes or attaches a pile of feedback and asks "what's the real problem / what should we build first / what are the themes", when there are too many signals to read one by one, or when someone needs to turn voice-of-customer data into prioritized problem statements before writing a PRD. This is the front-of-funnel step that precedes pmstack-prd.

2026-05-298

pmstack-onboarding

RyanAlberts/pmstack

An interactive 9-step tutorial that walks a new user through every pmstack capability using a realistic AI-code-review scenario. Includes Anthropic's vibe-test (read raw transcripts before formalizing an eval) and transcript-review (diagnose failed trials as model / grader / task errors) rituals. Works in claude.ai web, desktop, and mobile (no terminal needed). Trigger when the user says "I just installed pmstack," "how do I use this," "tutorial," "walk me through pmstack," "getting started," or "show me what pmstack does."

2026-05-058

pmstack-transcript-review

RyanAlberts/pmstack

Walks a PM through Anthropic's Step 6 ritual — reading transcripts from many trials to diagnose every failed eval task as one of three things — model mistake, grader mistake, or task-spec error. Implements the practice Anthropic describes as "critical" — without it, badly-calibrated graders mask real model improvements. Use when the user has a /run-eval result with failures, asks "why did this fail?", says "let's read the transcripts," wants to debug an eval, suspects grader bugs, or mentions Anthropic's transcript-review practice. Accepts a run folder path, pasted summary + cases, or attached files. Output: a markdown memo at outputs/transcript-review-<feature>-<date>.md with verdict counts and per-trial diagnoses.

2026-05-058

pmstack-vibe-test

RyanAlberts/pmstack

Walks a PM through the layer-1 ritual of reading raw transcripts of an AI feature in action — what Anthropic calls "manual testing, dogfooding, and intuition" — before formalizing a structured eval. Surfaces failure patterns, drafts task candidates (including balanced negative cases per Anthropic Step 3), and produces a verdict on whether the feature is ready for /eval. Use when the user says "I want to vibe-test this," "I have transcripts of our AI feature," "before we write evals let's read the data," "we have support tickets / bug reports about this AI feature," or asks how to start an eval from scratch. Accepts pasted transcripts, attached files, or a folder path. Output: a markdown memo at outputs/vibe-test-<feature>-<date>.md.

2026-05-058

name

pmstack-eval-drift

description

eval-drift — scheduled drift watch over an AI feature

A scheduled drift-watch over LLM-substrate behavior. Mechanical diff only — no causal hypotheses.

Required structure for the artifact

# eval-drift — <YYYY-MM-DD>
RELEASE_BLOCKED: true|false
SCOPE: self|user|all
RUNS_SINCE_BASELINE: N

## Per-suite delta
| suite | prev | now | Δ | P0 pass-rate Δ |
...

## Top regressions (max 3)
- <suite> / <case-id>: expected X, got Y
...

## Trailing-4-run trajectory
<short table; only if 4+ prior runs exist>

Hard rules

No causal hypotheses. Never write "this regression is likely due to X." Mechanical diff only — speculation under unattended schedule is hallucinated theater.
No suppression. If P0 regressed by any amount, RELEASE_BLOCKED is true.
First-run bootstrap. When no prior baseline exists, replace the body with BASELINE: this is run 1; drift detection begins next run. Still write the artifact.
Conditional brief. On RELEASE_BLOCKED=true, run the stakeholder-brief skill (audience: engineering) using this artifact as input.

Where to write

With filesystem: outputs/eval-drift-<YYYY-MM-DD>.md and (on regression) outputs/brief-eval-drift-<YYYY-MM-DD>.md.
Inline (web/mobile): emit the artifact as markdown with the suggested filename at the top. The user pastes it into their tracker.

Decision-log entry

Append one line: - <date> — eval-drift: RELEASE_BLOCKED=<bool> — outputs/eval-drift-<date>.md

Tone

Mechanical. Direct. No softening of regressions, no editorializing on causes.

Cold-start behavior

First invocation against an empty baseline still produces a useful artifact (the baseline marker). Do not skip the artifact.