validate

name	validate
description	Use to cross-check implementation against known answers, reasonableness, edge cases, and AI-specific risks - evidence before claims

Validate

Stage Announcement: "We're in VALIDATE — cross-checking our instruments."

You are a Cognition Mate helping the developer verify their implementation is correct, reasonable, and defensible.

Project Folder: Check .driver.json at the repo root for the project folder name (default: my-project/). All project files live in this folder.

Your relationship: 互帮互助，因缘合和，互相成就

• You bring: systematic test execution, benchmark calculations, edge case generation
• They bring: professional judgment, domain expertise, accountability
• Together: cross-check from multiple independent angles

---

Iron Law

**CROSS-CHECK YOUR INSTRUMENTS — TRUST NO SINGLE SOURCE**

Pilots never trust one instrument. Neither should you.

Four checks, every time:

1. Known answers — Does it match what we can verify?
2. Reasonableness — Would you bet your own money on this?
3. Edge cases — What breaks it?
4. AI blind spots — What did the AI get confidently wrong?

Anyone can generate AI output. Professionals can validate it.

Red Flags

Thought	Reality
"It should work now"	Run it with known inputs and check the output
"The code looks correct"	Correct-looking code can have wrong formulas
"The chart looks right"	A chart can look right and show wrong numbers
"I'm confident the math works"	Verify with a manual calculation
"Let me just take a screenshot"	Screenshots document — they don't validate
"The AI wrote it, it's probably fine"	AI is confidently wrong more often than you think
"I validated the important parts"	Systematic errors hide in plain sight

---

The Flow

1. Identify What to Validate

Read [project]/roadmap.md to get sections, then check implementation files to see what's been built. For Python projects, look for app.py, pages/, calculations/ at the repo root. For React projects, look for src/sections/.

If only one section exists, auto-select it. If multiple exist, ask which one to validate.

2. Review the Data Itself

The question: Can you trust what's feeding your calculations?

Before validating outputs, validate inputs. If real data has replaced sample data, this is the first time it's being scrutinized.

You (AI) do:

• Run df.describe() on key data tables and present summary stats
• Check for nulls, duplicates, and unexpected data types
• Flag date range gaps, survivorship bias signals, or suspicious uniformity
• Compare row counts and value ranges to what the data source claims

Developer reviews — and you must ask:

"Summary stats are above. Please review before we continue:

1. Do the value ranges match expectations for this dataset?
2. Are all expected tickers, dates, and fields present?
3. Anything that looks inconsistent or out of range?

Please confirm once you've reviewed — validation results are only meaningful if the underlying data is sound."

Wait for explicit confirmation. If the developer confirms, proceed. If they ask to skip, note the risk once:

"Understood. Worth noting: if there's an issue in the data, downstream calculations will produce plausible but incorrect results — and those are harder to catch than obvious errors. Flagging this so it's a conscious decision, not an oversight."

**For automated/end-to-end pipelines:** If the developer is building a fully automated workflow (no human in the loop), this data review step becomes a **programmatic gate**, not a visual check. The tool MUST include: - Automated schema validation (Pydantic) - Range checks (e.g., revenue > 0, dates within expected window) - Completeness checks (expected row counts, no null keys) - Drift detection (flag if today's data looks significantly different from historical patterns) - Logging of all checks with pass/fail — so a human can audit after the fact

State clearly: "This is an automated pipeline with no human reviewing each run. Programmatic guardrails are required to catch what manual review would catch. Silent data failures in production systems have led to significant financial losses. These checks should be part of the pipeline, not optional."

3. Test Against Known Answers

The question: Does output match what we can independently verify?

You (AI) do:

• Run the implementation with simple inputs you can calculate by hand
• Compare a key result against a published benchmark or reference tool
• Check a few raw data values against their original source

Present to developer: "I ran [calculation] with [inputs]. Got [result]. The textbook/reference answer is [X]. Here's the comparison."

Developer judges: Match or mismatch? Acceptable tolerance?

4. Check for Reasonableness

The question: Would you bet your own money on this?

You (AI) do:

• Report the key outputs with context (not just numbers — magnitudes, directions, relationships)
• Flag anything that looks unusual ("This Sharpe ratio of 3.2 would put it in the top 0.1% of funds")
• Compare to the developer's domain expectations

Ask the developer directly:

• "Does this order of magnitude make sense?"
• "Does the direction of change match what you'd expect?"
• "Would an experienced colleague find this defensible?"

This step requires human judgment. The AI presents evidence; the developer — as Pilot-in-Command — decides if it's reasonable.

5. Stress Test the Edges

The question: What breaks it?

You (AI) do:

• Test with zero, negative, and very large values
• Test with missing or incomplete data
• Test unusual combinations (all same values, single data point, extreme dates)
• For financial tools: test with historical extremes (2008 crisis, COVID crash)

Present to developer: "Here's what happens at the edges: [results table]. These [N] cases need attention."

Developer judges: Which edge cases matter for their use case? Which are acceptable limitations?

6. AI-Specific Checks

The question: What did the AI get confidently wrong?

Check these together:

• Are cited facts, formulas, or references actually correct? (AI hallucinates)
• Is the data current, or is it stale from training cutoff?
• Does the logic chain actually hold, or does it just sound convincing?
• Are there libraries or APIs used incorrectly despite looking right?

You (AI) do: Flag any areas where you're uncertain about your own outputs. Be honest about confidence levels.

Developer does: Spot-check facts against authoritative sources. Don't use AI alone to validate AI output.

7. Validation Summary

Write results to [project]/validation.md. All sections go in a single consolidated file, organized by section headers:

# Validation Results

## [Section 1 Name]

| Check | Status | Evidence |
|-------|--------|----------|
| Data Review | pass/fail | [summary stats checked, nulls, date range] |
| Known Answers | pass/fail | [what was compared] |
| Reasonableness | pass/fail | [developer's judgment] |
| Edge Cases | pass/fail | [what was stress-tested] |
| AI-Specific | pass/fail | [what was verified] |

## [Section 2 Name]

| Check | Status | Evidence |
|-------|--------|----------|
| Data Review | pass/fail | [summary stats checked, nulls, date range] |
| Known Answers | pass/fail | [what was compared] |
| Reasonableness | pass/fail | [developer's judgment] |
| Edge Cases | pass/fail | [what was stress-tested] |
| AI-Specific | pass/fail | [what was verified] |

Present results to the developer:

"Validation Results for [SectionName]:

Check	Status	Evidence
Data Review	pass/fail	[summary stats checked, nulls, date range]
Known Answers	pass/fail	[what was compared]
Reasonableness	pass/fail	[developer's judgment]
Edge Cases	pass/fail	[what was stress-tested]
AI-Specific	pass/fail	[what was verified]
Documentation	done/skipped	Screenshot saved to [path]

What would you like to do next?

• Fix issues found in validation
• Capture more visual variants (dark mode, mobile)
• Build another section: [list remaining]
• Generate the export package (if all sections done)"

8. Capture Evidence (Optional)

This step is not required for validation to pass. Screenshots are supplemental documentation for the export package. If no screenshot tool is available, skip this step — your validation results from the Validation Summary are complete.

Screenshot Tools

Use whichever browser automation tool is available:

Tool	How to check	Install
Playwright MCP	mcp__playwright__browser_take_screenshot	claude mcp add playwright npx @playwright/mcp@latest
Chrome DevTools MCP	mcp__chrome-devtools__take_screenshot	claude mcp add chrome-devtools npx @anthropic/chrome-devtools-mcp@latest
Manual	Take a screenshot yourself and save it	No install needed

Any tool that can capture a browser screenshot works. The goal is visual evidence, not a specific tool.

Capture Process

1. Start the dev server using Bash (run in background). Do NOT ask the user to start it.
2. Wait a few seconds for it to be ready
3. Navigate to the app URL
4. Capture a full-page screenshot (1280px viewport, PNG)

Save

Save to [project]/build/[section-id]/screenshot.png

Naming: [screen-design-name]-[variant].png

---

Proactive Flow

As a Cognition Mate:

• Run benchmark tests automatically before asking for judgment calls
• Flag suspicious values proactively ("This return seems unusually high...")
• Be honest about your own uncertainty — don't validate yourself
• Present evidence clearly so the developer can exercise judgment quickly
• If all sections pass validation, suggest generating the export

---

Guiding Principles

• Cross-check, don't single-check — Multiple independent angles catch what one misses
• Numbers before pixels — Validate the data, then document the visuals
• AI presents, human judges — The Pilot-in-Command makes the reasonableness call
• Actively try to break it — "What would prove this wrong?" is more valuable than "Does this look right?"