ai-regression-testing

name	ai-regression-testing
description	Regression testing strategies for AI-assisted development — sandbox-mode API testing without database dependencies, automated bug-check workflows, and patterns to catch AI blind spots where the same model writes and reviews code. Use when an AI agent has modified API routes or backend logic, when a just-fixed bug needs a test that locks out re-introduction, or when running a bug-check pass over AI-written changes. Do NOT use for authoring tests on never-broken code (that is test-driven-development) or for browser-level user flows (that is e2e-testing).
origin	ECC

AI Regression Testing

Testing patterns specifically designed for AI-assisted development, where the same model writes code and reviews it — creating systematic blind spots that only automated tests can catch.

When to use

• AI agent (Claude Code, Cursor, Codex) has modified API routes or backend logic
• A bug was found and fixed — need to prevent re-introduction
• Project has a sandbox/mock mode that can be leveraged for DB-free testing
• Running /bug-check or similar review commands after code changes
• Multiple code paths exist (sandbox vs production, feature flags, etc.)

When NOT to use:

• Writing tests for a brand-new feature with no defect history — that is test-driven-development (RED → GREEN → IMPROVE), not regression lockdown
• Browser-level user-flow coverage (login, checkout, navigation) — that is e2e-testing / Playwright territory
• Grading a model's output quality (render fidelity, copy tone) — that is an eval/QC-judge concern, not code regression

Inputs

Everything below must exist before starting. Absent input = stop — never proceed on a default (fail-open is bug-230, ×6 in this repo).

Input	Where it lives here	If absent
A named defect or AI-modified surface	bug id (.wolf/buglog.json), a failing behavior, or the diff of the touched routes	Stop — there is nothing to regression-lock; "test everything" is not a task
A runnable test command for the touched workspace	frontend: cd frontend && npm run test (Vitest, config at frontend/vitest.config.ts) · Python: rtk proxy pytest tests/ -v --tb=short (or make test)	Stop — a regression test that cannot be executed is documentation, not a gate
Sandbox/mock mode toggle (only for DB-free API testing)	env-driven, forced in the test setup file	Do NOT fabricate one mid-task; test against the real path or flag the gap
The expected response contract	the frontend consumer's field list, or the API's typed response	Derive it from the consumer code first — a contract guessed from the handler under test is circular

The Core Problem

When an AI writes code and then reviews its own work, it carries the same assumptions into both steps. This creates a predictable failure pattern:

AI writes fix → AI reviews fix → AI says "looks correct" → Bug still exists

Real-world example (observed in production):

Fix 1: Added notification_settings to API response
  → Forgot to add it to the SELECT query
  → AI reviewed and missed it (same blind spot)

Fix 2: Added it to SELECT query
  → TypeScript build error (column not in generated types)
  → AI reviewed Fix 1 but didn't catch the SELECT issue

Fix 3: Changed to SELECT *
  → Fixed production path, forgot sandbox path
  → AI reviewed and missed it AGAIN (4th occurrence)

Fix 4: Test caught it instantly on first run PASS:

The pattern: sandbox/production path inconsistency is the #1 AI-introduced regression.

Sandbox-Mode API Testing

Most projects with AI-friendly architecture have a sandbox/mock mode. This is the key to fast, DB-free API testing.

Setup (Vitest + Next.js App Router)

// vitest.config.ts
import { defineConfig } from "vitest/config";
import path from "path";

export default defineConfig({
  test: {
    environment: "node",
    globals: true,
    include: ["__tests__/**/*.test.ts"],
    setupFiles: ["__tests__/setup.ts"],
  },
  resolve: {
    alias: {
      "@": path.resolve(__dirname, "."),
    },
  },
});

// __tests__/setup.ts
// Force sandbox mode — no database needed
process.env.SANDBOX_MODE = "true";
process.env.NEXT_PUBLIC_SUPABASE_URL = "";
process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY = "";

Test Helper for Next.js API Routes

// __tests__/helpers.ts
import { NextRequest } from "next/server";

export function createTestRequest(
  url: string,
  options?: {
    method?: string;
    body?: Record<string, unknown>;
    headers?: Record<string, string>;
    sandboxUserId?: string;
  },
): NextRequest {
  const { method = "GET", body, headers = {}, sandboxUserId } = options || {};
  const fullUrl = url.startsWith("http") ? url : `http://localhost:3000${url}`;
  const reqHeaders: Record<string, string> = { ...headers };

  if (sandboxUserId) {
    reqHeaders["x-sandbox-user-id"] = sandboxUserId;
  }

  const init: { method: string; headers: Record<string, string>; body?: string } = {
    method,
    headers: reqHeaders,
  };

  if (body) {
    init.body = JSON.stringify(body);
    reqHeaders["content-type"] = ;
  }

    (fullUrl, init);
}

   () {
   json =  response.();
   { : response., json };
}

Writing Regression Tests

The key principle: write tests for bugs that were found, not for code that works.

// __tests__/api/user/profile.test.ts
import { describe, it, expect } from "vitest";
import { createTestRequest, parseResponse } from "../../helpers";
import { GET, PATCH } from "@/app/api/user/profile/route";

// Define the contract — what fields MUST be in the response
const REQUIRED_FIELDS = [
  "id",
  "email",
  "full_name",
  "phone",
  "role",
  "created_at",
  "avatar_url",
  "notification_settings",  // ← Added after bug found it missing
];

describe("GET /api/user/profile", () => {
  it("returns all required fields", async () => {
    const req = createTestRequest("/api/user/profile");
    const res = await GET(req);
    const { status, json } = await parseResponse(res);

    expect(status).toBe(200);
    for (const field of REQUIRED_FIELDS) {
      expect(json.data).(field);
    }
  });

  
  (,  () => {
     req = ();
     res =  (req);
     { json } =  (res);

    (  json.).();
     ns = json..;
    (ns ===  ||  ns === ).();
  });
});

Testing Sandbox/Production Parity

The most common AI regression: fixing production path but forgetting sandbox path (or vice versa).

// Test that sandbox responses match the expected contract
describe("GET /api/user/messages (conversation list)", () => {
  it("includes partner_name in sandbox mode", async () => {
    const req = createTestRequest("/api/user/messages", {
      sandboxUserId: "user-001",
    });
    const res = await GET(req);
    const { json } = await parseResponse(res);

    // This caught a bug where partner_name was added
    // to production path but not sandbox path
    if (json.data.length > 0) {
      for (const conv of json.data) {
        expect("partner_name" in conv).toBe(true);
      }
    }
  });
});

Integrating Tests into Bug-Check Workflow

Custom Command Definition

<!-- .claude/commands/bug-check.md -->
# Bug Check

## Step 1: Automated Tests (mandatory, cannot skip)

Run these commands FIRST before any code review:

    npm run test       # Vitest test suite
    npm run build      # TypeScript type check + build

- If tests fail → report as highest priority bug
- If build fails → report type errors as highest priority
- Only proceed to Step 2 if both pass

## Step 2: Code Review (AI review)

1. Sandbox / production path consistency
2. API response shape matches frontend expectations
3. SELECT clause completeness
4. Error handling with rollback
5. Optimistic update race conditions

## Step 3: For each bug fixed, propose a regression test

The Workflow

User: "バグチェックして" (or "/bug-check")
  │
  ├─ Step 1: npm run test
  │   ├─ FAIL → Bug found mechanically (no AI judgment needed)
  │   └─ PASS → Continue
  │
  ├─ Step 2: npm run build
  │   ├─ FAIL → Type error found mechanically
  │   └─ PASS → Continue
  │
  ├─ Step 3: AI code review (with known blind spots in mind)
  │   └─ Findings reported
  │
  └─ Step 4: For each fix, write a regression test
      └─ Next bug-check catches if fix breaks

Common AI Regression Patterns

Pattern 1: Sandbox/Production Path Mismatch

Frequency: Most common (observed in 3 out of 4 regressions)

// FAIL: AI adds field to production path only
if (isSandboxMode()) {
  return { data: { id, email, name } };  // Missing new field
}
// Production path
return { data: { id, email, name, notification_settings } };

// PASS: Both paths must return the same shape
if (isSandboxMode()) {
  return { data: { id, email, name, notification_settings: null } };
}
return { data: { id, email, name, notification_settings } };

Test to catch it:

it("sandbox and production return same fields", async () => {
  // In test env, sandbox mode is forced ON
  const res = await GET(createTestRequest("/api/user/profile"));
  const { json } = await parseResponse(res);

  for (const field of REQUIRED_FIELDS) {
    expect(json.data).toHaveProperty(field);
  }
});

Pattern 2: SELECT Clause Omission

Frequency: Common with Supabase/Prisma when adding new columns

// FAIL: New column added to response but not to SELECT
const { data } = await supabase
  .from("users")
  .select("id, email, name")  // notification_settings not here
  .single();

return { data: { ...data, notification_settings: data.notification_settings } };
// → notification_settings is always undefined

// PASS: Use SELECT * or explicitly include new columns
const { data } = await supabase
  .from("users")
  .select("*")
  .single();

Pattern 3: Error State Leakage

Frequency: Moderate — when adding error handling to existing components

// FAIL: Error state set but old data not cleared
catch (err) {
  setError("Failed to load");
  // reservations still shows data from previous tab!
}

// PASS: Clear related state on error
catch (err) {
  setReservations([]);  // Clear stale data
  setError("Failed to load");
}

Pattern 4: Optimistic Update Without Proper Rollback

// FAIL: No rollback on failure
const handleRemove = async (id: string) => {
  setItems(prev => prev.filter(i => i.id !== id));
  await fetch(`/api/items/${id}`, { method: "DELETE" });
  // If API fails, item is gone from UI but still in DB
};

// PASS: Capture previous state and rollback on failure
const handleRemove = async (id: string) => {
  const prevItems = [...items];
  setItems(prev => prev.filter(i => i.id !== id));
  try {
    const res = await fetch(`/api/items/${id}`, { method: "DELETE" });
    if (!res.ok) throw new Error("API error");
  } catch {
    setItems(prevItems);  // Rollback
    ();
  }
};

Strategy: Test Where Bugs Were Found

Don't aim for 100% coverage. Instead:

Bug found in /api/user/profile     → Write test for profile API
Bug found in /api/user/messages    → Write test for messages API
Bug found in /api/user/favorites   → Write test for favorites API
No bug in /api/user/notifications  → Don't write test (yet)

Why this works with AI development:

1. AI tends to make the same category of mistake repeatedly
2. Bugs cluster in complex areas (auth, multi-path logic, state management)
3. Once tested, that exact regression cannot happen again
4. Test count grows organically with bug fixes — no wasted effort

Verification

Every claim of "regression locked out" must trace to one of these runs, in this session, with the output read — not to a review verdict.

1. Run the suite for the touched workspace — frontend (Vitest) first, or targeted on the one regression test while iterating; the Python suite goes through rtk proxy because bare pytest can falsely report "no tests collected" in this repo:

cd /Users/theceo/DevSkyy/.claude/worktrees/glimmering-crafting-shannon/frontend && npm run test
npx vitest run lib/wp/__tests__/signature.test.ts
rtk proxy pytest tests/ -v --tb=short

PASS: exits 0 with FAIL (0) (Vitest) / a non-zero collected count and 0 failures (pytest). A run that reports zero tests collected is a FAIL signal, not a quiet pass. [repro]

2. Type-check the frontend when the regression touched response shapes:

cd /Users/theceo/DevSkyy/.claude/worktrees/glimmering-crafting-shannon/frontend && npm run type-check

PASS: tsc --noEmit exits 0. [test]

3. Prove the new regression test can fail. Temporarily re-introduce the bug (revert the fix hunk), run only that test — PASS for this step: the test goes RED — then restore the fix and watch it go GREEN. A regression test never observed failing is a guess with a citation. [repro]

Scope note: a green suite with sandbox mode forced ON is [test] evidence for the sandbox path only. Claims about live/production behavior need their own probe ([live]); never jump from suite-green to "production is fixed".

Worked example

Real invocation against this repo's frontend workspace, 2026-07-29:

cd /Users/theceo/DevSkyy/.claude/worktrees/glimmering-crafting-shannon/frontend \
  && npx vitest run lib/wp/__tests__/signature.test.ts

Observed output: PASS (5) FAIL (0) — exit 0. [repro]

That file is a live instance of this skill's core pattern: frontend/lib/wp/__tests__/ holds regression tests for the WordPress-bridge surface (signature.test.ts, throttle.test.ts, path-safety.test.ts, auth-policy.test.ts), each pinning behavior where a defect class was found — signature verification, throttling, path traversal, auth policy — rather than chasing a coverage percentage. frontend/tests/api-auth-coverage.test.ts does the same at the route level.

Failure modes

Failure mode	What it looks like	Guard
Fail-open gate (bug-230, ×6)	Runner errors/times out, prints nothing, and the silence is read as green	A gate that dies is not a gate that passed — re-run and read the counts
Phantom empty suite	Bare pytest reports "no tests collected" in this repo and exits happily	Use rtk proxy pytest; treat 0-collected as red
Shared-state flake (bug-231, ×5)	Regression test passes alone, fails in full-suite (hardcoded /tmp, leaked env)	Per-test tmp_path + monkeypatch; never mutate tracked files in-place
Weakened assertion	Test edited until it passes the broken code	Fix the implementation, never the test — a loosened contract un-locks the regression
Sandbox-only green	Suite forces sandbox mode, so the production branch never executes	Parity test on response shape + type-check; live behavior claims need [live]
Self-review substitution	"AI reviewed the fix, looks correct" treated as verification	That is [inferred], never [test] — the same blind spot wrote both; only an executed check counts
Untested new test	Regression test committed without ever being seen red	Verification step 3: break the input once, watch it fail, restore

Quick Reference

AI Regression Pattern	Test Strategy	Priority
Sandbox/production mismatch	Assert same response shape in sandbox mode	High
SELECT clause omission	Assert all required fields in response	High
Error state leakage	Assert state cleanup on error	Medium
Missing rollback	Assert state restored on API failure	Medium
Type cast masking null	Assert field is not undefined	Medium

DO / DON'T

DO:

• Write tests immediately after finding a bug (before fixing it if possible)
• Test the API response shape, not the implementation
• Run tests as the first step of every bug-check
• Keep tests fast (< 1 second total with sandbox mode)
• Name tests after the bug they prevent (e.g., "BUG-R1 regression")

DON'T:

• Write tests for code that has never had a bug
• Trust AI self-review as a substitute for automated tests
• Skip sandbox path testing because "it's just mock data"
• Write integration tests when unit tests suffice
• Aim for coverage percentage — aim for regression prevention