aiase-rpi

name	aiase-rpi
description	Research-Plan-Implement (RPI) workflow — the three-phase agentic SDLC for reliable AI-driven development, context window management, verifiability framework, and the autoresearch/gstack reference implementations. Load when the user asks about how to structure an AI coding session, how to plan before implementing, or the AIASE RPI framework.

Research-Plan-Implement (RPI) Workflow

From AIASE 2026 (NCKU), Weeks 5 & 7. Synthesized from Dex Horthy's "No Vibes Allowed" framework and Karpathy's autoresearch.

The core problem RPI solves: agents that jump straight to implementation build on misunderstanding and produce expensive rework. Agents that skip evaluation cannot tell if their output is correct.

The Three Phases

🔍 Phase 1: Research

• Use sub-agents for vertical slice exploration (one file/module depth), NOT full codebase scanning
• Produce a research document grounded in actual source code — not stale documentation
• If initial research quality is poor, discard and restart — don't patch bad research
• Human's highest-value intervention: Validate research accuracy before proceeding

Output: A research document with specific file paths, line numbers, and quotes from actual code.

📋 Phase 2: Plan

• Plan must be specific enough that "even a mediocre model won't fail"
• Include: exact file paths, line numbers, code snippets, the exact changes
• Transform vague intent ("fix auth module") into concrete actions ("edit src/auth/middleware.ts:47 to add token expiry check")
• Set quality gates before implementation begins
• Technical lead reviews the plan — not line-by-line code review, but plan review

Output: A structured plan document with checkboxes, file paths, and acceptance criteria.

⚙️ Phase 3: Implement

• Execute the plan with minimal context
• Pause and update plan if unexpected issues arise
• Don't let agents research mid-implementation — that's a context pollution path

Context Window Management

• Keep context under ~40% capacity; beyond that quality degrades ("dumb zone")
• Use intentional compaction: periodically compress current state into Markdown, review, start fresh context
• Isolate noisy work (searching, exploration) in sub-agents; return only conclusions to main context

Loss in the Middle

LLMs reliably retain content at the beginning and end of their context window, but degrade significantly on material in the middle — even if it technically fits within the window.

Symptoms:

• Agent "forgets" rules defined halfway through a long AGENTS.md
• Spec written at line 400 of 800 is silently ignored while line 1 and line 800 are followed
• Instructions that "should be obvious" are violated when they land in the middle of a large file

Mitigation: AGENTS.md as a Table of Contents, not a monolith

AGENTS.md             ← short directory file only (~50 lines)
  → docs/spec.md      ← specification detail
  → docs/security.md  ← security rules
  → docs/api.md       ← API contracts

AGENTS.md should say: "For security rules, read docs/security.md." The agent loads only what it needs for each task — progressive disclosure instead of dumping everything into a single file. This sidesteps Loss in the Middle by ensuring critical rules are always at the start of a fresh, short read.

Apply the same principle to research documents produced in Phase 1: keep them focused (one module, one file) rather than sprawling. A 20-line research note read completely beats a 200-line document partially retained.

Verifiability: The Hidden Boundary

A task is AI-optimizable if and only if it passes all three:

1. Observable — can see the result
2. Measurable — has a numeric or clear standard
3. Verifiable — can confirm "correct" vs. "looks correct"

Task	Verifiable?	Action
Data cleaning	✅ Yes	Automate fully with feedback loop
Performance tuning	✅ Yes	Let AI run benchmarks autonomously
Brand voice review	❌ No	Requires human taste judgment
Strategy direction	❌ No	Human-only decision

Professional leverage: Offload verifiable tasks; concentrate your effort on non-verifiable judgment.

Reference Implementations

autoresearch (Karpathy) — Vertical Iteration

Component	Role	Who Controls
train.py	Probabilistic core (LLM-modifiable)	Agent
prepare.py	Deterministic harness (fixed)	Human
program.md	Specification layer (iterative)	Human
val_bpb	Verifiability criterion	System design
5-min budget	Termination condition	Structural

Agent loops 100× overnight on same task, optimizing against bits-per-byte metric (not per-token loss — that can be gamed by vocab size changes).

Template for researchers:

1. Find repeating task with improvement potential
2. Design quantifiable success metric (must resist reward hacking)
3. Lock evaluation framework; make only core modifiable
4. Design agent instructions (spec as iterative document)
5. Run overnight loops

gstack (Garry Tan/YC) — Horizontal Pipeline

23 Markdown-defined slash commands, each a specialized role:

Phase	Example Command	What it Does
Think	/office-hours	6-question product deconstruction
Plan	/plan-eng-review	Data flow + threat model
Review	/cso	OWASP Top 10 + STRIDE security review
Ship	/ship	Test + PR automation

Contrast:

• autoresearch: Vertical (same task × 100 iterations)
• gstack: Horizontal (sequential phases × 1 complete project)
• Both encode decisions as executable Markdown specs

Eval-Driven Development Workflow

1. Collect real examples (Golden Dataset)
   ↓
2. Define scoring (Exact Match / Semantic / LLM-as-Judge)
   ↓
3. Automate eval runs
   ↓
4. Prompt modification → auto eval → measure impact
   ↓
5. Track eval scores over time (detect regressions)

"If you can't measure it, you can't improve it. Eval drives iteration, not intuition."

Metric Design: Avoid Reward Hacking

Anti-pattern: Per-token loss — agent can hack vocab size → spurious improvement.

Good pattern: Bits-per-byte — normalizes across tokenizer changes; forces genuine quality improvement.

General rule: Metric must be structured so shortcuts are impossible.

Anti-Patterns

Mistake	Impact	Fix
Skipping research phase	Poor plans built on misunderstanding	Always research first; discard bad research, don't patch
Context explosion	Falls into "dumb zone" quality cliff	Use sub-agents to isolate exploration; return only conclusions
Vague plans	High re-work rate in implementation	Plan must specify line numbers, file paths, exact changes
Autonomy without constraints	Slop multiplication	Enforce strict layering, dependency rules, Quality Gate agents

---

See also: [[aiase-sdd]] for spec writing before planning, [[aiase-harness]] for the harness environment RPI runs inside, [[aiase-multi-agent]] for using sub-agents in the Research phase.