prompt-lab

name	prompt-lab
description	Iterate on LLM prompts with structured evaluation and self-correction. Test prompts against ground truth, compare models, track version history. Self-correction loop sends invalid outputs back to LLM for fixing. Supports both taxonomy classification and QRA (Question-Reasoning-Answer) generation.
allowed-tools	["Bash","Read","Write"]
triggers	["prompt-lab","eval prompt","test prompt","compare models","compare models","extract prompts","test sparta qra","prompt evaluation","prompt-validator","qra evaluation","citation grounding"]
metadata	{"short-description":"Iterate and evaluate LLM prompts with self-correction"}
provides	["skill-creation"]
composes	["scillm","memory","task-monitor"]
taxonomy	["creation","iteration","self-improvement"]

Prompt Lab

Systematic prompt engineering with ground truth evaluation and self-correction loop.

Key Feature: Self-Correction Loop

Unlike simple validation that silently filters invalid outputs, this skill:

1. Presents vocabulary in prompt - LLM knows valid options upfront
2. Validates response - Pydantic catches invalid tags
3. Sends correction back to LLM - "You used invalid tags X. Valid options are Y. Please fix."
4. Tracks correction rounds - Metrics show how often LLM needed help

This gives the model a chance to self-correct rather than silently failing.

Quick Start

cd /home/graham/workspace/experiments/pi-mono/.pi/skills/prompt-lab

# Find the smallest model that works (NEW!)
./run.sh find-minimum --ground-truth queryspec.json --threshold 0.80

# Run evaluation with self-correction enabled (default)
./run.sh eval --prompt taxonomy_v1 --model deepseek

# Compare multiple models on same prompt
./run.sh compare --prompt taxonomy_v1 --models "deepseek,gpt-4o"



# View evaluation history
./run.sh history --prompt taxonomy_v1

Architecture

┌─────────────────────────────────────────────────────────────┐
│  Stage 1: LLM Call with Vocabulary in Prompt                │
│     - Vocabulary clearly defined (Tier 0, Tier 1 tags)      │
│     - JSON response format enforced                         │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│  Stage 2: Pydantic Validation                               │
│     - Parse JSON response                                   │
│     - Detect invalid/hallucinated tags                      │
│     - Track rejected tags for metrics                       │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│  Stage 3: Self-Correction Loop (if invalid tags detected)   │
│     - Send assistant message: "Invalid tags: X"             │
│     - Ask LLM to fix: "Valid options are: Y"                │
│     - Retry up to N times (default: 2)                      │
│     - Track correction rounds in metrics                    │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│  Stage 4: Evaluation Metrics                                │
│     - Precision/Recall/F1 vs ground truth                   │
│     - Correction rounds needed                              │
│     - Correction success rate                               │
└─────────────────────────────────────────────────────────────┘