| type | skill |
| name | agent-evals |
| description | Skill evaluation framework, self-evolving agent loop, multi-agent SDLC orchestration, and tool-use optimization for automated quality gates. |
| version | 1 |
Agent Evaluation & Self-Evolution System
Sources Researched
| Source | Pattern | Key Insight |
|---|
| LangChain Eval (Mar 2026) | Task → skill → compare | Define task, run without skill, run with skill, compare delta |
| Minko Gechev Skill Eval | evals.json assertions | Judge model scores output; pass@k for reliability |
| Future AGI (6 Dimensions) | Tool/Argument/Result/Recovery/Plan/Completion | Multi-dimensional scoring prevents gaming |
| Anthropic skill-creator | Trigger vs instruction skills | Skill design patterns for distinct use cases |
| GenericAgent | 5-layer memory + crystallization | From explore → execute → crystallize → reuse |
| Self-Evolving Agents Survey | Trace → Score → Optimize → Gate → Rollout | Continuous improvement loop with guardrails |
| Gem Team (16 agents) | PM → Architect → Dev → Reviewer → Security → ... | Spec-driven multi-agent SDLC |
| VALORA | Phased governance with verifiers | Each phase has entry/exit criteria |
| Zylos Tool Optimization | Tool RAG + trajectory pruning | Reduce cost with dynamic tool selection |
| Google ADK | Sequential/Parallel/Loop agents | Built-in orchestration patterns |
| Google Scaling Paper | 17.2x error amplification (independent) vs 4.4x (coordinated) | Always coordinate — never fully independent agents |
Architecture Overview
evals.json (skill definitions + test cases)
│
▼
RecordEvidence() ← Agent traces tool calls + outputs
│
▼
ScoreEval() ← Judge model scores each dimension
│
▼
CrystallizeSkill() ← Extract reusable pattern from successful trace
│
▼
Commit to .agents/skills/ ← CI gate blocks if score < threshold
1. evals.json Schema
File location: .agents/evals/evals.json
{
"schema_version": 2,
"last_updated": "2026-07-17T00:00:00Z",
"judge_model": "gemini-3.1-pro",
"skills": [
{
"name": "admin-ui-css",
"path": ".agents/skills/admin-ui",
"version": "1",
"trigger": "when editing admin CSS, views, or sidebar navigation",
"evals": ["eval-1", "eval-2"]
}
],
"evals": [
{
"id": "eval-1",
"name": "admin-css-class-coverage",
"description": "All CSS classes referenced in admin views exist in admin.css",
"type": "assertion",
"task": "Edit views/layouts/admin.php to add a new sidebar section",
"dimensions": {
"tool_selection": { "weight": 0.1 },
"argument_extraction": { "weight": 0.15 },
"result_utilization": { "weight": 0.2 },
"error_recovery": { "weight": 0.15 },
"plan_coherence": { "weight": 0.2 },
"task_completion": { "weight": 0.2 }
},
"assertions": [
"Every class in views/*.php exists in assets/css/admin.css",
"No var(--color-gold) in any admin view",
"SVG icons replace Unicode characters in sidebar",
"No inline style attributes in admin.php sidebar"
],
"threshold": 0.85,
"pass_count": 3,
"trial_count": 3
}
],
"scores": [
{
"eval_id": "eval-1",
"run_id": "run-001",
"timestamp": "2026-07-17T00:00:00Z",
"dimension_scores": {
"tool_selection": 0.9,
"argument_extraction": 0.85,
"result_utilization": 0.95,
"error_recovery": 0.7,
"plan_coherence": 1.0,
"task_completion": 1.0
},
"weighted_total": 0.91,
"assertions_passed": 4,
"assertions_total": 4,
"passed": true,
"model_used": "gemini-2.5-flash",
"duration_ms": 45000
}
],
"comparisons": [
{
"skill_path": ".agents/skills/admin-ui",
"eval_id": "eval-1",
"without_skill": { "score": 0.65, "pass_count": 1, "trial_count": 3 },
"with_skill": { "score": 0.91, "pass_count": 3, "trial_count": 3 },
"delta": { "score": 0.26, "pass_rate": 0.67 },
"timestamp": "2026-07-17T00:00:00Z"
}
]
}
Scoring Dimensions (Future AGI-based)
| Dimension | Weight | Measures |
|---|
| Tool Selection | 0.10 | Chooses the right tool (read vs grep, write vs edit) |
| Argument Extraction | 0.15 | Extracts correct file paths, CSS classes, parameters |
| Result Utilization | 0.20 | Uses tool output effectively (reads error, adjusts) |
| Error Recovery | 0.15 | Handles failures gracefully (retries, falls back) |
| Plan Coherence | 0.20 | Logical sequence of operations, no wasted steps |
| Task Completion | 0.20 | Fulfills all acceptance criteria, no regressions |
Judge Model Prompt Template
You are evaluating an agent's performance on this task:
{task_description}
The agent produced:
{trace_output}
Scoring rubric (0.0–1.0 per dimension):
- tool_selection: Did the agent use the right tool for each step?
- argument_extraction: Did it pass correct parameters?
- result_utilization: Did it read and act on tool output?
- error_recovery: How well did it handle failures?
- plan_coherence: Was the sequence of operations logical?
- task_completion: Were all acceptance criteria met?
Return a JSON object with dimension scores and weighted_total.
2. Self-Evolving Agent Loop
5-Layer Memory Architecture
┌─────────────────────────────────────┐
│ L1: Meta Rules (AGENTS.md) │ ← Immutable per session
├─────────────────────────────────────┤
│ L2: Insight Index (failures → │ ← Crystallized from evals
│ rules) │
├─────────────────────────────────────┤
│ L3: Global Facts (schema, map, │ ← bapXaura map + schema list
│ routes) │
├─────────────────────────────────────┤
│ L4: Task Skills (evals-optimized │ ← .agents/skills/ with evals
│ patterns) │
├─────────────────────────────────────┤
│ L5: Session Archive (current │ ← .agents/temp/ session memory
│ cycle trace) │
└─────────────────────────────────────┘
Crystallization Loop
1. Agent completes task → full trace captured
2. RecordEvidence() stores tool calls + results
3. ScoreEval() runs judge model against assertions
4. If score ≥ threshold (0.85):
a. CrystallizeSkill() extracts reusable sequence from trace
b. Store as .agents/skills/<domain>/references/<pattern>.md
c. Update evals.json with successful run
5. If score < threshold:
a. Analyze which dimensions failed
b. Update skill instruction in SKILL.md
c. Re-run eval with updated skill
6. If CI gate blocks (< threshold):
a. Block was correct — agent needs skill improvement
b. Do not override gate
Trace Format (RecordEvidence)
{
"trace_id": "trace-001",
"objective_id": "OBJ-12-3",
"steps": [
{
"step": 1,
"tool": "bash",
"command": "bapXaura map",
"result_summary": "Project map with 47 routes, 31 controllers, 18 services",
"duration_ms": 1200,
"error": null
},
{
"step": 2,
"tool": "grep",
"pattern": "var\\(--color-gold\\)",
"path": "views/admin",
"result_summary": "Found 6 occurrences in 3 files",
"duration_ms": 800,
"error": null
}
],
"result": {
"files_changed": ["views/admin/dashboard.php"],
"assertions_passed": 4,
"assertions_total": 4,
"score": 0.91
}
}
Token Savings with Crystallization
| Attempt | Tokens (input) | Improvement |
|---|
| 1st (no skill) | ~85,000 | Baseline |
| 2nd (with skill) | ~35,000 | 2.4× savings |
| 3rd (crystallized) | ~14,000 | 6× savings |
After 3+ successful crystallizations, the skill can be promoted to an automated workflow in .agents/workflows/.
3. Multi-Agent SDLC Orchestration
16 Specialist Agents (Gem Team Pattern)
| Agent | Role | When Delegated | Read-only? |
|---|
| PM (CTO) | Plan objectives, route | Every cycle | No |
| Architect | Design schema/changes | Schema changes | No |
| Developer (Worker) | Implement | Each objective | No |
| Reviewer | Verify evidence | After implementation | Yes |
| Tester | Run tests, validate | Before merge | Yes |
| Security | Scan for leaks/creds | PR review | Yes |
| Documenter | Update durable docs | After merge | No |
| Deployment | Push to hosting | On merge to main | No |
| UX | Design consistency check | UI changes | Yes |
| Data | Migration/schema safety | Schema changes | Yes |
| SEO | Meta tags, robots, sitemap | Content changes | Yes |
| Accessibility | a11y audit | UI changes | Yes |
| Performance | Lighthouse, Core Web Vitals | Every deploy | Yes |
| Analytics | Event tracking audit | Feature changes | Yes |
| Localization | i18n readiness | Content changes | Yes |
| Legal | Compliance check | Policy/terms changes | Yes |
Current repo: 6 roles active (CTO, Worker, Reviewer, Fixer, Documenter, Browser Tester). Expand as needed.
8-Phase Governance (VALORA)
Phase 1: Init
├── Issue created with evidence
├── CTO reads map + schema
└── Gate: Issue has acceptance criteria
Phase 2: Plan
├── CTO breaks into objectives
├── Each objective has single deliverable
└── Gate: Objectives are independent (no file overlap)
Phase 3: Implement
├── Worker executes one objective
├── CrystallizeSkill() on success
└── Gate: Tests pass + lint clean
Phase 4: Review
├── Reviewer verifies evidence vs acceptance criteria
├── Fresh context, read-only
└── Gate: All assertions pass
Phase 5: Fix (if review fails)
├── Findings documented with file+line references
├── Fixer modifies code, pushes
└── Gate: Re-review passes
Phase 6: Document
├── Update systematic-map.mmd if routes changed
├── Update schema if collections changed
└── Gate: Map valid + docs current
Phase 7: Verify
├── Final check: every objective met
├── Telemetry updated
└── Gate: Score ≥ 90
Phase 8: Deploy
├── Merge to main
├── Git push triggers auto-deploy
└── Verify deployed SHA matches
Parallel Fan-Out Safety Rules
Only parallelize when ALL conditions met:
- No file write overlap — different directories or extensions
- No git index contention — each writes to own branch segment
- No test interference — test suites are independent
- No schema contention — different collections
- Coordinator merges — never let sub-agents merge themselves
Parallel dispatch constraints for this repo:
app/ (backend) changes → serial only (service/controller coupling)views/ (templates) + assets/css/ (styles) → parallel allowed (different files)content/blog/ + storage/schema/ → serial (schema affects content display)tests/ + app/ → parallel allowed (test follows implementation)
4. Tool Use Optimization
Tool RAG (Dynamic Selection)
skill: admin-ui
tool_rules:
- pattern: "CSS class"
preferred_tool: grep
avoid: [bash, find]
- pattern: "sidebar navigation"
preferred_tool: read
context_needed: [views/layouts/admin.php, assets/css/admin.css]
- pattern: "new route"
preferred_tool: grep + read
files: [app/Routes.php, app/Controllers/]
- pattern: "schema change"
preferred_tool: read
files: [storage/schema/collections.php]
Trajectory Pruning
For tool calls returning >200 lines:
- Summarize to 3-5 key findings
- Store full result reference (file path + line range)
- Pass only summary to next step
function compress_tool_result($result, $max_lines = 50) {
if (count($result) <= $max_lines) return $result;
$summary = extract_head_tail($result, 10, 10);
$key_stats = extract_stats($result);
return array_merge($summary, ['summary' => $key_stats]);
}
Parallel Execution
When tool calls are independent:
Before (serial): read A → read B → read C → grep D → grep E = 5 sequential
After (parallel): [read A, read B, read C, grep D, grep E] = 1 batch
Savings: 5× latency reduction
5. Integration with Existing Systems
Telemetry Enrichment
New fields for .agents/ops/telemetry.json cycles:
{
"evals_run": 3,
"evals_passed": 3,
"evals_score": 0.91,
"crystallizations": 1,
"skill_refs_used": [".agents/skills/admin-ui"],
"tool_rag_hits": 12,
"tool_rag_misses": 2,
"trajectory_prunes": 4,
"parallel_batches": 2,
"tokens_saved_by_crystallization": 51000
}
Handoff JSON Extension
Add to existing handoff blocks:
{
"eval_id": "eval-1",
"required_score": 0.85,
"skill_path": ".agents/skills/admin-ui",
"crystallize_on_success": true
}
CI Gate Integration
In .github/workflows/ci.yml, add step:
- name: Agent Eval Gate
run: bapXaura eval --threshold=0.85
env:
JUDGE_MODEL: ${{ secrets.BAPX_AI_MODEL || 'gemini-3.1-pro' }}
If bapXaura eval fails (score < threshold), CI fails. No merge without passing evals.
6. Running Evals
CLI Commands (add to bapXaura)
bapXaura eval run <skill-name> # Run single eval
bapXaura eval all # Run all registered evals
bapXaura eval compare <skill-name> # Run with/without skill comparison
bapXaura eval crystallize <trace-id> # Extract pattern from successful trace
bapXaura eval status # Show latest scores and trends
Eval Pipeline
1. bapXaura eval run admin-ui-css
2. Agent completes task (with trace recording)
3. Judge model scores each dimension
4. Score written to evals.json
5. If score ≥ 0.85: crystallize path
6. If score < 0.85: log failure dimensions for skill improvement
7. Critical Mistakes
- ❌ Running evals without
bapXaura map + bapXaura schema list first
- ❌ Using the same model for task execution and judging (calibration bias)
- ❌ Judging on outcomes only, not process — rewards luck over reliability
- ❌ Crystallizing before 3 successful runs (premature optimization)
- ❌ Overriding CI eval gate without evidence
- ❌ Passing full traces to judge model (cost blowup)
- ❌ Writing tool rules that are too specific (no reuse across skills)
- ❌ Parallel dispatch without safety check
- ❌ Hardcoding eval thresholds — read from
evals.json always
References
