benchmark-improver

name	benchmark-improver
description	Wu Xing-based auto-improvement system for benchmarks. Detects anti-patterns via 相克 and generates improvements via 相生 pathways.
version	1
metadata	{"evolution-stats":{"total-experiments":870}}
level	molecule
atoms	["eight-keys-grader","elisp-expert","evolution-patterns"]

Benchmark Auto-Improver

Apply Wu Xing (Five Elements) theory to detect benchmark anti-patterns and generate targeted improvements.

Current Performance

• Keep rate when this skill is used: {{skill-keep-rate}}
• Experiments using this skill: {{skill-experiments}}
• Overall system keep rate: {{overall-keep-rate}}

Core Framework

相生 = Evolution Pathway (Improvements Grow)

Water → Wood → Fire → Earth → Metal → Water

Guides the sequence of improvements. When an element is deficient, strengthen it by enhancing its generator.

相克 = Anti-pattern Detection (Problems Found)

Wood → Earth → Water → Fire → Metal → Wood

Detects which element is in excess and applies the controlling element as remedy.

Improvement Rules

Wood (Operations) - Problem: Too Many Operations

Controlled by: Metal (coordination)
Generated by: Water (identity)

When benchmark shows excessive steps, redundant operations, or inefficiency:

• Threshold suggestions:
  • Reduce max_steps in success_criteria
  • Add tool_sequence requirements
  • Implement caching for repeated operations
• Test suggestions:
  • Split large tasks into subtasks
  • Add early termination conditions
  • Implement step budget tracking
• Prompt suggestions:
  • Add "be efficient" instruction
  • Add "prefer fewer tool calls" guideline
  • Add "check cache before operation" reminder

Fire (Intelligence) - Problem: Scattered, Reactive

Controlled by: Water (identity/principles)
Generated by: Wood (operations)

When benchmark shows lack of planning, reactive fixes, or scattered approach:

• Threshold suggestions:
  • Increase min_overall in eight_keys
  • Add phase_compliance requirements
  • Require explicit planning phase
• Test suggestions:
  • Add "must create plan before execution" test
  • Add "check if analysis needed" gate
  • Add "verify before act" checkpoint
• Prompt suggestions:
  • Add core principles to system prompt
  • Add "plan first, execute second" instruction
  • Add "ground decisions in evidence" guideline

Earth (Control) - Problem: Over-Constrained

Controlled by: Wood (operations/execution)
Generated by: Fire (intelligence)

When benchmark shows rigidity, excessive constraints, or inability to adapt:

• Threshold suggestions:
  • Relax max_steps constraint
  • Increase timeout_seconds
  • Reduce forbidden_tools list
• Test suggestions:
  • Remove redundant constraint checks
  • Allow continuation after constraint warning
  • Add "grace period" for constraint violations
• Prompt suggestions:
  • Add "be pragmatic about constraints" instruction
  • Add "explain constraint violation" option
  • Add "request constraint relaxation" capability

Metal (Coordination) - Problem: Too Rigid

Controlled by: Fire (intelligence/adaptation)
Generated by: Earth (control)

When benchmark shows bureaucratic overhead, inflexible rules, or coordination failures:

• Threshold suggestions:
  • Allow tool_sequence flexibility
  • Add "alternative_tools" option
  • Reduce required_tools strictness
• Test suggestions:
  • Add "equivalent tool" acceptance
  • Allow "custom tool" with approval
  • Implement "tool suggestion" mechanism
• Prompt suggestions:
  • Add "adapt tool usage to context" instruction
  • Add "explain tool choice" option
  • Add "suggest better tools" capability

Water (Identity) - Problem: Unclear Purpose

Controlled by: Earth (control/processes)
Generated by: Metal (coordination)

When benchmark shows vague goals, lack of direction, or identity crisis:

• Threshold suggestions:
  • Add explicit purpose statement requirement
  • Increase min_per_key for purpose-related keys
  • Add "goal_clarity" metric
• Test suggestions:
  • Add "state goal explicitly" test
  • Add "verify goal alignment" checkpoint
  • Add "measure outcome vs goal" evaluation
• Prompt suggestions:
  • Add "state your purpose" at start
  • Add "connect action to goal" instruction
  • Add "verify goal before proceed" checkpoint

Improvement Cycle

Observe → Detect → Generate → Apply → Verify → Feed Forward

1. Observe: Run benchmark and collect results
2. Detect: Identify anti-patterns via 相克 cycle
3. Generate: Create improvements via 相生 pathway
4. Apply: Edit files with actual changes (create checkpoint first)
5. Verify: Re-run benchmark, rollback if no improvement
6. Feed Forward: Record insights for next cycle

Verification Thresholds

• Minimum improvement to keep: 5% (0.05 score delta)
• Auto-improvement trigger: Score below 0.7
• Max retries before giving up: 3 attempts per improvement
• Rollback on failure: Always restore from checkpoint

Usage

;; Auto-improve a skill
(gptel-benchmark-auto-improve-skill "my-skill" benchmark-results)

;; Auto-improve a workflow
(gptel-benchmark-auto-improve-workflow "my-workflow" benchmark-results)

;; Batch improve multiple items
(gptel-benchmark-batch-improve
  '(("skill-a" skill results-a)
    ("workflow-b" workflow results-b)))

Anti-Pattern Detection

Anti-patterns are detected by analyzing benchmark results against Eight Keys scores:

• Vitality (φ) low: Mechanical repetition, no learning from feedback
• Clarity (fractal) low: Vague terms, untestable definitions
• Purpose (ε) low: No clear goals, abstract descriptions
• Wisdom (τ) low: No planning, reactive fixes
• Synthesis (π) low: Fragmented thinking, isolated facts
• Directness (μ) low: Polite evasion, unnecessary words
• Truth (∃) low: Assumptions over data, false confidence
• Vigilance (∀) low: Repeating mistakes, no error handling

Map low scores to their element and apply the corresponding improvement rules above.

Evolved Improvement Effectiveness

Based on analysis of which improvement types led to score increases.

Element	Effectiveness	Improvement Rate	Total	Improved	Worsened
Control (Earth)	marginally-effective	22%	18	4	5
Intelligence (Fire)	highly-effective	67%	3	2	0
Coordination (Metal)	ineffective	12%	77	9	15
Identity (Water)	ineffective	18%	148	27	27
Operations (Wood)	ineffective	8%	78	6	9

Recommendations

Prioritize these improvement types:

• Intelligence (Fire) (67% success rate)

Reconsider these improvement types:

• Control (Earth) (22% success rate, 5 worsened)
• Coordination (Metal) (12% success rate, 15 worsened)
• Identity (Water) (18% success rate, 27 worsened)
• Operations (Wood) (8% success rate, 9 worsened)

Evolved Recommendations

Based on analysis of 0 experiments.