perfguard-performance-aware-agent-visual

name

perfguard-performance-aware-agent-visual

description

Performance-aware multi-tool orchestration for visual content generation pipelines. Implements PerfGuard's three mechanisms (PASM, APU, CAPO) to select, score, and schedule AI image/video tools based on measured capability boundaries instead of generic descriptions. Use when: "build a visual generation pipeline", "orchestrate multiple image tools", "select the best AI model for this image task", "score and rank generation tools", "adaptive tool selection for AIGC", "performance-aware tool routing".

PerfGuard: Performance-Aware Agent for Visual Content Generation

This skill enables Claude to design and implement multi-tool orchestration systems for AI-generated visual content (AIGC) that select tools based on measured performance boundaries rather than static descriptions. Derived from the PerfGuard framework (ICLR 2026), the approach replaces the naive assumption that "all tools work equally well" with a structured scoring, ranking, and adaptive update system across three mechanisms: Performance-Aware Selection Modeling (PASM), Adaptive Preference Update (APU), and Capability-Aligned Planning Optimization (CAPO).

When to Use

• When the user is building a pipeline that routes tasks to multiple image generation or editing models (e.g., FLUX, SDXL, DALL-E, Midjourney APIs)
• When the user wants to automatically pick the best tool for a specific visual subtask (e.g., "which model handles spatial relationships better?")
• When the user needs a scoring/evaluation system for comparing AI-generated images across quality dimensions
• When the user asks to build a self-improving agent that learns which tools work best over time
• When the user wants to decompose complex visual tasks into subtasks matched to tool strengths
• When the user is implementing fallback/retry logic for visual generation where output quality is uncertain

Key Technique

The core insight: Generic tool descriptions ("generates high-quality images") are useless for selection. PerfGuard replaces them with multi-dimensional performance scores measured on fine-grained capability axes. For text-to-image tools, these axes are: color accuracy, shape fidelity, texture realism, 2D spatial, 3D spatial, numeracy (correct object count), and non-spatial semantics. For image editing tools: addition, removal, replacement, attribute alteration, motion change, style transfer, and background change. Each tool gets a score (0.0-1.0) per dimension, producing a capability fingerprint.

Selection works by weighted dot product. Given a task, the system identifies which dimensions matter (e.g., a request involving "three red cups arranged left to right" weights numeracy, color, and 2D-spatial heavily). The preference score for each candidate tool is sum(tool.score[dim] * task.weight[dim]). Tools are ranked by this score; the top-N candidates execute. Results are evaluated via a multi-dimensional quality assessment (object count, position, attributes, style, background, semantic alignment) scored at three levels (L1=1.0, L2=0.66, L3=0.33). If the composite score falls below a threshold (default 0.8), the system re-plans and retries with alternative tools.

The system learns from outcomes. APU compares the predicted tool ranking against the actual quality ranking from execution. When discrepancies occur (predicted #1 actually performed worst), preference scores are updated. This closes the loop: the system gets better at tool selection over time without manual re-tuning. CAPO feeds these updated scores back into the planner, biasing subtask decomposition toward operations where strong tools exist rather than generating subtasks no tool can reliably handle.

Step-by-Step Workflow

1. Define capability dimensions for your tool domain. For text-to-image: [color, shape, texture, 2d_spatial, 3d_spatial, numeracy, non_spatial]. For image editing: [addition, removal, replacement, attribute_alter, motion_change, style_transfer, background_change]. For video or other domains, define analogous axes.

2. Build a tool registry with performance scores. For each tool, populate a score dictionary across all dimensions. Initialize from benchmarks or manual evaluation. Store as JSON:

   {
     "flux": {"color": 0.74, "shape": 0.57, "texture": 0.69, "2d_spatial": 0.63, "numeracy": 0.52, "non_spatial": 0.92},
     "sd3":  {"color": 0.81, "shape": 0.59, "texture": 0.73, "2d_spatial": 0.61    
   
   

Concrete Examples

Example 1: Building a multi-model image generation router

User: "I have access to FLUX, SDXL, and DALL-E APIs. Build a system that automatically picks the best model for each image generation request."

Approach:

1. Define the tool registry with initial scores per capability dimension
2. Build a FastAPI service with a /generate endpoint
3. Implement request analysis to extract dimension weights
4. Compute ranked tool selection per request
5. Execute with the top tool, evaluate output, update scores

Output structure:

# tool_registry.py
TOOL_SCORES = {
    "flux": {
        "color": 0.74, "shape": 0.57, "texture": 0.69,
        "2d_spatial": 0.63, "3d_spatial": 0.45, "numeracy": 0.52,
        "non_spatial": 0.92
    },
    "sdxl": {
        "color": 0.81, "shape": 0.59, "texture": 0.73,
        "2d_spatial": 0.61, "3d_spatial": 0.50, "numeracy": 0.48,
        "non_spatial": 0.91
    },
    "dalle": {
        "color": 0.78, "shape": 0.65, "texture": 0.71,
        "2d_spatial": 0.58, "3d_spatial": 0.55, "numeracy": 0.60,
        "non_spatial": 0.88
    }
}

# selector.py
def select_tool(task_weights: dict[str, float], registry: dict) -> [[, ]]:
    
    rankings = []
     tool_name, scores  registry.items():
        total = (scores.get(dim, ) * w  dim, w  task_weights.items())
        rankings.append((tool_name, total))
     (rankings, key= x: x[], reverse=)


 () -> [, ]:
    
    weights = {dim:   dim  DIMENSIONS}  
     (w  prompt.lower()  w  [, , , , ]):
        weights[] = 
     (w  prompt.lower()  w  [, , , , ]):
        weights[] = 
     (w  prompt.lower()  w  [, , , , ]):
        weights[] = 
    
    total = (weights.values())
     {k: v / total  k, v  weights.items()}

Example 2: Self-improving image editing pipeline with fallback

User: "Build an editing pipeline that tries multiple editing models and learns which works best for different edit types."

Approach:

1. Define editing-specific dimensions: addition, removal, replacement, attribute_alter, style_transfer, background_change
2. Implement the edit classifier to determine edit type from instruction
3. Select top-2 tools, execute both, evaluate with multi-dimensional scoring
4. Update tool preferences based on observed quality
5. Persist updated scores to disk for future sessions

Output structure:

# evaluator.py
EVAL_DIMENSIONS = [
    "object_category", "object_count", "positional_accuracy",
    "attribute_binding", "style_match", "background_fidelity", "semantic_alignment"
]
TIER_SCORES = {"L1": 1.0, "L2": 0.66, "L3": 0.33}

def evaluate_output(original_prompt: str, image_path: str, llm_client) -> dict:
    """Score generated image across quality dimensions using VLM."""
    scores = {}
    for dim in EVAL_DIMENSIONS:
        response = llm_client.evaluate(
            image=image_path, prompt=original_prompt,
            question=f"Rate {dim}: L1 (fully correct), L2 (partially), L3 (present but wrong)"
        )
        scores[dim] = TIER_SCORES.get(response.tier, 0.33)

    non_semantic = [v for k, v in scores.items() if k != "semantic_alignment"]
    composite = (sum(non_semantic) / len(non_semantic) + scores["semantic_alignment"]) / 2
    return {"dimensions": scores, "composite": composite}

# updater.py
def update_preferences(tool_name: str, predicted_rank: int, actual_rank: int,
                       registry: dict, alpha:  = ) -> :
    
     predicted_rank != actual_rank:
        adjustment = alpha * (predicted_rank - actual_rank) / (predicted_rank, actual_rank)
         dim  registry[tool_name]:
            registry[tool_name][dim] = (, (,
                registry[tool_name][dim] + adjustment))
     registry

Example 3: Capability-aligned task decomposition

User: "I want to generate a complex scene: 'A medieval castle on a cliff at sunset with three knights on horseback in the foreground.' Break this into subtasks matched to the best available tools."

Approach:

1. Analyze the prompt for required capabilities: 3d_spatial (cliff perspective), numeracy (three knights), texture (medieval materials), color (sunset lighting)
2. Check tool registry for weak dimensions -- if no tool scores > 0.5 on numeracy, restructure to avoid relying on numeracy
3. Decompose into capability-aligned subtasks

Output:

Plan (CAPO-aligned):
  Subtask 1: Generate base scene "medieval castle on cliff at sunset"
    -> Best tool: SDXL (high color: 0.81, texture: 0.73)
    -> Avoids numeracy by not including knights yet

  Subtask 2: Generate single knight on horseback as reference
    -> Best tool: FLUX (high non_spatial: 0.92)
    -> Generates one high-quality knight to use as reference

  Subtask 3: Composite three knights into foreground using layout-guided generation
    -> Best tool: Layout_to_Image (uses bounding boxes for placement)
    -> Handles numeracy via explicit spatial layout rather than relying on model counting

  Subtask 4: Harmonize lighting and style across composited image
    -> Best tool: UltraEdit (style_transfer: 0.78)
    -> Ensures consistent sunset lighting across all elements

Quality gate: Evaluate composite score after each subtask. Threshold: 0.8.
Retry budget: up to 5 rounds with fallback to next-ranked tools.

Best Practices

• Do: Benchmark tools empirically before populating the score registry. Run each tool on 50-100 diverse prompts spanning all capability dimensions. PerfGuard's value comes from accurate scores, not guesses.
• Do: Use parallel execution of top-N tools when latency budget allows. Comparing actual outputs is far more reliable than relying on scores alone, especially early in deployment.
• Do: Keep the APU learning rate (alpha) conservative (0.1-0.2). Aggressive updates cause score oscillation when evaluation is noisy.
• Do: Store experience records with CLIP embeddings of the prompt for fast semantic retrieval. Past task outcomes are the highest-signal input for planning.
• Avoid: Using the same weight vector for all tasks. The entire point of PASM is that different tasks stress different dimensions. A uniform weighting collapses back to a generic "best overall model" selection.
• Avoid: Skipping the evaluation step. Without measured output quality, APU cannot update preferences and the system cannot self-improve. Even a rough VLM-based evaluation (GPT-4V, Gemini) is better than none.

Error Handling

• Tool execution failure (timeout, OOM, API error): Assign a composite score of 0.0, fall through to the next-ranked tool immediately. Log the failure mode in the experience record.
• All top-N tools score below threshold: After exhausting the retry budget (5 rounds), return the best result achieved with a quality warning. Do not loop indefinitely.
• Evaluation model disagrees with human judgment: Periodically calibrate VLM evaluation scores against human ratings. If the evaluator is consistently wrong on a dimension, add dimension-specific calibration offsets.
• Score drift from APU: Implement score bounds (never below 0.05, never above 0.99) and add periodic score decay toward benchmark baselines to prevent runaway drift.
• New tool added to registry: Initialize scores at the population mean across all dimensions, then run a quick benchmark pass (20-30 prompts) to establish a real fingerprint before using in production routing.

Limitations

• Requires a vision-language model for evaluation. The multi-dimensional scoring system depends on a capable VLM (GPT-4V-class or better) to assess generated images. Without it, APU cannot function.
• Initial scores need manual effort. The system is only as good as its initial benchmarks. Cold-start with uniform scores will produce poor tool selection until enough APU iterations accumulate.
• Dimension taxonomy is domain-specific. The text-to-image and image-editing dimensions from the paper may not transfer directly to video generation, 3D, or audio. You must define appropriate axes for your domain.
• Does not model tool interactions. PASM scores tools independently. If two tools combined produce better results than either alone (e.g., generate + refine), this must be captured in the experience/planning layer, not in individual tool scores.
• Evaluation latency adds overhead. Running VLM evaluation after every generation step increases end-to-end latency. For real-time applications, consider batch evaluation or sampling strategies.

Reference

• Paper: PerfGuard: A Performance-Aware Agent for Visual Content Generation (ICLR 2026)
• Code: github.com/FelixChan9527/PerfGuard
• What to look for: Section 3 details PASM scoring dimensions and the preference-weighted selection formula; Section 4 covers the APU update mechanism with predicted-vs-actual rank comparison; Section 5 describes CAPO's planner integration. The experience replay system using CLIP embeddings is in Section 3.3.