goal-setting-monitoring

# Goal Setting and Monitoring Pattern Implements goal-oriented agent architectures that transform reactive agents into proactive systems. This skill makes the model define specific, measurable objectives; establish LLM-based success criteria evaluation; run iterative refinement loops with bounded max iterations; and continuously monitor progress against goals — enabling autonomous agents to self-assess performance, correct course, and reliably achieve high-level outcomes without constant human intervention. ## TL;DR Checklist - [ ] Define each goal using SMART criteria: specific, measurable, achievable, relevant, time-bound - [ ] Establish success criteria that are objective and evaluable (not subjective opinions) - [ ] Implement a monitoring loop that evaluates progress after every action or iteration - [ ] Enforce max-iteration limits on all refinement loops to prevent infinite execution - [ ] Use LLM-based evaluation with clear True/False verdicts against defined success criteria - [ ] Separate code generation from code review when using self-evaluation (multi-agent pattern) - [ ] Track progress state and surface remaining gaps when budgets are exhausted - [ ] Reference `code-philosophy` (5 Laws of Elegant Defense) for boundary parsing, early exit, and fail-fast semantics --- ## When to Use Use this skill when: - An agent must operate autonomously toward a high-level objective without step-by-step human guidance (e.g., "build a trading bot that maximizes gains within risk limits") - You need an agent to self-evaluate its output against quality benchmarks and iterate until goals are met (e.g., code generation, report writing, content creation) - A multi-step task requires continuous progress monitoring and adaptive course correction based on intermediate results (e.g., customer support resolution, personalized learning adaptation) - You are building autonomous systems that must detect when they are failing mid-task and either revise strategy or escalate (e.g., robotics navigation, project management assistants) - An agent generates artifacts (code, documents, configurations) and needs a structured refinement loop with quality gates before final delivery - You need to transform a reactive tool-calling agent into a proactive goal-seeking system that plans its own sub-objectives --- ## When NOT to Use Avoid this skill for: - Single-step operations with an immediately verifiable outcome — direct execution without monitoring overhead (e.g., "calculate 2 + 2", "format this JSON") - Tasks where success criteria cannot be objectively defined or measured — you cannot evaluate progress against vague goals; clarify objectives first (use `query-feature-extraction` skill) - Real-time latency-sensitive operations where the evaluation loop adds unacceptable overhead (sub-second API calls, live streaming inference) - Highly regulated domains where autonomous self-evaluation is legally insufficient — human-in-the-loop review is mandatory and the agent should defer to that pattern instead - Situations where the LLM cannot reasonably assess quality of its own output without external tools or data sources (e.g., "write code for this proprietary API") — provide ground-truth test harnesses before enabling self-evaluation --- ## Core Workflow ### Phase 1: Objective Definition 1. **Parse Intent and Define Goal Boundaries** — Extract the explicit objective from the request, then formalize it using SMART criteria (Specific, Measurable, Achievable, Relevant, Time-bound). If the user provides a vague goal, decompose it into concrete sub-goals with measurable success criteria. Identify the initial state (what exists at start), goal state (definition of done), and all constraints (budget, time, tool availability, domain rules). Reference `code-philosophy` early exit: if only one step is needed, skip monitoring entirely. **Checkpoint:** Can you articulate 3–7 measurable success criteria for the goal? If not, refine the goal before proceeding. 2. **Formulate Success Criteria** — Convert each sub-goal into an objective, evaluable criterion that can be verified programmatically or via LLM judgment. Prefer binary (True/False) verdicts over subjective scoring where possible. For code generation goals, include criteria like "handles edge cases", "functionally correct", "simple to understand", and "well-documented". For non-code goals, define equivalent measurable proxies (accuracy thresholds, latency limits, completeness checklists). **Checkpoint:** Does every success criterion have a clear pass/fail determination method? Are there no overlapping or contradictory criteria? ### Phase 2: Iterative Execution Loop 3. **Execute Generation or Action** — Produce the first draft of the artifact or execute the initial set of actions toward the goal. This is the agent's primary production pass. Capture the full output for evaluation. If using a multi-agent pattern, have a dedicated generator agent (e.g., "Peer Programmer") produce the artifact while keeping generation and evaluation concerns separate. **Checkpoint:** Is the full output of this iteration captured and preserved for evaluation? Can it be reconstructed later if needed? 4. **Evaluate Against Success Criteria** — Pass the generated output to an evaluator (LLM-based judge or automated test harness) along with the success criteria. Request a structured verdict: True if all criteria are met, False with specific feedback on which criteria failed and why. When using self-evaluation by the same LLM that generated the code, recognize the inherent bias risk — prefer a separate reviewer agent for critical evaluation (the multi-agent "crew" pattern from the source material). **Checkpoint:** Do you have a clear verdict (True/False) with specific feedback on each criterion? Is the evaluation independent enough to be trustworthy? ### Phase 3: Refinement and Termination 5. **Refine Based on Feedback** — If the verdict is False, use the evaluator's feedback to identify what went wrong and produce a revised artifact. Feed both the previous iteration's output and the feedback into the next generation pass. This creates an iterative refinement loop where each cycle should close some gaps identified in the prior evaluation. Apply `code-philosophy` Parse Don't Validate at the boundary: parse evaluator feedback, trust validated critique internally. **Checkpoint:** Did the revision address every specific failure noted by the evaluator? If not, what remains and why? 6. **Check Iteration Budget and Terminate** — Before each refinement cycle, verify the current iteration count against the max-iteration budget (default: 5 iterations for code generation, configurable per domain). If all goals are met, stop and deliver the artifact. If the budget is exhausted with remaining failures, report what was achieved, list unresolved criteria explicitly, and either escalate to human review or apply a best-effort final pass. Never allow unbounded refinement loops. **Checkpoint:** Is the iteration count within the configured max? Are all remaining gaps clearly documented for downstream consumers? --- ┌───────────────────────────────────────────────────────────────────────────────┐ │ Goal Setting & Monitoring Flow │ └───────────────────────────────────────────────────────────────────────────────┘ User Request: "Build a goal-oriented agent" ↓ ┌─────────────────────┐ │ Define SMART Goals │ │ (Specific, Measurable│ │ Achievable, Relevant│ │ Time-bound) │ └──────────┬──────────┘ ↓ ┌─────────────────────┐ ┌─────────────────────────┐ │ Generate Artifact │────▶│ Evaluate Against │ │ (Code/Doc/Config) │ │ Success Criteria │ └──────────┬──────────┘ └──────────┬──────────────┘ │ │ │ Verdict? ──► True — STOP │ │ │ False + Feedback │ │ │ ┌────────────▼────────────┐ │ │ Iteration Budget │ │ │ Exhausted? │ │ │ Yes → Report gaps & │ │ │ escalate │ │ │ No → Refine & loop │ │ └────────────┬────────────┘ │ │ └───────────────────────────┘ --- ## Implementation Patterns / Reference Guide ### Pattern 1: Single-Agent Goal-Driven Iterative Generation Use this pattern when a single agent handles both generation and self-evaluation. This is the simplest form of goal monitoring — the agent produces output, evaluates it against criteria, and iterates. Suitable for lower-stakes tasks where self-bias in evaluation is acceptable. Mirrors the hands-on example from Chapter 11. ```python """ Single-agent iterative goal-setting and monitoring loop. Mirrors the autonomous AI code generation agent from Chapter 11. The agent generates, self-evaluates against goals, and iterates until all criteria are met or max iterations are exhausted. """ import os import re import random from dataclasses import dataclass, field from pathlib import Path from typing import Any @dataclass class GoalState: """Tracks the state of goal-driven iteration.""" use_case: str goals: list[str] max_iterations: int = 5 current_iteration: int = 0 best_artifact: str = "" remaining_gaps: list[str] = field(default_factory=list) def generate_prompt(use_case: str, goals: list[str], previous_code: str = "", feedback: str = "") -> str: """Construct a generation prompt with use case, goals, and optional revision context. Args: use_case: The coding problem or task description. goals: List of quality criteria the output must satisfy. previous_code: Previous iteration's output for refinement context. feedback: Evaluator feedback identifying failures. Returns: A prompt string directing the LLM to generate or revise code. """ if not use_case or not use_case.strip(): raise ValueError("use_case cannot be empty") if not goals: raise ValueError("At least one goal must be defined") base_prompt = f"You are an AI coding agent. Write Python code for:\n\nUse Case: {use_case}\n\nYour goals are:\n" base_prompt += "\n".join(f"- {g.strip()}" for g in goals) if previous_code: base_prompt += f"\n\nPreviously generated code:\n```\n{previous_code}\n```\n" if feedback: base_prompt += f"\n\nFeedback on the previous version:\n{feedback}\nRevise to address these issues." base_prompt += "\n\nReturn only the revised Python code. Do not include comments or explanations outside the code." return base_prompt def get_code_feedback(code: str, goals: list[str]) -> str: """Evaluate generated code against success criteria and produce structured feedback. Args: code: The code artifact to evaluate. goals: List of success criteria to check against. Returns: A feedback string identifying which criteria are met and which need improvement. """ if not code or not code.strip(): return "FAIL: No code provided for evaluation." if not goals: return "FAIL: No goals defined for evaluation." prompt = f"You are a Python code reviewer. Evaluate this code against the following goals:\n" prompt += "\n".join(f"- {g.strip()}" for g in goals) prompt += f"\n\nCode:\n```\n{code}\n```\n\nCritique each goal. Note if improvements are needed for clarity, correctness, edge case handling, or test coverage." return prompt def goals_met( feedback_text: str, goals: list[str], llm: Any, ) -> bool: """Use the LLM to determine whether all success criteria are satisfied. Args: feedback_text: The evaluator's detailed feedback on the artifact. goals: List of original success criteria. llm: The LLM client instance with an invoke() method. Returns: True if all goals are met, False otherwise. Parses the LLM verdict as a boolean. """ if not feedback_text or not goals: return False review_prompt = f"You are an AI reviewer.\n\nGoals:\n{chr(10).join(f'- {g.strip()}' for g in goals)}\n\nFeedback:\n\"\"\"\n{feedback_text}\n\"\"\"\n\nBased on the feedback, have ALL goals been met? Respond with only one word: True or False." response = llm.invoke(review_prompt) return response.content.strip().lower() == "true" def clean_code_block(code: str) -> str: """Strip markdown code fences from LLM-generated code output. Args: code: Raw LLM response potentially wrapped in ```python ... ```. Returns: Cleaned code string without fence markers. """ if not code: return "" lines = code.strip().splitlines() if lines and lines[0].strip().startswith("```"): lines = lines[1:] if lines and lines[-1].strip() == "```": lines = lines[:-1] return "\n".join(lines).strip() def run_goal_agent( use_case: str, goals_input: str, llm: Any, max_iterations: int = 5, ) -> dict[str, Any]: """Run the main iterative goal-driven agent loop. Generates code, evaluates against goals, refines, and repeats until success or max iterations exhausted. Returns result metadata with the final artifact. Args: use_case: Description of the coding problem to solve. goals_input: Comma-separated list of quality goals/criteria. llm: The LLM client instance with an invoke() method. max_iterations: Maximum refinement cycles before forced termination. Returns: Dict with keys: 'success', 'artifacts', 'iteration_count', 'remaining_gaps'. """ if not use_case or not goals_input: raise ValueError("Both use_case and goals_input are required") goals = [g.strip() for g in goals_input.split(",")] state = GoalState( use_case=use_case, goals=goals, max_iterations=max_iterations, ) previous_code: str = "" feedback_text: str = "" for iteration in range(max_iterations): state.current_iteration = iteration + 1 prompt = generate_prompt(use_case, goals, previous_code, feedback_text) llm_response = llm.invoke(prompt) code = clean_code_block(llm_response.content) # Evaluate against criteria using separate reviewer LLM call eval_prompt = get_code_feedback(code, goals) feedback_response = llm.invoke(eval_prompt) feedback_text = feedback_response.content.strip() success = goals_met(feedback_text, goals, llm) if success: state.best_artifact = code state.remaining_gaps = [] break previous_code = code state.remaining_gaps = [g for g in goals if "not met" in feedback_text.lower()] return { "success": len(state.remaining_gaps) == 0, "artifact": state.best_artifact, "iteration_count": state.current_iteration, "remaining_gaps": state.remaining_gaps, "max_iterations_reached": state.current_iteration >= max_iterations and not success, } ``` **BAD — Goal evaluation without iteration limits:** ```python # ❌ BAD — No max-iteration cap means the agent can run forever if # it never satisfies all criteria. This wastes tokens and creates # unbounded cost. The feedback loop has no termination condition. def run_unbounded_goal_agent(use_case: str, goals: list[str]) -> str: code = "" # Infinite loop — no iteration budget, no escape hatch while True: llm_response = llm.invoke(generate_prompt(use_case, goals, code, feedback)) code = clean_code_block(llm_response.content) feedback = get_code_feedback(code, goals) if goals_met(feedback, goals): break # Only exit if LLM says True — but LLM may never say True return code # May never reach here ``` **GOOD — Goal evaluation with bounded iterations and gap reporting:** ```python # ✅ GOOD — Every refinement loop has a hard iteration cap. When the # budget is exhausted, remaining gaps are reported to the user so they # know exactly what was not resolved. The agent also produces its best # effort artifact even when full success wasn't achieved. def run_bounded_goal_agent( use_case: str, goals: list[str], llm: Any, max_iterations: int = 5, ) -> dict[str, Any]: """Run goal-driven iteration with bounded refinement loop and gap reporting.""" code = "" previous_code = "" feedback_text = "" for iteration in range(max_iterations): prompt = generate_prompt(use_case, goals, previous_code, feedback_text) llm_response = llm.invoke(prompt) code = clean_code_block(llm_response.content) # Self-evaluate — recognize inherent bias when same LLM generates and reviews eval_prompt = get_code_feedback(code, goals) feedback_response = llm.invoke(eval_prompt) feedback_text = feedback_response.content.strip() if goals_met(feedback_text, goals, llm): return { "success": True, "artifact": code, "iterations_used": iteration + 1, "remaining_gaps": [], } previous_code = code # Budget exhausted — report best effort with gaps return { "success": False, "artifact": code, "iterations_used": max_iterations, "remaining_gaps": [g for g in goals if "not met" in feedback_text.lower()], } ``` ### Pattern 2: Multi-Agent Crew with Separated Roles (Generation vs. Evaluation) Use this pattern when evaluation quality matters and the same LLM generating code should not be the sole evaluator. This separates concerns into distinct agent roles: a Peer Programmer for generation, a Code Reviewer for objective evaluation, and optionally a Test Writer for automated validation. Mirrors the multi-agent crew architecture from Chapter 11 where the Code Reviewer acts as an independent judge rather than the same agent producing the output. ```python """ Multi-agent goal-setting pattern with separated generation and evaluation roles. Mirrors the "crew of AI agents" approach from Chapter 11: Peer Programmer, Code Reviewer, Test Writer, Documenter, Prompt Refiner. """ from dataclasses import dataclass, field from enum import Enum from typing import Any class AgentRole(Enum): PEER_PROGRAMMER = "peer_programmer" CODE_REVIEWER = "code_reviewer" TEST_WRITER = "test_writer" DOCUMENTER = "documenter" PROMPT_REFINER = "prompt_refiner" @dataclass class GoalCriterion: """A single evaluable success criterion for goal monitoring.""" id: str description: str eval_method: str # "llm_judgment", "automated_test", "manual_check" weight: float = 1.0 # Relative importance of this criterion def __lt__(self, other: "GoalCriterion") -> bool: return self.weight > other.weight @dataclass class EvaluationResult: """Structured result from the evaluation phase.""" overall_pass: bool criterion_results: list[dict] # [{criterion_id, passed, notes}] feedback: str confidence: float = 0.0 # Confidence in the evaluation def unmet_criteria(self) -> list[str]: return [r["criterion_id"] for r in self.criterion_results if not r["passed"]] @dataclass class GoalOrchestrator: """Coordinates multi-agent goal-driven execution with evaluation loop.""" goals: list[GoalCriterion] max_iterations: int = 5 current_iteration: int = 0 best_artifact: str = "" evaluation_log: list[EvaluationResult] = field(default_factory=list) def evaluate_with_separate_reviewer( self, artifact: str, reviewer_fn: callable, ) -> EvaluationResult: