check-trajectories-workflow

name	check-trajectories-workflow
description	Use Inspect Scout to analyze agent trajectories from evaluation log files. Runs default and custom scanners to detect external failures, formatting issues, reward hacking, and ethical refusals. Use when user asks to check/analyze agent trajectories. Trigger when the user asks you to run the "Check Agent Trajectories" workflow.

Check Agent Trajectories

This workflow uses Inspect Scout to automatically analyze agent trajectories. It's faster than manual analysis but may miss nuanced issues.

Workflow Steps

1. If the user does not give any indication of what log file to use, ask them for which log file or files they want the workflow performed on. You can show them the most recent log file in logs/ and ask them if that's what they want.

2. Determine the name of the evaluation the log file is used in, and create a folder titled agent_artefacts/trajectory_analysis/<eval_name>. Add a number to the end if the folder already exists.

3. Read agent_artefacts/trajectory_analysis/inspect_scout/scanners.py to see what the default scanners check for:

   1. outcome_summary: Brief summary of why the agent succeeded or failed.
   2. external_failure: Failed due to CAPTCHAs, rate limiting, network issues, missing dependencies.
   3. formatting_failure: Failed due to incorrect answer formatting despite correct answer.
   4. reward_hacking_success: Succeeded through reward hacking or unintended means.
   5. ethical_refusal: Failed because the agent refused on ethical or safety grounds.

   Additionally, if agent_artefacts/trajectory_analysis/<eval_name><version> exists already, check for any scanners contained in the latest version and include those.

4. Tell the user what will be checked by default and ask if they want to check for anything else.