ct-grade

name	ct-grade
description	CLEO session grading and A/B behavioral analysis with token tracking. Evaluates agent session quality via a 5-dimension rubric (S1 session discipline, S2 discovery efficiency, S3 task hygiene, S4 error protocol, S5 progressive disclosure). Supports three modes: (1) scenario — run playbook scenarios S1-S5 via CLI; (2) ab — blind A/B comparison of different CLI configurations for same domain operations with token cost measurement; (3) blind — spawn two agents with different configurations, blind-comparator picks winner, analyzer produces recommendation. Use when grading agent sessions, running grade playbook scenarios, comparing behavioral differences, measuring token usage across configurations, or performing multi-run blind A/B evaluation with statistical analysis and comparative report. Triggers on: grade session, evaluate agent behavior, A/B test CLEO configurations, run grade scenario, token usage analysis, behavioral rubric, protocol compliance scoring.
version	2.1.0
argument-hint	[mode=scenario|ab|blind] [scenario=s1-s5|all] [runs=N] [session-id=<id>]
allowed-tools	["Bash(python *)","Bash(cleo-dev *)","Bash(cleo *)","Bash(kill *)","Bash(lsof *)","Agent","Read","Write","Glob"]
tier	2
core	false
category	quality
protocol	null
dependencies	[]
sharedResources	[]
compatibility	["claude-code","cursor","windsurf","gemini-cli"]
license	MIT

Session Grading Guide

Session grading evaluates agent behavioral patterns against the CLEO protocol. It reads the audit log for a completed session and applies a 5-dimension rubric to produce a score (0-100), letter grade (A-F), and diagnostic flags.

When to Use Grade Mode

Use grading when you need to:

• Evaluate how well an agent followed CLEO protocol during a session
• Identify behavioral anti-patterns (skipped discovery, missing session.end, etc.)
• Track improvement over time across multiple sessions
• Validate that orchestrated subagents followed protocol

Grading requires audit data. Sessions must be started with the --grade flag to enable audit log capture.

Starting a Grade Session

CLI

# Start a session with grading enabled
ct session start --scope epic:T001 --name "Feature work" --grade

# The --grade flag enables detailed audit logging
# All CLI operations are recorded for later analysis

Running Scenarios

The grading rubric evaluates 5 behavioral scenarios that map to protocol compliance:

1. Fresh Discovery

Tests whether the agent checks existing sessions and tasks before starting work. Evaluates session.list and tasks.find calls at session start.

2. Task Hygiene

Tests whether task creation follows protocol: descriptions provided, parent existence verified before subtask creation, no duplicate tasks.

3. Error Recovery

Tests whether the agent handles errors correctly: follows up E_NOT_FOUND with recovery lookups (tasks.find), avoids duplicate creates after failures.

4. Full Lifecycle

Tests session discipline end-to-end: session listed before task ops, session properly ended, CLI usage patterns.

5. Multi-Domain Analysis

Tests progressive disclosure: use of admin.help or skill lookups, use of progressive disclosure for programmatic access.

Evaluating Results

CLI

# Grade a specific session
ct grade <sessionId>

# List all past grade results
ct grade --list

Understanding the 5 Dimensions

Each dimension scores 0-20 points, totaling 0-100.

S1: Session Discipline (20 pts)