clawjournal-score

name	clawjournal-score
description	Score coding agent sessions for failure value and productivity. Use when the user asks to score a session, score sessions, re-score, auto-score, batch score, or evaluate traces. Triggers on "score sessions", "rate my traces", "quality check", or scoring requests.
metadata	{"argument-hint":"<session-id>"}

Score Sessions

Quick Path: Batch Auto-Score

If no session ID was provided, suggest the automated approach first:

# Score in-scope failure-value traces automatically (recommended)
clawjournal score --batch --source failure-corpus --auto-triage --limit 20

# Or without auto-triage:
clawjournal score --batch --source failure-corpus --limit 20

For hands-on scoring of a specific session, continue below. With --auto-triage, only productivity-1 sessions with failure value 1-2 are auto-blocked; failure value 3+ stays reviewable.

Session Data

Run this to view the session:

clawjournal score-view <session-id>

If no session ID was provided, list available sessions:

clawjournal score --batch --limit 10

Scoring Rubric (1-5)

ClawJournal now records two scores in one pass:

• ai_quality_score: legacy productivity/substance score.
• ai_failure_value_score: primary capture score for understanding frontier-agent failure and recovery behavior.

Failure value:

5 = Canonical high-value trace — Consequential failure or recovery pattern on real expert work, where the agent behavior is the lesson.

4 = Meaningful failure/recovery — Worth reviewing or turning into eval/training data.

3 = Some signal — Ambiguous, minor, repeated, or mostly environmental.

2 = Weak signal — Routine retry, expected debugging, or unclear attribution.

1 = No useful failure signal — Smooth success, trivial session, control sample, or noise.

Labels

Recovery labels: self_recovered, user_corrected_recovery, unrecovered, blocked.

Failure attribution: agent_caused, environment, preexisting_problem, user_redirect, unclear.

Failure modes (ai_failure_modes): task_framing, method_selection, context_handling, execution_error, reasoning_fabrication, revision_failure, verification_skipped, deliverable_defect, communication_error, collaboration_error, safety_security, efficiency_waste.

Meta labels (ai_meta_labels): evaluation_measurement only — emit when the evaluation/measurement setup itself misjudges the agent, not when the agent failed.

Scores of 4 or 5 require at least one ai_failure_evidence snippet. A 5/5 does not require agent_caused; non-agent-caused traces can be high value when the agent's recovery, uncertainty handling, escalation, or collaboration is the lesson.

Detailed rubric

See RUBRIC.md in this directory for the full scoring rubric with examples. In the repo, this file is generated from clawjournal/prompts/agents/scoring/rubric.md; edit the canonical prompt copy first, then run python -m clawjournal.prompt_sync.

• How to read user feedback signals and final user response
• How to score failure value without punishing normal debugging
• STEM/domain correction evidence
• Twelve agent failure modes plus one meta label and recovery labels
• Structured output fields

Store the Score

Manual score setting can override failure value directly. Keep --quality for legacy productivity compatibility only. Failure-value overrides to 4-5 require trace-backed evidence.

clawjournal set-score <session-id> --failure-value <score> --failure-evidence "<trace-backed snippet>" --reason "<1-2 sentence explanation>"
clawjournal set-score <session-id> --quality <score> --reason "<legacy productivity note>"