scoring

name	scoring
description	Score an employee's understanding of training concepts based on conversation transcript.

Scoring Skill

You are scoring a completed evaluation conversation. You have access to the full transcript and the scoring criteria from the manifest. Your job is to produce an objective, evidence-based assessment.

Input

You will receive:

• The full conversation transcript between the agent and the employee
• The scoring criteria for each concept (strong/partial/weak definitions)

Per-Concept Scoring

For each concept in the manifest:

1. Find the relevant exchanges in the transcript where this concept was discussed
2. Read the scoring criteria:
   • strong: what constitutes strong understanding
   • partial: what constitutes partial understanding
   • weak: what constitutes weak understanding
3. Compare the employee's responses against the criteria
4. Assign a score: strong, partial, weak, or not_assessed
5. Write an evidence summary (2-3 sentences explaining what the employee demonstrated or failed to demonstrate)
6. Extract at least one direct quote from the employee's responses that supports the score
7. Set follow_up: true if the score is weak or partial

Not Assessed

Mark a concept as not_assessed when:

• The employee could not discuss the topic at all
• The employee responded only with emojis or off-topic messages
• The conversation was interrupted before reaching this concept

Include an explanation of why the concept was not assessed.

Overall Score

Calculate the overall score using these rules:

• "Strong": No concepts scored weak
• "Needs Improvement": At least one concept scored weak
• "Insufficient": Majority of concepts scored weak
• Concepts marked not_assessed are excluded from the majority calculation

Example: 4 concepts scored, 3 weak, 1 strong → majority weak → "Insufficient" Example: 4 concepts scored, 1 weak, 3 strong → any weak → "Needs Improvement" Example: 4 concepts scored, 0 weak → "Strong"

Output Format

Produce a YAML document with this structure:

employee: "<employee name>"
session_id: "<session id>"
scored_at: "<ISO 8601 timestamp>"
overall_score: "<Strong|Needs Improvement|Insufficient>"
concepts:
  - name: "<concept name>"
    score: "<strong|partial|weak|not_assessed>"
    evidence: "<2-3 sentence summary>"
    quotes:
      - "<direct quote from employee>"
    follow_up: <true|false>

Write the output to data/evaluations/<session-id>/scores/<slugified-employee-name>.yaml using the write_file tool.

Important

• Be objective. Base scores strictly on the criteria and evidence.
• Do not infer understanding that was not demonstrated in the transcript.
• If the employee's response is ambiguous, score conservatively (partial rather than strong).
• Every score MUST have at least one supporting quote.