build-scoring

name	build-scoring
description	Design multi-dimensional evaluation rubrics with calibrated scales, thresholds, and function-adaptive weights.

/build-scoring - Scoring Rubric Design

When to Use

• After /define-audiences (scoring rubrics serve the audiences)
• When designing evaluation frameworks for the target OS
• When domain-input/scoring-rubrics.md needs to be filled

Inputs

• Auto-loaded: domain-input/audiences.md (what audiences evaluate)
• Auto-loaded: domain-input/domain-workflow.md (pipeline stages and healthy metrics)
• Auto-loaded: domain-input/domain-knowledge.md (existing domain frameworks)

Process

Step 1: Identify What Gets Scored

From the workflow and audience maps, identify every point where the OS needs to evaluate something:

1. Triage scoring: Should the user pursue this opportunity? (equivalent to job-fit-scorer)
2. Output quality scoring: Is this artifact good enough to send? (equivalent to recruiter-reviewer)
3. Performance scoring: How well did the user perform in an interaction? (equivalent to Three Laws grading)
4. Progress scoring: Is the overall pipeline healthy? (equivalent to weekly-retro metrics)

For each scoring context, ask the designer:

Step 2: Dimension Design

For each scoring context:

1. What are the 3-5 dimensions? Named, specific evaluation axes.
2. For each dimension, what's the scale? (Recommend 0-10 or 0-20)
3. For each dimension, calibrate with examples:
   • What does 9-10 look like? (Describe a real example)
   • What does 7-8 look like?
   • What does 5-6 look like?
   • What does 3-4 look like?
   • What does 1-2 look like?
4. Are dimensions equally weighted? If not, what shifts the weights? (e.g., user type, context, stage)

Structure into:

## Scoring Context: [Name] (e.g., "Opportunity Triage")

### Dimensions

| Dimension | Scale | Weight (Default) | What It Measures |
|-----------|-------|-------------------|-----------------|
| [X] | 0-20 | [X]% | [one line] |
| [X] | 0-20 | [X]% | [one line] |

### Calibration: [Dimension 1]

| Score | Description | Example |
|-------|-------------|---------|
| 9-10 | [X] | [real example] |
| 7-8 | [X] | [real example] |
| 5-6 | [X] | [real example] |
| 3-4 | [X] | [real example] |
| 1-2 | [X] | [real example] |

Step 3: Thresholds and Recommendations

For each scoring context, define what the total score means:

1. What score means STOP? (Don't pursue, don't send, not ready)
2. What score means PROCEED WITH CAUTION? (Pursue only with a mitigant, needs revision)
3. What score means GO? (Strong match, ready to send, well-prepared)

Ask the designer: "If I score something 72/100, what should the user do?" This calibrates whether the thresholds are right.

### Thresholds

| Range | Recommendation | User Action |
|-------|---------------|-------------|
| Below [X] | SKIP / STOP | [what to do instead] |
| [X]-[Y] | PROCEED WITH CAUTION | [what mitigant is needed] |
| [Y]-100 | GO | [next step] |

### Boundary Cases
- Score of exactly [X]: [which band and why]
- Score of exactly [Y]: [which band and why]

Step 4: Adaptive Weights

Ask the designer if weights shift based on context:

1. By user type / persona: Do early-career users need different weights than veterans? (equivalent to early-career dimension weighting)
2. By sub-domain / function: Does a SWE need different triage weights than a PM?
3. By stage: Do weights change as something moves through the pipeline?
4. By special conditions: Are there override conditions? (equivalent to visa dealbreaker, remote-only penalty)

For each variant:

### Weight Variants

| Condition | Dimension Adjustments | Rationale |
|-----------|----------------------|-----------|
| [User type X] | [Dim A] from 20% → 30%, [Dim B] from 20% → 10% | [why] |
| [Special condition] | Override to SKIP regardless of score | [why] |

Step 5: Score Inflation Guard

Design the sanity check for each scoring context:

1. What combination of low sub-scores should cap the total? (e.g., "if 2+ required dimensions score below 5, total should never exceed 65")
2. What's the realistic score distribution? (Most things should score 50-80. Scores above 90 should be rare.)
3. What's the "too generous" indicator? If Claude is scoring everything above 75, something is wrong.

Output

Write the complete scoring framework to domain-input/scoring-rubrics.md.

Tell the designer:

Scoring rubrics designed:
- [N] scoring contexts: [list]
- [N] total dimensions across all contexts
- Thresholds: SKIP below [X], CAUTION [X]-[Y], GO above [Y]
- [N] weight variants for different user types/conditions

Key calibration check: A [typical example] should score approximately [X]. Does that feel right?

Next: Run /design-identity to define the user context model and persona variants.

Quality Checks

Good scoring rubrics:

• Every dimension has calibrated examples at each score level, not just numbers
• Thresholds produce actionable recommendations, not just labels
• Weight variants are justified with rationale, not arbitrary
• Score inflation guard is defined
• The designer confirmed the calibration feels right against a real example

Bad scoring rubrics:

• Dimensions are vague ("quality," "fit," "alignment")
• No examples — just score ranges with descriptions
• All dimensions equally weighted with no consideration of context
• No score inflation guard
• Thresholds are arbitrary round numbers without calibration