build-scoring

name	build-scoring
description	Design multi-dimensional evaluation rubrics with calibrated scales, thresholds, and function-adaptive weights.

/build-scoring - Scoring Rubric Design

When to Use

• After /define-audiences (scoring rubrics serve the audiences)
• When designing evaluation frameworks for the target OS
• When domain-input/scoring-rubrics.md needs to be filled

Inputs

• Auto-loaded: domain-input/audiences.md (what audiences evaluate)
• Auto-loaded: domain-input/domain-workflow.md (pipeline stages and healthy metrics)
• Auto-loaded: domain-input/domain-knowledge.md (existing domain frameworks)

Process

Step 1: Identify What Gets Scored

From the workflow and audience maps, identify every point where the OS needs to evaluate something:

1. Triage scoring: Should the user pursue this opportunity? (equivalent to job-fit-scorer)
2. Output quality scoring: Is this artifact good enough to send? (equivalent to recruiter-reviewer)
3. Performance scoring: How well did the user perform in an interaction? (equivalent to Three Laws grading)
4. Progress scoring: Is the overall pipeline healthy? (equivalent to weekly-retro metrics)

For each scoring context, ask the designer:

Step 2: Dimension Design

For each scoring context:

1. What are the 3-5 dimensions? Named, specific evaluation axes.
2. For each dimension, what's the scale? (Recommend 0-10 or 0-20)
3. For each dimension, calibrate with examples:
   • What does 9-10 look like? (Describe a real example)
   • What does 7-8 look like?
   • What does 5-6 look like?
   • What does 3-4 look like?
   • What does 1-2 look like?
4. Are dimensions equally weighted? If not, what shifts the weights? (e.g., user type, context, stage)

Structure into:

## Scoring Context: [Name] (e.g., "Opportunity Triage")

### Dimensions

| Dimension | Scale | Weight (Default) | What It Measures |
|-----------|-------|-------------------|-----------------|
| [X] | 0-20 | [X]% | [one line] |
| [X] | 0-20 | [X]% | [one line] |

### Calibration: [Dimension 1]

| Score | Description | Example |
|-------|-------------|---------|
| 9-10 | [X] | [real example] |
| 7-8 | [X] | [real example] |
| 5-6 | [X] | [real example] |
| 3-4 | [X] | [real example] |
| 1-2 | [X] | [real example] |

Principle 10 (Orthogonal Evaluation Axes): If two dimensions are logically independent (e.g., quality vs. urgency, fit vs. legitimacy), score them on separate axes rather than combining into one rubric. Each independent axis gets its own thresholds and recommendations.

Step 3: Thresholds and Recommendations

For each scoring context, define what the total score means:

1. What score means STOP? (Don't pursue, don't send, not ready)
2. What score means PROCEED WITH CAUTION? (Pursue only with a mitigant, needs revision)
3. What score means GO? (Strong match, ready to send, well-prepared)

Ask the designer: "If I score something 72/100, what should the user do?" This calibrates whether the thresholds are right.

### Thresholds

| Range | Recommendation | User Action |
|-------|---------------|-------------|
| Below [X] | SKIP / STOP | [what to do instead] |
| [X]-[Y] | PROCEED WITH CAUTION | [what mitigant is needed] |
| [Y]-100 | GO | [next step] |

### Boundary Cases
- Score of exactly [X]: [which band and why]
- Score of exactly [Y]: [which band and why]

Step 4: Adaptive Weights

Ask the designer if weights shift based on context:

1. By user type / persona: Do early-career users need different weights than veterans? (equivalent to early-career dimension weighting)
2. By sub-domain / function: Does a SWE need different triage weights than a PM?
3. By stage: Do weights change as something moves through the pipeline?
4. By special conditions: Are there override conditions? (equivalent to visa dealbreaker, remote-only penalty)

For each variant:

### Weight Variants

| Condition | Dimension Adjustments | Rationale |
|-----------|----------------------|-----------|
| [User type X] | [Dim A] from 20% → 30%, [Dim B] from 20% → 10% | [why] |
| [Special condition] | Override to SKIP regardless of score | [why] |

Step 5: Score Inflation Guard

Design the sanity check for each scoring context:

1. What combination of low sub-scores should cap the total? (e.g., "if 2+ required dimensions score below 5, total should never exceed 65")
2. What's the realistic score distribution? (Most things should score 50-80. Scores above 90 should be rare.)
3. What's the "too generous" indicator? If Claude is scoring everything above 75, something is wrong.

Output

Write the complete scoring framework to domain-input/scoring-rubrics.md.

Tell the designer:

Scoring rubrics designed:
- [N] scoring contexts: [list]
- [N] total dimensions across all contexts
- Thresholds: SKIP below [X], CAUTION [X]-[Y], GO above [Y]
- [N] weight variants for different user types/conditions

Key calibration check: A [typical example] should score approximately [X]. Does that feel right?

Next: Run /design-identity to define the user context model and persona variants.

Quality Checks

Good scoring rubrics:

• Every dimension has calibrated examples at each score level, not just numbers
• Thresholds produce actionable recommendations, not just labels
• Weight variants are justified with rationale, not arbitrary
• Score inflation guard is defined
• The designer confirmed the calibration feels right against a real example

Bad scoring rubrics:

• Dimensions are vague ("quality," "fit," "alignment")
• No examples — just score ranges with descriptions
• All dimensions equally weighted with no consideration of context
• No score inflation guard
• Thresholds are arbitrary round numbers without calibration