| name | weighted-scorer |
| description | Builds weighted scoring decision matrices for technology selection, vendor evaluation, and multi-criteria option comparisons. Guides you through defining criteria, assigning weights, scoring options, and validating results with sensitivity analysis. Trigger when users ask about decision matrices, weighted scoring, comparing options, vendor selection, or build vs. buy choices. |
| license | MIT |
Weighted Scorer — Weighted Scoring Decision Framework
A systematic multi-criteria decision methodology. Suitable for technology selection, vendor evaluation, Build vs Buy decisions, tool comparison, hiring assessments, and any scenario requiring comparison of multiple options across multiple dimensions.
Quick Start
Guide the user through 5 steps:
- List candidate options (2–7)
- Define evaluation criteria (4–8)
- Assign weights (must total 100%)
- Score each option per criterion (1–5 scale)
- Calculate weighted totals + sensitivity analysis
The user just needs to say:
"Help me do a technology selection comparing PostgreSQL, MySQL, and MongoDB for our e-commerce order system"
The agent will guide the user step by step through the entire decision process.
I. Full Process Guide
Step 1: Clarify Decision Context
Before scoring, clarify the following (use questions to guide the user):
| Question | Purpose |
|---|
| What problem is this decision trying to solve? | Focus on real needs; avoid comparing for comparison's sake |
| What are the candidate options? | Identify what's being compared (recommend 2–7 options) |
| Who are the key stakeholders? | Different roles care about different criteria |
| When does the decision need to be made? | Affects depth of information gathering |
| Are there any dealbreakers? | Identify hard constraints to filter out infeasible options first |
Hard constraint filtering: If any option fails a dealbreaker criterion (e.g., budget cap, compliance requirement, technical compatibility), eliminate it immediately — it does not enter the scoring stage.
Step 2: Define Evaluation Criteria
Criteria Selection Principles
- MECE principle: Criteria should be mutually exclusive and collectively exhaustive
- Scorability: Each criterion must allow differentiated scoring across options
- Keep it manageable: 4–8 criteria is optimal. Fewer than 4 makes the decision too coarse; more than 8 dilutes signal with noise
Common Criteria Templates
Technology Selection:
| Criterion | Description | How to Evaluate |
|---|
| Feature fit | How well it meets core requirements | List requirements and compare item by item |
| Performance | Latency, throughput, concurrency | Benchmark data or official documentation |
| Scalability | Horizontal/vertical scaling capability | Architecture design analysis |
| Community & ecosystem | Documentation, community activity, third-party integrations | GitHub stars/issues, Stack Overflow activity |
| Learning curve | Difficulty for the team to adopt | Match against team's existing skill set |
| Ops complexity | Difficulty of deployment, monitoring, troubleshooting | Operational experience or documentation review |
| Cost | Licensing, infrastructure, personnel costs | TCO (Total Cost of Ownership) estimate |
| Long-term risk | Vendor lock-in, technology obsolescence risk | Open source vs. proprietary, market trends |
Vendor Evaluation:
| Criterion | Description |
|---|
| Product capability | Feature coverage, usability, customizability |
| Pricing & terms | Total cost, payment terms, discount conditions |
| Technical support | SLA, response time, support channels |
| Security & compliance | Data security, certifications, regulatory compliance |
| Integration capability | APIs, SDKs, compatibility with existing systems |
| Company strength | Financial stability, market share, customer references |
Build vs Buy:
| Criterion | Description |
|---|
| Requirements fit | Degree to which customization needs are met |
| Time to launch | Time from decision to production readiness |
| Total cost (3 years) | Development/purchase + maintenance + opportunity cost |
| Team capability match | Whether the team can realistically build it in-house |
| Strategic value | Whether it builds core competitive advantage |
| Maintenance burden | Long-term maintenance effort and complexity |
Step 3: Assign Weights
Weights reflect the relative importance of each criterion. They must total 100%.
Method A: Direct Assignment (Quick)
Best when there are few criteria (≤5) or the decision-maker has a clear sense of priorities.
How: Ask the user to assign a percentage to each criterion, totaling 100%.
Prompt to guide the user:
"Distribute 100 points across these criteria. Give more points to the most important ones. Don't worry about precision — go with your gut, and we can adjust later."
Method B: Rank-Order Allocation (Recommended)
Best when the user doesn't have a strong intuition for numeric weights.
Steps:
- Ask the user to rank the criteria by importance (most important first)
- Automatically calculate weights using the ROC (Rank Order Centroid) formula:
Weight for rank k = (1/k + 1/(k+1) + ... + 1/n) / n
where n = total number of criteria, k = rank position (1 = most important)
| Rank | 4 criteria | 6 criteria | 8 criteria |
|---|
| 1 | 52.1% | 40.8% | 34.0% |
| 2 | 27.1% | 24.2% | 21.5% |
| 3 | 14.6% | 15.8% | 15.2% |
| 4 | 6.3% | 10.3% | 11.1% |
| 5 | — | 6.1% | 7.9% |
| 6 | — | 2.8% | 5.4% |
| 7 | — | — | 3.3% |
| 8 | — | — | 1.6% |
No manual calculation needed — use scripts/decision_matrix.py to generate automatically.
Method C: Pairwise Comparison (Precise)
Best for team decisions, many criteria (>5), or when you need to reduce subjective bias.
Steps:
- Create all pairwise combinations of criteria
- For each pair, judge which is more important (win = 1, loss = 0, tie = 0.5)
- Each criterion's score = number of wins / total number of comparisons
Example (4 criteria: A, B, C, D):
| Comparison | Result |
|---|
| A vs B | A wins |
| A vs C | C wins |
| A vs D | A wins |
| B vs C | C wins |
| B vs D | Tie |
| C vs D | C wins |
Result: A=2/6=33%, B=0.5/6=8%, C=3/6=50%, D=0.5/6=8%. Normalize to 100%.
Weight Quality Checks
After assigning weights, run these sanity checks:
Step 4: Score Each Option
Scoring Scale (1–5)
You must define what each score means before scoring begins — otherwise, different scorers apply inconsistent standards:
| Score | General Meaning | Example (for "Performance" criterion) |
|---|
| 5 | Outstanding; significantly exceeds requirements | Latency <10ms, easily handles 10x expected traffic |
| 4 | Good; meets requirements with headroom | Latency <50ms, handles 3x expected traffic |
| 3 | Adequate; barely meets current requirements | Latency <100ms, handles current traffic |
| 2 | Insufficient; requires additional effort to meet requirements | Latency 100–500ms, needs optimization |
| 1 | Severely lacking; essentially unusable | Latency >500ms, cannot meet basic requirements |
Scoring Guidelines
- Set anchors first: For each criterion, quickly identify the best (5) and worst (1–2) option, then score the rest
- Score by criterion: Score column by column (not row by row) — this makes cross-option comparison fairer
- Provide rationale: Attach a one-sentence justification to each score to prevent forgetting the basis later
- Check for discrimination: If all options score the same on a criterion, that criterion adds no value — consider removing it
Step 5: Calculate & Analyze
Weighted Total Calculation
Total score for Option X = Σ(weight of criterion i × Option X's score on criterion i)
Output Format
Present results in a standard table:
| Criterion | Weight | Option A | Option B | Option C |
|-----------|--------|----------|----------|----------|
| Feature fit | 30% | 4 (1.20) | 5 (1.50) | 3 (0.90) |
| Performance | 25% | 5 (1.25) | 3 (0.75) | 4 (1.00) |
| Cost | 25% | 3 (0.75) | 4 (1.00) | 5 (1.25) |
| Learning curve | 20% | 4 (0.80) | 2 (0.40) | 3 (0.60) |
| **Weighted Total** | **100%** | **4.00** | **3.65** | **3.75** |
| **Rank** | | **1** | **3** | **2** |
Values in parentheses are the weighted score for that cell (weight × score).
II. Sensitivity Analysis (Critical — Do Not Skip)
Weighted scoring results depend on assumptions about weights and scores. If the gap between 1st and 2nd place is less than 0.3 points (on a 5-point scale), sensitivity analysis is mandatory.
Analysis Methods
1. Weight Perturbation Test
For each key criterion, shift its weight by ±10% (adjust other weights proportionally) and check if the ranking changes:
Original weights: Feature 30%, Performance 25%, Cost 25%, Learning 20%
Test 1: Feature 40%, Performance 21.4%, Cost 21.4%, Learning 17.1% → Ranking changed?
Test 2: Feature 20%, Performance 28.6%, Cost 28.6%, Learning 22.9% → Ranking changed?
2. Score Perturbation Test
For any score that was debatable, adjust it by ±1 point and check if the ranking changes.
3. Reversal Point Analysis
Calculate: "How much would a specific weight or score need to change for the 2nd-place option to overtake 1st place?" If only a tiny change is needed, the two options are essentially too close to call.
Conclusion Template
Conclusion: Option A leads with 4.00 points vs. Option C at 3.75 (gap of 0.25). Sensitivity analysis shows that when the "Cost" criterion weight increases to 35% (+10%), Option C overtakes Option A. Recommend reviewing the appropriateness of the Cost weight, or gathering more cost data to improve scoring accuracy.
III. Cognitive Bias Checklist
After completing scoring, check for these common biases:
| Bias | Manifestation | Countermeasure |
|---|
| Anchoring | The first option evaluated tends to score higher | After scoring all options, revisit the first one |
| Halo effect | Positive overall impression inflates all criterion scores | Score by criterion (column), not by option (row) |
| Status quo bias | Tendency to favor the option currently in use | Imagine choosing from scratch, ignoring migration costs |
| Confirmation bias | Starting with a conclusion, then finding evidence to support it | Have people with different perspectives score independently and compare |
| Sunk cost fallacy | Reluctance to switch because of prior investment | Clearly separate "past investment" from "future value" |
| Availability bias | Recent positive/negative news about an option skews judgment | Require objective evidence for every score |
IV. Script Tool
The scripts/decision_matrix.py script automates calculations:
Features:
- Input criteria, weights, and scores; output a weighted scoring table (Markdown format)
- Automatic sensitivity analysis (weight ±10% perturbation test)
- ROC rank-order method for automatic weight calculation
- Detection of scoring consistency issues (e.g., zero discrimination on a criterion)
Usage:
python3 scripts/decision_matrix.py --json '{
"dimensions": ["Features", "Performance", "Cost", "Learning Curve"],
"weights": [30, 25, 25, 20],
"options": ["PostgreSQL", "MySQL", "MongoDB"],
"scores": {
"PostgreSQL": [4, 5, 3, 4],
"MySQL": [5, 3, 4, 2],
"MongoDB": [3, 4, 5, 3]
}
}'
python3 scripts/decision_matrix.py --roc '["Features", "Performance", "Cost", "Learning Curve"]'
V. Full Example: Database Selection for an E-commerce System
Background
- E-commerce order system, 100K orders per day
- Team of 5, primarily experienced with relational databases
- Limited budget, preference for open-source solutions
Step 1: Candidate Options
PostgreSQL, MySQL, MongoDB (Hard constraint filter: Oracle eliminated due to licensing cost exceeding budget)
Step 2: Criteria Definition
Feature fit, Performance, Scalability, Community & ecosystem, Learning curve, Ops complexity, Cost
Step 3: Weights (Rank-Order Method)
User ranking: Features > Performance > Cost > Learning curve > Community > Ops > Scalability
ROC weights: Features 37.0%, Performance 22.8%, Cost 15.6%, Learning 10.9%, Community 7.3%, Ops 4.4%, Scalability 2.0%
Step 4: Scoring
| Criterion | Weight | PostgreSQL | MySQL | MongoDB |
|---|
| Feature fit | 37.0% | 5 | 4 | 3 |
| Performance | 22.8% | 4 | 4 | 5 |
| Cost | 15.6% | 5 | 5 | 4 |
| Learning curve | 10.9% | 3 | 4 | 2 |
| Community & ecosystem | 7.3% | 5 | 5 | 4 |
| Ops complexity | 4.4% | 3 | 4 | 2 |
| Scalability | 2.0% | 4 | 3 | 5 |
Step 5: Results
- PostgreSQL: 4.45 points → Rank 1
- MySQL: 4.21 points → Rank 2
- MongoDB: 3.57 points → Rank 3
Sensitivity analysis: The gap between PostgreSQL and MySQL is 0.24 points. Sensitivity analysis shows rankings remain unchanged after ±10% weight perturbation across all criteria. Although the margin is small, the conclusion is not sensitive to weight changes, indicating high confidence.
VI. When to Use & Limitations
Good Fit
- 2–7 candidate options, 4–8 evaluation criteria
- Decision involves multiple participants or requires documentation
- You need to explain "why this option" to stakeholders
Not a Good Fit
- Only 2 options with obvious differences → a simple Pros/Cons list is more efficient
- Decision is dominated by a single factor → no need for a multi-criteria framework
- Information is extremely incomplete → scoring is meaningless; gather information first
- Highly emotional personal decisions → a rational framework may mask true preferences
Common Pitfalls
- Too many criteria: More than 8 criteria means each carries little weight, reducing discriminating power
- False precision: 3.72 vs. 3.68 does not mean A is truly better than B — pay attention to sensitivity analysis
- Ignoring qualitative judgment: The matrix is a decision aid, not a replacement for intuition
- Set it and forget it: Re-evaluate when decision conditions change
References
- Keeney, R. & Raiffa, H. — Decisions with Multiple Objectives (classic text on multi-objective decision-making)
- Belton, V. & Stewart, T. — Multiple Criteria Decision Analysis (practical MCDA guide)
- ROC weighting method — Barron & Barrett (1996), "Decision quality using ranked attribute weights"
