Weighted Scorer

Weighted Scorer — Weighted Scoring Decision Framework

A systematic multi-criteria decision methodology. Suitable for technology selection, vendor evaluation, Build vs Buy decisions, tool comparison, hiring assessments, and any scenario requiring comparison of multiple options across multiple dimensions.

Quick Start

Guide the user through 5 steps:

1. List candidate options (2–7)
2. Define evaluation criteria (4–8)
3. Assign weights (must total 100%)
4. Score each option per criterion (1–5 scale)
5. Calculate weighted totals + sensitivity analysis

The user just needs to say:

"Help me do a technology selection comparing PostgreSQL, MySQL, and MongoDB for our e-commerce order system"

The agent will guide the user step by step through the entire decision process.

I. Full Process Guide

Step 1: Clarify Decision Context

Before scoring, clarify the following (use questions to guide the user):

Hard constraint filtering: If any option fails a dealbreaker criterion (e.g., budget cap, compliance requirement, technical compatibility), eliminate it immediately — it does not enter the scoring stage.

Step 2: Define Evaluation Criteria

Criteria Selection Principles

• MECE principle: Criteria should be mutually exclusive and collectively exhaustive
• Scorability: Each criterion must allow differentiated scoring across options
• Keep it manageable: 4–8 criteria is optimal. Fewer than 4 makes the decision too coarse; more than 8 dilutes signal with noise

Common Criteria Templates

Technology Selection:

Vendor Evaluation:

Build vs Buy:

Step 3: Assign Weights

Weights reflect the relative importance of each criterion. They must total 100%.

Method A: Direct Assignment (Quick)

Best when there are few criteria (≤5) or the decision-maker has a clear sense of priorities.

How: Ask the user to assign a percentage to each criterion, totaling 100%.

Prompt to guide the user:

"Distribute 100 points across these criteria. Give more points to the most important ones. Don't worry about precision — go with your gut, and we can adjust later."

Method B: Rank-Order Allocation (Recommended)

Best when the user doesn't have a strong intuition for numeric weights.

Steps:

1. Ask the user to rank the criteria by importance (most important first)
2. Automatically calculate weights using the ROC (Rank Order Centroid) formula:

Weight for rank k = (1/k + 1/(k+1) + ... + 1/n) / n

where n = total number of criteria, k = rank position (1 = most important)

No manual calculation needed — use scripts/decision_matrix.py to generate automatically.

Method C: Pairwise Comparison (Precise)

Best for team decisions, many criteria (>5), or when you need to reduce subjective bias.

Steps:

1. Create all pairwise combinations of criteria
2. For each pair, judge which is more important (win = 1, loss = 0, tie = 0.5)
3. Each criterion's score = number of wins / total number of comparisons

Example (4 criteria: A, B, C, D):

Result: A=2/6=33%, B=0.5/6=8%, C=3/6=50%, D=0.5/6=8%. Normalize to 100%.

Weight Quality Checks

After assigning weights, run these sanity checks:

• Extremes test: Is the highest-weighted criterion really that much more important than the lowest?
• Discriminating power test: Is any criterion weighted below 5%? If so, consider merging or removing it
• Concentration test: Do the top two criteria account for more than 70% combined? If so, are the remaining criteria truly unimportant?
• Stakeholder test: Would a different stakeholder assign significantly different weights?

Step 4: Score Each Option

Scoring Scale (1–5)

You must define what each score means before scoring begins — otherwise, different scorers apply inconsistent standards:

Scoring Guidelines

• Set anchors first: For each criterion, quickly identify the best (5) and worst (1–2) option, then score the rest
• Score by criterion: Score column by column (not row by row) — this makes cross-option comparison fairer
• Provide rationale: Attach a one-sentence justification to each score to prevent forgetting the basis later
• Check for discrimination: If all options score the same on a criterion, that criterion adds no value — consider removing it

Step 5: Calculate & Analyze

Weighted Total Calculation

Total score for Option X = Σ(weight of criterion i × Option X's score on criterion i)

Output Format

Present results in a standard table:

| Criterion | Weight | Option A | Option B | Option C |
|-----------|--------|----------|----------|----------|
| Feature fit | 30% | 4 (1.20) | 5 (1.50) | 3 (0.90) |
| Performance | 25% | 5 (1.25) | 3 (0.75) | 4 (1.00) |
| Cost | 25% | 3 (0.75) | 4 (1.00) | 5 (1.25) |
| Learning curve | 20% | 4 (0.80) | 2 (0.40) | 3 (0.60) |
| **Weighted Total** | **100%** | **4.00** | **3.65** | **3.75** |
| **Rank** | | **1** | **3** | **2** |

Values in parentheses are the weighted score for that cell (weight × score).

II. Sensitivity Analysis (Critical — Do Not Skip)

Weighted scoring results depend on assumptions about weights and scores. If the gap between 1st and 2nd place is less than 0.3 points (on a 5-point scale), sensitivity analysis is mandatory.

Analysis Methods

1. Weight Perturbation Test

For each key criterion, shift its weight by ±10% (adjust other weights proportionally) and check if the ranking changes:

Original weights: Feature 30%, Performance 25%, Cost 25%, Learning 20%
Test 1: Feature 40%, Performance 21.4%, Cost 21.4%, Learning 17.1%  → Ranking changed?
Test 2: Feature 20%, Performance 28.6%, Cost 28.6%, Learning 22.9%  → Ranking changed?

2. Score Perturbation Test

For any score that was debatable, adjust it by ±1 point and check if the ranking changes.

3. Reversal Point Analysis

Calculate: "How much would a specific weight or score need to change for the 2nd-place option to overtake 1st place?" If only a tiny change is needed, the two options are essentially too close to call.

Conclusion Template

Conclusion: Option A leads with 4.00 points vs. Option C at 3.75 (gap of 0.25). Sensitivity analysis shows that when the "Cost" criterion weight increases to 35% (+10%), Option C overtakes Option A. Recommend reviewing the appropriateness of the Cost weight, or gathering more cost data to improve scoring accuracy.

III. Cognitive Bias Checklist

After completing scoring, check for these common biases:

IV. Script Tool

The scripts/decision_matrix.py script automates calculations:

Features:

• Input criteria, weights, and scores; output a weighted scoring table (Markdown format)
• Automatic sensitivity analysis (weight ±10% perturbation test)
• ROC rank-order method for automatic weight calculation
• Detection of scoring consistency issues (e.g., zero discrimination on a criterion)

Usage:

# Full scoring calculation + sensitivity analysis
python3 scripts/decision_matrix.py --json '{
  "dimensions": ["Features", "Performance", "Cost", "Learning Curve"],
  "weights": [30, 25, 25, 20],
  "options": ["PostgreSQL", "MySQL", "MongoDB"],
  "scores": {
    "PostgreSQL": [4, 5, 3, 4],
    "MySQL":      [5, 3, 4, 2],
    "MongoDB":    [3, 4, 5, 3]
  }
}'

# ROC weight calculation: just provide criteria ranked from most to least important
python3 scripts/decision_matrix.py --roc '["Features", "Performance", "Cost", "Learning Curve"]'

V. Full Example: Database Selection for an E-commerce System

Background

• E-commerce order system, 100K orders per day
• Team of 5, primarily experienced with relational databases
• Limited budget, preference for open-source solutions

Step 1: Candidate Options

PostgreSQL, MySQL, MongoDB (Hard constraint filter: Oracle eliminated due to licensing cost exceeding budget)

Step 2: Criteria Definition

Feature fit, Performance, Scalability, Community & ecosystem, Learning curve, Ops complexity, Cost

Step 3: Weights (Rank-Order Method)

User ranking: Features > Performance > Cost > Learning curve > Community > Ops > Scalability

ROC weights: Features 37.0%, Performance 22.8%, Cost 15.6%, Learning 10.9%, Community 7.3%, Ops 4.4%, Scalability 2.0%

Step 4: Scoring

Step 5: Results

• PostgreSQL: 4.45 points → Rank 1
• MySQL: 4.21 points → Rank 2
• MongoDB: 3.57 points → Rank 3

Sensitivity analysis: The gap between PostgreSQL and MySQL is 0.24 points. Sensitivity analysis shows rankings remain unchanged after ±10% weight perturbation across all criteria. Although the margin is small, the conclusion is not sensitive to weight changes, indicating high confidence.

VI. When to Use & Limitations

Good Fit

• 2–7 candidate options, 4–8 evaluation criteria
• Decision involves multiple participants or requires documentation
• You need to explain "why this option" to stakeholders

Not a Good Fit

• Only 2 options with obvious differences → a simple Pros/Cons list is more efficient
• Decision is dominated by a single factor → no need for a multi-criteria framework
• Information is extremely incomplete → scoring is meaningless; gather information first
• Highly emotional personal decisions → a rational framework may mask true preferences

Common Pitfalls

1. Too many criteria: More than 8 criteria means each carries little weight, reducing discriminating power
2. False precision: 3.72 vs. 3.68 does not mean A is truly better than B — pay attention to sensitivity analysis
3. Ignoring qualitative judgment: The matrix is a decision aid, not a replacement for intuition
4. Set it and forget it: Re-evaluate when decision conditions change

References

• Keeney, R. & Raiffa, H. — Decisions with Multiple Objectives (classic text on multi-objective decision-making)
• Belton, V. & Stewart, T. — Multiple Criteria Decision Analysis (practical MCDA guide)
• ROC weighting method — Barron & Barrett (1996), "Decision quality using ranked attribute weights"