Evaluative Research

Evaluative Research Methodology

A structured approach to evaluating options and making evidence-based decisions between alternatives. Use this methodology when the core question is "which option should we choose?" rather than "what happened?" or "how does this work?"

When to Use

Use this methodology when you need to:

• Compare libraries, frameworks, or services for a specific use case
• Evaluate migration paths between technologies
• Assess build vs. buy decisions
• Choose between architectural approaches
• Analyze trade-offs between competing solutions
• Conduct technology landscape analysis

Key indicator: There are multiple viable options and you need a structured way to decide between them.

Not for: Debugging or root cause analysis (use investigation-methodology), finding what's missing from a spec (use gap-analysis), or understanding how something works (use general research).

Evaluative Research Framework

Phase 1: Define the Decision

Before comparing options, articulate what you're deciding:

• Decision statement: Frame as a clear question ("Which editor library should we use for rich text editing?")
• Decision drivers: What matters most? (performance, DX, community, maintenance, cost, time-to-implement)
• Constraints: Non-negotiable requirements that eliminate options early
• Context: Current architecture, team expertise, existing patterns
• Stakeholders: Who is affected by this decision?

Stopping point: You can state the decision, its drivers, and constraints in 2-3 sentences.

Phase 2: Landscape Scan

Cast a wide net to identify all viable options:

• Identify candidates: Research the space broadly before narrowing
• Quick-filter: Eliminate options that violate hard constraints
• Categorize: Group options by approach (e.g., "full framework" vs. "lightweight library" vs. "build custom")
• Select finalists: 2-4 options for deep evaluation (more than 4 becomes unwieldy)

Sources to check:

• Official documentation and getting-started guides
• GitHub: stars, issues, commit frequency, release cadence
• npm/package registry: download trends, bundle size, dependency count
• Community: Stack Overflow activity, Discord/forum presence
• Existing codebase: similar patterns already implemented

Stopping point: You have 2-4 finalists and can explain why others were eliminated.

Phase 3: Deep Evaluation

Evaluate each finalist against your decision drivers:

• Hands-on assessment: Read docs, review API surface, check examples
• Codebase fit: How does each option integrate with existing architecture?
• Effort estimate: What does adoption look like? (learning curve, migration effort, boilerplate)
• Risk assessment: What could go wrong? (abandonment, breaking changes, scaling limits)
• Long-term trajectory: Is the project growing, stable, or declining?

For each option, document:

• Description (what it is, approach it takes)
• Strengths (evidence-backed, not just marketing)
• Weaknesses (real limitations, not hypothetical)
• Effort estimate (Low/Medium/High with rationale)
• Risk level (Low/Medium/High with rationale)

Stopping point: You can articulate the genuine trade-offs between finalists.

Phase 4: Comparative Analysis

Synthesize findings into a decision:

• Build comparison matrix: Rate each option against each decision driver
• Identify key trade-offs: What do you give up with each choice?
• Consider second-order effects: How does each option affect adjacent systems?
• Weight by priority: Not all criteria are equal — weight by what matters most
• Formulate recommendation: Clear primary recommendation with confidence level

Comparison Matrix Format:

Stopping point: You have a clear recommendation with evidence-backed rationale.

Output Format

Structure evaluative research outputs as:

# Investigation: [Decision Topic]

## Summary

[2-3 sentences: what decision was needed, what was evaluated, what's
recommended]

## Decision Context

- **Decision**: [Clear question being answered]
- **Drivers**: [What matters most, in priority order]
- **Constraints**: [Hard requirements]
- **Current state**: [Relevant existing architecture/tools]

## Options Evaluated

### Option 1: [Name]

**What it is**: [Brief description] **Strengths**:

- [Evidence-backed strength]

**Weaknesses**:

- [Evidence-backed weakness]

**Effort**: [Low/Medium/High] — [rationale] **Risk**: [Low/Medium/High] —
[rationale]

[Repeat for each option]

### Options Eliminated Early

- [Option X]: Eliminated because [reason]

## Comparison

| Criteria | Option 1 | Option 2 | Option 3 |
| -------- | -------- | -------- | -------- |
| ...      | ...      | ...      | ...      |

### Key Trade-offs

[The 2-3 most significant trade-offs between the top options]

## Recommendation

**Primary**: [Option name] — [Confidence: High/Medium/Low]

**Rationale**: [Why this option, tied to decision drivers and evidence]

**When to reconsider**: [Under what circumstances would you choose differently?]

## Next Steps

1. [Immediate action]
2. [Validation step]
3. [Follow-up if needed]

## References

- [Links to docs, benchmarks, repos evaluated]

Methodological Principles

Evidence over opinion: Every rating in the comparison matrix should trace back to something concrete — documentation, benchmarks, code review, or real-world usage data.

Honest about uncertainty: If you can't properly evaluate a criterion, say so. "Unable to assess without prototyping" is better than a guess.

Decision drivers first: Start from what matters, not from what's easy to compare. Don't let available data drive the criteria.

Disconfirm actively: For your recommended option, try hardest to find its weaknesses. For the runner-up, try hardest to find its strengths.

State the trade-off clearly: The best decisions acknowledge what you're giving up, not just what you're gaining.

Quality Checklist

Before concluding evaluative research:

• Decision statement is clear and specific
• Decision drivers reflect actual priorities (not generic criteria)
• At least 2-4 options were seriously evaluated
• Eliminated options have documented rationale
• Strengths and weaknesses are evidence-backed
• Comparison matrix uses weighted criteria
• Key trade-offs are explicitly stated
• Recommendation includes confidence level
• "When to reconsider" conditions are specified
• References are included for verification

Scaling to Complexity

Quick evaluation (clear winner, limited scope):

• 2-3 options, 3-4 criteria
• Focus on the one criterion that differentiates
• Stop when one option clearly dominates

Standard evaluation (multiple good options):

• 3-4 options, 5-7 criteria
• Full comparison matrix with weights
• Stop when trade-offs are clear and recommendation is defensible

Strategic evaluation (high-stakes, long-term impact):

• 3-5 options, 7+ criteria
• Prototype or proof-of-concept for top candidates
• Stop when you have high confidence or documented why certainty isn't possible