paulpas Framework Evaluation Criteria
Structures and elicits evaluation criteria across eight dimensions to prevent hype-driven framework selection. When loaded, this skill makes the model act as a senior engineering evaluator — systematically extracting requirements, separating non-negotiables from scored criteria, building weighted evaluation matrices, analyzing failure modes, and producing decision records with documented trade-offs for any technology comparison.
TL;DR Checklist
- Elicit requirements across all 8 dimensions (technical, team skills, ecosystem, security, deployment, integration, cost, viability) — never skip a dimension
- Separate knock-out criteria from scored criteria before evaluating any candidate
- Normalize weights so they sum to exactly 1.0 across all scoring criteria
- Run failure mode analysis on the top candidate(s) before finalizing
- Document the decision with an ADR including context, drivers, weighted scores, and consequences
- Involve at least two stakeholders in the evaluation — no solo framework selections
- Record "why not" for rejected candidates, not just "why" for the winner
- Validate knock-out criteria by running concrete verification steps (not opinions)
When to Use
Use this skill when:
- Starting a framework comparison before scoring any candidates (this is always step zero)
- Selecting a new library or framework for a project with long-term maintenance needs
- During a technology spike where you need structured evaluation criteria up front
- Leading an architectural review that includes technology stack decisions
- The team has conflicting preferences and needs objective, requirement-driven criteria to resolve disagreements
- Before adopting a framework recommended solely because it is popular, new, or used at another company
- When migrating from an existing framework and need criteria to evaluate replacement candidates
- For build-vs-buy decisions where you must assess whether a framework fits your specific constraints
When NOT to Use
Avoid this skill for:
- During active implementation of a framework already selected — use
framework-utilizationinstead - For throwaway prototypes or weekend projects where framework choice is irrelevant
- When requirements are already fully defined by another process or skill (go back to those, do not re-elicitation)
- For one-off script decisions where the same tool would work equally well (overhead outweighs benefit)
Core Workflow
Step 1: Elicit Requirements Across All 8 Dimensions
Systematically go through each evaluation dimension. For each dimension, write down specific, verifiable requirements — not preferences. Use the questions listed in the Evaluation Dimensions section below to guide elicitation. Checkpoint: Every dimension must have at least one requirement written down before proceeding. If a dimension is genuinely irrelevant (e.g., cost for an internal open-source tool), document why it was excluded explicitly.
Step 2: Identify Knock-out Criteria
From the requirements collected in Step 1, identify which ones are non-negotiables. A knock-out criterion disqualifies any framework that fails it — there is no partial credit. Examples: "Must support Python 3.10+", "Must use an OSI-approved license", "Must run on Linux x86_64". Checkpoint: Every knock-out criterion must have a concrete, testable verification method (e.g., python -c "import framework; print(framework.__version__)").
Step 3: Construct Weighted Evaluation Matrix
Assign normalized weights to all non-knock-out criteria. Weights must sum to exactly 1.0. Use the Importance enum (LOW=1 through ESSENTIAL=5) to assign raw importance, then normalize. Score each candidate framework against every criterion on a 1–10 scale with evidence — no gut feelings. Checkpoint: The weighted sum for each candidate must be calculated and all candidates must pass knock-out checks before scoring is considered complete.
Step 4: Analyze Failure Modes
For the top-scoring candidate(s), run failure mode analysis against all six documented failure modes. For each failure mode, determine whether the risk is present, its likelihood, and what mitigation applies. Checkpoint: If any high-severity failure mode is identified for the leading candidate with no viable mitigation, reconsider the selection before proceeding.
Step 5: Document Decision with ADR
Produce a structured Architecture Decision Record in MADR format (see Decision Documentation Patterns below). The ADR must include context, decision drivers, knock-out criteria results, weighted scores for all candidates, and positive/negative consequences of the chosen direction. Checkpoint: All stakeholders have reviewed the ADR before status moves from proposed to accepted.
Evaluation Dimensions (The Core Content)
Each dimension below describes what to assess, key questions to answer, common pitfalls, and a real example drawn from actual framework selection decisions.
1. Technical Capability Fit
Assess how well the framework's features align with your project's functional requirements. This is the most commonly evaluated dimension but also the most often done incompletely — teams compare feature lists without considering depth of capability.
Key questions:
- Does the framework support all required use cases, or only a subset?
- How deep is the API surface for each feature you need? (Can it do simple things easily AND complex things if needed?)
- What is the framework's extensibility model? Can you add features not built in?
- Are there performance characteristics that could become bottlenecks?
Common pitfalls:
- "Feature parity" thinking — assuming a feature exists because other frameworks have it
- Not testing the specific use case under realistic conditions (a POC exercise prevents this)
- Over-valuing "batteries included" when your project needs only one or two components
Example: When the Playwright team evaluated migrating from Selenium, they didn't just check "can it automate a browser?" — they assessed specific capabilities: auto-wait mechanisms, network interception, multi-tab handling, and mobile emulation. The deep API surface for Playwright's locator strategies (CSS, text, role, test-id) was a decisive factor because their tests needed multiple reliable targeting strategies.
2. Team Skills & Learning Curve
Evaluate the team's existing knowledge of the framework and estimate the time-to-productivity for unfamiliar ones. This dimension is frequently underestimated — it is one of the top causes of framework selection failure. A technically superior framework that your team cannot use effectively is worse than a good-enough framework they can master quickly.
Key questions:
- What fraction of the team has production experience with this framework?
- How many weeks until a mid-level developer on the team is productive (not expert)?
- Are there training resources, books, or community support for learning gaps?
- Is knowledge concentrated in one person (bus factor risk)?
Common pitfalls:
- Assuming "everyone can pick it up quickly" without evidence
- Ignoring the learning curve impact on delivery timelines for the current sprint/quarter
- Not distinguishing between initial learning and mastery depth
Example: When a team chose Go over Rust for a networking service, the deciding factor was that several engineers already had Go experience from internal tooling. The decision sustainability criteria explicitly noted: "The team can contribute meaningfully within 1–2 weeks without external training." A framework with zero team familiarity would require 40–80 hours of ramp-up per developer.
3. Ecosystem & Maintainership
Assess the health and trajectory of the framework's ecosystem — not just current activity but sustainability. A framework with a dead repository is a ticking time bomb regardless of how good it is today.
Key questions:
- What is the release cadence? (Weekly, monthly, quarterly, annually?)
- How many contributors are active in the last 90 days?
- How large and healthy is the third-party package/plugin ecosystem?
- Is documentation current and does it include migration guides from popular alternatives?
Common pitfalls:
- Looking at total GitHub stars instead of recent activity (stars don't equal health)
- Assuming a well-known framework name guarantees active maintenance
- Not checking if the primary maintainer is employed by a company that could abandon it
Example: The ThoughtWorks Technology Radar categorizes frameworks into "Assess," "Trial," and "Adopt" rings precisely to surface ecosystem maturity. A framework in "Trial" might have strong technical merit but insufficient production validation or community breadth. The radar approach explicitly rewards frameworks that demonstrate real-world adoption beyond the core team.
4. Security Posture
Evaluate the framework's security track record, dependency graph risk, and licensing compliance. This dimension should not be an afterthought — security flaws in a foundational framework cascade into every module that uses it.
Key questions:
- What is the CVE history? Are there known unpatched vulnerabilities?
- How deep is the dependency tree? (More dependencies = larger attack surface)
- Does the framework verify signatures on published artifacts?
- Is the license OSI-approved and compatible with your project's distribution model?
Common pitfalls:
- Assuming well-known frameworks have no security issues ("big projects can't be vulnerable")
- Ignoring transitive dependency risks (the framework itself may be clean, but its dependencies might not be)
- Not checking if the license has copyleft clauses that affect your distribution
Example: When selecting a CSS framework for a public-facing application, the team evaluated whether Tailwind CSS had any known XSS vectors in its utility class processing. They also verified that its MIT license was compatible with distributing compiled stylesheets without source disclosure — a concern with GPL-licensed alternatives.
5. Deployment Environment Fit
Assess how well the framework integrates with your deployment infrastructure, resource constraints, and operational requirements. A framework that works beautifully on your local machine may fail in production due to platform incompatibilities or resource demands.
Key questions:
- Does the framework support your target platforms (Linux, Windows, macOS, containerized)?
- What are the minimum and recommended resource requirements?
- Does it support containerization (Docker images, multi-stage builds)?
- Are there built-in monitoring/integration hooks (OpenTelemetry, Prometheus metrics, structured logging)?
Common pitfalls:
- Assuming cross-platform because marketing says so (verify with actual CI tests)
- Ignoring startup time requirements for serverless or edge deployment scenarios
- Not evaluating whether the framework's logging format matches your log aggregation pipeline
Example: A team evaluated Next.js versus a custom React SSR setup for deployment to Vercel and AWS Lambda. The decision hinged on Next.js's built-in support for edge runtime, automatic code splitting, and Image component optimization — all critical for their serverless deployment target with cold-start sensitivity.
6. Integration Requirements
Evaluate how the framework connects with your existing systems, data formats, authentication patterns, and CI/CD pipelines. Frameworks don't exist in isolation — integration friction is a real cost that compounds over time.
Key questions:
- Does the framework's API align with existing service contracts (REST, GraphQL, gRPC)?
- What data formats does it natively support (JSON, Protocol Buffers, Avro)?
- Can it integrate with your authentication system (OAuth2, JWT, SAML, LDAP)?
- Is there CI/CD plugin or CLI tooling for automated testing and deployment?
Common pitfalls:
- Assuming standard protocols "just work" without verifying specific version compatibility
- Overlooking auth pattern alignment (a framework may support OAuth2 but not the specific grant flow your identity provider uses)
- Not evaluating whether the framework's test runner integrates with your existing CI pipeline
Example: When selecting a GraphQL client, Apollo Client won partly because it integrated cleanly with their existing JWT-based auth flow and had a caching strategy compatible with their Redux state management. The alternative frameworks either required significant adapter code or lacked the specific cache configuration patterns the team needed.
7. Cost & Total Cost of Ownership
Go beyond licensing fees to calculate the true cost: infrastructure, training, support contracts, and opportunity cost of developer time spent on framework-related work versus business logic.
Key questions:
- What is the licensing model? (MIT, Apache 2.0, GPL, commercial, dual-license?)
- Does the framework increase infrastructure costs (e.g., requires more compute, specialized hardware)?
- Will you need paid support contracts or managed service subscriptions?
- What is the opportunity cost — how much developer time will framework mastery vs. business development consume?
Common pitfalls:
- Focusing only on license fees while ignoring infrastructure and training costs
- Assuming all open-source licenses are equally "free" (GPL copyleft has real legal costs)
- Not estimating support burden (a framework with no commercial backing means your team absorbs all support)
Example: PostgreSQL was selected over a proprietary database partly because the total cost of ownership — including zero licensing fees, existing team expertise, and active community support — was dramatically lower despite comparable performance. The absence of vendor lock-in meant future negotiation leverage on any managed service pricing.
8. Long-Term Viability
Assess whether the framework is likely to remain relevant, maintained, and compatible with your roadmap over the expected lifecycle of the project (typically 3–7 years). A framework's trajectory matters as much as its current features.
Key questions:
- Is there a public roadmap aligned with your projected needs?
- What is the adoption trend — growing, plateaued, or declining?
- How complex would an exit strategy be if you need to migrate away later?
- Is the license stable (no history of license changes)?
Common pitfalls:
- Choosing a framework because it has 50k GitHub stars when the star count peaked two years ago
- Ignoring the migration cost — a framework with deep coupling makes exit difficult
- Assuming current maintainer commitment guarantees future stability (check who pays for development)
Example: The adr/adr decision sustainability criteria explicitly evaluate: "Will this framework likely still be maintained in 3 years?" and "What is our exit path?" A framework backed by a single startup with no alternative funding source received lower viability scores than one with multiple contributing companies, regardless of feature parity.
Criteria Data Structure
Represent evaluation criteria programmatically to enable consistent scoring and automated matrix calculation:
from dataclasses import dataclass, field
from enum import IntEnum
from typing import Optional
class Importance(IntEnum):
"""Importance levels for evaluation criteria. Higher = more weight."""
LOW = 1
MODERATE = 2
HIGH = 3
CRITICAL = 4
ESSENTIAL = 5
@dataclass
class EvaluationCriterion:
"""A single evaluation criterion for framework selection."""
name: str
dimension: str # One of the 8 dimensions
importance: Importance
weight: float # Normalized weight (sums to 1.0 across all scoring criteria)
is_knockout: bool = False
description: str = ""
validation_method: Optional[str] = None # How to verify compliance
def __post_init__(self):
if self.is_knockout and self.weight != 0.0:
raise ValueError(
f"Knock-out criterion '{self.name}' must have weight=0. "
"Knock-outs disqualify; scoring criteria receive weights."
)
if self.is_knockout and not self.validation_method:
raise ValueError(
f"Knock-out criterion '{self.name}' requires a validation_method. "
"A knock-out must be concretely testable."
)
@dataclass
class FrameworkCandidate:
"""A framework being evaluated against all criteria."""
name: str
version: str
license_type: str
criteria_scores: dict[str, float] = field(default_factory=dict) # criterion_name -> score (1-10)
knockout_failures: list[str] = field(default_factory=list)
@property
def weighted_score(self) -> float:
"""Calculate total weighted score. Only scoring criteria contribute."""
return sum(
self.criteria_scores.get(c.name, 0.0) * c.weight
for c in FrameworkEvaluator.all_criteria
)
@property
def is_viable(self) -> bool:
"""Check if framework passes all knock-out criteria."""
return len(self.knockout_failures) == 0
def add_score(self, criterion_name: str, score: float, max_score: float = 10.0) -> None:
"""Add a scored criterion result (clamped to valid range)."""
if not (1.0 <= score <= max_score):
raise ValueError(f"Score {score} for '{criterion_name}' must be between 1 and {max_score}")
self.criteria_scores[criterion_name] = round(score / max_score * 10, 2)
def add_knockout_failure(self, criterion_name: str, reason: str) -> None:
"""Record a knock-out failure with explanatory context."""
self.knockout_failures.append(f"{criterion_name}: {reason}")
class FrameworkEvaluator:
"""Framework evaluation engine implementing the weighted matrix pattern."""
# Dimension groupings for structured elicitation
DIMENSIONS = {
"technical_capability": [
"Core feature completeness",
"API design quality",
"Performance characteristics",
"Extensibility points",
],
"team_skills": [
"Current team familiarity",
"Learning curve estimate",
"Training resource availability",
"Knowledge distribution risk",
],
"ecosystem": [
"Active development velocity",
"Community size",
"Third-party package ecosystem",
"Documentation quality",
],
"security": [
"CVE history",
"Dependency graph depth",
"Code signing practices",
"Supply chain security",
],
"deployment_fit": [
"Platform compatibility",
"Resource requirements",
"Containerization support",
"Monitoring integrations",
],
"integration": [
"API compatibility with existing systems",
"Data format support",
"Auth pattern alignment",
"CI/CD pipeline integration",
],
"cost": [
"Licensing model",
"Infrastructure cost impact",
"Training cost",
"Support contract needs",
],
"long_term_viability": [
"Roadmap alignment",
"Adoption trend",
"Exit strategy complexity",
"License stability",
],
}
all_criteria: list[EvaluationCriterion] = []
@classmethod
def normalize_weights(cls, raw_importances: dict[str, Importance]) -> dict[str, float]:
"""Convert importance scores to normalized weights summing to 1.0."""
total = sum(imp.value for imp in raw_importances.values())
if total == 0:
raise ValueError("Sum of importance values cannot be zero")
return {name: round(imp.value / total, 4) for name, imp in raw_importances.items()}
@classmethod
def run_evaluation(
cls,
criteria: list[EvaluationCriterion],
candidates: list[FrameworkCandidate],
) -> dict[str, FrameworkCandidate]:
"""Run complete evaluation and return scored candidates sorted by score descending."""
cls.all_criteria = criteria
results: dict[str, FrameworkCandidate] = {}
for candidate in candidates:
# Knock-out check first — non-negotiables are binary pass/fail
if not candidate.is_viable:
continue # Excluded entirely
# Calculate weighted score from scoring criteria only
candidate.weighted_score = sum(
candidate.criteria_scores.get(c.name, 0.0) * c.weight
for c in criteria
)
results[candidate.name] = candidate
return dict(
sorted(results.items(), key=lambda item: item[1].weighted_score, reverse=True)
)
@classmethod
def generate_summary(cls, candidates: list[FrameworkCandidate]) -> str:
"""Generate a human-readable evaluation summary table."""
lines = [
"=" * 70,
"FRAMEWORK EVALUATION SUMMARY",
"=" * 70,
f"{'Candidate':<25} {'Version':<12} {'Score':>8} {'Viable':>6}",
"-" * 70,
]
for candidate in sorted(candidates, key=lambda c: getattr(c, "weighted_score", 0), reverse=True):
score = getattr(candidate, "weighted_score", 0)
viable = "Yes" if candidate.is_viable else "No"
lines.append(f"{candidate.name:<25} {candidate.version:<12} {score:>8.3f} {viable:>6}")
if not candidate.is_viable:
for failure in candidate.knockout_failures:
lines.append(f" ❌ Knock-out failed: {failure}")
lines.append("=" * 70)
return "\n".join(lines)
Example: Evaluating Three Framework Candidates
# Define knock-out criteria (weight=0, disqualify on failure)
knockout_criteria = [
EvaluationCriterion(
name="OSI-approved license",
dimension="security",
importance=Importance.ESSENTIAL,
weight=0.0,
is_knockout=True,
description="Must use an OSI-approved open-source license (MIT, Apache 2.0, BSD)",
validation_method="Check LICENSE file in repository for SPDX identifier",
),
EvaluationCriterion(
name="Python 3.10+ support",
dimension="technical_capability",
importance=Importance.CRITICAL,
weight=0.0,
is_knockout=True,
description="Must support Python 3.10 or higher (project minimum)",
validation_method="pip install <framework> && python -c 'import framework; print(framework.__version__)'",
),
]
# Define scoring criteria (with normalized weights)
scoring_criteria = [
EvaluationCriterion(
name="Core feature completeness",
dimension="technical_capability",
importance=Importance.HIGH,
weight=0.15, # Normalized: 3 / (3+2+2+...+1) = 0.15
description="Does the framework cover all required use cases with sufficient depth?",
),
EvaluationCriterion(
name="Team familiarity",
dimension="team_skills",
importance=Importance.HIGH,
weight=0.12,
description="Current team's production experience level with this framework",
),
EvaluationCriterion(
name="Active development velocity",
dimension="ecosystem",
importance=Importance.CRITICAL,
weight=0.18,
description="Release cadence and active contributor count in last 90 days",
),
EvaluationCriterion(
name="Documentation quality",
dimension="ecosystem",
importance=Importance.MODERATE,
weight=0.08,
description="Clarity, completeness, and searchability of documentation",
),
EvaluationCriterion(
name="Deployment compatibility",
dimension="deployment_fit",
importance=Importance.HIGH,
weight=0.13,
description="Supports containerization and target deployment platforms",
),
EvaluationCriterion(
name="Integration ease",
dimension="integration",
importance=Importance.MODERATE,
weight=0.09,
description="Fits with existing auth, data formats, and CI/CD pipeline",
),
EvaluationCriterion(
name="Total cost of ownership",
dimension="cost",
importance=Importance.HIGH,
weight=0.12,
description="Licensing + infrastructure + training costs over 3-year horizon",
),
EvaluationCriterion(
name="Long-term viability",
dimension="long_term_viability",
importance=Importance.CRITICAL,
weight=0.13,
description="Adoption trend, roadmap stability, and exit strategy complexity",
),
]
# Verify weights sum to 1.0
total_weight = sum(c.weight for c in scoring_criteria)
assert total_weight == 1.0, f"Weights must sum to 1.0, got {total_weight}"
# Define candidates
candidates = [
FrameworkCandidate(name="FastAPI", version="0.104.1", license_type="MIT"),
FrameworkCandidate(name="Django", version="4.2.3", license_type="BSD-3-Clause"),
FrameworkCandidate(name="Flask", version="3.0.0", license_type="BSD-3-Clause"),
]
# Score each candidate (1-10 scale, evidence-based)
# FastAPI scores
fastapi = candidates[0]
fastapi.add_score("Core feature completeness", 9.0)
fastapi.add_score("Team familiarity", 5.0) # Team new to it
fastapi.add_score("Active development velocity", 9.0)
fastapi.add_score("Documentation quality", 9.0)
fastapi.add_score("Deployment compatibility", 8.0)
fastapi.add_score("Integration ease", 7.0)
fastapi.add_score("Total cost of ownership", 8.0)
fastapi.add_score("Long-term viability", 8.0)
# Django scores
django = candidates[1]
django.add_score("Core feature completeness", 9.5)
django.add_score("Team familiarity", 7.0) # Moderate experience
django.add_score("Active development velocity", 8.0)
django.add_score("Documentation quality", 10.0)
django.add_score("Deployment compatibility", 8.0)
django.add_score("Integration ease", 8.0)
django.add_score("Total cost of ownership", 6.0) # Heavier, more infra cost
django.add_score("Long-term viability", 9.5)
# Flask scores
flask = candidates[2]
flask.add_score("Core feature completeness", 7.0)
flask.add_score("Team familiarity", 8.0) # High experience
flask.add_score("Active development velocity", 7.0)
flask.add_score("Documentation quality", 7.0)
flask.add_score("Deployment compatibility", 9.0)
flask.add_score("Integration ease", 9.0) # Minimal, fits anything
flask.add_score("Total cost of ownership", 7.0)
flask.add_score("Long-term viability", 8.5)
# Run evaluation
results = FrameworkEvaluator.run_evaluation(scoring_criteria, candidates)
print(FrameworkEvaluator.generate_summary(candidates))
Decision Documentation Patterns
Use the MADR (Milestone-driven Architecture Decision Records) template adapted for framework selection decisions. Every framework comparison must produce an ADR — even if the decision is to stay with the current framework.
ADR Generation Class
from datetime import date
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class FrameworkDecisionADR:
"""Architecture Decision Record for framework selection."""
title: str
status: str = "proposed" # proposed | rejected | accepted | deprecated | superseded
deciders: list[str] = field(default_factory=list)
date: str = field(default_factory=lambda: date.today().isoformat())
context_description: str = ""
decision_drivers: list[str] = field(default_factory=list)
# Evaluation data
knock_out_criteria: list[str] = field(default_factory=list)
weighted_criteria: dict[str, float] = field(default_factory=dict)
# Results
candidates_evaluated: list[str] = field(default_factory=list)
selected_framework: Optional[str] = None
positive_consequences: list[str] = field(default_factory=list)
negative_consequences: list[str] = field(default_factory=list)
@property
def markdown(self) -> str:
"""Generate the ADR in MADR-compatible markdown format."""
sections = [
f"# {self.title}",
"",
f"* Status: {self.status}",
f"* Deciders: {', '.join(self.deciders)}",
f"* Date: {self.date}",
"",
"---",
"",
"## Context and Problem Statement",
self.context_description,
"",
]
if self.decision_drivers:
sections.append("## Decision Drivers")
for driver in self.decision_drivers:
sections.append(f"* {driver}")
sections.append("")
if self.knock_out_criteria:
sections.append("## Knock-out Criteria")
for criterion in self.knock_out_criteria:
sections.append(f"* [ ] {criterion}")
sections.append("")
if self.weighted_criteria:
sections.append("## Weighted Evaluation Criteria")
total_weight = sum(self.weighted_criteria.values()) or 1.0
for criterion, weight in sorted(
self.weighted_criteria.items(), key=lambda x: x[1], reverse=True
):
normalized_pct = (weight / total_weight) * 100
marker = " **KNOCK-OUT**" if criterion in self.knock_out_criteria else ""
sections.append(
f"* **{criterion}**: {normalized_pct:.0f}% weight{marker}"
)
sections.append("")
if self.candidates_evaluated:
sections.append("## Candidates Evaluated")
for candidate in self.candidates_evaluated:
marker = " <-- SELECTED" if candidate == self.selected_framework else ""
sections.append(f"* {candidate}{marker}")
sections.append("")
if self.positive_consequences or self.negative_consequences:
sections.append("## Consequences")
sections.append("")
if self.positive_consequences:
sections.append("### Positive (Benefits)")
for consequence in self.positive_consequences:
sections.append(f"+ {consequence}")
sections.append("")
if self.negative_consequences:
sections.append("### Negative (Drawbacks / Risks)")
for consequence in self.negative_consequences:
sections.append(f"- {consequence}")
sections.append("")
sections.append("---")
return "\n".join(sections)
# Example: Generate an ADR from evaluation results
adr = FrameworkDecisionADR(
title="Select FastAPI for Internal API Service Framework",
status="proposed",
deciders=["alice (tech lead)", "bob (backend engineer)", "carol (platform engineer)"],
context_description=(
"Our internal services require a lightweight, high-performance HTTP framework "
"for building REST APIs and microservice endpoints. The current service uses "
"a legacy framework with slow startup times and limited async support. We need "
"a framework that supports Python 3.10+, runs efficiently in containers, and "
"integrates with our existing OpenTelemetry observability stack."
),
decision_drivers=[
"Must support async/await natively for high-concurrency workloads",
"Must run on Linux x86_64 in Docker containers",
"Must use an OSI-approved license (MIT or Apache 2.0 preferred)",
"Team needs to achieve productivity within 2 weeks of introduction",
"Must integrate with existing JWT-based authentication",
],
knock_out_criteria=[
"OSI-approved license",
"Python 3.10+ support",
"Linux container compatibility",
],
weighted_criteria={
"Core feature completeness": 0.15,
"Team familiarity": 0.12,
"Active development velocity": 0.18,
"Documentation quality": 0.08,
"Deployment compatibility": 0.13,
"Integration ease": 0.09,
"Total cost of ownership": 0.12,
"Long-term viability": 0.13,
},
candidates_evaluated=["FastAPI", "Django", "Flask"],
selected_framework="FastAPI",
positive_consequences=[
"4x faster request throughput compared to current legacy framework",
"Automatic OpenAPI/Swagger documentation generation reduces API docs burden",
"Built-in dependency injection simplifies testing and mocking",
"Async-native design enables efficient concurrent request handling",
],
negative_consequences=[
"Smaller ecosystem of third-party extensions compared to Django",
"Team has limited prior async experience — requires learning sprint",
"No built-in ORM — must select and integrate a separate database layer",
"Rapidly evolving API may introduce breaking changes between minor versions",
],
)
print(adr.markdown)
ADR Workflow for Framework Selection
- Draft the ADR before evaluation begins — Set context and decision drivers upfront so they don't change based on results (prevents confirmation bias).
- Fill in knock-out criteria with validation methods — Each must be concretely testable.
- Populate weighted criteria and scores during evaluation — Use the matrix from Step 3 above.
- Document consequences explicitly — Both positive and negative. The MADR pattern treats consequences as equally important to the decision itself.
- Set status to
acceptedonly after stakeholder review — Keep itproposeduntil all deciders have reviewed and agreed.
Failure Mode Analysis
Research into real-world framework selection failures identifies six common failure modes. For each, this section provides detection heuristics, prevention strategies, and a real example from ADR repositories.
1. Hype-Driven / Trend-Following Selection
Detection heuristics:
- The primary justification is "it's popular" or "everyone is using it"
- No specific project requirements were elicited before comparing candidates
- The recommendation comes from outside the team without stakeholder validation
- GitHub stars are cited as the primary evidence of quality
Prevention strategy: Anchor every evaluation decision to requirement-derived criteria (Step 1 above). A framework that scores highly on requirement-weighted criteria may differ significantly from one chosen based on popularity. If you cannot articulate three specific project requirements that this framework satisfies better than alternatives, the selection is likely hype-driven.
Real example: An adr/adr decision record shows a team that chose React because it was trending in 2014, then documented in retrospective: "We adopted React before validating that its component model fit our data-intensive dashboard needs. We spent six months refactoring when we realized a simpler templating approach would have been more productive." The ADR later became a reference for "hype detection" in framework selection decisions.
2. Team Skill Mismatch
Detection heuristics:
- Zero team members have production experience with the framework
- The estimated learning curve is not factored into sprint timelines
- No training budget or resources are allocated for ramp-up
- The framework's documentation assumes prior knowledge of concepts the team lacks
Prevention strategy: Conduct a skill gap analysis matrix. For each team member, rate their familiarity (none / basic / intermediate / expert) with every candidate framework. Summarize as an average and compare against the estimated productivity timeline. If the weighted average familiarity is "none" for the leading candidate, mandate a learning sprint or select a more familiar alternative.
Real example: A Go language selection ADR documented: "The team chose Go over Rust even though Rust had superior memory safety guarantees. Three engineers had production Go experience from internal CLI tooling, while zero had shipped Rust in production. The decision sustainability criteria rated Go's 'team productivity within 1–2 weeks' as critical and Rust's as uncertain."
3. Ecosystem Immaturity
Detection heuristics:
- Fewer than 50 contributors with activity in the last 90 days
- Releases are spaced more than 6 months apart
- Documentation is incomplete or contains outdated examples
- The framework is newer than 12 months without significant production adoption evidence
Prevention strategy: Track GitHub metrics over time (not just current snapshot). Look at release cadence history, issue resolution times, and contributor retention. Use the ThoughtWorks Technology Radar classification as an external validation — frameworks in "Assess" should be treated as higher risk than those in "Trial" or "Adopt." Require at least one proof-of-concept exercise that stresses real-world usage before adoption.
Real example: A CSS framework comparison ADR noted: "We evaluated three modern CSS frameworks. One had an innovative architecture but only 12 contributors and releases every 8 months. We rated it 'high risk' on ecosystem maturity despite superior technical design. The selected framework had a larger team and weekly release cadence, which mattered for long-term support."
4. Vendor Lock-In
Detection heuristics:
- The framework's proprietary features are central to the architecture (not optional plugins)
- Data formats or APIs are not publicly documented
- Migration would require significant rewrites because of deep coupling
- The licensing terms change unexpectedly or include usage-based fees
Prevention strategy: Conduct a portability assessment. Map every proprietary feature used in the architecture to a potential alternative. If you cannot describe an exit path with concrete steps, flag it as high lock-in risk. Prefer frameworks with open standards and well-documented data formats. For managed services, verify that data export tools exist and are maintained independently of the service itself.
Real example: An adr/adr record documented: "We chose PostgreSQL over a proprietary time-series database because PostgreSQL's license (PostgreSQL License) allows unrestricted use, modification, and redistribution. The proprietary alternative required annual per-node licensing and had no documented data export path for migration. Even though the proprietary option had slightly better query performance, lock-in risk was rated 'critical' in our decision matrix."
5. Over-Engineering vs Under-Specification
Detection heuristics:
- The selected framework includes features you will never use (bloat)
- OR: the framework requires building too much custom infrastructure that should be built-in
- Architecture decisions are made based on anticipated future needs rather than current requirements
- There is no "good enough" threshold defined for any evaluation dimension
Prevention strategy: Define a "good enough" threshold per criterion — the minimum acceptable level that satisfies your current needs. Any framework meeting this threshold receives maximum score regardless of how much it exceeds it. Conversely, frameworks that force you to build custom infrastructure for basic functionality should be penalized. Balance is achieved when the framework's capability surface matches your requirement surface without significant overhang or gaps.
Real example: A team evaluating backend frameworks documented: "Django offered a complete ORM, admin panel, and authentication system out of the box — we only needed the REST API layer. The extra featu
…(truncated)