1115 Research 1 Evaluation 44dae600

Research: Creative Idea Evaluation Frameworks

Compiled: 2026-01-29

1. Idea Evaluation Criteria & Frameworks

1.1 Core Dimensions of Creative Ideas

Most frameworks agree on two to three fundamental dimensions:

These three dimensions appear repeatedly across the literature, from Besemer & Treffinger (1981) to modern AI creativity research.

1.2 NAF Framework (Novelty, Attractiveness, Feasibility)

The NAF Technique is a quick idea selection method developed from the Synectics creative problem-solving tradition (George Prince, founder of Synectics Inc.).

How it works:

• Create a table with ideas in rows, three criteria in columns
• Score each idea 1-10 on each criterion
• Sum scores and rank ideas from high to low

The three criteria:

1. Novelty -- How novel is the idea? If it is not novel for this situation, it probably is not very creative.
2. Attractiveness -- How attractive is this as a solution? Does it completely solve the problem, or is it only partial?
3. Feasibility -- How feasible is it to put into practice?

Interpreting scores:

• If feasibility is 80%+ (8/10), the idea is worth trying -- the remaining 20% relates to implementation issues (getting agreement, funding, time)
• If feasibility is below 50% but novelty and appeal are high, turn concerns into new wishes or problem definitions to build in more feasibility
• If an idea is neither new, attractive, nor feasible, its implementation will likely be disappointing

Origin note: NAF was developed because while suspending judgment worked for generating ideas, people would slip into "either/or thinking" when selecting, and this mental attitude got in the way of idea development.

Sources:

• The NAF Technique -- TICON
• NAF - Mycoted
• NAF/NUF Test - CreativeSolvers
• Tool - NAF Technique - RIKON

1.3 Creative Product Analysis Matrix (CPAM)

Proposed by Besemer & Treffinger (1981), the CPAM is a systematic model for evaluating creative products.

Three-factor model:

1. Novelty -- Originality of the concepts and the methods used to present the product
2. Resolution -- How well the product fits into its context: functionality, usability, accessibility
3. Elaboration & Synthesis -- Aesthetic and stylistic qualities of the product

CPSS Instrument (Creative Product Semantic Scale):

• Developed by Besemer & O'Quin (1986) based on CPAM
• Uses 7-point bipolar adjective scales
• 55 items structured into 11 sub-factors across the 3 main criteria
• Validated with confirmatory factor analysis

Key benefit: Helps non-practitioners evaluate creative products systematically. Especially useful for increasing generalizability of judgments by untrained judges.

Applications: Design education, advertising, engineering, consumer products, art.

Sources:

• Creative Product Analysis Matrix: Testing the Model Structure (Creativity Research Journal)
• Fostering Creativity in Design Education: Using CPAM (ERIC)
• Creative Product Analysis to Foster Innovation (ResearchGate)

1.4 Consensual Assessment Technique (CAT)

Developed by Teresa Amabile (Harvard Business School, 1982), the CAT is considered the "gold standard" of creativity assessment (Carson, 2006).

How it works:

• A panel of expert judges independently rates creative products
• Judges assess products in isolation, without conferring
• Judges are NOT trained by the researcher -- ratings are based on domain familiarity
• Individual ratings are collected and aggregated for an overall measure

Core assumptions:

1. It is possible to obtain reliable judgments of product creativity given appropriate judges -- people can recognize creativity when they see it
2. Creativity has degrees, and experts can agree that some products are more or less creative than others

Key properties:

• Theory-independent -- Unlike divergent-thinking tests, the CAT is not based on any particular theory of creativity
• Domain-flexible -- Can be applied to poems, stories, art, business solutions, marketing plans, engineering designs, music
• Expert-based -- Uses domain experts rather than standardized questionnaires

Strengths:

• Most similar to how real-life creativity and innovation is conceptualized in organizations
• Flexible across disciplines and types of creative work

Limitations:

• Resource-intensive and time-consuming
• Finding qualified expert judges can be difficult
• Inherently subjective (though inter-rater reliability is typically measured)

Recent development: Large language models (LLMs) are being explored as automated alternatives to human expert judges for CAT-style assessments.

Sources:

• Consensual Assessment Technique - ScienceDirect Topics
• The Consensual Assessment Technique - Creative Huddle
• Assessing Creativity Using CAT (ResearchGate)
• The Social Psychology of Creativity: CAT - Harvard Business School
• Assessing Creativity with CAT - Springer

1.5 Accidental Creative's Three-Part Framework

A practitioner-friendly framework evaluating ideas on three criteria:

1. Effectiveness -- Will it work? Will it solve the problem?
2. Practicality -- Can we actually do this with available resources and constraints?
3. Team Energy -- Is the team excited about this idea? Will they rally behind it?

This framework emphasizes that evaluation should enable meaningful conversation rather than people arguing from gut instinct.

Source: A Framework For Choosing The Best Idea - Accidental Creative

2. Idea Scoring Systems & Rubrics

2.1 RICE Scoring Model

Formula: Score = (Reach x Impact x Confidence) / Effort

Best for: Mature, data-backed decision-making, especially in product/digital innovation.

2.2 ICE Scoring Model

Formula: Score = Impact x Confidence x Ease

Simpler and faster than RICE. Uses "Ease" as a rough feasibility indicator instead of precise effort estimates. Best for quick, early-stage screening.

2.3 Weighted Decision Matrix

Step-by-step process:

1. Define evaluation criteria (e.g., strategic fit, feasibility, impact, novelty, cost, time, risk)
2. Assign a "criteria coefficient" for each criterion: 1 (low importance) to 5 (high importance)
3. Score each idea 1-5 on each criterion
4. Multiply scores by weights
5. Sum weighted scores across criteria for each idea
6. Rank ideas by total weighted score

Ensuring fairness:

• Use shared rubrics that define what each score means (a "5" for impact should mean the same thing to everyone)
• Gather multiple ratings per idea to average out personal preferences
• Watch for biases: overrating ideas from leadership, overvaluing large audiences, underestimating implementation challenges

2.4 Common Evaluation Criteria for Scorecards

2.5 Quick vs. Thorough Methods

2.6 Solo vs. Group Evaluation

Solo evaluation:

• Less susceptible to groupthink
• Faster for initial screening
• Good for CAT-style independent ratings

Group evaluation methods:

• Dot voting: Each person gets 3-5 dots to place on favorite ideas. Fast and democratic.
• Consensus discussion: Team discusses until agreement. Best for small teams; ensures commitment.
• Independent then aggregate: Rate individually first, then discuss discrepancies. Best of both worlds.

Sources:

• Idea Evaluation Process & Criteria Guide - Ideawake
• 7 Factors to Help You Evaluate Innovation Ideas - ITONICS
• Idea Evaluation Matrix: Step-by-Step Guide - Wazoku
• The best methods to score and evaluate innovation ideas - Accept Mission
• How to evaluate brainstorm ideas - HatRabbits
• Ideas Evaluation - Alcor Fund

3. When to Evaluate: Timing & Process

3.1 Divergent vs. Convergent Thinking

The psychologist J.P. Guilford (1956) coined these terms:

• Divergent thinking: Generate multiple novel ideas in an open-ended, free-flowing, non-linear manner. Explore many solutions. Suspend judgment.
• Convergent thinking: Select, evaluate, and refine the most promising ideas. Narrow down possibilities. Apply judgment.

Critical rule: Apply each type of thinking at the appropriate point in the process. Never mix them.

3.2 The Separation Principle

Generate first, evaluate later. This is the single most important timing principle.

• During divergent phase: No idea should have any value associated with it. Reserve judgment entirely.
• Create a transitional phase between generating and judging. People must switch from creative to critical thinking -- a completely different mindset.
• Give participants time to adjust: introduce a short break, or evaluate ideas the next day.

3.3 Why Premature Evaluation Kills Creativity

• Evaluating too early causes people to self-censor
• Teams get stuck on one idea instead of exploring the space
• Feasible/safe ideas get selected over novel ones
• Research shows individuals tend to choose more feasible and desirable ideas over truly creative ones when forced to evaluate early

3.4 Stage-Gate for Ideas

A structured approach to idea development:

1. Stage 1: Diverge -- Generate as many ideas as possible without judgment
2. Gate 1: Quick screen -- Use NAF or dot voting to shortlist (remove obvious non-starters)
3. Stage 2: Develop -- Flesh out shortlisted ideas, combine and refine
4. Gate 2: Thorough evaluation -- Use decision matrix or RICE scoring
5. Stage 3: Prototype/Test -- Build minimal versions of top ideas
6. Gate 3: Go/No-go -- Final decision based on evidence

3.5 Handling Dealbreaker Criteria

Some criteria are absolute must-haves (e.g., safety in nuclear plants, legal compliance):

• Use a checkbox approach to remove all ideas that fail dealbreaker criteria
• Alternatively, check top-ranked ideas against dealbreakers and apply creative thinking to see if they can be made to work
• If you still cannot fix the idea, discard it

3.6 What NOT to Evaluate Early

Do not place too much emphasis on projected revenue, ROI, or NPV for early-stage ideas, because most assumptions underpinning the ideas are yet to be determined. Save financial modeling for later stages when ideas are more developed.

3.7 Different Ideas Need Different Processes

Ideas aimed at improving existing products cannot be evaluated using the same approach as ideas aiming to introduce something completely new. Organizations need multiple idea evaluation processes adapted to different types of ideas -- with varying criteria. Standard metrics cannot always capture the true potential of groundbreaking, disruptive ideas.

Sources:

• Convergent vs. Divergent Thinking - Asana (2026)
• Divergent vs Convergent Thinking - Mural
• How to evaluate brainstorm ideas - HatRabbits
• Divergent vs Convergent Thinking - Creately
• Two Thinking Caps - McCombs School of Business
• The best methods to score and evaluate innovation ideas - Accept Mission

4. AI-Specific Idea Evaluation

4.1 Novelty vs. Feasibility Trade-off in AI Ideas

Research reveals a tension between novelty and feasibility in AI-generated ideas:

Study 1 (arXiv, Si et al.): AI-generated research ideas were judged as more novel than human expert ideas (p<0.05), but showed signs that these gains may come at the slight expense of feasibility. The study proposed that human judgment of novelty is difficult even for experts.

Study 2 (Harvard Business School): Found the opposite pattern -- evaluators judged human solutions as more novel (employing more "out of the box" thinking), while AI-generated ideas were found to be more valuable and feasible. Conclusion: the most productive approach is to combine human novelty with AI practicality.

Key insight: The direction of the novelty-feasibility trade-off may depend on the domain, the specific AI system, and how prompts are constructed.

Source: Can LLMs Generate Novel Research Ideas? (arXiv) Source: Can AI Match Human Ingenuity in Creative Problem-Solving? - HBS

4.2 Evaluating Novelty in AI-Generated Research Plans (2025-2026)

Research from BITS Pilani investigated whether agentic workflows (multi-step systems using iterative reasoning, evolutionary search, and recursive decomposition) generate more novel and feasible research plans.

Findings:

• Google Co-Scientist and GPT Deep Research proposals were praised for methodological clarity and literature alignment
• Decomposition-based methods in AI and multi-agent domains were described as "technically feasible"
• Climate and biotech proposals sometimes lacked sufficient grounding in experimental protocols
• Concern about "smart plagiarism" in single-step prompting approaches

Source: Evaluating Novelty in AI-Generated Research Plans (arXiv, 2025)

4.3 AI's Impact on Innovation Evaluation

A Harvard Business School field experiment on how AI recommendations affect human evaluation found:

• When evaluators received predictive AI recommendations first, they selected solutions with higher average innovation scores
• When evaluators received generative AI recommendations first, the solutions they picked varied more widely

Key trade-off:

• Average quality matters because solutions must satisfy stakeholder needs
• Variety matters because breakthroughs often emerge from the tails -- the atypical ideas many evaluators miss
• The way organizations orchestrate AI tools significantly impacts creative outcomes

Source: AI Trends for 2026 - HBS Working Knowledge

4.4 The Problem of AI "Safe" Ideas

AI-generated ideas tend toward the mean -- they can be highly competent but lack the extreme outlier quality that drives breakthroughs. This is sometimes called the homogenization effect.

Why this happens:

• LLMs are trained on vast amounts of existing text, so they synthesize known patterns
• Without specific prompting for novelty, AI defaults to high-feasibility, moderate-novelty ideas
• AI lacks the lived experience and domain intuition that drives truly novel human insights

Strategies for filtering AI safe ideas:

• Explicitly prompt for novelty and unconventional thinking
• Use AI for quantity, then apply human judgment for quality
• Combine multiple AI approaches (different prompts, different models) for diversity
• Apply the NAF framework to separate genuinely novel AI ideas from merely competent ones

4.5 Diversity Metrics for AI-Generated Ideas

When evaluating a batch of AI-generated ideas, measure not just individual quality but set-level diversity:

• Semantic diversity: How different are the ideas from each other? (measured via embeddings)
• Category coverage: Do the ideas span multiple categories or approaches?
• Novelty relative to training data: Are the ideas genuinely new or recombinations of known solutions?
• Outlier presence: Does the set contain any high-risk/high-reward ideas, or is everything clustered around the safe center?

4.6 The Human-AI Collaboration Model

The emerging consensus in 2026 is that AI should augment rather than replace human creativity:

• AI excels at generating quantity and feasible ideas quickly
• Humans excel at recognizing novelty, applying domain intuition, and making judgment calls
• The best outcomes come from iterative collaboration: AI generates, human evaluates, AI refines, human selects
• Domain expertise remains crucial for evaluation -- expertise is necessary for selecting/refining ideas and avoiding issues like hallucinations and homogenization

Source: Beyond idea generation: The importance of idea evaluation in Human-AI collaborative creativity (ScienceDirect, DiStefano & Beaty, 2026) Source: Creative Innovation Guide 2026 - Six Paths Consulting

5. Summary: Choosing an Evaluation Approach

Key Principles

1. Separate generation from evaluation -- always diverge before converging
2. Use structured criteria -- reduces bias vs. gut-feel decisions
3. Match the method to the stakes -- quick methods for early stages, thorough for final decisions
4. Different ideas need different processes -- incremental improvements vs. disruptive innovation
5. For AI ideas: check for homogenization -- explicitly measure novelty and diversity, not just feasibility
6. Multiple raters reduce bias -- aggregate independent judgments for best results

6. All Sources

Frameworks & Techniques

• The NAF Technique -- TICON
• NAF - Mycoted
• NAF/NUF Test - CreativeSolvers
• A Framework For Choosing The Best Idea - Accidental Creative
• Idea evaluation: the other side of the creativity coin - Ness Labs

CPAM & CPSS

• Creative Product Analysis Matrix: Testing the Model Structure (Creativity Research Journal)
• Fostering Creativity in Design Education: Using CPAM (ERIC)
• Creative Product Analysis to Foster Innovation (ResearchGate)

Consensual Assessment Technique

• Consensual Assessment Technique - ScienceDirect Topics
• The Consensual Assessment Technique - Creative Huddle
• Assessing Creativity Using CAT - ResearchGate
• The Social Psychology of Creativity: CAT - Harvard Business School

Scoring & Rubrics

• Idea Evaluation Process & Criteria Guide - Ideawake
• 7 Factors to Help You Evaluate Innovation Ideas - ITONICS
• Idea Evaluation Matrix: Step-by-Step Guide - Wazoku
• The best methods to score and evaluate innovation ideas - Accept Mission
• Ideas Evaluation - Alcor Fund

Brainstorming & Timing

• How to evaluate brainstorm ideas - HatRabbits
• How to Evaluate Ideas - Collective Campus
• How to Evaluate Ideas - Destination Innovation
• Convergent vs. Divergent Thinking - Asana (2026)
• Divergent vs Convergent Thinking - Mural

AI-Specific

• Can LLMs Generate Novel Research Ideas? (arXiv)
• Evaluating Novelty in AI-Generated Research Plans (arXiv, 2025)
• Can AI Match Human Ingenuity? - HBS Working Knowledge
• AI Trends for 2026 - HBS Working Knowledge
• Beyond idea generation: Human-AI collaborative creativity (ScienceDirect, 2026)
• Creative Innovation Guide 2026 - Six Paths Consulting
• Assessment of creative potential via novelty and usefulness - Cambridge Core

Innovation & 2026 Context

• Successful Inventing 2026 - USPTO
• Adobe 2026 Creative Trends