Product Eval

Product Evaluation Skill

Evaluate the product from a real user's perspective. Produce a numerical score across 4 dimensions (total /100). Threshold to pass: 70/100.

Prerequisites

1. If ux-audit has already been run for this PR, use its results (screenshots, performance metrics, interaction findings) as input. Do NOT re-run the same checks — build on what ux-audit already verified.
2. Start the dev server or confirm the deployed app URL is accessible.
3. Use Playwright MCP to navigate, interact, and screenshot every screen evaluated.
4. Adopt the target user's persona — their technical level, goals, context. Do NOT evaluate as a developer.

Phase 0: User Research (before scoring)

Conduct user research BEFORE scoring dimensions. Real user data prevents you from evaluating in a vacuum.

The Mom Test (Rob Fitzpatrick)

Never ask "Would you use this?" or "Do you think this is a good idea?" — people lie to be polite. Instead, ask about their life, their problems, their current behavior. Talk about specifics in the past, not hypotheticals about the future. Talk less, listen more.

5 Interview Questions That Work

1. "What's the hardest part about [task this product solves]?"
2. "Tell me about the last time that happened — walk me through it"
3. "What have you tried so far to solve it?"
4. "What don't you love about [their current solution]?"
5. "If you had a magic wand, what would it do?"

Avoid: generic claims ("I usually..."), future promises ("I would..."), and hypothetical maybes ("I might..."). These are worthless signals.

Guerrilla Usability Testing

Give 3-5 users a specific task (e.g., "sign up and create your first project"). Watch silently — do not help. Note where they hesitate, click wrong things, or ask questions. Each confusion point = a usability issue. Sessions should be under 15 minutes. You only need 3-5 participants to surface the most critical problems.

When to Run User Research

• Before product-eval scoring — use findings to inform dimension scores
• After major feature releases — validate that the feature solves the intended problem
• When churn spikes — exit interviews reveal why users leave

Store all findings in auto memory (memory/user_research.md) with: date, participant count, method (interview/usability test), and top 3 findings.

Dimension 1: Usability (30 points) — Nielsen's 10 Heuristics

Source: Jakob Nielsen, "10 Usability Heuristics for User Interface Design" (1994, updated 2020).

For each heuristic, navigate the app, check for violations, and rate severity on Nielsen's 0-4 scale:

• 0 = not a problem, 1 = cosmetic, 2 = minor, 3 = major, 4 = catastrophe

Score: 30 minus the sum of all severity ratings. If any single heuristic scores severity 4 (catastrophe), the entire dimension scores 0/30 regardless of the sum.

Dimension 2: Learnability (25 points) — Cognitive Walkthrough

Source: Wharton, Rieman, Lewis & Polson, "The Cognitive Walkthrough Method" (1994).

1. Pick 3-5 critical user tasks (the tasks a new user must complete to get value).
2. For each task, list every discrete step a novice user must take.
3. At each step, answer these 4 questions:
   • Q1: Will the user try to achieve the right effect?
   • Q2: Will the user notice the correct action is available?
   • Q3: Will the user associate the correct action with the desired effect?
   • Q4: Will the user see that progress is being made toward their goal?
4. A step passes only if all 4 answers are Yes.

Score: (passing steps / total steps) x 25. Target: 80%+ steps pass.

Dimension 3: First-Time Experience (25 points) — FTUE Audit

Evaluate as a brand-new user arriving with zero context.

Score: Sum of all four criteria (max 25).

Dimension 4: Engagement Signals (20 points) — Google HEART Framework

Source: Google HEART framework (Rodden, Hutchinson & Fu, 2010). This is a design-time assessment, not analytics-based.

Rate each signal 1-5 based on what you observe in the product:

• Happiness (1-5): Would a user feel satisfied after using this? Would they recommend it?
• Engagement (1-5): Is there a reason to come back? Is the core loop compelling?
• Adoption (1-5): Would new users complete onboarding? Is the barrier low enough?
• Retention (1-5): What would bring users back tomorrow? Next week? Is there a hook?
• Task Success (1-5): Can users complete their primary task without frustration or confusion?

Score: (sum of all 5 ratings / 25) x 20. Max 20 points.

Scoring and Grading

Total: Usability + Learnability + FTUE + Engagement = X / 100

Pass threshold: 70/100. Below 70 → enter the Iteration Protocol below.

Output Format

## Product Evaluation Report
Dimension 1: Usability — X/30
  [list each violated heuristic with severity rating and evidence]
Dimension 2: Learnability — X/25
  [list each task, steps, and which steps failed with reason]
Dimension 3: First-Time Experience — X/25
  [time-to-value measurement, steps to first success count, onboarding assessment]
Dimension 4: Engagement Signals — X/20
  [each HEART signal with rating and justification]

TOTAL: X/100 (Grade: X)
Priority issues: [top 3 by severity — what to fix first]

After Scoring

Store the evaluation result in auto memory (memory/product_scores.md) with: timestamp, total score, per-dimension scores, grade, and the top 3 priority issues.

Iteration Protocol (if score < 70)

When the score is below 70, enter this auto-fix loop. Max 3 rounds.

Each Round:

1. PRIORITIZE — Take top 3 issues by severity from the evaluation report. For each: concrete action, files to touch, expected score impact.

2. FIX — One issue at a time:

   • Write/edit the code
   • Invoke self-review on changed files
   • Run test-gate to verify no regression
   • If tests fail → revert immediately, move to next issue
   • Commit via pre-commit — one atomic commit per fix

3. RE-EVALUATE — Re-run the 4 dimensions above on the updated product. Compare: new score vs previous. Log delta per dimension.

4. LEARN — Store in auto memory (memory/product_scores.md): issue fixed, change made, score impact (e.g., "+5 on FTUE"). Builds a reusable knowledge base of high-impact improvements.

5. CHECK — Score ≥ 70 → exit. Score < 70 AND round < 3 → continue. Round = 3 → exit with remaining issues listed.

Exit Report

## Iteration Summary
Rounds completed: N
Starting score: X/100 → Final score: Y/100
Improvements made:
  1. [issue] → [fix] → [score impact]
Remaining issues (if still < 70):
  1. [issue] — reason not fixed

Safety Rules

• Max 3 rounds — hard cap, prevents infinite loops
• Self-review every fix — no quality shortcuts under automation
• Revert on test failure — undo immediately if anything breaks
• Stop on user-input issues — business decisions, missing credentials require human input

Next Steps

Report to user: "Product score: [N]/100. Grade: [letter]. Top issues: [list]"

Suggested next steps (user decides):

• Score ≥ 70 → "Run copy-craft for launch page"
• Score < 70 → "Fix top issues, then re-evaluate"
• Ready to announce → "Run content-create"