Roast My Product

Roast My Product

Your product, roasted by a panel of experts before your users find the flaws. Maximize user experience and product quality at any stage. This method was forged across three real audit rounds on a production Chrome extension: journey lenses found the catastrophic breaks, role lenses found the professional debt, and a journey re-run on the already-fixed codebase surfaced four more P0s — so both lens families are mandatory, plus a live walkthrough and hard regression gates.

Project facts (fill in for YOUR product, keep current)

Replace this block with your product's ground truth. Examples of what belongs here: production identifiers the code must match, the single owner of every threshold/rubric/schema other surfaces display, deploy commands and who runs them, quota/limit values, and any "never do X" rules your history has earned. An audit without this block re-litigates settled facts and misses drift.

Phase 0 — Scope

1. git log <last-release>..HEAD --oneline + git diff --stat: know what changed.
2. Read every prior report (keep them in docs/audits/) and build the exclusion list (fixed + explicitly deferred). Agents must be told what NOT to re-report; novelty is the whole value of a new round.
3. Code written since the last audit is the highest-defect-density target — name those files explicitly in the engineer and journey prompts.
4. Never audit a working tree another session/person is actively editing; audit a dedicated worktree or a clean checkout.

Phase 1 — Dual-lens agent fleet (launch all in ONE message, in background)

Journey agents — each walks a named user path step by step, "reading the code as the product"; every finding must name the exact step where it bites:

• First-run: discover → install/sign up → first setup → first core action → understanding the result
• Heavy week: day 1 intensive use → day 2 return (caches! persisted state!) → day 3 secondary platforms/surfaces → day N hitting limits → comeback after absence
• Create-and-deliver: produce the core artifact → review it → export/share it → the artifact's fate after the session ends → second use the same day
• Acquisition: ad/social click as desktop AND mobile personas → landing → store/app listing → decision moment → what happens to the 90% who don't convert

Role agents — professional checklists:

• PM: funnel completeness, day-2 retention hooks, monetization runway, measurement within your privacy promises, listing/landing conversion, competitive positioning
• Engineer: correctness of the NEWEST code first; injection/XSS on every sink fed by external or model-generated text; abuse and cost exposure of every public endpoint; platform lifecycle traps; validator false-positive rates; the minimal unit-test set worth building
• UX: state coverage per surface (empty/loading/error/stale); copy register consistency; competing callouts; keyboard and assistive-tech access; click-cost of the core loop
• UI: cross-surface tokens (color/radius/type scale/iconography); embedded UI quality inside host contexts incl. dark mode; whether new elements join the existing design system

Prompt contract for every agent: fully self-contained context, the exclusion list ("do NOT re-report"), and a structured output requirement: Return ONLY a JSON array: [{"area","priority":"P0|P1|P2","problem","impact":"one sentence","fix":"concrete","evidence":"file:line"}] — 4-10 findings you genuinely believe in; no filler.

Phase 2 — Live walkthrough (while agents run)

Real browser or device, real product: trigger the core action, screenshot the response, time it, read the console/logs. Watch for: output distributions that cluster suspiciously, states that never resolve, and anything on screen that contradicts the data underneath. Record what is LIVE versus what the branch holds — deployed drift is itself a finding.

Phase 3 — Backend smoke (deployed API)

A small script, a handful of calls, synthetic data only (never a real user's data). Exercise the cheap path against deliberately dissimilar inputs (assert a sane output spread and contract/schema invariants) and the expensive path once (assert content invariants survive end to end — e.g. every synthetic metric present verbatim in the output).

Phase 4 — Regression gates (hard, before any ship decision)

1. The build/package step runs clean; version bumped BEFORE packaging; never rebuild a version already submitted for review.
2. Your end-to-end smoke (the one that loads the real artifact under production identifiers) passes.
3. Syntax/type checks on every changed file.
4. Reference crosschecks between layers (every DOM id / route / config key a changed file references actually exists on the other side) — this bug class ships silently and costs releases.

Phase 5 — Synthesize, report, decide

1. Merge, dedupe, and VERIFY surprising agent claims against the code before reporting them.
2. Write docs/audits/<YYYY-MM-DD>-<label>.md: a 3-theme executive summary, then P0 (users lost today) / P1 (fix within two weeks) / P2 (backlog); each item = problem → impact → fix → file:line evidence. End with an execution-order table and an execution-status section.
3. Every finding carries its recommended change, written directly under the problem — the fix field is mandatory and concrete (file, approach, and when helpful the exact new copy/value). Never present a problem without a proposal; when the owner asks "how would you change it?", the answer must already be in the report.
4. Deliver the report, then walk the owner through it item by item — approve / explain / defer per item, recording every decision in the status section and moving deferred items to a persistent docs/audits/TODO.md so they survive across sessions. Explanations must be plain-language: analogies beat terminology, and every technical term gets one sentence of grounding.

Phase 6 — Fix loop

• Bugs in code this audit's own sprint produced: fix on the spot, mark "fixed during audit" in the report.
• Approved items: implement in coherent batches per deployable unit, with checks + build + commit per batch and explanatory commit messages.
• Every novel bug class found earns a unit test before the next round.
• Anything deployed re-runs Phase 3 against production.

House rules (paid for in blood)

1. Two truths drifting is the house failure mode — any "X hardcodes what Y owns" finding (thresholds, identifiers, schemas, copy) is automatically ≥P1, and the fix is always "one owner, everyone else reads it".
2. Honesty findings are P0 — anything the product displays that its own data contradicts (inflated labels, success messages over failures, swallowed warnings, marketing assets from an older ruleset). Trust is the moat.
3. The audit audits itself — this sprint's fixes are next round's first suspects.
4. Journey beats checklist for depth; run both.
5. Live state ≠ repo state — diff production before editing anything.
6. Every bound or denial the product imposes must explain itself to the user in plain words (what happened, why, what to do next).