# Brutal Idea Eval > Comprehensive IDEA evaluation combining Reality's Moat scar tissue framework, Revenue Reality Check, VC Power-Law filter, and structured deep research pipeline. Point it at an IDEA.md for end-to-end analysis: defensibility ratio, revenue viability, VC scalability, validated research, and synthesized verdict with pivot recommendations. - Skill: `sssemil/brutal-idea-eval` (Agent Skill) - Install (CLI): `npx skillmds@latest add sssemil/brutal-idea-eval` - Raw SKILL.md: https://api.skillmd.com/api/skills/sssemil/brutal-idea-eval/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: sssemil (https://skillmd.com/u/sssemil) - Updated: 2026-09-10 - Page: https://skillmd.com/skills/sssemil/brutal-idea-eval --- Comprehensive IDEA evaluation engine. Evaluates a single `IDEA.md` end-to-end using: 1. Reality's Moat scar tissue framework (defensibility) 2. Revenue Reality Check R1-R6 (short-term viability) 3. VC Power-Law + Control Filter (scalability ceiling) 4. Structured Deep Research pipeline (claim validation, resume support, JSON output) Produces a validated brutal reality analysis, an optimistic opportunity lens, a structured deep research report, a final synthesized verdict, and a pivot recommendation. Agent assumptions (applies to all agents and subagents): - All tools are functional and will work without error. Do not test tools or make exploratory calls. - Only call a tool if it is required to complete the task. Every tool call should have a clear purpose. --- # Brutal Idea Evaluation Process ## Overview When pointed at an `IDEA.md`, execute these phases in order: 1. **Phase 1**: Load and parse IDEA.md, create session directory 2. **Phase 2**: Reality's Moat analysis (Steps 1-5) → writes `phase-2-moat.md` 3. **Phase 3**: VC Power-Law + Control Filter → writes `phase-3-vc.md` 4. **Phase 4**: Revenue Reality Check (R1-R6) → writes `phase-4-revenue.md` 5. **Phase 5**: Synthesize initial verdict → writes `phase-5-verdict.md` 6. **Phase 6**: Structured Deep Research (claim validation) → writes `outline.yaml`, `fields.yaml`, `results/*.json`, `report.md` 7. **Phase 7**: Final synthesis and pivot recommendation → writes `verdict.md` Each phase builds on the previous. Do not skip phases. **Every phase writes its output to a file in the session directory and updates `state.yaml` before moving to the next phase.** This enables resume from zero context at any point. ### Session Directory Structure ``` workspace/idea-eval/IE--/ ├── state.yaml # Phase completion tracking (source of truth for resume) ├── phase-2-moat.md # Phase 2 output ├── phase-3-vc.md # Phase 3 output ├── phase-4-revenue.md # Phase 4 output ├── phase-5-verdict.md # Phase 5 output (includes claims catalog) ├── outline.yaml # Phase 6 research outline ├── fields.yaml # Phase 6 field definitions ├── progress.yaml # Phase 6 research progress ├── results/ # Phase 6 research results (one JSON per item) ├── generate_report.py # Phase 6 report script ├── report.md # Phase 6 deep research report └── verdict.md # Phase 7 final synthesis ``` ### state.yaml Format ```yaml session: "IE--" idea_source: "" created: "" phases: phase_1_context: completed phase_2_moat: completed phase_3_vc: in_progress phase_4_revenue: pending phase_5_verdict: pending phase_6_research: pending phase_6_report: pending phase_7_synthesis: pending ``` **Rules:** - Update `state.yaml` AFTER writing the phase output file, not before. - A phase is `completed` only when its output file exists AND `state.yaml` says so. - `in_progress` means the phase was started but not finished (crash recovery target). - `pending` means not yet started. --- # PHASE 1 — Load Context & Create Session ## Step 0: Resume Check (BEFORE anything else) Before starting a new session, check if the user provided a path to an existing session directory or if one can be auto-detected. **Auto-detection**: Check for existing `workspace/idea-eval/` directories. If the user provides a session path or says "resume," read `state.yaml` from that directory and skip to the Resume Logic section at the bottom of this document. If no resume is detected, proceed with a fresh session. ## Step 0.1: Load IDEA.md Read `IDEA.md` from the project root or path provided by user. Extract and catalog: - Core business idea description - Market size claims - Growth rate claims - Competitor claims - Regulatory assumptions - Technology assumptions - Revenue model and pricing assumptions - Target customer profile - Distribution assumptions If IDEA.md is missing or empty, stop and ask the user to provide one. ## Step 0.2: Get Current Date ```bash date +%Y-%m-%d ``` ## Step 0.3: Determine Session Number Find the highest existing session number: ```bash ls workspace/idea-eval/ 2>/dev/null | grep -oE '[0-9]{4}' | sort -rn | head -1 ``` New session number = highest + 1. If none exist, start at `0001`. ## Step 0.4: Generate Session Slug Generate from the idea name: - Lowercase - Replace spaces and special characters with hyphens - Max 40 characters - Remove trailing hyphens Session directory: `workspace/idea-eval/IE--/` ## Step 0.5: Create Session Directory & Initialize State Create the directory immediately: ```bash mkdir -p workspace/idea-eval/IE--/results ``` Write initial `state.yaml`: ```yaml session: "IE--" idea_source: "" created: "" phases: phase_1_context: completed phase_2_moat: pending phase_3_vc: pending phase_4_revenue: pending phase_5_verdict: pending phase_6_research: pending phase_6_report: pending phase_7_synthesis: pending ``` Phase 1 is marked `completed` because IDEA.md has been read and the session is initialized. --- # PHASE 2 — Reality's Moat Analysis ## Defensibility Formula ``` defensibility = scar_tissue / specifiable_code ``` - **Scar tissue** = operational knowledge earned by acting in a system that keeps changing. - **Specifiable code** = anything describable enough for AI to build. - Build cost trends toward zero. Verification cost does not. Most ideas are majority specifiable code. Assume low ratio unless proven otherwise. --- ## Step 1: Identify the Ratio Split the idea into two piles. ### Specifiable Parts APIs, dashboards, auth, billing logic, SDKs, ML training loops, UI components, orchestration layers, data pipelines, CRUD systems. If AI can reproduce it from a description, it belongs here. Be generous — most software is specifiable. ### Scar Tissue Parts Knowledge earned only by acting in the system: - Filing and seeing what gets rejected - Executing transactions and learning failure modes - Processing claims and resolving disputes - Negotiating regulators - Handling edge cases discovered only in production Ask: Could a well-funded competitor converge on this knowledge using public information? If yes, it is not scar tissue — it is a stockpile. State the ratio honestly: - **Low** (<30% scar tissue) - **Medium** (30-60% scar tissue) - **High** (>60% scar tissue) --- ## Step 2: The Three Survival Questions Run in order. Each is a gate. ### Q1 — Interventional or Observational? Can equivalent knowledge be generated by watching data? - **Observational knowledge** = collected by watching. Medical images, transaction logs, call recordings, public filings, market data. Any large enough sample converges on the same patterns. This is a stockpile, not a moat. - **Interventional knowledge** = generated by acting. Filing a document and seeing what gets rejected. Sending a transaction and learning how a specific bank handles errors. Processing a claim and discovering how payer rules interact. This can only be learned by doing. Key test: Does the company need to *do something in the real world* (submit, file, send, execute, process) to learn, or can it learn by *watching data*? If the answer is "AI can learn this from data," the moat dissolves. Say so clearly. ### Q2 — Is the System Changing? - **Converging** = rules stabilizing. AI converges on answers. No moat. - **Stable** = rules fixed. Knowledge is stockpilable. - **Increasing complexity** = new rules interact with old ones. Changes compound. Moat compounds. - **Extreme volatility** = system changes so fast that accumulated knowledge expires rapidly. Moats compound only in non-stationary systems. State the direction. ### Q3 — Is the System Adversarial? Does the system adapt against what you know? - **Non-adversarial** (bureaucracies, regulations, physical infrastructure) = knowledge compounds durably. What you learn in 2026 still helps in 2028. The system evolves but does not fight back. **Stripe pattern — durable.** - **Adversarial** (cybersecurity, trading, SEO, ad fraud) = the system actively designs around what you know. Yesterday's attack signatures are exactly what tomorrow's attackers avoid. **Treadmill pattern — exhausting.** --- ## Step 3: Interaction Effects ### Width Test Does operating for more customers generate combinatorial knowledge? - **Independent streams** = each customer adds knowledge linearly. 1,000 customers = 1,000 learnings. Any competitor with enough data converges. - **Crossing streams** = learning from Customer A reshapes how you serve Customer B. Interactions encode the specific history of interventions. Only the company that acted has them. Crossing streams create compounding moats. Independent streams create stockpiles. --- ## Step 4: Competitive Compression Test If a well-funded competitor started today with the best AI available, how long before they match your operational knowledge? - **Days to weeks** = no moat. Pure specifiable code. - **Months** = weak moat. Probably a stockpile, not scar tissue. - **Years** = real moat. Check whether it is genuinely path-dependent or just slow data collection. - **Cannot be reproduced** = strong moat. Knowledge was created by specific history of interventions in a system that has since changed. Be concrete about what the competitor would need to rebuild and how long it would take. --- ## Step 5: System Replacement Risk Is the system you are earning scar tissue in going to keep existing? Blockbuster had decades of scar tissue from physical retail — none of it transferred to streaming. The knowledge was real but the system was replaced. Ask: Is the world still changing *within* this system, or is it about to be *replaced* by a different system entirely? If there is a plausible system replacement on the horizon, flag it. Scar tissue in a dying system is a liability, not an asset. ## Phase 2 Checkpoint: Write Output & Update State Write the complete Phase 2 analysis to `phase-2-moat.md` in the session directory. Include all five steps: ratio, survival questions, interaction effects, competitive compression, and system replacement risk. Then update `state.yaml`: set `phase_2_moat: completed`. --- # PHASE 3 — VC Power-Law & Control Filter After moat analysis, apply these filters. These determine whether the idea is investable at scale, separate from whether it is defensible. ## A) Ceiling Test Can this plausibly reach $500M+ ARR? Evaluate: - Total addressable market (TAM) — use bottom-up, not top-down - Realistic capture rate given competition - Pricing power trajectory as market matures - Geographic or regulatory caps on expansion If structurally capped (geography, regulation, niche TAM), state the ceiling clearly. A $50M ARR ceiling is still a real business — just not a venture-scale one. ## B) Inevitable Touchpoint Test Do customers *have* to pass through this layer? Or can they route around it? Does replacing you require: - Re-verification of production edge cases - Re-integration with downstream systems - Regulatory re-approval - Operational rework across customer base If switching is easy (export data, plug in competitor), control is weak. If switching requires re-earning scar tissue, control is strong. ## C) Distribution Asymmetry Does distribution compound automatically? Or does growth require linear outbound effort forever? Evaluate: - Network effects (real or aspirational?) - Viral mechanics (does usage create awareness?) - Platform lock-in (do integrations create switching costs?) - Content/SEO compounding (does organic grow over time?) Distribution leverage often matters more than scar tissue. A strong moat with no distribution compounds slowly. Weak moat with strong distribution can reach revenue fast. ## Phase 3 Checkpoint: Write Output & Update State Write the complete Phase 3 analysis to `phase-3-vc.md` in the session directory. Include all three tests: ceiling, inevitable touchpoint, and distribution asymmetry. Then update `state.yaml`: set `phase_3_vc: completed`. --- # PHASE 4 — Revenue Reality Check (R1-R6) This axis is separate from defensibility. A low-ratio idea that hits $10k MRR fast is a legitimate strategy: get revenue flowing, use it to fund the search for scar tissue, and build defensibility while customers are already paying. Do not use this phase to dismiss or filter out ideas — use it to add information. ## R1: Revenue Math Work backward from $10,000/month at three price points: - **Low ($25-50/mo):** How many customers? (200-400.) Is this a self-serve volume play? Where do they come from? - **Mid ($200-500/mo):** How many customers? (20-50.) Can the founder close these with light-touch sales? - **High ($2,000-5,000/mo):** How many customers? (2-5.) Is this enterprise? What is the sales cycle? State which price point is most realistic for this idea and why. ## R2: Time to First Paying Customer How long from "product is built" to "first dollar received"? Map the steps: Build -> Find customer -> Get attention -> Demo/trial -> Close -> Payment received If time-to-first-dollar exceeds 3 months, the 6-month target is at serious risk. ## R3: Sales Cycle vs. 6-Month Constraint What is the realistic sales cycle at the identified price point? How many complete deal cycles fit in 6 months? Subtract onboarding time, pilot periods, and procurement delays. Does the math still work? ## R4: Distribution Channel Which channel can the founder realistically use from day one? - **Content/SEO:** Slow build (3-6 months to traction). Only viable if founder has existing audience. - **Outbound (cold email/LinkedIn):** Fast for high-ACV products. Requires clear ICP and compelling pain point. - **Community/word-of-mouth:** Requires existing audience or network. Slow to start, compounds over time. - **Marketplace/platform:** Instant distribution but platform dependency and margin pressure. - **Partnerships/integrations:** Slow to establish, fast once live. Gatekept by partner priorities. State which channel is most realistic and what that implies for the ramp. ## R5: Willingness to Pay Three key signals: - Is anyone *currently paying* for a solution to this problem (even a bad one)? - Is anyone *currently solving this manually* with people (spreadsheets, VAs, consultants)? - Is this a "must have" (blocks revenue, creates legal risk, prevents operations) or "nice to have" (saves time, improves metrics)? If nobody is paying or solving manually, willingness-to-pay risk is high. State this clearly. ## R6: Revenue Ramp Shape Which pattern describes the most likely revenue trajectory? - **Linear:** One customer at a time, steady climb. Typical for consultative sales. - **Step-function:** A few large accounts land at discrete intervals. Typical for enterprise/mid-market. - **Exponential:** Product-led growth with viral or network mechanics. Rare but powerful when real. - **Front-loaded (services wedge):** Start with high-touch services revenue, transition to product. Fast revenue, slower margin improvement. State the shape and whether it can reach $10k MRR within the 6-month window. ## Revenue Verdict - **Likely** = clear demand, short sales cycle, realistic channel, math works at achievable price point. - **Possible** = demand signals exist but sales cycle is tight, or channel requires building, or price point needs validation. - **Unlikely** = significant unknowns in demand, distribution, or sales cycle. Multiple things must go right. - **Near-impossible** = long sales cycles, unproven demand, no clear distribution, or math requires unrealistic customer volume. State the single biggest risk to hitting $10k MRR in 6 months. ## Revenue-First Strategy Note If the idea is low-ratio on defensibility but scores well on revenue, call this out explicitly as a viable path. Revenue buys time and funding to discover where the scar tissue lives. The strategic question the founder must answer: *"What operational knowledge will you accumulate while serving these customers that a competitor cannot?"* If they have a credible answer, revenue-first is not a consolation prize — it is a strategy. ## Phase 4 Checkpoint: Write Output & Update State Write the complete Phase 4 analysis to `phase-4-revenue.md` in the session directory. Include all six sub-questions (R1-R6), the revenue verdict, and the revenue-first strategy note if applicable. Then update `state.yaml`: set `phase_4_revenue: completed`. --- # PHASE 5 — Initial Verdict Synthesis Before launching deep research, synthesize what you know into an initial verdict. This frames the research questions. ## Verdict Dashboard ``` **Ratio:** [High / Medium / Low] — X% specifiable, Y% scar tissue **Volatility:** [Converging / Stable / Increasing / Extreme] — direction of system complexity **Interventional:** [Yes / Partial / No] — does operating generate knowledge that watching cannot? **Adversarial:** [Yes / No] — does the system fight back? **Interaction effects:** [Strong / Weak / None] — do customer streams cross? **Time to scar tissue:** [Immediate / Months / Years / Never] — how fast can you start accumulating? **System replacement risk:** [Low / Medium / High] — could the whole domain get disrupted? **VC ceiling:** [$XB / $XM / Capped] — maximum plausible ARR **Inevitable touchpoint:** [Strong / Weak / None] — can customers route around you? **Distribution asymmetry:** [Compounding / Linear / None] — does growth feed itself? **$10k MRR in 6 months:** [Likely / Possible / Unlikely / Near-impossible] — [biggest risk or enabler] ``` ## Initial One-Line Verdict Choose the most honest characterization: - "This gets commoditized when building is free." - "Real scar tissue, but it is a treadmill — you never rest." - "Durable compounding moat. This is the Stripe pattern." - "The knowledge is real but the system might get replaced." - "High ceiling but distribution is fragile." - "Low ratio but fast revenue — viable as a revenue-first play." ## Claims to Validate List every factual claim from IDEA.md that the deep research phase should verify: - Market size claims - Growth rate claims - Competitor landscape claims - Regulatory assumptions - Technical feasibility claims - Revenue model assumptions - Distribution assumptions These become research items for Phase 6. ## Phase 5 Checkpoint: Write Output & Update State Write the complete Phase 5 analysis to `phase-5-verdict.md` in the session directory. Include: - The full verdict dashboard - The initial one-line verdict - The complete claims-to-validate list (this is critical — Phase 6 reads it) Then update `state.yaml`: set `phase_5_verdict: completed`. --- # PHASE 6 — Structured Deep Research Pipeline This phase validates claims from IDEA.md using the deep research engine. It runs fully autonomously with resume support. --- ## Step 6.0: Load Prior Phase Context On resume, re-read prior phase outputs to reconstruct context: 1. Read `IDEA.md` from path stored in `state.yaml` → `idea_source` 2. Read `phase-5-verdict.md` → get claims-to-validate list The session directory already exists (created in Phase 1). Update `state.yaml`: set `phase_6_research: in_progress`. --- ## Step 6.1: Generate Research Framework Using the claims catalog from Phase 5 (or `phase-5-verdict.md` on resume), generate: ### Items List Research objects derived from IDEA.md claims: - TAM validation (verify market size claims) - Competitor landscape (verify competitor claims, find unlisted competitors) - Funding levels (how much capital is flowing into this space?) - Regulatory barriers (verify regulatory assumptions) - Technical feasibility (validate technology claims) - Distribution assumptions (verify channel viability) - Switching cost evidence (validate or challenge lock-in claims) - Pricing benchmarks (what are competitors charging? what are customers paying?) - Growth trajectory evidence (verify growth rate claims) Each item includes: - `name`: Item name - `category`: Classification - `description`: What this item validates and why it matters ### Field Framework Per category, define research fields: - `market_size`, `cagr`, `data_source` - `incumbent_players`, `funding_raised`, `market_share` - `switching_costs`, `integration_depth` - `regulatory_barriers`, `compliance_timeline` - `moat_indicators`, `compression_estimate` - `uncertainty_flags` Proceed directly to web search supplement. Do not ask the user for confirmation. --- ## Step 6.2: Web Search Supplement Launch 1 web-search-agent (background) using the Task tool with `model: sonnet` and `max_turns: 20`. **Parameter Retrieval**: - `{topic}`: The idea name/domain from IDEA.md - `{YYYY-MM-DD}`: Current date from Step 0.2 - `{step1_output}`: Complete output from Step 6.1 (items list + field framework) - `{time_range}`: "Since 2024" **Hard Constraint**: The following prompt must be strictly reproduced, only replacing variables in `{xxx}`. Do not modify structure or wording. **Prompt Template**: ``` You are an elite internet researcher. Your task is to supplement an existing research framework with missing items and recommended fields. ## Research Methodology Before searching, determine which search strategies apply to this topic. Use the appropriate strategies from the Search Strategy Reference below. Get today's date first: date +%Y-%m-%d Generate 5-10 different search query variations to maximize coverage: - Include technical terms, product names, and common variations - Think of how different people might describe the same topic - Use exact phrases in quotes for specific names - Include version numbers and dates when relevant ## Information Gathering Standards - Read beyond the first few results - valuable information is often buried - Look for patterns across different sources - Pay attention to dates to ensure relevance - Note different approaches and their trade-offs - Identify authoritative sources and experienced contributors - Check for updated information or superseded approaches - Verify across multiple sources when possible ## Task Research topic: {topic} Current date: {YYYY-MM-DD} Based on the following initial framework, supplement latest items and recommended research fields. ## Existing Framework {step1_output} ## Goals 1. Verify if existing items are missing important objects 2. Supplement items based on missing objects 3. Continue searching for {topic} related items within {time_range} and supplement 4. Supplement new fields ## Output Requirements Return structured results directly (do not write files): ### Supplementary Items - item_name: Brief explanation (why it should be added) ... ### Recommended Supplementary Fields - field_name: Field description (why this dimension is needed) ... ### Sources - [Source1](url1) - [Source2](url2) ``` ### 6.2.1 Merge Findings After the web search agent completes, merge findings with the initial framework: - Add supplementary items (avoid duplicates) - Add recommended fields - Note sources for traceability --- ### 6.2.2 Write Session Files After merging, immediately write: `outline.yaml`: ```yaml topic: "" session: "IE--" created: "" source: "IDEA.md" items: - name: "" category: "" description: "" # ... more items output_dir: "./results" ``` `fields.yaml`: ```yaml categories: : fields: - name: "" description: "" detail_level: "" # ... more fields # ... more categories ``` --- ## Step 6.3: Deep Research Preparation ### 6.3.1 Read Outline Read `workspace/idea-eval/IE--/outline.yaml` to get items list. ### 6.3.2 Resume Check Check for completed and in-progress results: ```bash ls workspace/idea-eval/IE--/results/*.json 2>/dev/null ls workspace/idea-eval/IE--/results/*.started 2>/dev/null ``` Determine item status: - `.json` exists = **completed** (skip) - `.started` exists but no `.json` = **interrupted** (re-research) - Neither exists = **never started** (research) ### 6.3.3 Prepare Execution Plan Calculate total remaining items. Display which will be researched vs skipped. --- ## Step 6.4: Execute Deep Research ### 6.4.1 Launch Research Agents Launch remaining agents using the Task tool with `model: sonnet`, `run_in_background: true`, and `max_turns: 25`. **Batching strategy**: - **10 or fewer items**: Launch ALL agents in a single parallel batch. - **More than 10 items**: Split into batches of 10. Launch each batch in parallel, wait for completion via filesystem polling, then launch next batch. Each agent researches one item and outputs JSON. **Agent Prompt Template** (per item): **Hard Constraint**: The following prompt must be strictly reproduced, only replacing variables in `{xxx}`. Do not modify structure or wording. ``` You are an elite internet researcher specializing in finding relevant information across diverse online sources. Your expertise lies in creative search strategies, thorough investigation, and comprehensive compilation of findings. ## Progress Tracking Before starting research, write a marker file to signal that this agent has started: Write an empty file to {started_path} After self-validation passes and the JSON result is confirmed correct, delete the marker file: rm {started_path} ## Research Methodology Get today's date first: date +%Y-%m-%d Generate 5-10 different search query variations to maximize coverage: - Include technical terms, product names, and common variations - Think of how different people might describe the same topic - Use exact phrases in quotes for specific names - Include version numbers and dates when relevant ### Search Strategy Reference Use the following search strategies based on what is relevant to the research topic: **General Web Strategy** (for broad information gathering): Sources: Reddit, official documentation, blog posts, Hacker News, Dev.to, Medium, Discord, X/Twitter - Look for official recommendations first - Cross-reference with community consensus - Find examples from production use - Identify anti-patterns and common pitfalls - Note evolving best practices - Create structured comparisons with clear criteria - Find real-world usage examples and case studies - Look for performance benchmarks and user experiences **Academic Papers Strategy** (for research, algorithms, scientific topics): Sources: Google Scholar, arXiv, Hugging Face Papers, bioRxiv, ResearchGate, Semantic Scholar, ACM Digital Library, IEEE Xplore - Use Google Scholar as primary source with advanced search operators - Search by author names, paper titles, DOI numbers - Include year ranges to find seminal works and recent publications - Look for related papers and citation patterns - Search for preprints on arXiv and bioRxiv - Track citation networks to understand research evolution ## Information Gathering Standards - Read beyond the first few results - valuable information is often buried - Look for patterns in solutions across different sources - Pay attention to dates to ensure relevance (note if information is outdated) - Note different approaches and their trade-offs - Identify authoritative sources and experienced contributors - Verify information across multiple sources when possible - Clearly indicate when information is speculative or unverified ## Task Research {item_related_info}, output structured JSON to {output_path} ## Field Definitions Read {fields_path} to get all field definitions ## Output Requirements 1. Output JSON according to fields defined in fields.yaml 2. Mark uncertain field values with [uncertain] 3. Add uncertain array at the end of JSON, listing all uncertain field names 4. All field values must be in English ## Self-Validation After writing the JSON file, read it back and verify: 1. Every field defined in fields.yaml has a corresponding entry in the JSON 2. The JSON is valid (properly formatted) 3. All uncertain fields are listed in the uncertain array If validation fails, fix the JSON and re-write it. ## Output Path {output_path} ``` ### 6.4.2 Parameter Construction For each item: - `{item_related_info}`: Item's complete YAML content (name + category + description) - `{output_path}`: Absolute path to `workspace/idea-eval/IE--/results/.json` - Slugify: replace spaces with `_`, remove special characters - `{fields_path}`: Absolute path to `workspace/idea-eval/IE--/fields.yaml` - `{started_path}`: Absolute path to `workspace/idea-eval/IE--/results/.started` ### 6.4.3 Write progress.yaml Immediately after launching all agents: ```yaml status: in_progress started: "" total_items: items: - name: "" slug: "" status: pending # ... all items being researched ``` ### 6.4.4 Monitor Progress (Filesystem-Based) **CRITICAL**: Do NOT use TaskOutput to read agent results. Agent outputs are large and reading them into the orchestrator context will cause context window exhaustion. All results are persisted to disk as JSON — the orchestrator only needs to check file existence. **Polling loop** — repeat until all items are resolved: 1. Check completion: ```bash ls /results/*.json 2>/dev/null | wc -l ls /results/*.started 2>/dev/null | wc -l ``` 2. Calculate: `completed` = .json count, `in_progress` = .started without .json, `remaining` = total - completed 3. Display: "Progress: X/Y items completed, Z still running." 4. If `in_progress > 0`, wait ~30 seconds (`sleep 30`) then poll again 5. If `in_progress == 0`, exit loop **After loop completes:** 1. Update `progress.yaml` with final status per item 2. Report: "All agents complete. X/Y items researched successfully." 3. If any failed, list them and suggest resume --- ## Step 6.5: Report Configuration ### 6.5.1 Scan Summary Fields Read all completed JSON results and identify fields suitable for TOC display: - Numeric fields - Short metric fields - Fields appearing across most items ### 6.5.2 Select Summary Fields Auto-select the most informative summary fields for the TOC: - Prefer numeric fields, short metric fields, and fields appearing across most items - Select 2-4 fields maximum - Do not ask the user — choose automatically based on data coverage and informativeness --- ## Step 6.6: Generate Report ### 6.6.1 Generate Report Script Generate `generate_report.py` in the session directory. The script must handle: **1. JSON Structure Compatibility** Support two structures: - Flat: `{"name": "xxx", "release_date": "xxx"}` - Nested: `{"basic_info": {"name": "xxx"}, "technical_features": {...}}` Field lookup order: Top level -> category mapping key -> traverse all nested dicts **2. Complex Value Formatting** - List of dicts: format each dict as one line, separate kv with ` | ` - Normal list: short lists joined with comma, long lists with line breaks - Nested dict: recursive formatting - Long text (>100 chars): add `
` or blockquote **3. Extra Fields Collection** Collect fields in JSON but not in fields.yaml, put in "Other Info." Filter out internal fields (`_source_file`, `uncertain`). **4. Uncertain Value Skipping** Skip if: - Value contains `[uncertain]` - Field name in `uncertain` array - Value is None or empty **5. Report Format** - **TOC**: Every item with number, name (anchor link), and selected summary fields - **Detailed Sections**: One per item, organized by field category ### 6.6.2 Execute Script ```bash python workspace/idea-eval/IE--/generate_report.py ``` ## Phase 6 Checkpoint: Update State After research agents complete: update `state.yaml`: set `phase_6_research: completed`. After report is generated: update `state.yaml`: set `phase_6_report: completed`. --- # PHASE 7 — Final Synthesis Update `state.yaml`: set `phase_7_synthesis: in_progress`. ### Load Context for Synthesis On resume (zero context), re-read all prior outputs: 1. `IDEA.md` from `idea_source` in `state.yaml` 2. `phase-2-moat.md` — defensibility analysis 3. `phase-3-vc.md` — VC filter results 4. `phase-4-revenue.md` — revenue reality check 5. `phase-5-verdict.md` — initial verdict and claims list 6. `report.md` — deep research report (summary, not raw JSON) Then read research results from `workspace/idea-eval/IE--/results/`. Compare validated findings against IDEA.md claims. Note where claims were confirmed, corrected, or invalidated. ## Final Output Format ``` ## [Idea Name] **What it is:** One sentence. --- ### Defensibility Analysis **The Ratio:** X% scar tissue / Y% specifiable code **Survival Questions:** - Q1 (Interventional?): [Answer with specifics] - Q2 (System changing?): [Answer with direction] - Q3 (Adversarial?): [Answer with pattern name] **Interaction Effects:** [How customer streams interact] **Competitive Compression:** [How fast a funded competitor catches up] **System Replacement Risk:** [Assessment] --- ### VC Filter - **Ceiling:** [$XB / $XM / Capped] — [reasoning] - **Inevitable Touchpoint:** [Strong / Weak / None] — [what makes switching hard or easy] - **Distribution Asymmetry:** [Compounding / Linear / None] — [mechanism] --- ### Revenue Reality - **Price tier:** [Low / Mid / High] — [amount] x [customers needed] - **Time to first dollar:** [estimate] - **Sales cycle fit:** [how many cycles in 6 months] - **Distribution:** [most realistic channel] - **Willingness to pay:** [must-have / nice-to-have / unproven] - **Ramp shape:** [linear / step-function / exponential / front-loaded] - **$10k MRR in 6 months:** [Likely / Possible / Unlikely / Near-impossible] — [biggest risk] - **Revenue-first strategy:** [If applicable — how revenue buys time to find scar tissue] --- ### Deep Research Summary - **Claims validated:** [list] - **Claims corrected:** [list with corrections] - **New risks discovered:** [list] - **New opportunities discovered:** [list] - **Areas of high uncertainty:** [list] --- ### Verdict Dashboard **Ratio:** [High / Medium / Low] — X% specifiable, Y% scar tissue **Volatility:** [Converging / Stable / Increasing / Extreme] **Interventional:** [Yes / Partial / No] **Adversarial:** [Yes / No] **Interaction effects:** [Strong / Weak / None] **Time to scar tissue:** [Immediate / Months / Years / Never] **System replacement risk:** [Low / Medium / High] **VC ceiling:** [$XB / $XM / Capped] **Inevitable touchpoint:** [Strong / Weak / None] **Distribution asymmetry:** [Compounding / Linear / None] **$10k MRR in 6 months:** [Likely / Possible / Unlikely / Near-impossible] ### Final One-Line Verdict [The most honest single-sentence characterization of this idea] --- ### The Pivot [If the idea scores poorly on any axis, show the strongest alternative version.] Consider how the pivot affects: - **Moat:** Does the interventional version create deeper scar tissue? - **Revenue speed:** Does it make $10k MRR easier or harder? - **Distribution:** Does it unlock a better channel? - **Compression:** Does it change the competitive timeline? If the idea is observational, show the interventional version. If the idea has weak distribution, show the version with stronger distribution. If the idea has a low ceiling, show the version that unlocks a larger TAM. ``` Write the final synthesis to `workspace/idea-eval/IE--/verdict.md`. Update `state.yaml`: set `phase_7_synthesis: completed`. --- ## Completion Summary Present: ``` ## Idea Evaluation Complete: **Session**: workspace/idea-eval/IE--/ ### Output Files - state.yaml — Phase completion tracking (resume source of truth) - phase-2-moat.md — Defensibility analysis - phase-3-vc.md — VC Power-Law filter - phase-4-revenue.md — Revenue Reality Check - phase-5-verdict.md — Initial verdict + claims catalog - outline.yaml — Research outline and items list - fields.yaml — Field definitions - progress.yaml — Research execution progress - results/ — JSON results per item ( files) - generate_report.py — Report generation script - report.md — Deep research report - verdict.md — Final synthesized verdict ### Key Findings - Defensibility: [one line] - Revenue: [one line] - VC scalability: [one line] - Biggest risk: [one line] - Recommended next step: [one line] ``` --- # Common Patterns Reference Reference these when you see them: | Pattern | Example | Defensibility | Revenue | |---------|---------|---------------|---------| | **The Stripe** | Payments, ground station brokerage, claims processing | Durable compounding. Non-adversarial, interventional, crossing streams. | Slow ramp — enterprise sales cycles, compliance gates, long integration timelines. $10k MRR in 6 months unlikely without services wedge. | | **The Treadmill** | Cybersecurity, trading alpha, SEO, ad fraud | Real scar tissue but adversarial. You run to stay in place. | Variable — cybersecurity sells on fear and closes fast; trading alpha monetizes immediately but is volatile; SEO/ad fraud often self-serve with fast ramp. | | **The Stockpile** | Medical imaging datasets, content libraries, public market data | Observational. AI synthesizes equivalents. Moat dissolves. | Often fast — proven demand, self-serve, but race to bottom on price as competitors replicate. | | **The Experiment** | Drug development, clinical trials, molecule screening | Path-dependent but rate-limited by experiment speed. Slow durable moat. | Very slow — regulatory timelines, long sales cycles, high ACV but few deals. $10k MRR in 6 months near-impossible without services revenue. | | **The CRM** | Generic SaaS, dashboards, developer tools, auth libraries | Mostly specifiable code. Low ratio. Commoditized when building is free. | Often fast — proven demand, self-serve signup, low friction. But margins compress as competitors multiply. | | **The Blockbuster** | Scar tissue in a system being replaced | Real knowledge in a dying domain. Liability, not asset. | May still be extractable short-term from slow-migrating incumbents. Declining market = declining revenue ceiling. | --- # Anti-Patterns to Watch For - **"We have data"** — Data collected by watching is a stockpile. Only data generated by acting is scar tissue. - **"We have network effects"** — Network effects amplify whatever they are built around. If the product is stationary, the network dissolves. When switching is free, the crowd moves. - **"It's hard to build"** — Irrelevant when building is free. The question is not "can someone build this?" but "can someone verify it works in production without your operational knowledge?" - **"We're first to market"** — First-mover in specifiable software is temporary. First-mover in scar tissue accumulation is permanent — but only if you are actually accumulating scar tissue. - **"AI can't do this yet"** — Capabilities change fast. If the task is stationary, AI will converge. Time your analysis to "when AI can do this," not "right now." - **"Our TAM is $100B"** — Top-down TAM is fantasy. Bottom-up TAM is reality. How many specific buyers at what specific price? - **"We'll figure out distribution later"** — Distribution is not an afterthought. If you cannot name your first 10 customers and how you will reach them, distribution is a risk. --- # Principles 1. **Be honest, not reassuring.** The goal is to save the founder time and money by identifying weak spots before they invest years. Frame honesty as respect. 2. **Be specific, not generic.** Never say "this could be commoditized." Say exactly what is specifiable, what is scar tissue, and how thin the scar tissue layer is. 3. **Most ideas are low-ratio. That does not mean they cannot make money.** Defensibility and revenue speed are orthogon …(truncated)