Search Eval

Search Result Quality Evaluation

Every time you process search results, evaluate before using. Do NOT treat search results as truth. Apply these filters to every source before citing, recommending, or acting on it.

Source Credibility (check every result)

• Official docs / GitHub (stars > 1K) / academic papers → trustworthy
• Known media (TechCrunch, HN, Verge, official blogs) → reference-worthy
• Personal blog / Medium / content farm → must cross-verify with at least one other independent source
• SEO articles ("Top 10 Best...", "Ultimate Guide to...") → assume untrustworthy unless backed by verifiable data
• Forum answers (Stack Overflow, Reddit) → check vote count, date, and whether the answer was accepted; cross-verify for critical decisions

Evidence Check

• Has code / has data / has verifiable claims → trustworthy
• Pure opinions / pure advice / no concrete examples → lower weight
• Claims about a tool or project → verify by checking the actual repo (stars, activity, issues)
• Benchmarks without methodology → treat as anecdotal
• "Works for me" without reproduction steps → not evidence

Multi-Source Consistency

• Only one source says it → treat as unverified, flag it
• 2 sources agree → plausible but not confirmed
• 3+ independent sources agree → can be trusted
• If sources contradict each other → flag the contradiction, present both sides

Freshness

• Tech articles older than 2 years → may be outdated, check for newer alternatives
• GitHub repos with no updates in 6+ months → flag as potentially abandoned
• API documentation → always prefer the latest version; old docs may reference deprecated endpoints
• Framework guides → check which version they target; a React 16 guide is not valid for React 19

GitHub Project Evaluation (before recommending or installing)

Run this checklist before recommending any library, tool, or dependency:

• Stars < 500 → flag as unproven, warn the user explicitly
• No recent commits (6+ months) → flag as potentially abandoned
• README has no real usage examples → suspicious; may be vaporware
• Open issues with no maintainer response → sign of an unmaintained project
• Always check: stars, last commit date, number of contributors, open issues count
• Compare alternatives → never recommend the first result; check if there is a more popular or better-maintained option
• License → verify the license is compatible with the project's needs

When in Doubt

Say "I couldn't find enough trustworthy information" — do NOT fabricate an answer from low-quality sources. It is always better to be honest about uncertainty than to present unreliable information as fact.

Dependency Installation Rule

Before installing any dependency:

1. Run the GitHub evaluation checklist above
2. Check if the standard library or an existing dependency already solves the problem
3. Justify why this dependency is necessary — "it's popular" is not justification
4. Prefer dependencies with zero or minimal transitive dependencies

Learning

After evaluating a dependency or tool, store the decision in auto memory (memory/dependency_decisions.md):

• Tool/library name
• Verdict (adopted/rejected), reason, date, alternatives considered
• This prevents re-evaluating the same tool in future conversations.

Research Anti-Patterns

Proven failure patterns in research. Check for ALL of these during every research task.

Unverified Data

• Never cite a claim without checking the original source. A blog post citing a blog post is not a source. Find the actual person, the actual data, the actual context.
• When a claim seems too good to be true ("launched to $X with no audience"), verify the full picture. Often the person had a pre-existing network, audience, or unfair advantage not mentioned in the summary.
• Rule: One source = unverified. Period. Don't cite it as fact. Three independent sources = potentially reliable.

SEO Content Disguised as Research

• Listicle articles ("Top N [category] Ideas") exist to generate search traffic, not to inform. They are written by SEO specialists, not domain experts.
• Rule: Check the author's credibility. Do they have direct experience with what they recommend? If their business model is "writing content for traffic," the advice is unreliable.
• Red flags: numbered lists of ideas without data, generic advice applicable to any domain, no specific metrics or sources cited.

Survivorship Bias

• Searching for success stories only shows winners. You never see the majority that failed or earned nothing.
• Rule: For every success story, actively search for failure data. Search "[topic] failed" or "[topic] shut down" or "[category] failure rate."
• Rule: Always look up base rates. What percentage of products in this category earn meaningful revenue? Your default assumption should be the base rate, not the outlier.

Surface-Level Validation

• Knowing a product's revenue number is not validation. If you haven't looked at what the product actually does, how it's priced, what users say about it, and what the UX feels like, you know nothing meaningful.
• Rule: For every product you cite as a comparable, you MUST visit the actual product. Navigate it. Read the pricing page. Check reviews.
• Check: Can you describe the product's UX, pricing tiers, and one specific user complaint? If not, your "research" is incomplete.

Next Steps

Report to user: "Evaluation: [approved/rejected]. Tool: [name]. Reason: [one line]"

Suggested next steps (user decides):

• Approved → "Resume implementation"
• Rejected → "Search for alternatives"