GOAT Synthesis Output - 15-Test Comparison and CROSS Improvement

GOAT Synthesis Output - 15-Test Comparison and CROSS Improvement

Overview

The synthesis output - the "What I learned" / "I'm now an expert on X" narrative - is the ONLY thing that matters. All data collection, scoring, deduplication, and rendering exist solely to produce the best possible synthesis. This plan runs 15 synthesis comparisons (5 topics x 3 versions), judges which version produces the best final output, and creates concrete changes to make CROSS the GOAT.

Problem Statement / Motivation

The previous 15-test comparison analyzed the data layer (JSON files, item counts, YouTube scores, cross-ref counts). That analysis showed CROSS wins on data quality metrics. But data quality != output quality. The user's exact words: "that's all that matters... all DATA/methods/etc. should be about making it the best fucking results ever."

The paper.design example shows what great output looks like:

• "Paper Desktop + MCP is the big story" - a specific, grounded finding
• "per @stephenhaney" / "per r/UXDesign" - actual source citations
• "698 likes on X" - real engagement numbers woven into narrative
• "KEY PATTERNS from the research:" - structured patterns, not vague summaries
• "---I'm now an expert on paper.design" - confident, actionable invitation

We need to see what Base, HN, and CROSS actually produce as narratives, side-by-side, for the same topics. Then fix CROSS until its output is undeniably the best.

Approach: Controlled Synthesis Comparison

Why not re-run claude --print "/last30days X" 15 times?

Three problems:

1. WebSearch non-determinism - Claude's WebSearch tool returns different results per run, confounding version comparison
2. QUERY_TYPE non-determinism - Claude classifies "best rap songs 2026" as RECOMMENDATIONS one run and GENERAL the next, producing structurally different outputs
3. Reddit API non-determinism - The existing JSON data shows Base got 4 Reddit items for Claude Code while CROSS got 10, purely from API timing

The controlled approach

Reuse the 15 JSON files already captured (same-day, topic-sequential, verified clean). For each:

1. Convert JSON to compact markdown using each version's render_compact() (preserving version-specific rendering like CROSS's [xref:] tags)
2. Feed the compact markdown to Claude with the synthesis instructions from each version's SKILL.md
3. Hardcode QUERY_TYPE per topic to eliminate classification variance
4. Exclude WebSearch to isolate Python pipeline differences
5. Save all 15 synthesis outputs for side-by-side comparison

This gives us 15 synthesis narratives produced from identical underlying data, with only the version's rendering and synthesis instructions varying. The fairest possible comparison.

The 5 Topics and Their QUERY_TYPEs

Technical Approach

Phase 1: Setup and Data Preservation

1a. Copy JSON data to repo (CRITICAL - /tmp is ephemeral)

mkdir -p docs/comparison-results/json
cp /tmp/last30days-comparison/full/*.json docs/comparison-results/json/

• Copy all 15 JSON files to docs/comparison-results/json/
• Verify all 15 files are present and non-empty

1b. Build JSON-to-compact converter

The render pipeline differs per version. We need each version's render_compact() to produce the compact markdown that version would actually show Claude.

Script: scripts/generate-synthesis-inputs.py

"""
For each of the 15 JSON result files, render compact markdown
using that version's render_compact() function.

The JSON files store raw data. Each version's render.py formats
it differently (CROSS adds [xref:] tags, relevance scores, etc.).
"""
import json
import sys
sys.path.insert(0, 'scripts')
from lib.render import render_compact
from lib.schema import Report

for json_file in sorted(glob('docs/comparison-results/json/*.json')):
    data = json.load(open(json_file))
    report = Report.from_dict(data)
    compact = render_compact(report)
    # Save as .md alongside the JSON
    md_file = json_file.replace('/json/', '/compact/').replace('.json', '.md')
    open(md_file, 'w').write(compact)

Problem: render_compact() differs across Base/HN/CROSS. Solution: run the converter 3 times, once per git checkout, only for that version's files.

• Write scripts/generate-synthesis-inputs.py
• Run on Base checkout for base-*.json files -> docs/comparison-results/compact/base-*.md
• Run on HN checkout for hn-*.json files -> docs/comparison-results/compact/hn-*.md
• Run on CROSS checkout for cross-*.json files -> docs/comparison-results/compact/cross-*.md
• Clean __pycache__ between every git checkout

1c. Extract synthesis prompts from each version's SKILL.md

Each version's SKILL.md contains the instructions Claude uses to synthesize the compact data into the narrative. These differ across versions (CROSS has cross-ref instructions, HN has HN citation rules, etc.).

• Check out Base (427a4e4), copy SKILL.md synthesis section (Judge Agent + Display phases) to docs/comparison-results/prompts/base-synthesis-prompt.md
• Check out HN (main), copy to docs/comparison-results/prompts/hn-synthesis-prompt.md
• Check out CROSS (feat/youtube-relevance-cross-source), copy to docs/comparison-results/prompts/cross-synthesis-prompt.md
• For each, prepend the hardcoded QUERY_TYPE and topic name

Phase 2: Generate 15 Synthesis Outputs

2a. Write the synthesis runner

Script: scripts/run-synthesis-comparison.py

For each of the 15 (topic, version) pairs:

1. Read the compact markdown from docs/comparison-results/compact/{version}-{n}-{slug}.md
2. Read the synthesis prompt from docs/comparison-results/prompts/{version}-synthesis-prompt.md
3. Call the Anthropic API (Claude Sonnet 4.6 for cost/speed, or Opus 4.6 for max quality - configurable)
4. Template: "You are running /last30days. The user asked about '{topic}'. QUERY_TYPE = {query_type}. Here is the research data:\n\n{compact_markdown}\n\n{synthesis_prompt}"
5. Save response to docs/comparison-results/synthesis/{version}-{n}-{slug}.md

"""
Generate synthesis narratives for all 15 test cases.
Uses Anthropic API directly to control the synthesis environment.
"""
import anthropic
import os
from pathlib import Path

client = anthropic.Anthropic()

TOPICS = [
    (1, 'claude-code', 'Claude Code skills and MCP servers', 'GENERAL'),
    (2, 'seedance', 'Seedance AI video generation', 'NEWS'),
    (3, 'macbook', 'M4 MacBook Pro review', 'RECOMMENDATIONS'),
    (4, 'rap', 'best rap songs 2026', 'RECOMMENDATIONS'),
    (5, 'react-svelte', 'React vs Svelte 2026', 'GENERAL'),
]
VERSIONS = ['base', 'hn', 'cross']

for version in VERSIONS:
    prompt_text = Path(f'docs/comparison-results/prompts/{version}-synthesis-prompt.md').read_text()
    for num, slug, topic, qtype in TOPICS:
        compact = Path(f'docs/comparison-results/compact/{version}-{num}-{slug}.md').read_text()

        user_msg = f"""You are the /last30days skill. The user asked: "{topic}"

Parsed intent:
- TOPIC = {topic}
- TARGET_TOOL = unknown
- QUERY_TYPE = {qtype}

Here is the research output from the Python pipeline:

{compact}

Now synthesize this research into your expert narrative following these instructions:

{prompt_text}"""

        response = client.messages.create(
            model=os.environ.get('SYNTHESIS_MODEL', 'claude-sonnet-4-6-20250514'),
            max_tokens=4096,
            messages=[{"role": "user", "content": user_msg}]
        )

        output = response.content[0].text
        out_path = Path(f'docs/comparison-results/synthesis/{version}-{num}-{slug}.md')
        out_path.parent.mkdir(parents=True, exist_ok=True)
        out_path.write_text(output)
        print(f'  {version}-{num}-{slug}: {len(output)} chars')

• Write scripts/run-synthesis-comparison.py
• Run it (15 API calls, ~2-3 min total)
• Verify all 15 synthesis files exist and are non-empty

2b. Quality check synthesis outputs

Spot-check 3 outputs (one per version) for:

• Did Claude actually produce a "What I learned" narrative?
• Are there real citations (per @handle, per r/subreddit)?
• Is the stats block present?
• Is there an invitation at the end?

If any output is broken (e.g., Claude refused or produced meta-commentary instead of synthesis), adjust the prompt and re-run that one.

• Spot-check cross-1-claude-code.md, hn-2-seedance.md, base-4-rap.md
• Fix any prompt issues and re-run failed outputs

Phase 3: Evaluate - Which Version Produces the Best Output?

3a. Define evaluation rubric (BEFORE reading any outputs)

Score each synthesis 1-5 on these dimensions:

Total: weighted average, 1.0 to 5.0

3b. LLM-as-judge evaluation (blinded)

Script: scripts/evaluate-synthesis.py

For each topic (5 total), present all 3 versions to an evaluator LLM with version labels stripped. Ask it to score each on the rubric above.

"""
Blinded LLM evaluation of synthesis outputs.
Strips version labels, presents as Version A/B/C in random order.
"""
import anthropic
import random

RUBRIC = """Score each version 1-5 on these dimensions:

1. GROUNDEDNESS (30%): Does the narrative cite specific sources?
   Look for: "per @handle", "per r/subreddit", "per [channel] on YouTube"
   1 = generic, no citations. 5 = every finding has a source.

2. SPECIFICITY (25%): Are findings specific or vague?
   1 = "AI video is trending". 5 = "Seedance 2.0 added lip sync, 698 likes per @paper"

3. COVERAGE (20%): Does it represent findings from all available sources?
   1 = only Reddit mentioned. 5 = Reddit, X, YouTube, HN woven naturally.

4. ACTIONABILITY (15%): Does the invitation give specific next steps based on research?
   1 = "let me know if you want more". 5 = "I can walk you through Seedance 2.0's workflow"

5. FORMAT COMPLIANCE (10%): Stats block present with real counts? Citation format correct?
   1 = missing stats. 5 = perfect stats block + source counts + top voices.

For each version, output:
- Groundedness: X/5
- Specificity: X/5
- Coverage: X/5
- Actionability: X/5
- Format: X/5
- Weighted Total: X.X/5.0
- One sentence on what makes this version better or worse than the others.
"""

# For each topic, randomly shuffle version order to prevent position bias
for topic in TOPICS:
    versions = ['base', 'hn', 'cross']
    random.shuffle(versions)
    label_map = {v: chr(65+i) for i, v in enumerate(versions)}  # A, B, C

    # Present to evaluator
    prompt = f"Topic: {topic}\n\n"
    for v in versions:
        text = read_synthesis(v, topic)
        prompt += f"=== VERSION {label_map[v]} ===\n{text}\n\n"
    prompt += RUBRIC

    # Call evaluator (use Opus for best judgment)
    response = evaluate(prompt)
    # Map labels back to versions
    save_evaluation(topic, response, label_map)

• Write scripts/evaluate-synthesis.py
• Run evaluation (5 API calls, one per topic)
• Collect scores into summary table

3c. Human spot-check

Read 3 synthesis outputs yourself (one per version, same topic) and verify the LLM evaluation makes sense. If the LLM scores don't match your gut, investigate.

• Read all 3 versions for topic 1 (Claude Code) side-by-side
• Read all 3 versions for topic 2 (Seedance) side-by-side
• Verify LLM scores align with human judgment

3d. Compile verdict

Plus per-dimension breakdown:

• Which version is best at Groundedness?

• Which version is best at Specificity?

• Which version is best at Coverage?

• Which version is best at Actionability?

• Which version is best at Format Compliance?

• Build summary table with scores

• Identify per-dimension winners

• Write verdict paragraph

Phase 4: Make CROSS the GOAT

Based on the evaluation, identify the specific synthesis weaknesses and fix them. Changes fall into two buckets:

Bucket A: Data pipeline changes (score.py, dedupe.py, render.py, youtube_yt.py)

These affect what compact markdown Claude sees. From the previous JSON analysis, known issues:

1. Cross-source linking nearly broken (3/178 items linked at 0.5 threshold)

   • Fix: hybrid similarity (token + trigram Jaccard) at 0.40 threshold
   • File: scripts/lib/dedupe.py
   • Expected impact: 26 items linked instead of 3

2. Cross-ref rendering cryptic ([xref: HN5, HN4] is meaningless)

   • Fix: Show [also on: HN, Reddit] with source names
   • File: scripts/lib/render.py
   • Expected impact: Claude can naturally say "discussed on both Reddit and HN"

3. YouTube synonym gap ("hip hop" != "rap" in relevance scoring)

   • Fix: SYNONYMS dict in youtube_yt.py
   • File: scripts/lib/youtube_yt.py
   • Expected impact: "Lit Hip Hop Mix 2026" gets 0.67+ instead of 0.33

4. HN search too narrow (React/Svelte returns 0 HN items)

   • Fix: Split multi-keyword topics into OR queries for HN Algolia
   • File: scripts/lib/hackernews.py (or wherever HN search lives)
   • Expected impact: Framework debate topics get HN coverage

Bucket B: Synthesis instruction changes (SKILL.md)

These affect how Claude interprets the data. Specific changes to discover from the 15-output comparison:

5. Cross-source narrative instruction - If CROSS data has [also on: HN, Reddit] tags but Claude ignores them, add explicit instruction: "When items appear across multiple platforms, lead with that cross-platform signal - it's the strongest evidence of importance."

6. YouTube transcript utilization - CROSS provides transcript snippets. If Claude's synthesis doesn't use them, add: "YouTube transcripts contain direct quotes and technical details. Weave 1-2 transcript quotes into your synthesis."

7. Source weighting for synthesis - Current: Reddit/X > YouTube > Web. Should HN be weighted higher for tech topics? Should YouTube transcript content elevate YouTube's synthesis weight?

8. Citation density tuning - The paper.design example has ~1 citation per paragraph. If any version over-cites (every sentence) or under-cites (no sources), adjust the citation frequency instruction.

9. Stats block accuracy - Verify each version's stats block matches actual data. If counts are wrong, the render output may need to include pre-computed stats that Claude can copy directly.

10. Invitation quality - The best invitations reference specific things from the research ("I can walk you through Seedance 2.0's lip sync workflow"). If a version produces generic invitations ("let me know if you want to know more"), strengthen the instruction.

Changes 5-10 are discovered from Phase 3 analysis. They may or may not apply.

• Implement Bucket A changes (data pipeline fixes from JSON analysis)
• Analyze Phase 3 results to identify Bucket B changes needed
• Implement Bucket B changes (SKILL.md synthesis instruction improvements)
• Run bash scripts/sync.sh to deploy updated skill

Phase 5: Validate

Re-run 5 synthesis outputs (one per topic, CROSS-after only) using the improved CROSS code, and compare against CROSS-before.

• Generate 5 new compact markdowns from JSON data using improved render.py
• Generate 5 new synthesis outputs using improved SKILL.md
• Score with same rubric (LLM-as-judge, blinded against CROSS-before)
• Verify improvement on every dimension, or iterate

File Structure

docs/comparison-results/
  json/                          # 15 JSON files (already exist at /tmp, need copying)
    base-1-claude-code.json
    hn-1-claude-code.json
    cross-1-claude-code.json
    ... (15 total)
  compact/                       # 15 compact markdown files (rendered per-version)
    base-1-claude-code.md
    hn-1-claude-code.md
    cross-1-claude-code.md
    ... (15 total)
  prompts/                       # 3 synthesis prompts (extracted from each SKILL.md)
    base-synthesis-prompt.md
    hn-synthesis-prompt.md
    cross-synthesis-prompt.md
  synthesis/                     # 15 synthesis narratives (the actual output!)
    base-1-claude-code.md
    hn-1-claude-code.md
    cross-1-claude-code.md
    ... (15 total)
  evaluation/                    # 5 evaluation results (one per topic, blinded)
    eval-1-claude-code.md
    eval-2-seedance.md
    eval-3-macbook.md
    eval-4-rap.md
    eval-5-react-svelte.md
  summary.md                     # Final verdict with all scores and improvement plan

scripts/
  generate-synthesis-inputs.py   # JSON -> compact converter
  run-synthesis-comparison.py    # Compact + prompt -> synthesis via Claude API
  evaluate-synthesis.py          # Blinded LLM evaluation

Acceptance Criteria

• All 15 JSON files preserved in docs/comparison-results/json/
• All 15 compact markdowns generated (version-specific rendering preserved)
• All 3 synthesis prompts extracted from version-specific SKILL.md files
• All 15 synthesis narratives generated and saved
• All 15 syntheses scored on 5-dimension rubric (blinded, LLM-as-judge)
• Summary table shows per-topic and overall winner
• At least 3 synthesis outputs human-spot-checked against LLM scores
• CROSS improvement plan includes both data pipeline (Bucket A) and synthesis instruction (Bucket B) changes
• Bucket A changes implemented with tests
• Bucket B changes implemented and deployed via sync.sh
• 5 validation synthesis outputs show improvement over CROSS-before
• Final summary.md documents everything: scores, verdict, changes, validation

Dependencies & Risks

Technical Considerations

Why Anthropic API instead of claude --print?

1. Controlled environment - Same model, same temperature, same max_tokens for all 15
2. No WebSearch confound - API calls don't have tool access unless we grant it
3. Reproducible - Can re-run with different models or prompts
4. Faster - API call takes ~10s vs ~3min for full claude --print with tools

Why reuse JSON data instead of re-running the pipeline?

1. Eliminates temporal non-determinism - Same Reddit/X/YouTube data for all comparisons
2. No rate limiting - Zero API calls to source platforms
3. Already validated - 15 files verified clean, zero errors
4. Still tests rendering differences - Each version's render_compact() runs on its checkout

What about the WebSearch step?

The full skill pipeline includes Claude doing WebSearch after the Python script. We deliberately exclude this because:

1. WebSearch results vary per run (different web results each time)
2. WebSearch is identical across all 3 versions (no version difference to test)
3. Including it would confound the comparison with noise
4. The Python pipeline data is where version differences live

Sources & References

• Previous JSON analysis: docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md - data-layer comparison with scoring verdict
• Previous test plan: docs/plans/2026-02-25-feat-15-test-comparison-make-cross-goat-plan.md - execution protocol for 15 JSON test runs
• SKILL.md synthesis instructions: SKILL.md lines 165-320 (Judge Agent + Display phases)
• Scoring weights: scripts/lib/score.py - 45% relevance + 25% recency + 30% engagement
• Render pipeline: scripts/lib/render.py:57 - render_compact() function
• Cross-source linking: scripts/lib/dedupe.py:160 - cross_source_link() function
• YouTube relevance: scripts/lib/youtube_yt.py - _compute_relevance() token overlap
• Existing test harness: scripts/test-v1-vs-v2.sh:104 - claude --print approach
• YouTube display bug: docs/plans/2026-02-15-fix-youtube-display-and-search-quality-plan.md
• Output formatting fixes: docs/plans/2026-02-06-fix-last30days-v2-formatting-reddit-citations-plan.md