Eval Projects

Eval Projects — Review Ralph Loops and Suggest Improvements

Review the evaluation tracking from all ralph loops and their logs, then produce a report of critical and nice-to-have improvements for loop structure, skill combinations, tooling, and operations.

Data sources

1. GENERATED.md — List of projects and their loop dirs. Use it to know which loops/<project>/ entries exist and each project’s Skills used.
2. loops/ — One subdir per project. In each loops/<project-name>/ read:
   • Loop structure: PLAN.md, PROMPT.md, AGENT.md (phases, steps, instructions).
   • Progress and skill combinations: PROGRESS.md (parse structured entries: <!-- STEP step_id="..." label="..." skills="..." outcome="..." at="..." -->), progress.json, status.json.
   • Run output: report_*.json, report_history_*.jsonl (iteration, cost, plan_done/plan_pending, exit_code, event), logs/*.log when present, screenshots/ when present.

If a project has no loops/<project>/ yet or files are missing, note it as a gap (no evaluation data).

What to analyze

• Loop structure: Phase/step layout, step granularity, ordering vs scaffold DAG, redundant or missing steps.
• Skill combinations: Which skills run together per step, order across steps, failure hotspots by skill or combination.
• Outcomes: PASS/FAIL/PARTIAL from PROGRESS.md; success/failure/circuit_breaker/budget_exceeded from report history; no-progress runs.
• Cost and iteration count: From report_history_*.jsonl (cost_usd, iteration, plan_done, plan_total).
• Logs: Errors, timeouts, flaky steps, repeated failures in logs.
• Consistency: Missing or malformed PROGRESS entries, missing report history, loops not updated after runs (stale catalog).

Output format

Produce a single report with two sections.

Critical improvements

Issues that block success, waste budget, or prevent reliable evaluation. Examples:

• Steps that consistently FAIL or trigger circuit breaker; suggest split step, different skill order, or dependency fix.
• Loop dirs never updated after runs (evaluation data stale or missing).
• PROGRESS.md missing structured entries (step_id, skills, outcome, timestamp) so evaluations cannot parse.
• Same error repeated across iterations (same_error circuit breaker); suggest root-cause fix.
• Cost or iteration blow-up on specific steps or skills; suggest smaller steps or different model.
• Missing or broken logs/report files so runs cannot be evaluated.

List each with: Project (or “all” / “multiple”), Finding, Suggestion.

Nice-to-have improvements

Improvements that would help quality, observability, or reuse but are not blocking. Examples:

• Align PLAN step order with scaffold DAG for a project that diverges.
• Add checkpoint steps (e.g. “Validate Phase N”) to narrow failure scope.
• Standardize PROGRESS entry format across projects.
• Sync loops/ from CI after each run so catalog is always current.
• Add screenshots or log excerpts for failed steps to loops/ for debugging.
• Skill combinations that often run together: consider documenting as a “pack” in SCAFFOLD.md.
• Report history trends: suggest budget or iteration limits per project type.

List each with: Project (or “all”), Idea, Benefit.

Process

1. Read GENERATED.md and list every project that has a Ralph loop link (e.g. loops/<project>/).
2. For each such project, read loops/<project-name>/ if it exists. Parse PLAN.md, PROGRESS.md, report_*.json, report_history_*.jsonl, and logs when present.
3. If loops/<project-name>/ is missing or empty, record that and treat as “no evaluation data” for that project.
4. Aggregate findings across projects: common failure patterns, skill-combo hotspots, structural issues, missing or stale data.
5. Write the report: Critical improvements first, then Nice-to-have improvements, with clear project scope and actionable suggestions.

Acceptance criteria

• GENERATED.md and all referenced loops/ dirs were read.
• PROGRESS.md (and report history where present) were used to infer outcomes and skill combinations.
• Report has a Critical improvements section and a Nice-to-have improvements section.
• Each item has project scope and a concrete suggestion.
• Gaps (missing loop dir, missing files, no runs) are called out.