Agentv Prompt Optimizer

AgentV Prompt Optimizer

Input Variables

• eval-path: Path or glob pattern to the AgentV evaluation file(s) to optimize against
• optimization-log-path (optional): Path where optimization progress should be logged

Workflow

1. Initialize

   • Verify <eval-path> (file or glob) targets the correct system.
   • Identify Prompt Files:
     • Infer prompt files from the eval file content (look for file: references in input that match these patterns).
     • Recursively check referenced prompt files for other prompt references (dependencies).
     • If multiple prompts are found, consider ALL of them as candidates for optimization.
   • Identify Optimization Log:
     • If <optimization-log-path> is provided, use it.
     • If not, create a new one in the parent directory of the eval files: optimization-[timestamp].md.
   • Read content of the identified prompt file.

2. Optimization Loop (Max 10 iterations)

   • Execute (The Generator): Run agentv eval <eval-path>.
     • Targeted Run: If iterating on specific stubborn failures, use --test-id <test_id> to run only the relevant tests.
   • Analyze (The Reflector):
     • Locate the results file path from the console output (e.g., .agentv/results/eval_...jsonl).
     • Orchestrate Subagent: Use runSubagent to analyze the results.
       • Task: Read the results file, calculate pass rate, and perform root cause analysis.
       • Output: Return a structured analysis including:
         • Score: Current pass rate.
         • Root Cause: Why failures occurred (e.g., "Ambiguous definition", "Hallucination").
         • Insight: Key learning or pattern identified from the failures.
         • Strategy: High-level plan to fix the prompt (e.g., "Clarify section X", "Add negative constraint").
   • Decide:
     • If 100% pass: STOP and report success.
     • If Score decreased: Revert last change, try different approach.
     • If No improvement (2x): STOP and report stagnation.
   • Refine (The Curator):
     • Orchestrate Subagent: Use runSubagent to apply the fix.
       • Task: Read the relevant prompt file(s), apply the Strategy from the Reflector, and generate the log entry.
       • Output: The Log Entry describing the specific operation performed. markdown ### Iteration [N] - **Operation**: [ADD / UPDATE / DELETE] - **Target**: [Section Name] - **Change**: [Specific text added/modified] - **Trigger**: [Specific failing test case or error pattern] - **Rationale**: [From Reflector: Root Cause] - **Score**: [From Reflector: Current Pass Rate] - **Insight**: [From Reflector: Key Learning]
     • Strategy: Treat the prompt as a structured set of rules. Execute atomic operations:
       • ADD: Insert a new rule if a constraint was missed.
       • UPDATE: Refine an existing rule to be clearer or more general.
         • Clarify: Make ambiguous instructions specific.
         • Generalize: Refactor specific fixes into high-level principles (First Principles).
       • DELETE: Remove obsolete, redundant, or harmful rules.
         • Prune: If a general rule covers specific cases, delete the specific ones.
       • Negative Constraint: If hallucinating, explicitly state what NOT to do. Prefer generalized prohibitions over specific forbidden tokens where possible.
       • Safety Check: Ensure new rules don't contradict existing ones (unless intended).
     • Constraint: Avoid rewriting large sections. Make surgical, additive changes to preserve existing behavior.
   • Log Result:
     • Append the Log Entry returned by the Curator to the optimization log file.

3. Completion

   • Report final score.
   • Summarize key changes made to the prompt.
   • Finalize Optimization Log: Add a summary header to the optimization log file indicating the session completion and final score.

Guidelines

• Generalization First: Prefer broad, principle-based guidelines over specific examples or "hotfixes". Only use specific rules if generalized instructions fail to achieve the desired score.
• Simplicity ("Less is More"): Avoid overfitting to the test set. If a specific rule doesn't significantly improve the score compared to a general one, choose the general one.
• Structure: Maintain existing Markdown headers/sections.
• Progressive Disclosure: If the prompt grows too large (>200 lines), consider moving specialized logic into a separate file or skill.
• Quality Criteria: Ensure the prompt defines a clear persona, specific task, and measurable success criteria.