Foundry Agentic Eval

Agentic evaluation patterns

Apply structured evaluation loops that generate, evaluate, critique, refine, and stop when objective criteria pass or iteration limits prevent further useful improvement.

When to invoke

• "Add self-critique to this agent workflow."
• "Build an evaluator-optimizer loop for generated reports."
• "Create a rubric-based LLM-as-judge evaluator."
• "Improve generated code by running tests and fixing failures."
• "Measure and improve agent response quality."

Evaluation loop patterns

Generate -> Evaluate -> Critique -> Refine -> Output
    ^                              |
    +------------------------------+

Implementation details

Use clear criteria before generation starts. Set max_iterations to 3 by default and 3-5 only when cost and latency are acceptable. Stop when all criteria pass, overall_score >= score_threshold, tests pass, or scores stop improving.

Preserve these implementation names when translating examples into code: reflect_and_refine, EvaluatorOptimizer, CodeReflector, evaluate_outcome, llm_judge, evaluate_with_rubric, generate, evaluate, optimize, run, run_tests, max_iterations, score_threshold, overall_score, criteria, critique_data, all_pass, failed, RUBRIC, accuracy, clarity, and completeness.

Basic reflection skeleton:

def reflect_and_refine(task: str, criteria: list[str], max_iterations: int = 3) -> str:
    output = llm(f"Complete this task:
{task}")
    for i in range(max_iterations):
        critique = llm(f"Evaluate this output against criteria: {criteria}
Output: {output}
Rate each: PASS/FAIL with feedback as JSON.")
        critique_data = json.loads(critique)
        all_pass = all(c["status"] == "PASS" for c in critique_data.values())
        if all_pass:
            return output
        failed = {k: v["feedback"] for k, v in critique_data.items() if v["status"] == "FAIL"}
        output = llm(f"Improve to address: {failed}
Original: {output}")
    return output

Evaluator-optimizer skeleton:

class EvaluatorOptimizer:
    def __init__(self, score_threshold: float = 0.8):
        self.score_threshold = score_threshold

    def generate(self, task: str) -> str:
        return llm(f"Complete: {task}")

    def evaluate(self, output: str, task: str) -> dict:
        return json.loads(llm(f"Evaluate output for task: {task}
Output: {output}
Return JSON with overall_score and dimensions."))

    def optimize(self, output: str, feedback: dict) -> str:
        return llm(f"Improve based on feedback: {feedback}
Output: {output}")

    def run(self, task: str, max_iterations: int = 3) -> str:
        output = self.generate(task)
        for _ in range(max_iterations):
            evaluation = self.evaluate(output, task)
            if evaluation["overall_score"] >= self.score_threshold:
                break
            output = self.optimize(output, evaluation)
        return output

Code reflection skeleton:

class CodeReflector:
    def reflect_and_fix(self, spec: str, max_iterations: int = 3) -> str:
        code = llm(f"Write Python code for: {spec}")
        tests = llm(f"Generate pytest tests for: {spec}
Code: {code}")
        for _ in range(max_iterations):
            result = run_tests(code, tests)
            if result["success"]:
                return code
            code = llm(f"Fix error: {result['error']}
Code: {code}")
        return code

Evaluation criteria

Rubric example:

RUBRIC = {
    "accuracy": {"weight": 0.4},
    "clarity": {"weight": 0.3},
    "completeness": {"weight": 0.3},
}

Compatibility vocabulary

Preserve these legacy terms, API names, command placeholders, and literal phrases when applying or migrating this skill:

• criteria/rubric
• quality-critical
• self-improvement
• single-shot

Open Horizons integration

• Scope evaluation to the Agent IDP objective, approved datasets, and current Horizon stage.
• Preserve Open Horizons Foundry deployment, tracing, safety, and evidence boundaries where applicable.
• Route cross-domain sequencing through open-horizons-orchestration (skill).

Output template

## Agentic evaluation design

**Status:** designed | implemented | blocked
**Pattern:** Basic Reflection | Evaluator-Optimizer | Code-Specific Reflection | Outcome-Based | LLM-as-Judge | Rubric-Based

| Component | Responsibility | Inputs | Outputs | Stop condition |
| --- | --- | --- | --- | --- |
| Generator | <what creates the draft> | <task/spec> | <output> | <n/a> |
| Evaluator | <criteria or tests> | <output> | <JSON score/critique> | <threshold/pass> |
| Optimizer | <refinement action> | <feedback> | <revised output> | <max iterations/convergence> |

### Rubric or criteria
- <criterion>: <pass rule or weight>

### Validation
- <test, parse, score, or convergence check>

Quality gate

• Criteria or rubric are defined before generation.
• Evaluator output is structured and parseable, preferably JSON.
• max_iterations and convergence checks prevent infinite refinement.
• Stop conditions are explicit and testable.
• Evaluation history is logged for debugging.
• Parse failures and non-improving iterations have fallback behavior.
• The optimizer preserves original task requirements rather than gaming the evaluator.