nafeeur Prompt and Agent Evaluations
- Define representative tasks with frozen repositories, initial state, expected behavior, and prohibited regressions.
- Score outcome correctness, tests/build, diff quality, tool efficiency, context size, latency, and recovery from failures.
- Capture full trajectories and normalize nondeterministic fields.
- Include adversarial cases: misleading files, large MCP catalogs, unavailable tools, malformed model calls, and interrupted runs.
- Compare changes against a baseline over multiple seeds/models when variability matters.
- Do not optimize solely for judge prose; verify executable artifacts.