Azure Foundry Evaluations — OpenAI Evals API
You can evaluate agent and model quality using the Foundry OpenAI Evals API. Create evaluation definitions with testing criteria, run them against models, and analyze pass/fail results.
Endpoint
All requests: http://localhost:8443 with ?api-version=2025-11-15-preview. Auth is automatic.
Operations
List evaluations
curl -s 'http://localhost:8443/openai/evals?api-version=2025-11-15-preview'
Create an evaluation
curl -s -X POST 'http://localhost:8443/openai/evals?api-version=2025-11-15-preview' \
-H 'Content-Type: application/json' \
-d '{"name":"quality-check","data_source_config":{"type":"custom","item_schema":{"type":"object","properties":{"input":{"type":"string"},"expected":{"type":"string"}},"required":["input","expected"]}},"testing_criteria":[{"type":"string_check","name":"exact-match","input":"{{sample.output_text}}","reference":"{{item.expected}}","operation":"eq"}]}'
Run an evaluation
curl -s -X POST 'http://localhost:8443/openai/evals/eval_abc123/runs?api-version=2025-11-15-preview' \
-H 'Content-Type: application/json' \
-d '{"name":"run-1","data_source":{"type":"jsonl","source":{"type":"file_content","content":[{"item":{"input":"2+2","expected":"4"}}]}}}'
List evaluators (built-in + custom)
curl -s 'http://localhost:8443/evaluators?api-version=2025-11-15-preview'
List evaluation rules
curl -s 'http://localhost:8443/evaluationrules?api-version=2025-11-15-preview'
When to use
- Measuring agent quality (accuracy, safety, groundedness)
- A/B testing model versions or prompt changes
- Continuous evaluation of production agent responses
- Red-teaming and safety testing
When NOT to use
- For runtime inference (use /v1/chat/completions or /openai/responses)
- For conversation management (use foundry-conversations skill)