Liber1917 Skill Behavior Test Guide
Define and run behavior tests for Agent Skills using the evals.json format.
evals.json Schema
{
"$schema": "http://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"skill_name": { "type": "string" },
"skill_version": { "type": "string" },
"evals": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": { "type": "string" },
"name": { "type": "string" },
"trigger": {
"type": "object",
"properties": {
"input": { "type": "string" },
"should_trigger": { "type": "boolean" }
},
"required": ["input", "should_trigger"]
},
"flow": {
"type": "object",
"properties": {
"expected_tools": { "type": "array", "items": { "type": "string" } },
"expected_sequence": { "type": "boolean" }
}
},
"output": {
"type": "object",
"properties": {
"contains": { "type": "array", "items": { "type": "string" } },
"not_contains": { "type": "array", "items": { "type": "string" } },
"format": { "type": "string", "enum": ["text", "json", "markdown", "table", "code"] }
}
}
},
"required": ["id", "name", "trigger"]
}
}
}
}
Judgment Criteria
Trigger Judgment
- Input matches skill description → skill should activate
- Input does not match → skill should NOT activate
- Edge cases: ambiguous inputs, multi-language, partial matches
Flow Judgment (optional)
- Which tools should the skill call?
- In what order?
- Are there unnecessary tool calls?
Output Judgment (optional)
- Must contain specific keywords/phrases
- Must NOT contain certain content
- Expected output format (text/json/markdown/table/code)
Writing Guidelines
- Each eval tests ONE behavior aspect (trigger, flow, or output)
- Use descriptive IDs:
trigger-positive-basic,output-format-json,flow-tool-sequence - Include at least 2 positive trigger tests and 1 negative trigger test per skill
- Test edge cases: empty input, very long input, multi-language
- Save evals.json alongside the SKILL.md being tested