Tool Categories (Flexible Matching)
Problem: Your test expects read_file. Agent uses bash cat. Test fails. Both are correct.
Solution: Test by intent, not exact tool name.
Before (Brittle)
expected:
tools:
- read_file # Fails if agent uses bash, text_editor, etc.
After (Flexible)
expected:
categories:
- file_read # Passes for read_file, bash cat, text_editor, etc.
Built-in Categories
| Category |
Matches |
file_read |
read_file, bash, text_editor, cat, view, str_replace_editor |
file_write |
write_file, bash, text_editor, edit_file, create_file |
file_list |
list_directory, bash, ls, find, directory_tree |
search |
grep, ripgrep, bash, search_files, code_search |
shell |
bash, shell, terminal, execute, run_command |
web |
web_search, browse, fetch_url, http_request, curl |
git |
git, bash, git_commit, git_push, github |
python |
python, bash, python_repl, execute_python, jupyter |
Custom Categories
Add project-specific categories in config.yaml:
# .evalview/config.yaml
tool_categories:
database:
- postgres_query
- mysql_execute
- sql_run
my_custom_api:
- internal_api_call
- legacy_endpoint
Why This Matters
Different agents use different tools for the same task. Categories let you test behavior, not implementation.
For example, all of these accomplish "read a file":
- Claude Code:
read_file
- OpenAI:
bash with cat
- Custom agent:
text_editor
With categories, your test passes for all of them:
expected:
categories:
- file_read # All three approaches pass
Combining Tools and Categories
You can mix exact tool names and categories:
expected:
tools:
- my_specific_tool # Must use this exact tool
categories:
- file_read # Plus any file reading approach
Related Documentation
- Evaluation Metrics
- CLI Reference
1---2name: tool-categories-flexible-matching3description: Different agents use different tools for the same task. Categories let you test behavior, not implementation.4---5# Tool Categories (Flexible Matching)67**Problem:** Your test expects `read_file`. Agent uses `bash cat`. Test fails. Both are correct.89**Solution:** Test by *intent*, not exact tool name.1011---1213## Before (Brittle)1415```yaml16expected:17 tools:18 - read_file # Fails if agent uses bash, text_editor, etc.19```2021## After (Flexible)2223```yaml24expected:25 categories:26 - file_read # Passes for read_file, bash cat, text_editor, etc.27```2829---3031## Built-in Categories3233| Category | Matches |34|----------|---------|35| `file_read` | read_file, bash, text_editor, cat, view, str_replace_editor |36| `file_write` | write_file, bash, text_editor, edit_file, create_file |37| `file_list` | list_directory, bash, ls, find, directory_tree |38| `search` | grep, ripgrep, bash, search_files, code_search |39| `shell` | bash, shell, terminal, execute, run_command |40| `web` | web_search, browse, fetch_url, http_request, curl |41| `git` | git, bash, git_commit, git_push, github |42| `python` | python, bash, python_repl, execute_python, jupyter |4344---4546## Custom Categories4748Add project-specific categories in `config.yaml`:4950```yaml51# .evalview/config.yaml52tool_categories:53 database:54 - postgres_query55 - mysql_execute56 - sql_run57 my_custom_api:58 - internal_api_call59 - legacy_endpoint60```6162---6364## Why This Matters6566Different agents use different tools for the same task. Categories let you test **behavior**, not **implementation**.6768For example, all of these accomplish "read a file":69- Claude Code: `read_file`70- OpenAI: `bash` with `cat`71- Custom agent: `text_editor`7273With categories, your test passes for all of them:7475```yaml76expected:77 categories:78 - file_read # All three approaches pass79```8081---8283## Combining Tools and Categories8485You can mix exact tool names and categories:8687```yaml88expected:89 tools:90 - my_specific_tool # Must use this exact tool91 categories:92 - file_read # Plus any file reading approach93```9495---9697## Related Documentation9899- [Evaluation Metrics](EVALUATION_METRICS.md)100- [CLI Reference](CLI_REFERENCE.md)
tools-only