Agent Learner
An AI toolkit for configuring, benchmarking, comparing, and optimizing agent prompts and evaluation results. Agent Learner provides persistent, file-based logging for each command category with timestamped entries, summary statistics, multi-format export, and full-text search across all records.
Commands
| Command |
Description |
configure |
Configure agent settings — log configuration entries or view recent ones |
benchmark |
Benchmark agent performance — log benchmark results or view history |
compare |
Compare agent outputs — log comparison data or view recent comparisons |
prompt |
Prompt management — log prompt variations or view recent prompts |
evaluate |
Evaluate agent outputs — log evaluation results or view history |
fine-tune |
Fine-tune parameters — log fine-tuning sessions or view recent ones |
analyze |
Analyze agent behavior — log analysis entries or view recent analyses |
cost |
Cost tracking — log cost data or view recent cost entries |
usage |
Usage monitoring — log usage metrics or view recent usage data |
optimize |
Optimize configurations — log optimization runs or view history |
test |
Test agent behavior — log test results or view recent tests |
report |
Report generation — log report entries or view recent reports |
stats |
Show summary statistics across all log categories (entry counts, data size, first entry date) |
export <fmt> |
Export all data in json, csv, or txt format to the data directory |
search <term> |
Full-text search across all log files (case-insensitive) |
recent |
Show the 20 most recent entries from the activity history log |
status |
Health check — show version, data directory, total entries, disk usage, and last activity |
help |
Show the full help message with all available commands |
version |
Print the current version string |
Each data command (configure, benchmark, compare, etc.) works in two modes:
- Without arguments: displays the 20 most recent entries from that category
- With arguments: saves the input as a new timestamped entry and reports the total count
Data Storage
All data is stored in plain text files under the data directory:
- Category logs:
$DATA_DIR/<command>.log — one file per command (e.g., configure.log, benchmark.log, prompt.log), each entry is timestamp|value
- History log:
$DATA_DIR/history.log — audit trail of every command executed with timestamps
- Export files:
$DATA_DIR/export.<fmt> — generated by the export command in json, csv, or txt format
Default data directory: ~/.local/share/agent-learner/
Requirements
- Bash (with
set -euo pipefail support)
- Standard Unix utilities:
grep, cat, date, echo, wc, du, head, tail, basename
- No external dependencies or API keys required
When to Use
- Benchmarking agent performance — When you need to track and compare benchmark results across different agent configurations, models, or prompt strategies
- Prompt engineering iteration — When you're testing multiple prompt variations and want to log each version with results for later comparison
- Cost and usage tracking — When you need to monitor API costs and usage metrics over time to optimize spending
- Fine-tuning experiments — When running fine-tuning sessions and you want to log parameters, results, and observations for reproducibility
- Cross-category analysis — When you need to search across all logged data (benchmarks, prompts, evaluations, costs) to find patterns or specific entries
Examples
# Initialize and check status
agent-learner status
# Log a benchmark result
agent-learner benchmark "GPT-4o on MMLU: 88.7% accuracy, 1.2s avg latency"
# Log a prompt variation
agent-learner prompt "System: You are a helpful coding assistant. Always explain your reasoning step by step."
# Compare two configurations
agent-learner compare "GPT-4o vs Claude-3.5: GPT-4o 12% faster, Claude 5% more accurate on code tasks"
# Track costs
agent-learner cost "March batch: 12,450 tokens input, 3,200 tokens output, $0.47 total"
# View all recent benchmarks
agent-learner benchmark
# Search across all logs for a specific term
agent-learner search "accuracy"
# Export all data as JSON
agent-learner export json
# View summary statistics
agent-learner stats
# Show recent activity
agent-learner recent
Output
All commands return output to stdout. Export files are written to the data directory:
agent-learner export json # → ~/.local/share/agent-learner/export.json
agent-learner export csv # → ~/.local/share/agent-learner/export.csv
agent-learner export txt # → ~/.local/share/agent-learner/export.txt
Every command execution is logged to $DATA_DIR/history.log for auditing purposes.
Powered by BytesAgain | bytesagain.com | hello@bytesagain.com
1---2name: agent-learner3description: Benchmark and compare agent prompts and evaluation results. Use when tuning strategies, evaluating outputs, or comparing configurations.4---5
6# Agent Learner
7
8An AI toolkit for configuring, benchmarking, comparing, and optimizing agent prompts and evaluation results. Agent Learner provides persistent, file-based logging for each command category with timestamped entries, summary statistics, multi-format export, and full-text search across all records.
9
10## Commands
11
12| Command | Description |
13|---------|-------------|
14| `configure` | Configure agent settings — log configuration entries or view recent ones |
15| `benchmark` | Benchmark agent performance — log benchmark results or view history |
16| `compare` | Compare agent outputs — log comparison data or view recent comparisons |
17| `prompt` | Prompt management — log prompt variations or view recent prompts |
18| `evaluate` | Evaluate agent outputs — log evaluation results or view history |
19| `fine-tune` | Fine-tune parameters — log fine-tuning sessions or view recent ones |
20| `analyze` | Analyze agent behavior — log analysis entries or view recent analyses |
21| `cost` | Cost tracking — log cost data or view recent cost entries |
22| `usage` | Usage monitoring — log usage metrics or view recent usage data |
23| `optimize` | Optimize configurations — log optimization runs or view history |
24| `test` | Test agent behavior — log test results or view recent tests |
25| `report` | Report generation — log report entries or view recent reports |
26| `stats` | Show summary statistics across all log categories (entry counts, data size, first entry date) |
27| `export <fmt>` | Export all data in json, csv, or txt format to the data directory |
28| `search <term>` | Full-text search across all log files (case-insensitive) |
29| `recent` | Show the 20 most recent entries from the activity history log |
30| `status` | Health check — show version, data directory, total entries, disk usage, and last activity |
31| `help` | Show the full help message with all available commands |
32| `version` | Print the current version string |
33
34Each data command (configure, benchmark, compare, etc.) works in two modes:
35- **Without arguments**: displays the 20 most recent entries from that category
36- **With arguments**: saves the input as a new timestamped entry and reports the total count
37
38## Data Storage
39
40All data is stored in plain text files under the data directory:
41
42- **Category logs**: `$DATA_DIR/<command>.log` — one file per command (e.g., `configure.log`, `benchmark.log`, `prompt.log`), each entry is `timestamp|value`
43- **History log**: `$DATA_DIR/history.log` — audit trail of every command executed with timestamps
44- **Export files**: `$DATA_DIR/export.<fmt>` — generated by the `export` command in json, csv, or txt format
45
46Default data directory: `~/.local/share/agent-learner/`
47
48## Requirements
49
50- Bash (with `set -euo pipefail` support)
51- Standard Unix utilities: `grep`, `cat`, `date`, `echo`, `wc`, `du`, `head`, `tail`, `basename`
52- No external dependencies or API keys required
53
54## When to Use
55
561. **Benchmarking agent performance** — When you need to track and compare benchmark results across different agent configurations, models, or prompt strategies
572. **Prompt engineering iteration** — When you're testing multiple prompt variations and want to log each version with results for later comparison
583. **Cost and usage tracking** — When you need to monitor API costs and usage metrics over time to optimize spending
594. **Fine-tuning experiments** — When running fine-tuning sessions and you want to log parameters, results, and observations for reproducibility
605. **Cross-category analysis** — When you need to search across all logged data (benchmarks, prompts, evaluations, costs) to find patterns or specific entries
61
62## Examples
63
64```bash
65# Initialize and check status
66agent-learner status
67
68# Log a benchmark result
69agent-learner benchmark "GPT-4o on MMLU: 88.7% accuracy, 1.2s avg latency"
70
71# Log a prompt variation
72agent-learner prompt "System: You are a helpful coding assistant. Always explain your reasoning step by step."
73
74# Compare two configurations
75agent-learner compare "GPT-4o vs Claude-3.5: GPT-4o 12% faster, Claude 5% more accurate on code tasks"
76
77# Track costs
78agent-learner cost "March batch: 12,450 tokens input, 3,200 tokens output, $0.47 total"
79
80# View all recent benchmarks
81agent-learner benchmark
82
83# Search across all logs for a specific term
84agent-learner search "accuracy"
85
86# Export all data as JSON
87agent-learner export json
88
89# View summary statistics
90agent-learner stats
91
92# Show recent activity
93agent-learner recent
94```
95
96## Output
97
98All commands return output to stdout. Export files are written to the data directory:
99
100```bash
101agent-learner export json # → ~/.local/share/agent-learner/export.json
102agent-learner export csv # → ~/.local/share/agent-learner/export.csv
103agent-learner export txt # → ~/.local/share/agent-learner/export.txt
104```
105
106Every command execution is logged to `$DATA_DIR/history.log` for auditing purposes.
107
108---
109
110Powered by BytesAgain | bytesagain.com | hello@bytesagain.com
bytesagain