bytesagain LLM Flow
An AI toolkit for configuring, benchmarking, comparing, prompting, evaluating, fine-tuning, analyzing, and optimizing LLM workflows. Each command logs timestamped entries to local files with full export, search, and statistics support.
Commands
Core AI Operations
| Command | Description |
|---|---|
llm-flow configure <input> |
Record a configuration change (or view recent configs with no args) |
llm-flow benchmark <input> |
Log a benchmark run and its results |
llm-flow compare <input> |
Record a model or output comparison |
llm-flow prompt <input> |
Log a prompt template or prompt engineering note |
llm-flow evaluate <input> |
Record an evaluation result or metric |
llm-flow fine-tune <input> |
Log a fine-tuning session or parameters |
llm-flow analyze <input> |
Record an analysis observation |
llm-flow cost <input> |
Log cost tracking data (tokens, dollars, etc.) |
llm-flow usage <input> |
Record API usage metrics |
llm-flow optimize <input> |
Log an optimization attempt and outcome |
llm-flow test <input> |
Record a test case or test result |
llm-flow report <input> |
Log a report entry or summary |
Utility Commands
| Command | Description |
|---|---|
llm-flow stats |
Show summary statistics across all log files |
llm-flow export <fmt> |
Export all data in json, csv, or txt format |
llm-flow search <term> |
Search all entries for a keyword (case-insensitive) |
llm-flow recent |
Show the 20 most recent activity log entries |
llm-flow status |
Health check: version, entry count, disk usage, last activity |
llm-flow help |
Display full command reference |
llm-flow version |
Print current version (v2.0.0) |
How It Works
Every core command accepts free-text input. When called with arguments, LLM Flow:
- Timestamps the entry (
YYYY-MM-DD HH:MM) - Appends it to the command-specific log file (e.g.
benchmark.log,cost.log) - Records the action in a central
history.log - Reports the saved entry and running total
When called with no arguments, each command displays the 20 most recent entries from its log file.
Data Storage
All data is stored locally in plain-text log files:
~/.local/share/llm-flow/
├── configure.log # Configuration changes
├── benchmark.log # Benchmark results
├── compare.log # Model comparisons
├── prompt.log # Prompt templates & notes
├── evaluate.log # Evaluation metrics
├── fine-tune.log # Fine-tuning sessions
├── analyze.log # Analysis observations
├── cost.log # Cost tracking
├── usage.log # API usage metrics
├── optimize.log # Optimization attempts
├── test.log # Test cases & results
├── report.log # Report entries
├── history.log # Central activity log
└── export.{json,csv,txt} # Exported snapshots
Each log uses pipe-delimited format: timestamp|value.
Requirements
- Bash 4.0+ with
set -euo pipefail - Standard Unix utilities:
wc,du,grep,tail,date,sed - No external dependencies — pure bash
When to Use
- Building AI agent workflows — log each step of your agent pipeline (configure → prompt → evaluate → optimize) with full traceability
- Tracking LLM costs and usage — record per-request costs, token counts, and API usage to monitor spending across providers
- Benchmarking and comparing models — log benchmark metrics side-by-side to make data-driven model selection decisions
- Fine-tuning experiment tracking — capture hyperparameters, dataset details, and evaluation scores for every fine-tuning run
- Generating compliance reports — export all logged activity to JSON/CSV for audits, SOC reviews, or stakeholder reporting
Examples
# Configure a new workflow
llm-flow configure "workflow: summarize → classify → respond, model=claude-3.5"
# Benchmark a model
llm-flow benchmark "claude-3.5-sonnet: 94% accuracy, 0.8s p50 latency, $0.003/req"
# Log a prompt template
llm-flow prompt "system: You are a helpful assistant. Always cite sources."
# Track API costs
llm-flow cost "March week 3: 890k tokens in, 210k tokens out, $12.40 total"
# Evaluate output quality
llm-flow evaluate "human eval score: 4.2/5.0 across 50 samples"
# Search across all logs
llm-flow search "claude"
# Export to CSV for analysis
llm-flow export csv
# Quick health check
llm-flow status
Configuration
Set the DATA_DIR variable in the script or modify the default path to change storage location. Default: ~/.local/share/llm-flow/
Powered by BytesAgain | bytesagain.com | hello@bytesagain.com