Eval

Eval Skill

Structured evaluation of everything the agent manages.

⚡ Execution Architecture (Anti-Timeout)

Eval is split into 3 independent subagents that run in parallel. Each subagent handles one domain, returns a short result block, and exits. Main agent collects results and formats the final report.

On-demand (user asks): Always run fresh — never read from cache. Cron/heartbeat: Run async, save to memory/eval-cache.json, report only if issues found.

Subagent Split

Timeout: Spawn all subagents with runTimeoutSeconds=60. Do NOT use openclaw status — it is slow and causes SIGKILL. Use only fast targeted commands (see templates below).

How to Run (on-demand)

1. Spawn SA-1, SA-2, SA-3 simultaneously (sessions_spawn, runtime=subagent)
2. Each returns a JSON block with its section
3. Main agent merges and formats the final report
4. Send to owner

Subagent Task Templates

SA-1 — System Health:

Run system health check using ONLY fast targeted commands (do NOT run openclaw status):
1. /path/to/ocana/bifrost/vertex-ctl.sh status 2>&1 | head -3
2. curl -s --max-time 5 http://127.0.0.1:18789/ | head -1 || echo 'gateway unreachable'
3. git -C /path/to/openclaw/workspace log -1 --format="%ar"
4. python3 -c "import os,subprocess; db=os.environ.get('PA_DB_URL',''); r=subprocess.run(['psql',db,'-c','SELECT COUNT(*) FROM messages;','-t'],capture_output=True,text=True,timeout=5) if db else None; print(r.stdout.strip() if r else 'no DB')"
Return JSON: {"vertex": "RUNNING/DOWN", "gateway": "reachable/unreachable", "backup": "X ago", "db": "X msgs"}

SA-2 — PA Network:

Read /path/to/workspace/contact-list.md
Find all PA entries (lines with "PA:" or under WhatsApp Groups / PAs section)
Count total PAs, check for any notes about issues/offline/billing
Return JSON: {"total": N, "online": N, "issues": [{"pa": "name", "issue": "description"}]}

SA-3 — Tasks + Memory + Skills:

Check:
1. wc -l /path/to/workspace/MEMORY.md
2. ls /path/to/workspace/skills | wc -l
3. Check today's daily note exists
4. grep tasks.md for open/done count
Return JSON: {"memory_lines": N, "skills": N, "daily_note": true/false, "tasks_open": N, "tasks_done": N}

When to Use

Trigger phrases:

• "תעשי eval" / "run eval"
• "מה עובד ומה לא" / "what's working and what isn't"
• "תדרגי את עצמך" / "rate yourself"
• "בדקי הכל" / "check everything"

Eval Report Format

📋 Full Eval — [DATE]

━━━ SELF PERFORMANCE ━━━
Execution:      [1-5] [comment]
Accuracy:       [1-5] [comment]
Memory:         [1-5] [comment]
Proactivity:    [1-5] [comment]
Communication:  [1-5] [comment]
TOTAL: [X]/25

━━━ ACTIVE TASKS ━━━
✅ Done today:   [count]
🟡 In progress:  [count]
❌ Stalled:      [count] — [list stalled tasks]

━━━ PA NETWORK ━━━
✅ Working:  [list]
⚠️ Issues:   [list with issue]
❌ Down:     [list]

━━━ SKILLS ━━━
Installed: [count]
Used today: [list]
Unused (7+ days): [list]

━━━ INTEGRATIONS ━━━
Calendar (owner):     [connected ✅ / broken ❌ / unknown ?]
monday.com:           [connected ✅ / broken ❌]
Email (gog):          [connected ✅ / broken ❌]
GitHub backup:        [last push: X ago]
WhatsApp:             [connected ✅ / disconnected ❌]

━━━ MEMORY HEALTH ━━━
Daily notes:     [today's file exists? ✅/❌]
Long-term:       [MEMORY.md size — OK / bloated]
Learnings:       [count this week]
Last backup:     [X ago]

━━━ RECOMMENDATIONS ━━━
1. [Most important thing to fix]
2. [Second priority]
3. [Optional improvement]

Running the Eval

Step 1 — Self Performance Score

Score each dimension 1–5 based on today's activity:

Execution (1–5):
- 5: All tasks completed without reminders
- 3: Most tasks done, some follow-up needed
- 1: Multiple tasks missed or forgotten

Accuracy (1–5):
- 5: No corrections from owner
- 3: 1–2 corrections
- 1: Multiple errors or wrong outputs

Memory (1–5):
- 5: Recalled context correctly every time
- 3: Missed some context, caught on
- 1: Repeated the same mistakes

Proactivity (1–5):
- 5: Acted before being asked multiple times
- 3: Responded to requests, minimal initiative
- 1: Only reacted, no proactive actions

Communication (1–5):
- 5: Clear, concise, no unnecessary narration
- 3: Occasionally verbose or unclear
- 1: Shared reasoning, listed options, narrated steps

Step 2 — Task Audit

TASKS_FILE="$HOME/.openclaw/workspace/memory/tasks.md"

echo "Tasks done:"
grep -c "\[x\]" "$TASKS_FILE" 2>/dev/null || echo 0

echo "Tasks in progress:"
grep -c "\[ \]" "$TASKS_FILE" 2>/dev/null || echo 0

# Stalled = in progress for 2+ days
echo "Stalled tasks (2+ days old):"
grep "\[ \]" "$TASKS_FILE" | grep -v "$(date +%Y-%m-%d)" | grep -v "$(date -u -d '1 day ago' +%Y-%m-%d 2>/dev/null)" || echo "none"

Step 3 — PA Network Health

# PA network is tracked in contact-list.md (not billing-status.json)
grep -i "PA:" /path/to/workspace/contact-list.md | head -30

Step 4 — Skills Audit

SKILLS_DIR="$HOME/.openclaw/workspace/skills"

echo "Installed skills:"
ls "$SKILLS_DIR" | grep -v README | wc -l

echo "Skills list:"
ls "$SKILLS_DIR" | grep -v README

Step 5 — Integration Health

# Test Anthropic billing
API_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -H "x-api-key: ${ANTHROPIC_API_KEY:-none}" \
  -H "anthropic-version: 2023-06-01" \
  https://api.anthropic.com/v1/models 2>/dev/null)

# Interpret result
if [ "$API_STATUS" = "200" ]; then echo "Billing: ✅ OK"
elif [ "$API_STATUS" = "402" ]; then echo "Billing: ❌ OUT OF CREDITS"
elif [ "$API_STATUS" = "401" ]; then echo "Billing: ❌ Invalid key"
else echo "Billing: ? HTTP $API_STATUS"
fi

# Test GitHub backup
LAST_PUSH=$(git -C "$HOME/.openclaw/workspace" log -1 --format="%ar" 2>/dev/null)
echo "Last backup: $LAST_PUSH"

# Test monday.com
if [ -f "$HOME/.credentials/monday-api-token.txt" ]; then
  MONDAY_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
    -X POST https://api.monday.com/v2 \
    -H "Authorization: $(cat $HOME/.credentials/monday-api-token.txt)" \
    -H "Content-Type: application/json" \
    -d '{"query": "{ me { id } }"}' 2>/dev/null)
  [ "$MONDAY_STATUS" = "200" ] && echo "monday.com: ✅" || echo "monday.com: ❌ ($MONDAY_STATUS)"
else
  echo "monday.com: ? (no token found)"
fi

Step 6 — Memory Health (All 4 Layers)

TODAY=$(date -u +%Y-%m-%d)
WORKSPACE="/path/to/openclaw/workspace"

# Layer 1: MEMORY.md (long-term rules)
MEMORY_LINES=$(wc -l < "$WORKSPACE/MEMORY.md" 2>/dev/null || echo 0)
if [ "$MEMORY_LINES" -gt 200 ]; then
  echo "Layer 1 MEMORY.md: ⚠️ Large ($MEMORY_LINES lines) — consider pruning"
elif [ "$MEMORY_LINES" -gt 0 ]; then
  echo "Layer 1 MEMORY.md: ✅ ($MEMORY_LINES lines)"
else
  echo "Layer 1 MEMORY.md: ❌ empty or missing"
fi

# Layer 2: Daily notes
[ -f "$WORKSPACE/memory/daily/$TODAY.md" ] \
  && echo "Layer 2 Daily notes: ✅ ($TODAY exists)" \
  || echo "Layer 2 Daily notes: ❌ not created yet"

# Layer 3: PostgreSQL (WhatsApp history)
python3 << 'PYEOF'
import os, subprocess
db_url = os.environ.get('PA_DB_URL', '')
if not db_url:
    print('Layer 3 PostgreSQL: ❌ PA_DB_URL not set')
else:
    try:
        result = subprocess.run(
            ['psql', db_url, '-c', 'SELECT COUNT(*) FROM messages;', '-t'],
            capture_output=True, text=True, timeout=5
        )
        count = result.stdout.strip()
        if count.isdigit() and int(count) > 0:
            print(f'Layer 3 PostgreSQL: ✅ ({count} messages)')
        else:
            print(f'Layer 3 PostgreSQL: ⚠️ connected but empty ({count})')
    except Exception as e:
        print(f'Layer 3 PostgreSQL: ❌ {e}')
PYEOF

# Layer 4: SQLite semantic (vector memory)
# IMPORTANT: correct path is /path/to/openclaw/memory/main.sqlite (NOT workspace)
python3 << 'PYEOF'
try:
    import sqlite3, sqlite_vec
    db = sqlite3.connect('/path/to/openclaw/memory/main.sqlite')
    db.enable_load_extension(True)
    sqlite_vec.load(db)
    db.enable_load_extension(False)
    files = db.execute('SELECT COUNT(*) FROM files').fetchone()[0]
    chunks = db.execute('SELECT COUNT(*) FROM chunks').fetchone()[0]
    vectors = db.execute('SELECT COUNT(*) FROM chunks_vec').fetchone()[0]
    cache = db.execute('SELECT COUNT(*) FROM embedding_cache').fetchone()[0]
    if chunks > 1 and vectors > 1:
        print(f'Layer 4 Semantic SQLite: ✅ ({files} files, {chunks} chunks, {vectors} vectors, {cache} cached embeddings)')
    else:
        print(f'Layer 4 Semantic SQLite: ⚠️ low data ({chunks} chunks, {vectors} vectors) — may not be indexed')
except Exception as e:
    print(f'Layer 4 Semantic SQLite: ❌ {e}')
PYEOF

Memory layer legend:

• Layer 1: MEMORY.md — curated rules and context
• Layer 2: Daily notes — raw session log
• Layer 3: PostgreSQL — full WhatsApp history (2,700+ msgs)
• Layer 4: SQLite+vec — semantic/vector search across workspace files

Recommendations Logic

After running all steps, generate recommendations:

If any PA has billing_error AND status != resolved:
  → "Fix billing for [PA list] — they can't function"

If any task has status in_progress for 2+ days:
  → "Follow up on stalled task: [task name]"

If MEMORY.md > 200 lines:
  → "Prune MEMORY.md — it's getting bloated"

If daily notes don't exist:
  → "Create today's memory file"

If last backup > 6 hours ago:
  → "Run git backup"

If API billing = 402:
  → "My own API key is out of credits — alert the admin immediately"

Scheduling

Run eval:

• On demand — when owner asks
• Weekly — every Sunday at 09:00
• After major incidents — billing crisis, WA disconnect, etc.

Cost Tips

• Cheap: Reading files, scoring, formatting — any small model
• Expensive: Summarizing large memory files — skip if not asked
• Avoid: Running all API health checks every hour — cache for 30 min
• Batch: Run all health checks in one pass, not one at a time

Minimum Model

Any model that can:

1. Read files
2. Apply if/then scoring rules
3. Format a structured report

No advanced reasoning needed.

PA Performance Scoring (Merged from pa-eval skill)

Use this section when evaluating individual PA agents (weekly self-eval or on-demand when owner gives feedback).

Scoring Dimensions (1–5 each, max 40 points)

Grade: A (36–40), B (28–35), C (20–27), D (<20)

Owner Feedback Signals

Log these automatically when detected:

Rule: Log feedback signals immediately — don't batch them.

Weekly Eval File

Save to .learnings/eval/YYYY-MM-DD.md with: scores table, owner feedback, tasks completed/failed, what went well, what to improve, actions for next week.

Benchmark Tests (Run Monthly)

• Task Completion Rate: completed / assigned × 100% — Target: >90%
• Accuracy Rate: (tasks - corrections) / tasks × 100% — Target: >95%
• Memory Retention: Ask about something discussed 7+ days ago — Target: >80% recall

Skill Usage (from gateway logs)

# Which skills were used in recent sessions
for f in /tmp/openclaw/openclaw-*.log; do
  echo "=== $(basename $f) ==="
  grep -o "skills/[a-z_-]*/SKILL\.md" "$f" 2>/dev/null | sort | uniq -c | sort -rn | head -10
done