Rl Status

/rl-status — Agent Reward & Value Snapshot

Shows the per-agent reward trajectory built up by the RL hooks (emit_task_reward, emit_correction_signal, emit_session_rewards).

Output

# RL Status — <date>

## Agent value table (sorted by 7d mean, ascending = worst first)
| Agent | n_total | mean (all) | 7d n | 7d mean | 7d-30d delta | trend |
|---|---|---|---|---|---|---|
| flake-detector | 18 | 0.42 | 5 | -0.20 | ↓ -0.62 | DEGRADING |
| verifier | 142 | 0.81 | 28 | 0.85 | ↑ +0.04 | stable |
...

## Anomalies (last 7d)
- 3 sessions with cache hit rate below 70%
- 2 corrections matched against `agent-tuner` (false positives?)

## Top tuning candidates (≥2 negative occurrences from same agent)
1. flake-detector — 4 verifier_fail in 7d. Run `/tune-agent flake-detector`.
2. ...

## Recent reward distribution (24h)
{ verifier_pass: 12, verifier_fail: 1, critic_clean: 8, critic_findings: 3, correction_detected: 2 }

## Verifier gain (precision − solver baseline; arXiv 2512.02304)
| Agent | window | n_pass | precision | FP | gain |
|---|---|---|---|---|---|
| verifier | 30d | 26 | 96.2% | 1 | +2.1% |
| critic | 30d | 9 | 100% | 0 | +5.9% |
Gain ~0 or negative = verdicts add nothing beyond the solver base rate
(rubber-stamping — the self-/intra-family verification failure mode).
Baseline measured 2026-07-11 (pre-cross-family-judge): verifier all-time
gain was **−12.6%**. Track whether the cross-family judge moves this.

How

# Verifier gain (30d window + all-time)
python3 ~/.claude/rl/verifier_gain.py --days 30
python3 ~/.claude/rl/verifier_gain.py

# Score due predictions (decision observability — every landed harness
# change carries a falsifiable prediction; this checks them when due)
python3 ~/.claude/rl/prediction_ledger.py score

# Per-agent 7d/30d windows — RECOMPUTE FROM THE EVENT LOG, relative to today.
# Do NOT trust value/*.json rolling_7d/rolling_30d: those fields are frozen at
# each agent's last_updated, so a dormant agent whose last run was a May
# correction reads as a live -2.0 failure. (This footgun produced a false-alarm
# read on 2026-07-06 before recompute was made the default.)
python3 ~/.claude/rl/window_stats.py 2>/dev/null || python3 - <<'PY'
import json
from datetime import datetime, timezone, timedelta
now = datetime.now(timezone.utc)
c7, c30 = now - timedelta(days=7), now - timedelta(days=30)
NEG = {'verifier_fail', 'correction_detected'}
A = {}
for line in open(f"{__import__('os').path.expanduser('~')}/.claude/rl/rewards.jsonl"):
    line = line.strip()
    if not line: continue
    try: r = json.loads(line)
    except Exception: continue
    d = A.setdefault(r.get('agent','?'), {'all':[], 'd7':[], 'd30':[], 'neg7':0})
    rew = float(r.get('reward', 0)); ts = datetime.fromisoformat(r['ts'].replace('Z','+00:00'))
    d['all'].append(rew)
    if ts > c30: d['d30'].append(rew)
    if ts > c7:
        d['d7'].append(rew)
        if r.get('signal') in NEG: d['neg7'] += 1
mean = lambda x: sum(x)/len(x) if x else None
rows = []
for a, d in A.items():
    m7, m30 = mean(d['d7']), mean(d['d30'])
    delta = (m7 - m30) if (m7 is not None and m30 is not None) else None
    rows.append((a, len(d['all']), mean(d['all']), len(d['d7']), m7, len(d['d30']), m30, delta, d['neg7']))
rows.sort(key=lambda r: (r[4] if r[4] is not None else 99))  # active-worst first; dormant (n/a) sink last
f = lambda v: 'n/a' if v is None else f'{v:+.3f}'
print(f"{'AGENT':34s}| n_all| mean_all| 7d_n| 7d_mean| 30d_n|30d_mean| delta| neg7")
for a,na,ma,n7,m7,n30,m30,dl,neg in rows:
    print(f"{a:34s}|{na:5d} | {f(ma):>7}|{n7:4d} | {f(m7):>7}|{n30:5d}|{f(m30):>7}|{f(dl):>6}|{neg:4d}")
print("\nFLAGS (criteria below; DORMANT = 0 events in 7d is NOT a flag):")
flag = False
for a,na,ma,n7,m7,n30,m30,dl,neg in rows:
    why = []
    if n7 > 0 and m7 is not None and m7 < 0: why.append(f"7d {m7:+.2f}<0")
    if dl is not None and dl < -0.3: why.append(f"drop {dl:+.2f}")
    if neg >= 2: why.append(f"{neg} fail/corr in 7d")
    if why: flag = True; print(f"  /tune-agent {a}   ({'; '.join(why)})")
if not flag: print("  none — all agents with 7d activity are healthy.")
PY

# Frozen snapshot read (LEGACY — kept only for cross-checking the recompute
# above; the rolling_* fields are stale for any agent not updated in <window>).
ls ~/.claude/rl/value/*.json | xargs -I {} cat {} | jq -s 'sort_by(.rolling_7d.mean)'

# Reward histogram
jq -s 'group_by(.signal) | map({signal: .[0].signal, n: length, mean_reward: ([.[] | .reward] | add / length)})' \
  ~/.claude/rl/rewards.jsonl

# Recent corrections (24h)
python3 -c "
import json, sys
from datetime import datetime, timezone, timedelta
cutoff = datetime.now(timezone.utc) - timedelta(days=1)
for line in open('$HOME/.claude/rl/rewards.jsonl'):
    r = json.loads(line)
    ts = datetime.fromisoformat(r['ts'].replace('Z','+00:00'))
    if ts > cutoff and r.get('source') == 'user_correction':
        print(r['agent'], '—', r.get('matched_pattern',''))
"

Tuning candidate criteria

An agent appears as a tuning candidate when:

• rolling_7d.mean < 0 (more failures than successes recently), OR
• rolling_7d.mean - rolling_30d.mean < -0.3 (sharp drop), OR
• ≥2 verifier_fail OR correction_detected events from this agent in 7d

Surface these with the recommended /tune-agent <name> command.

What this is NOT

• Real fine-tuning (we only adjust prompts via Reflexion)
• Statistical-significance-tested A/B results (use /ab-test for that)
• Cost reporting (use /cache-report)

This is operational telemetry for "is the fleet healthy?"