Predictions

Predictions

When you make a forward-looking claim — "Tim will find this interesting", "the wiki cleanup will reduce orphan pages", "this new prompt will make tool errors drop" — that's a prediction, and you can test yourself against reality if you write it down with enough structure to be checked later.

This is a sibling to the applied-proposals audit (FUTURE_WORK §12.2). That one tracks operator-merged policy changes. This one tracks YOUR in-the-moment guesses. Both close double-loops; they just operate at different scopes.

Contract

Trigger: Making a forward-looking claim with a checkable outcome (1-7 day horizon typical) — "the wiki cleanup will reduce orphan pages", "Tim will reply within 24h", "this prompt change will drop tool errors". Also fires when opening a protected-surface proposal PR through open_proposal / submit_proposal — attach the prediction to the proposal rationale or PR body so the applied-proposals audit can verify it.

Requires: A specific check time / horizon; a claim narrow enough to be unambiguously right-or-wrong at horizon (not "things will go well"); mimir predictions CLI on PATH.

Guarantees:

• Each prediction lands as a structured state/predictions.jsonl entry: claim, horizon, optional context — never as free-form prose.
• Reflection's grading pass picks up due predictions and scores them; results feed the agent's own calibration over time.
• The audit loop closes: predicted-vs-actual deltas surface in reflection turns and applied-proposals audits.

Does not: Predict trivially-true / trivially-checkable things (the test: would you be willing to be wrong, and would being wrong teach you something?); auto-grade — that's reflection's job; chain predictions (each is independent — "if X then Y" splits into one prediction per claim).

When to write a prediction

• The operator just asked you to predict something explicitly ("what do you think will happen if…").
• You're about to surface something speculative to the operator (a Bluesky post, a research finding, a flagged anomaly) and there's a clear "did this land or not?" question 1-7 days out.
• You're proposing a protected-surface change via open_proposal / submit_proposal — include a prediction in the proposal rationale or PR body so the audit can verify it after merge.
• You're noticing a pattern and want to test whether it holds.

Don't predict trivially-true or trivially-checkable things. The test is: would you be willing to be wrong, and would being wrong teach you something?

Format

Use the bundled CLI:

mimir predictions add \
  --claim "Tim will reply to my deep-agents post within 24h" \
  --kind binary \
  --horizon-hours 24 \
  --verifiable-by operator_review \
  --rationale "Tim has replied to 80% of similar posts within 12h"

Returns the prediction id (e.g. pred-2026-05-02-a1b2). Record it in your reasoning so the operator can find it later.

Kinds

• binary — true/false claim. Marked correct/wrong by review.
• numeric — "X will be ~N". Use --target N and optional --tolerance T.
• tool_freq — "Tool/Skill X will be invoked ≥N times in window". Auto-verifiable: counts tool_call events in turns.jsonl over the horizon. Use --target-tool Read --target N.
• error_rate — "errors of class X will drop". Auto-verifiable: counts tool_call_denied / error events over before/after windows split at made_at. Use --target N for the threshold ratio (e.g. 0.5 = halved).

Verifiable-by

• operator_review — operator marks via mimir predictions mark. Default for binary predictions about external behavior.
• events_jsonl — auto-verifiable from events.jsonl. Default for error_rate and similar event-derived signals.
• turns_jsonl — auto-verifiable from turns.jsonl. Default for tool_freq and similar turn-derived signals.

Workflow

When you write a prediction:

1. State the claim out loud in your reply (so the operator sees it).
2. Call mimir predictions add ... via Bash.
3. Note the returned id. If the operator might want to mark it later, surface the id in your reply.
4. End the turn.

Reviewing predictions

The reflection skill calls mimir predictions review weekly to surface past-horizon predictions ready for evaluation. You don't need to invoke that yourself unless an operator asks "how did your predictions do?"

If asked directly:

mimir predictions review --horizon-elapsed-only

Auto-verifiable predictions (events_jsonl / turns_jsonl kinds) score themselves on review. For operator_review predictions, the operator marks them via:

mimir predictions mark <id> --status correct \
  --actual "Tim replied at 18:42 with positive comment" \
  --lesson ""

When --status wrong, the --lesson is required: trace the incorrect assumption to a memory block and update it.

Stats

mimir predictions stats --days 30

Shows accuracy by kind, by author (agent/operator), and a calibration curve when there are enough samples.

What NOT to predict

• Shield-tic predictions. Predictions whose options all have similar probabilities and similar shapes — Meta-1 silence-correct 0.95, Meta-2 logging-correct 0.95, Meta-3 verification-correct 0.95 — are calibration noise that drowns real signal. The test: would the resolution of this prediction change what you'd do next? If no, it's filler — drop it. "Calibrated at 95%" inflated by shield-tics is worse than no calibration data because it lies to the reflection pass.
• Re-narrating the structured field as prose. If you wrote predictions: a 0.45, b 0.30, c 0.25 in the structured field, don't then say in the body "I think a is most likely." Pick one surface — the structured field is what review reads.
• Predictions you'd have made without thinking. "The build will probably pass" without a specific reason isn't a calibration data point; it's reflex. A real prediction names why you expect the outcome, so a wrong call traces back to a broken assumption.

Self-reminders

• Be specific. "Things will be better" is unverifiable. "Read errors will drop below 5/day in the next week" is verifiable.
• Record the rationale. When the prediction is wrong, the rationale is what gets traced back to the broken assumption.
• Don't over-predict. A few well-formed predictions per week beats a flood of vague ones. The point is calibration, not output volume.
• Past-horizon counts. If you see N predictions past their horizon in ## Recent feedback signals, work through them before adding more.