Prediction Toolkit

Prediction Toolkit

Purpose: When a user question involves predicting the future, quantifying uncertainty, updating a belief with evidence, or inferring what would happen under a different decision, this skill picks the right tool from seven families and runs it. It is a decision router + a set of runnable methods.

Without this skill, Claude defaults to linear prose reasoning and quietly skips quantitative methods. With this skill, Claude reaches for the method that actually fits the shape of the problem.

When to use this skill

Load this skill when any of these apply:

• The question is about the future ("how long will...", "what's the chance...", "forecast X")
• The question asks for uncertainty quantification ("what's the range of likely outcomes", "what's the P95 worst case")
• The question involves updating a belief ("given this new data, should I change my estimate")
• The question involves comparing decisions under uncertainty ("should we do A or B")
• The question asks why something happened or what would happen if we changed X (causal)
• The question involves small data and confidence bounds ("5 out of 100 — how confident can I be")
• The question involves aggregating expert opinions or market signals

Do NOT load this skill for:

• Deterministic calculations (arithmetic, currency conversion, geometry) — just do the math
• Questions with explicit known answers (look them up)
• Tasks where there's no uncertainty at all

The seven method families

Each family solves a different shape of prediction problem. The router below maps problem shape → method.

Where "guide" is listed, the toolkit tells you when to reach for that family and what library/approach to use, rather than duplicating production libraries. This is deliberate — bad implementations of these methods are worse than pointing at the right library.

How to pick a method (decision tree)

Is this about predicting a FUTURE outcome?
├── YES
│   ├── Do you have historical time-series data?
│   │   ├── YES → statistical forecasting
│   │   │         • short series, want simple forecast → exponential_smoothing
│   │   │         • long series, need confidence bands → ARIMA (external / scipy)
│   │   │         • many features, nonlinear → machine learning
│   │   └── NO  → do you have a model + uncertain inputs?
│   │             ├── YES → Monte Carlo (use monte-carlo-predictor skill)
│   │             └── NO  → crowd aggregation / expert elicitation
│   │
│   └── Need to combine multiple methods?
│       → Monte Carlo for uncertainty + ML point forecast as an input
│
└── NO — this is about explaining the past or updating a belief
    ├── Updating a probability with new evidence?
    │   → bayesian_update
    │
    ├── Small sample, need CI on a mean/median/etc.?
    │   → bootstrap
    │
    ├── "Would X have changed the outcome?" (counterfactual)
    │   → causal inference (read methods/causal-inference.md)
    │
    └── Need the underlying physics to extrapolate?
        → first-principles (read methods/first-principles.md)

Use python -m engine.router recommend "<problem description>" to get an automated recommendation.

Workflow

When a user asks a prediction/forecasting question:

1. Classify the shape of the problem using the decision tree above (or run router.py recommend).
2. Reach for the matching method:
   • Monte Carlo: invoke the monte-carlo-predictor skill
   • Bootstrap: from engine.bootstrap import bootstrap_ci
   • Bayesian: from engine.bayesian_update import beta_binomial_update
   • Exponential smoothing: from engine.exponential_smoothing import forecast
   • For ML/crowd/causal/first-principles: read the corresponding methods/*.md guide — they document when to escalate to external tooling or specific libraries
3. Combine when needed:
   • Bayesian prior + Monte Carlo forward simulation
   • Bootstrap CI on a ML model's predictions
   • Exponential smoothing forecast as a Monte Carlo input
4. Report honestly:
   • State which method was used and why
   • Name the key assumptions (distributions, priors, smoothing params)
   • Quote uncertainty ranges, not just point estimates
   • Flag when the method might be wrong (tiny sample, untested prior, structural break risk)

Interfaces

All four integration paths, same as monte-carlo-predictor:

# Python API
from engine import bootstrap_ci, bayesian_update, forecast, recommend_method

# CLI
python -m engine router recommend "forecast next quarter's revenue"
python -m engine bootstrap --data data.csv --stat mean --ci 0.95
python -m engine bayes --prior "beta(2,8)" --successes 5 --trials 20
python -m engine forecast --series data.csv --periods 4

# JSON stdin/stdout
echo '{"method":"bootstrap","data":[1,2,3,4,5],"stat":"mean"}' | python -m engine run -

# MCP server
python -m engine.mcp_server

See engine/agent_contract.md for the stable interface spec.

Design rules

1. No raw eval. All user-supplied expressions go through the safe evaluator from monte-carlo-predictor (or equivalent AST whitelist).
2. Pure data in, pure data out. Specs are JSON/YAML; outputs are JSON. Safe for LLM agents to generate.
3. Pure Python fallbacks. Methods work without numpy/scipy; they use optimized libraries when available.
4. Each method is independently callable. No "god-mode" function — the agent composes methods.
5. Honest uncertainty. Every output includes confidence bounds and caveats. No false precision.

Working examples

• examples/ab-test-bayesian/ — Bayesian A/B test update
• examples/sales-forecast-smoothing/ — Time-series forecast with exponential smoothing
• examples/bootstrap-ci/ — Confidence intervals from a tiny sample

Tests

cd skills/prediction-toolkit && python3 -m engine.tests.test_toolkit

All tests must pass before shipping any changes.

Related skills

• monte-carlo-predictor — The Monte Carlo family lives there. This skill routes to it for simulation problems. Do not duplicate.

Honest limits

• The router is a heuristic. It suggests, it doesn't decide. Override when you have context.
• ML/crowd/causal/first-principles are guides, not implementations in this skill. They tell you when and how to reach for external tooling. Building them in Python without scipy/sklearn/pyro would be dishonest about quality.
• The value of the toolkit is 80% knowing which tool to reach for and 20% the specific implementation. Treat the router as the main product.