Rule110 Prediction Eval

rule110-prediction-eval

The Need for a Big World Simulator: A Scientific Challenge for Continual Learning — Kumar et al. (2024) (arXiv:2408.02930, 2024)

What this evaluates

Probes a model's continual learning capability in a partially observable, non-stationary synthetic environment based on the Rule 110 cellular automaton. It measures how well capacity-constrained agents adapt to gradual distribution shifts induced by increasing prediction horizons and evolving task parameters.

Datasets

• Rule 110 Prediction Environment — total ?; splits: test (-1)

Metrics

• online accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted binary state cells at horizon K over the episode length T=100. Computed as 1 - (number of mispredicted cells / total predicted cells).

Input / output format

Input: Sequential observations of the first 16 dimensions of a 32-dimensional binary state vector from a Rule 110 cellular automaton, provided step-by-step over an episode length of T=100.

Output: Binary prediction of the full 32-dimensional state vector at a specified prediction horizon K ∈ {1, 2, 4, 8, 16}.

Scoring recipe

def compute_online_accuracy(predictions, targets, horizon_k):
    correct = 0
    total = 0
    for pred, target in zip(predictions, targets):
        if pred[horizon_k] == target[horizon_k]:
            correct += 1
        total += 1
    return correct / total if total > 0 else 0.0

Common pitfalls

• Partial observability means unobserved cells influence predictions at the observable boundary, making larger K significantly harder due to information propagation delay.
• The environment induces gradual non-stationarity by incrementing the initial state τ every T=100 steps, requiring continual adaptation rather than static fine-tuning.

Evidence (verbatim from paper)

Figure 5: Top: Online accuracy for the medium-sized neural network without regularization (left) and with regenerative regularization (right). On the bottom figure, we see that doubling the capacity leads to approximately half the error at larger prediction horizons suggesting that this indeed simulates the big world properties that we have outlined.

Citation

@misc{kumar2024bigworldsimulator,
  title={The Need for a Big World Simulator: A Scientific Challenge for Continual Learning},
  author={Kumar et al. (2024)},
  year={2024},
  note={arXiv:2408.02930}
}

• arXiv: 2408.02930