Ludii Rbg Eval

ludii-rbg-eval

An Empirical Evaluation of Two General Game Systems: Ludii and RBG — Piette et al. (2019) (arXiv:1907.00244, 2019)

What this evaluates

Evaluates the computational efficiency and human-readability of two General Game Playing systems (Ludii and RBG) by measuring their playout throughput and the token count required to define game rules.

Datasets

• Unspecified game rule suite — total ?; splits: (unstated)

Metrics

• playouts (primary) — range: other
  • Number of flat Monte Carlo random playouts executed per second by the system's reasoning engine.
• token_count — range: other
  • Total number of tokens required to define a game's rules in the system's language.

Input / output format

Input: Game rule definitions formatted in either Ludii's ludeme-based syntax or RBG's two-tiered language.

Output: System-reported token count for the definition, and the total number of random playouts completed within a fixed 10-minute runtime.

Scoring recipe

def evaluate_system(game_rules, system, time_limit_sec=600):
    token_count = count_tokens(system.parse(game_rules))
    start_time = time.time()
    playout_count = 0
    while time.time() - start_time < time_limit_sec:
        playout_count += system.run_flat_mcts_playout()
    playouts_per_sec = playout_count / (time.time() - start_time)
    return playouts_per_sec, token_count

Common pitfalls

• RBG has both an interpreter and a compiler; the paper compares Ludii to both, so results must be clearly attributed to the correct RBG mode.
• Token counting methodology is not formally defined in the excerpt, making cross-system readability comparisons sensitive to tokenizer choices.
• The 10-minute fixed runtime may not allow MCTS to converge for complex games, so playout counts reflect raw engine speed rather than search quality.

Evidence (verbatim from paper)

Hence, we use flat Monte Carlo playouts as the metric for comparing the efficiency between Ludii and RBG. In order to make a comparison between the human-readability of the two systems we use two main criteria: clarity and simplicity. Clarity refers to the degree to which game descriptions would be self-explanatory to non-specialist readers, and simplicity refers to the ease with which game descriptions can be created and modified, and can be estimated by the number of tokens required to define games.

Citation

@misc{piette2019ludii,
  title={An Empirical Evaluation of Two General Game Systems: Ludii and RBG},
  author={Piette et al. (2019)},
  year={2019},
  note={arXiv:1907.00244}
}

• arXiv: 1907.00244