Ncrs Body Brain Coevolution Eval

ncrs-body-brain-coevolution-eval

A Unified Substrate for Body-Brain Co-evolution — Pontes-Filho et al. (2022) (arXiv:2203.12066, 2022)

What this evaluates

This evaluation probes the ability of a unified neural cellular automaton substrate to simultaneously evolve robot morphology and control policies for navigation and manipulation tasks. It measures how well decentralized, local interactions enable robots to chase light, navigate obstacles, and carry objects in custom simulation environments.

Datasets

• NCRS Custom Simulation Environments (LC, LCO, CBT) — total ?; splits: test (-1); repo https://github.com/sidneyp/neural-cellular-robot-substrate

Metrics

• fitness score (primary) — range: [0, 1]
  • Custom reward function from the simulation environment that aggregates task-specific objectives (e.g., distance to target, ball proximity, movement efficiency). Higher values indicate better performance.
• QD-score — range: [0, 1]
  • Quality-Diversity metric that measures the volume of the behavior space covered by the evolved population, weighted by the fitness of each behavior configuration.
• success rate — range: percent
  • Percentage of successful task completions averaged over 100 independent simulation runs per model.

Input / output format

Input: Initial seed state for the neural cellular automaton (NCA), sensor inputs (e.g., light position), and environment configuration (grid size, obstacles, target/ball positions).

Output: Per time-step: NCA cell state updates during the developmental phase, followed by actuator/control signals during the operational phase.

Scoring recipe

def compute_fitness(env, agent):
    return env.get_reward() # Custom reward function

def compute_qd_score(elites, feature_extractor, fitness_func):
    behaviors = [feature_extractor(e) for e in elites]
    return quality_diversity_metric(behaviors, [fitness_func(e) for e in elites])

def compute_success_rate(agent, env, num_runs=100):
    successes = sum(1 for _ in range(num_runs) if env.run_and_check_success(agent))
    return (successes / num_runs) * 100

Common pitfalls

• Higher fitness scores do not necessarily correlate with higher success rates in the simulation (e.g., CMA-ME achieved higher fitness but lower success rate in LC).
• Success rate is highly sensitive to the specific definition of task completion (e.g., passing the center of the body through an obstacle passage vs. just reaching the target).
• QD-score values are only comparable if the same feature space, emitter configuration, and elite selection criteria are used.

Evidence (verbatim from paper)

The results indicate that CMA-ES and CMA-ME can reach almost the same fitness scores after training. However, CMA-ME has fewer generations for the 15 emitters (4,000 generations per emitter).

Citation

@misc{pontesfilho2022unified,
  title={A Unified Substrate for Body-Brain Co-evolution},
  author={Pontes-Filho et al. (2022)},
  year={2022},
  note={arXiv:2203.12066}
}

• arXiv: 2203.12066