terraLingua-eval

terraLingua-eval

TerraLingua: Emergence and Analysis of Open-endedness in LLM Ecologies — Giuseppe Paolo et al. (2026) (arXiv:2603.16910, 2026)

What this evaluates

Evaluates the emergence of open-ended dynamics, sustained novelty, and social organization in a persistent multi-agent LLM ecology. It probes how environmental constraints, agent personality, and artifact persistence shape cumulative cultural evolution and cooperative norms.

Datasets

• TerraLingua Simulation Environment — total ?; splits: (unstated); repo https://github.com/cognizant-ai-lab/terralingua

Metrics

• artifact novelty score (primary) — range: other
  • Generated by the AI Anthropologist (Claude Sonnet 4.5) based on artifact content and evolution. Averaged over N=5 independent samples to reduce variance.
• AI Anthropologist vs human assessment agreement — range: percent
  • Qualitative and quantitative comparison between the AI Anthropologist's systematic analyses of agent behavior, group structure, and artifact evolution, and independent human expert evaluations.

Input / output format

Input: Agents receive: current timestep observation, internal memory (soft-limited to 150 tokens, hard-limited to 250), previous action, and a personality genome (OCEAN+ traits in [-1,1]). The AI Anthropologist receives: group logs (split into overlapping segments if exceeding context limits), artifact data, and task-specific prompts.

Output: Agents produce: discrete actions and text-based artifacts (capped at 500 tokens). The AI Anthropologist produces: structured analyses of behavior/group structure, artifact classifications, phylogeny reconstructions, and novelty scores.

Scoring recipe

def compute_metrics(artifacts, group_logs, human_evals):
    novelty_scores = []
    for _ in range(5):
        score = anthropologist_llm.evaluate_novelty(artifacts)
        novelty_scores.append(score)
    artifact_novelty = mean(novelty_scores)
    ai_analysis = anthropologist_llm.analyze(group_logs)
    agreement = compare(ai_analysis, human_evals)
    return artifact_novelty, agreement

Common pitfalls

• Runs terminate early upon population extinction, causing variable simulation durations that must be normalized or accounted for in cross-condition comparisons.
• Context window limits require splitting group logs into overlapping segments for the AI Anthropologist, which may introduce boundary effects in analysis.
• LLM alignment biases toward cooperative defaults are counterbalanced by the OCEAN+ personality extension, which must be correctly initialized to avoid skewed baselines.

Evidence (verbatim from paper)

The study also evaluated the AI Anthropologist by comparing its analyses with human assessments. ... Artifact novelty scores were averaged over N=5 samples.

Citation

@misc{paolo2026terraLingua,
  title={TerraLingua: Emergence and Analysis of Open-endedness in LLM Ecologies},
  author={Giuseppe Paolo et al. (2026)},
  year={2026},
  note={arXiv:2603.16910}
}

• arXiv: 2603.16910