Researchtown Eval

researchtown-eval

ResearchTown: Simulator of Human Research Community — Yu et al. (2024) (arXiv:2412.17767, 2024)

What this evaluates

Evaluates whether a multi-agent research simulator can accurately reconstruct masked research nodes (papers and reviews) from their local neighborhood context in a collaborative graph. It probes the model's ability to capture interdisciplinary collaboration patterns and realistic academic writing styles.

Datasets

• ResearchTown simulated community graph — total ?; splits: test (-1); repo https://github.com/ulab-uiuc/research-town

Metrics

• reconstruction_similarity (primary) — range: [0, 1]
  • Cosine similarity between text embeddings of the predicted node (paper or review) and its ground-truth counterpart, computed using OpenAI's text-embedding-large-3 model. Scores range from 0 (no similarity) to 1 (identical).

Input / output format

Input: A community graph G(V, E) with a masked target node v, along with the text-form features x_u of all neighboring nodes u in N(v).

Output: Text-form hidden states h_v for paper writing, or text-form review predictions r_v for review writing.

Scoring recipe

def compute_similarity(pred_text, gold_text):
    pred_emb = get_embedding(pred_text, model='text-embedding-large-3')
    gold_emb = get_embedding(gold_text, model='text-embedding-large-3')
    return cosine_similarity(pred_emb, gold_emb)

# For paper writing:
paper_score = mean([compute_similarity(h_v, h_v_star) for v in test_nodes])
# For review writing (using gold papers to avoid error accumulation):
review_score = mean([compute_similarity(r_v, r_v_star) for v in test_nodes])

Common pitfalls

• Using predicted paper text instead of ground-truth paper text to generate review predictions, which accumulates errors and unfairly penalizes the model.
• Assuming ground-truth papers/reviews are perfect; the paper notes they are used as proxies for human research community behavior.
• Not properly masking the target node's hidden state before neighborhood aggregation, which leaks the answer.

Evidence (verbatim from paper)

validated via a scalable, objective node-masking benchmark (ResearchBench) measuring reconstruction similarity, achieving average scores of 0.68 (paper writing) and 0.49 (review writing). We utilize state-of-the-art embedding models like text-embedding-large-3 to build distance function for d_p(h_v, h_v^) and d_r(r_v, r_v^).

Citation

@misc{yu2024researchtown,
  title={ResearchTown: Simulator of Human Research Community},
  author={Yu et al. (2024)},
  year={2024},
  note={arXiv:2412.17767}
}

• arXiv: 2412.17767