# Sciarena Evaluation Platform

> Build community-driven evaluation platforms for scientific tasks using pairwise model comparisons and human voting. Assess foundation models on literature-grounded reasoning without automated metrics.

- Skill: `adu2021/sciarena-evaluation-platform` (Agent Skill)
- Install (CLI): `npx skillmds@latest add adu2021/sciarena-evaluation-platform`
- Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/sciarena-evaluation-platform/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: adu2021 (https://skillmd.com/u/adu2021)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/adu2021/sciarena-evaluation-platform

---


# SciArena: Community-Driven Evaluation for Scientific Reasoning

Evaluating language models on scientific literature tasks is challenging because correct answers aren't verifiable through simple string matching or automated metrics. A paper's technical contribution is nuanced; a model's explanation of a methodology can be correct in multiple ways; reasoning about research significance requires human judgment. Automated evaluation metrics (BLEU, ROUGE) fail because they don't capture semantic correctness of scientific reasoning. The traditional solution—hiring annotators to score outputs—is expensive and doesn't scale to many models.

SciArena solves this through community voting: researchers evaluate model outputs on scientific tasks by comparing two models side-by-side and voting for which response is better. This crowdsourced approach leverages existing expertise (the research community) and scales to 47 models. The platform also provides SciArena-Eval, a meta-evaluation benchmark measuring how well automated systems can predict which model researchers would prefer.

## Core Concept

The key insight is that **pairwise comparisons are easier than absolute scoring**. Instead of asking "rate this explanation on a scale of 1-5," ask "which explanation better explains the research methodology?" Humans naturally prefer comparisons because they can identify relative strengths. By collecting many pairwise votes (20,000+ across the platform), you build a leaderboard that reflects community consensus.

The approach has three components:

1. **Pairwise evaluation interface**: Show two model outputs; voters choose which is better
2. **Aggregation algorithm**: Convert pairwise votes into model rankings (similar to Elo ratings)
3. **Meta-evaluation benchmark**: Use human votes as ground truth to test automated evaluators

## Architecture Overview

The evaluation platform consists of:

- **Task Pool**: Scientific literature questions requiring understanding of papers, methods, and contributions
- **Model Submission System**: Interface for evaluating new models on the task pool
- **Voting Interface**: Pairwise comparison tool allowing researchers to vote on response quality
- **Ranking System**: Aggregates votes into a leaderboard using ranking algorithms (Bradley-Terry, Elo)
- **Meta-evaluation Suite**: Automated metrics tested against human preferences to identify the best evaluation method
- **Leaderboard Dashboard**: Real-time ranking of models with confidence intervals

## Implementation

**Step 1: Prepare task dataset with scientific questions**

Create a pool of questions grounded in scientific literature that require deep understanding.

```python
def create_scientific_task_dataset(papers, num_questions_per_paper=3):
    """
    Generate evaluation questions from research papers.
    Each question requires understanding the paper's methodology, results, or contributions.
    """
    tasks = []

    for paper in papers:
        # Extract key aspects from paper
        abstract = paper['abstract']
        methods = paper['methods']
        results = paper['results']
        title = paper['title']

        # Generate different types of questions
        questions = [
            {
                'type': 'methodology',
                'question': f"Explain the key methodology in this paper: {title}. "
                           f"What are the main technical contributions?",
                'context': methods,
                'paper_id': paper['id']
            },
            {
                'type': 'interpretation',
                'question': f"What do the experimental results tell us about {title}? "
                           f"What are the key findings?",
                'context': results,
                'paper_id': paper['id']
            },
            {
                'type': 'significance',
                'question': f"Why is {title} important to the field? "
                           f"How does it advance our understanding?",
                'context': abstract,
                'paper_id': paper['id']
            }
        ]

        for q in questions:
            tasks.append(q)

    return tasks
```

**Step 2: Implement the pairwise voting interface**

Structure the evaluation to collect human preferences between model pairs.

```python
class PairwiseVotingTask:
    """
    Represents a single pairwise comparison task.
    One research question, two model outputs, human vote.
    """

    def __init__(self, task_id, question, model_a_output, model_b_output):
        self.task_id = task_id
        self.question = question
        self.model_a_output = model_a_output
        self.model_b_output = model_b_output
        self.vote = None  # 'a', 'b', or 'tie'
        self.voter_expertise = None
        self.confidence = None

    def to_dict(self):
        """Format for JSON serialization or database storage."""
        return {
            'task_id': self.task_id,
            'question': self.question,
            'model_a': self.model_a_output,
            'model_b': self.model_b_output,
            'vote': self.vote,
            'expertise': self.voter_expertise,
            'confidence': self.confidence
        }

def generate_pairwise_tasks(models, task_dataset, pairs_per_task=3):
    """
    Create pairwise comparison tasks from all model outputs.
    Each question is evaluated by multiple model pairs for statistical robustness.
    """
    # First, generate outputs from all models
    model_outputs = {}

    for model in models:
        model_outputs[model['name']] = []

        for task in task_dataset:
            output = model['generate_fn'](task['question'], task['context'])
            model_outputs[model['name']].append(output)

    # Now create pairwise tasks
    pairwise_tasks = []

    for task_idx, task in enumerate(task_dataset):
        # Select pairs of models to compare (round-robin or random)
        model_names = list(model_outputs.keys())
        pairs = []

        # Each task compared by multiple pairs for robustness
        for i in range(pairs_per_task):
            # Random pair selection
            pair = random.sample(model_names, 2)
            pairs.append(pair)

        for model_a, model_b in pairs:
            pairwise_task = PairwiseVotingTask(
                task_id=f"{task_idx}_{model_a}_vs_{model_b}",
                question=task['question'],
                model_a_output=model_outputs[model_a][task_idx],
                model_b_output=model_outputs[model_b][task_idx]
            )
            pairwise_tasks.append(pairwise_task)

    return pairwise_tasks

def collect_vote(task, voter_id, vote, confidence=0.5):
    """
    Record a vote with optional confidence score.
    Confidence helps weight reliable voters more heavily.
    """
    task.vote = vote  # 'a', 'b', or 'tie'
    task.voter_expertise = get_voter_expertise(voter_id)
    task.confidence = confidence  # 0.0 to 1.0

    # Store in database
    store_vote_in_database(task.to_dict(), voter_id)
```

**Step 3: Aggregate votes into model rankings**

Convert pairwise votes into a leaderboard using ranking algorithms.

```python
def aggregate_votes_bradley_terry(pairwise_votes, models, max_iterations=100):
    """
    Use Bradley-Terry model to convert pairwise comparisons into scalar scores.
    This is the standard method for ranking from pairwise preferences.
    """
    # Initialize scores
    scores = {model: 1.0 for model in models}

    for iteration in range(max_iterations):
        # Update each model's score based on votes
        new_scores = {}

        for model in models:
            # Count wins and losses
            total_comparisons = 0
            weighted_wins = 0

            for vote in pairwise_votes:
                # Did this model appear in this comparison?
                if vote['model_a'] == model:
                    opponent = vote['model_b']
                    is_winner = vote['vote'] == 'a'
                elif vote['model_b'] == model:
                    opponent = vote['model_a']
                    is_winner = vote['vote'] == 'b'
                else:
                    continue

                # Weight by voter confidence and expertise
                weight = vote['confidence'] * vote['expertise_weight']
                total_comparisons += weight

                if is_winner:
                    weighted_wins += weight
                    # Bradley-Terry: account for opponent strength
                    weighted_wins += weight * scores[opponent] / (scores[model] + scores[opponent])
                else:
                    # Loss: decrease proportionally to opponent strength
                    weighted_wins -= weight * scores[opponent] / (scores[model] + scores[opponent])

            # Update score: empirical win rate adjusted for opponent strength
            if total_comparisons > 0:
                new_scores[model] = scores[model] * (1 + 0.01 * weighted_wins / total_comparisons)
            else:
                new_scores[model] = scores[model]

        # Check for convergence
        score_change = sum(abs(new_scores[m] - scores[m]) for m in models) / len(models)
        scores = new_scores

        if score_change < 0.001:
            break

    # Normalize to 0-100 scale
    min_score = min(scores.values())
    max_score = max(scores.values())
    normalized_scores = {
        model: 100 * (scores[model] - min_score) / (max_score - min_score)
        for model in models
    }

    return normalized_scores

def compute_leaderboard_with_confidence(votes, models):
    """
    Compute leaderboard with 95% confidence intervals using bootstrap.
    This shows ranking uncertainty.
    """
    # Bootstrap: resample votes with replacement
    bootstrap_scores = []

    for bootstrap_iteration in range(1000):
        resampled_votes = [random.choice(votes) for _ in range(len(votes))]
        scores = aggregate_votes_bradley_terry(resampled_votes, models)
        bootstrap_scores.append(scores)

    # Compute confidence intervals
    leaderboard = []

    for model in models:
        model_scores = [scores[model] for scores in bootstrap_scores]
        mean_score = np.mean(model_scores)
        ci_low = np.percentile(model_scores, 2.5)
        ci_high = np.percentile(model_scores, 97.5)

        leaderboard.append({
            'model': model,
            'score': mean_score,
            'ci_low': ci_low,
            'ci_high': ci_high
        })

    # Sort by score
    return sorted(leaderboard, key=lambda x: x['score'], reverse=True)
```

**Step 4: Create meta-evaluation benchmark**

Use human votes as ground truth to evaluate automated metrics.

```python
def evaluate_automated_metric(metric_function, human_votes, models):
    """
    Test how well an automated metric predicts human preferences.
    Compute correlation between metric rankings and human consensus.
    """
    # Get human consensus ranking
    human_scores = aggregate_votes_bradley_terry(human_votes, models)
    human_ranking = sorted(models, key=lambda m: human_scores[m], reverse=True)

    # Get automated metric ranking
    metric_scores = {}

    for model in models:
        score = metric_function(model)
        metric_scores[model] = score

    metric_ranking = sorted(models, key=lambda m: metric_scores[m], reverse=True)

    # Compute Spearman correlation between rankings
    human_ranks = {model: i for i, model in enumerate(human_ranking)}
    metric_ranks = {model: i for i, model in enumerate(metric_ranking)}

    correlation = spearmanr(
        [human_ranks[m] for m in models],
        [metric_ranks[m] for m in models]
    )[0]

    # Also compute accuracy: how often metric predicts the winner?
    metric_correct = 0
    total_comparisons = 0

    for vote in human_votes:
        model_a, model_b = vote['model_a'], vote['model_b']
        human_preference = vote['vote']

        metric_preference = 'a' if metric_scores[model_a] > metric_scores[model_b] else 'b'

        if metric_preference == human_preference:
            metric_correct += 1
        total_comparisons += 1

    metric_accuracy = metric_correct / total_comparisons

    return {
        'spearman_correlation': correlation,
        'prediction_accuracy': metric_accuracy,
        'is_reliable': correlation > 0.75
    }
```

## Practical Guidance

| Aspect | Recommended Value | Notes |
|---|---|---|
| Votes per task | 3-5 | Balances coverage and robustness |
| Total tasks | 100-500 | 100 minimum for statistical significance |
| Total votes collected | 5,000-20,000 | More votes = more stable rankings |
| Expert voter weight | 1.5-2.0x | Upweight domain experts |
| Confidence threshold | 0.3+ | Filter out uncertain votes |

**When to use community-driven evaluation:**
- You have tasks where correctness is subjective or open-ended (literary analysis, scientific reasoning)
- You want to evaluate many models without hiring expensive annotators
- You have an active community (researchers, practitioners)
- You need to assess relative model quality rather than absolute correctness

**When NOT to use community voting:**
- You need deterministic, verifiable correctness (math problems, fact-checking)
- Your community is too small to provide enough votes
- You need rapid evaluation (voting takes time to collect votes)
- Your evaluation criteria are highly specialized and hard to explain to voters

**Common pitfalls:**
- **Insufficient votes**: If a model pair is compared only 1-2 times, rankings are unreliable. Aim for 3-5 votes per comparison.
- **Voter bias**: Researchers might prefer papers from their own field. Use stratified sampling to expose voters to diverse domains.
- **Non-convergence**: Bradley-Terry assumes consistency; if many "ties" or contradictory votes appear, the model may not have a clear skill difference. Investigate specific examples.
- **Leaderboard gaming**: Models might be optimized to win against specific competitors rather than improving genuinely. Rotate model pairs to prevent this.

## Reference

SciArena: An Open Evaluation Platform for Foundation Models in Scientific Literature
https://arxiv.org/abs/2507.01001