Multi Agent Evolve

Multi-Agent Evolve: Self-Improvement via Co-evolutionary RL

LLM reasoning improvement typically requires external verifiers (Python, human feedback). Multi-Agent Evolve eliminates this dependency through three co-evolving agents that provide internal supervision.

The proposer-solver-judge framework creates a closed learning loop where agents improve by interacting with each other.

Core Concept

Three specialized agents from single base LLM:

• Proposer: generates diverse questions
• Solver: attempts to solve them
• Judge: evaluates both question quality and solution correctness

All three undergo RL simultaneously, creating adaptive curriculum as proposer generates harder questions.

Architecture Overview

• Three agent roles instantiated from same LLM
• Shared parameter backbone with role-specific prompting
• Reward signals derived from agent interactions
• Co-evolutionary training loop with alternating optimization

Implementation Steps

Define agent roles with role-specific prompting. Each agent is a view of the same LLM with different prompts:

class MultiAgentSystem:
    def __init__(self, base_llm, model_size='3B'):
        self.base_llm = base_llm
        self.model_size = model_size

        # Role-specific system prompts
        self.proposer_prompt = (
            "You are a question generator. Generate challenging questions "
            "that test reasoning. Format: QUESTION: [question]"
        )

        self.solver_prompt = (
            "You are a problem solver. Given a question, provide step-by-step "
            "reasoning and a final answer. Format: REASONING: [steps]\nANSWER: [answer]"
        )

        self.judge_prompt = (
            "You are an evaluator. Rate question quality (1-5) and solution "
            "correctness (1-5). Format: QUESTION_SCORE: [score]\nSOLUTION_SCORE: [score]"
        )

    def generate_question(self, context=""):
        """Proposer generates question."""
        prompt = self.proposer_prompt + f"\nContext: {context}\n"
        question = self.base_llm.generate(prompt, max_tokens=100)
        return question

    def solve_question(self, question):
        """Solver generates solution."""
        prompt = self.solver_prompt + f"\nQuestion: {question}\n"
        solution = self.base_llm.generate(prompt, max_tokens=500)
        return solution

    def judge_quality(self, question, solution):
        """Judge evaluates both."""
        prompt = self.judge_prompt
        prompt += f"\nQuestion: {question}\nSolution: {solution}\n"
        evaluation = self.base_llm.generate(prompt, max_tokens=50)

        # Parse scores
        scores = self._parse_scores(evaluation)
        return scores

    def _parse_scores(self, evaluation):
        """Extract question and solution scores."""
        # Simple parsing - in practice use more robust extraction
        question_score = 3.0  # Default
        solution_score = 3.0

        import re
        q_match = re.search(r'QUESTION_SCORE:\s*(\d)', evaluation)
        s_match = re.search(r'SOLUTION_SCORE:\s*(\d)', evaluation)

        if q_match:
            question_score = float(q_match.group(1))
        if s_match:
            solution_score = float(s_match.group(1))

        return question_score, solution_score

Implement the co-evolution training loop where agents interact and learn:

def coevolution_training_step(agents, rl_trainer, num_interactions=100):
    """Single training step with agent interaction and RL updates."""
    experiences = []

    for _ in range(num_interactions):
        # Proposer generates question
        context = "Mathematics, Reasoning"
        question = agents.generate_question(context)

        # Solver solves it
        solution = agents.solve_question(question)

        # Judge evaluates both
        q_score, s_score = agents.judge_quality(question, solution)

        # Record experience
        experience = {
            'question': question,
            'solution': solution,
            'question_score': q_score,
            'solution_score': s_score,
            'total_reward': 0.6 * q_score + 0.4 * s_score
        }
        experiences.append(experience)

    # RL training: optimize all three roles
    proposer_loss = rl_trainer.optimize_proposer(experiences)
    solver_loss = rl_trainer.optimize_solver(experiences)
    judge_loss = rl_trainer.optimize_judge(experiences)

    return {
        'proposer_loss': proposer_loss,
        'solver_loss': solver_loss,
        'judge_loss': judge_loss,
        'avg_reward': sum(e['total_reward'] for e in experiences) / len(experiences)
    }

def train_multi_agent_evolve(agents, num_rounds=1000, interaction_per_round=100):
    """Full training loop."""
    rl_trainer = RLTrainer(agents.base_llm)

    for round_idx in range(num_rounds):
        metrics = coevolution_training_step(agents, rl_trainer, interaction_per_round)

        if round_idx % 100 == 0:
            print(f"Round {round_idx}: "
                  f"Avg Reward: {metrics['avg_reward']:.3f}, "
                  f"Proposer Loss: {metrics['proposer_loss']:.4f}")

    return agents

Practical Guidance

When to use:

• Self-improvement without external verifiers
• Closed-loop reasoning training
• Domains with no ground-truth evaluation functions
• Scaling reasoning with cheaper models

When NOT to use:

• When accurate external verification exists (more stable)
• Real-time applications (training overhead)
• Tasks requiring factual correctness (no ground truth in loop)

Common pitfalls:

• Judge becoming too lenient (reward inflation)
• Proposer generating trivial questions (solver overfits)
• Unbalanced role optimization (one dominates)
• Insufficient diversity in proposals (stuck in local optimum)

Reference: Multi-Agent Evolve on arXiv