# Agent Long Context Benchmark > Build controllable benchmarks for evaluating long-context agents using environment rollouts. Generate diverse multi-step agent tasks that require maintaining context across extended interaction sequences, enabling evaluation of agent reasoning quality in scenarios with long history requirements. - Skill: `adu2021/agent-long-context-benchmark` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/agent-long-context-benchmark` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/agent-long-context-benchmark/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/agent-long-context-benchmark --- ## Problem Existing agent benchmarks typically use short, isolated tasks that don't reflect real-world requirements where agents must maintain context and reason over long interaction histories. Long-context agents need benchmarks that can systematically vary task complexity, history length, and environmental conditions to properly evaluate their capabilities. ## Solution Create AgentLongBench: a controllable benchmark framework that generates synthetic long-context agent tasks through environment rollouts. This approach allows you to: 1. **Control Task Complexity**: Vary the number of steps, branching decisions, and tool interactions 2. **Simulate Extended Histories**: Generate long chains of observations and actions that agents must reason over 3. **Test Context Window Usage**: Evaluate how agents handle increasing history lengths 4. **Generate Diverse Scenarios**: Create varied environmental states and task requirements programmatically ## When to Use - Evaluating long-context language models on agent tasks - Testing agent memory and reasoning capabilities with extended histories - Benchmarking context window limitations and efficiency - Developing agents for complex multi-step workflows (code generation, research, planning) - Analyzing how agent performance degrades with longer interaction histories ## When NOT to Use - Short, single-turn interaction tasks (standard benchmarks suffice) - Real-world task evaluation (use human-curated benchmarks) - Tasks requiring specialized domain knowledge (this is synthetic evaluation) ## Implementation ### Step 1: Define the Environment Simulator Create a controllable environment that generates long-context agent interactions. ```python class LongContextEnvironment: """Simulation environment for generating long interaction sequences""" def __init__(self, complexity_level=1, max_steps=50): self.complexity = complexity_level # 1-5 scale self.max_steps = max_steps self.state = self.initialize_state() self.step_count = 0 self.history = [] def initialize_state(self): """Set up initial task and environment state""" return { "goal": self.generate_goal(self.complexity), "environment_objects": self.generate_objects(self.complexity), "available_actions": self.available_actions(), "current_progress": 0.0 } def step(self, agent_action): """ Execute one agent action and return observation + reward Returns: (observation, reward, done, info) """ self.step_count += 1 # Validate and execute action is_valid = self.validate_action(agent_action) if not is_valid: observation = self.get_observation() return observation, -1.0, False, {"error": "invalid_action"} # Apply action to environment result = self.apply_action(agent_action) observation = self.get_observation() # Track in history self.history.append({ "step": self.step_count, "action": agent_action, "result": result, "state_snapshot": self.state.copy() }) # Compute reward progress_delta = self.get_progress() - self.state["current_progress"] reward = progress_delta * 10 # Scale reward signal self.state["current_progress"] += progress_delta done = (self.step_count >= self.max_steps) or self.goal_reached() return observation, reward, done, {"history_length": len(self.history)} def get_observation(self): """Return current observation including history summary""" return { "step": self.step_count, "goal": self.state["goal"], "visible_objects": self.get_visible_objects(), "recent_actions": self.history[-5:], # Last 5 actions "progress_toward_goal": self.state["current_progress"], "available_actions": self.state["available_actions"] } ``` ### Step 2: Implement Task Generation Create diverse tasks at varying complexity levels. ```python def generate_goal(complexity_level): """Generate task goal based on complexity""" if complexity_level == 1: return "Find and retrieve item X" elif complexity_level == 2: return "Complete sequence: find X, transform with Y, store in Z" elif complexity_level == 3: return "Multi-constraint planning: achieve A while maintaining B, avoid C" elif complexity_level == 4: return "Complex reasoning: infer hidden rules from observations, then achieve goal" else: return "Open-ended exploration and adaptation to changing goals" def generate_environment_rollout(num_steps, complexity): """Generate a full trajectory of environment interactions""" env = LongContextEnvironment(complexity_level=complexity, max_steps=num_steps) trajectory = [] observation = env.get_observation() done = False while not done and env.step_count < num_steps: # Simulate a sequence of reasonable actions action = select_trajectory_action(observation, env.state["goal"]) observation, reward, done, info = env.step(action) trajectory.append({ "observation": observation, "action": action, "reward": reward, "done": done }) return { "trajectory": trajectory, "final_success": env.goal_reached(), "steps_taken": env.step_count, "context_length": len(env.history) } ``` ### Step 3: Build the Benchmark Dataset Generate a collection of tasks at different complexity and length levels. ```python class AgentLongBench: """Full benchmark for long-context agent evaluation""" def __init__(self, num_tasks_per_level=10, max_context_length=1000): self.num_tasks = num_tasks_per_level self.max_context_length = max_context_length self.tasks = self.generate_benchmark() def generate_benchmark(self): """Create diverse tasks across complexity and context length dimensions""" benchmark = { "easy": [], "medium": [], "hard": [], "very_hard": [], "expert": [] } # Easy: short context, simple goals for i in range(self.num_tasks): rollout = generate_environment_rollout(num_steps=10, complexity=1) benchmark["easy"].append(rollout) # Medium: longer context, multi-step goals for i in range(self.num_tasks): rollout = generate_environment_rollout(num_steps=30, complexity=2) benchmark["medium"].append(rollout) # Hard: long context, complex reasoning for i in range(self.num_tasks): rollout = generate_environment_rollout(num_steps=50, complexity=3) benchmark["hard"].append(rollout) # Very Hard: very long context with constraints for i in range(self.num_tasks): rollout = generate_environment_rollout(num_steps=100, complexity=4) benchmark["very_hard"].append(rollout) # Expert: longest context, open-ended adaptation for i in range(self.num_tasks): rollout = generate_environment_rollout( num_steps=min(150, self.max_context_length), complexity=5 ) benchmark["expert"].append(rollout) return benchmark def evaluate_agent(self, agent, difficulty="easy"): """ Run agent on benchmark tasks and compute metrics """ tasks = self.tasks[difficulty] results = { "success_rate": 0.0, "avg_steps": 0.0, "avg_history_length": 0.0, "context_efficiency": 0.0 } successes = 0 total_steps = 0 total_history_length = 0 for task in tasks: # Reset agent for task agent.reset() # Run agent through task trajectory = task["trajectory"] task_success = False for step_data in trajectory: observation = step_data["observation"] action = agent.act(observation) # (evaluate action against expected behavior) if action == step_data["action"]: task_success = True if task_success: successes += 1 total_steps += task["steps_taken"] total_history_length += task["context_length"] results["success_rate"] = successes / len(tasks) results["avg_steps"] = total_steps / len(tasks) results["avg_history_length"] = total_history_length / len(tasks) results["context_efficiency"] = results["success_rate"] / ( results["avg_history_length"] / self.max_context_length ) return results ``` ### Step 4: Integrate with Agent Training Loop Use the benchmark for iterative agent improvement. ```python def train_agent_with_benchmark(agent, benchmark, num_epochs=10): """ Train agent using AgentLongBench with curriculum learning Start with easy tasks, progress to harder ones """ difficulties = ["easy", "medium", "hard", "very_hard", "expert"] for epoch in range(num_epochs): for difficulty in difficulties: # Evaluate on current difficulty metrics = benchmark.evaluate_agent(agent, difficulty) # Stop if success rate drops too low if metrics["success_rate"] < 0.3: continue # Train agent on this difficulty level agent.train_on_tasks(benchmark.tasks[difficulty], epochs=1) print(f"Epoch {epoch}, Difficulty {difficulty}: " f"Success={metrics['success_rate']:.2f}, " f"ContextEff={metrics['context_efficiency']:.2f}") ``` ## Key Insights - **Controllability**: Environment rollouts allow precise control over task difficulty and context length - **Curriculum Learning**: Start with easy tasks, progress systematically to harder ones - **Context Efficiency**: Measure not just success but how efficiently agents use available context window - **Scalability**: Benchmark can generate unlimited synthetic tasks at any complexity level ## Benchmark Dimensions - **Complexity**: 1-5 scale from simple retrieval to open-ended adaptation - **Context Length**: 10-1000+ steps, testing how agents handle extended histories - **Task Types**: Retrieval, transformation, multi-step planning, constrained reasoning, open exploration ## References - arXiv:2601.20730: AgentLongBench framework and evaluation results - Controllable generation of long-context agent evaluation tasks