Ready Jurist One Eval

ready-jurist-one-eval

Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments — Zheng Jia et al. (2025) (arXiv:2507.04037, 2025)

What this evaluates

Evaluates the ability of LLM-based agents to perform interactive, procedural legal tasks in dynamic, multi-turn Chinese legal environments. It probes knowledge retrieval, document drafting, and court proceeding navigation, measuring both task completion and adherence to legal procedures.

Datasets

• J1-ENVS — total ?; splits: test (-1)

Metrics

• average scores (primary) — range: percent
  • Average of task scores across all J1-ENVS environments except VER. Computed by evaluating task completion per scenario and averaging the results.
• completion rate — range: percent
  • Proportion of samples that successfully reach designated procedural stages in civil or criminal court environments.
• behavior consistency — range: other
  • Average rating on a 1 to 10 scale assessing how well an environment role's behavior matches its predefined profile during multi-turn interactions.

Input / output format

Input: Multi-turn interactive legal scenarios in Chinese, including case backgrounds, procedural rules, and role-specific instructions. Agents receive environment state updates and task prompts.

Output: Agent actions/responses in natural language or structured commands to interact with the environment, complete tasks, and navigate court stages.

Scoring recipe

def compute_metrics(predictions, gold, env_states):
    # 1. Average Score (excluding VER)
    env_scores = []
    for env in environments:
        if env == 'VER': continue
        score = evaluate_task_completion(predictions[env], gold[env])
        env_scores.append(score)
    avg_score = sum(env_scores) / len(env_scores)

    # 2. Completion Rate
    completed = sum(1 for s in samples if s.reaches_designated_stage)
    completion_rate = completed / len(samples)

    # 3. Behavior Consistency (1-10 scale)
    consistency = [rater.rate(profile, behavior) for rater, profile, behavior in interactions]
    avg_consistency = sum(consistency) / len(consistency)
    return avg_score, completion_rate, avg_consistency

Common pitfalls

• Legal-specific LLMs often underperform general-purpose models in dynamic settings despite strong static benchmark scores, as they lack interactive procedural capabilities.
• Evaluation relies on LLM-based scoring (GPT-4o) for most metrics, with human evaluation only covering 10% of samples, which may limit reliability compared to full human annotation.
• Metrics like P_FS and completion rates are environment-specific (civil vs. criminal court), making direct cross-environment comparison difficult without normalization.

Evidence (verbatim from paper)

To assess overall performance, we compute the average scores across all environments except V ER, and rank the models by size. ... measure the completion rate by computing how many samples successfully reach designated stages. ... Both GPT-4o and human evaluators rate the consistency between each role’s profile and its behavior on a scale of 1 to 10

Citation

@misc{zheng2025readyjuristone,
  title={Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments},
  author={Zheng Jia et al. (2025)},
  year={2025},
  note={arXiv:2507.04037}
}

• arXiv: 2507.04037