Swe Chat Eval

swe-chat-eval

SWE-chat: Coding Agent Interactions From Real Users in the Wild — Baumann et al. (2026) (arXiv:2604.20779, 2026)

What this evaluates

Evaluates real-world coding agent interactions by measuring how much agent-generated code survives into final commits, alongside efficiency metrics like token usage, cost, and runtime per committed line.

Datasets

• SWE-chat — total ?; splits: test (-1); repo https://github.com/SALT-NLP/SWE-chat

Metrics

• Code survival rate (primary) — range: percent
  • The fraction of the agent’s net output (after self-overwrites) that the human kept unchanged, computed as (agent lines survived / agent lines in final state) * 100.
• Coding efficiency — range: percent
  • The fraction of the agent’s total effort (including lines it later rewrote) that ended up in the commit, computed as (agent lines survived / agent cumulative lines produced) * 100.
• Agent-authored % — range: percent
  • The percentage of committed lines that were authored by the agent, computed as (agent lines survived / total committed lines) * 100.

Input / output format

Input: Raw session logs from the Entire platform, including user prompts, assistant responses, thinking traces, tool calls, tool results, per-turn token usage, and commit metadata.

Output: Computed metric values (percentages or ratios) representing code attribution, survival rates, and efficiency scores per session or commit.

Scoring recipe

# Replay file-modifying tool calls chronologically to track file states
for tool_call in session.tool_calls:
    apply(tool_call)
    diff = compute_line_diff(previous_state, current_state)
    tag_lines(diff, provenance="agent")

# Aggregate at commit time
agent_lines_survived = count_lines_in_final_commit(provenance="agent")
agent_lines_final = count_lines_in_final_state(provenance="agent")
total_committed_lines = count_lines_in_final_commit()

# Compute metrics
code_survival_rate = (agent_lines_survived / agent_lines_final) * 100
coding_efficiency = (agent_lines_survived / agent_cumulative_lines) * 100
agent_authored_pct = (agent_lines_survived / total_committed_lines) * 100

Common pitfalls

• Concurrent human-agent edits to the same file can cause transcript inconsistencies and misattribution of line provenance.
• Sessions often span multiple commits or multiple sessions contribute to one commit, requiring strict filtering (only ~48.6% have unambiguous attribution).

Evidence (verbatim from paper)

We quantify coding agent efficiency using several complementary approaches, all computed from raw data without the need for annotations. ... Coding efficiency measures the fraction of the agent’s total effort (including lines it later rewrote) that ended up in the commit. The Code survival rate measures the fraction of the agent’s net output (after self-overwrites) that the human kept unchanged.

Citation

@misc{baumann2026swechat,
  title={SWE-chat: Coding Agent Interactions From Real Users in the Wild},
  author={Baumann et al. (2026)},
  year={2026},
  note={arXiv:2604.20779}
}

• arXiv: 2604.20779