Codechat Conversation Analysis Eval

codechat-conversation-analysis-eval

Developer-LLM Conversations: An Empirical Study of Interactions and Generated Code Quality — Zhong et al. (2025) (arXiv:2509.10402, 2025)

What this evaluates

This protocol evaluates the structural and linguistic characteristics of real-world developer-LLM conversational interactions. It measures conversational efficiency, turn-taking dynamics, code snippet complexity, and programming language distribution to understand how developers prompt LLMs and how models respond in iterative coding workflows.

Datasets

• CodeChat — total 82845; splits: full (82845); repo https://github.com/Software-Evolution-Analytics-Lab-SEAL/CodeChat.git

Metrics

• Token Ratio (TR) (primary) — range: ratio
  • The number of LLM-generated tokens divided by the number of developer prompt tokens. Calculated using the GPT-4 tokenizer for consistency across all conversations.
• Turn Count (TC) — range: integer
  • The total number of prompt–response pairs (turns) in a single conversation. Used to classify interactions as single-turn or multi-turn.
• Lines of Code (LOC) — range: integer
  • The number of non-blank lines in each LLM-generated code snippet. Used to measure code snippet size and complexity across programming languages.
• Programming Language Rate (PL-Rate) — range: [0, 1]
  • The proportion of LLM responses generating code in a specific programming language X, calculated as the count of snippets tagged with X divided by the total number of snippets with explicit language tags.

Input / output format

Input: Sequences of developer prompts and corresponding LLM-generated text/code responses within a single conversation session, along with optional programming language tags.

Output: Computed metric values per conversation (e.g., token counts, turn counts, LOC, language tags) and aggregated statistical results (e.g., rates, frequencies, p-values from Wilcoxon or Kruskal-Wallis tests).

Scoring recipe

def compute_metrics(conversation):
    tr = len(gpt4_tokenize(conversation.llm_response)) / len(gpt4_tokenize(conversation.developer_prompt))
    tc = len(conversation.turns)
    loc = sum(1 for line in conversation.code_snippet.split('\n') if line.strip())
    pl_rate = conversation.language_tag_count / total_snippets_with_tags
    return {'TR': tr, 'TC': tc, 'LOC': loc, 'PL-Rate': pl_rate}

Common pitfalls

• Using a tokenizer other than GPT-4 for Token Ratio calculation, which breaks cross-study comparability and violates the protocol.
• Counting blank lines when computing Lines of Code (LOC), as the protocol explicitly requires non-blank lines only.
• Failing to normalize programming language aliases (e.g., 'js' to 'javascript') before computing PL-Rate, leading to fragmented language distributions.

Evidence (verbatim from paper)

We define the Token Ratio (TR) as the number of LLM-generated tokens divided by the number of developer prompt tokens, using the GPT-4 tokenizer for consistency. To compare token lengths between developer prompts and LLM responses across CodeChat, we employ the Wilcoxon signed-rank test, a non-parametric statistical test for paired samples, with a significance level of p=0.05.

Citation

@misc{zhong2025developerllmconversations,
  title={Developer-LLM Conversations: An Empirical Study of Interactions and Generated Code Quality},
  author={Zhong et al. (2025)},
  year={2025},
  note={arXiv:2509.10402}
}

• arXiv: 2509.10402