Mti Temperament Eval

mti-temperament-eval

MTI: A Behavior-Based Temperament Profiling System for AI Agents — Jeong (2026) (arXiv:2604.02145, 2026)

What this evaluates

Measures four independent behavioral axes—Reactivity, Compliance, Sociality, and Resilience—in AI agents to distinguish intrinsic dispositional traits from raw capability. It evaluates how models respond to structured behavioral protocols under baseline and stress conditions, and how alignment techniques like RLHF alter these dispositions.

Datasets

• MTI Behavioral Battery — total 1930; splits: test (1930)

Metrics

• MTI Temperament Axes (Reactivity, Compliance, Sociality, Resilience) (primary) — range: other (axis-specific scales)
  • Reactivity uses pairwise Likert similarity delta; Compliance uses instruction flip rate + NoF score; Sociality uses emotional context delta ratio; Resilience uses PM = quality_stress / quality_baseline. All scores are derived via automated keyword counting, heuristic stance detection, and rule-based quality assessment without human or LLM judges.

Input / output format

Input: Structured behavioral prompts and stress conditions delivered via Ollama with temperature=0 and default system prompts. Models are tested sequentially under baseline and stress conditions.

Output: Text responses from the LLM, processed through deterministic scripts for keyword counting, heuristic stance detection, and rule-based quality scoring.

Scoring recipe

def score_mtii(predictions, conditions):
    scores = {}
    for axis in ['Reactivity', 'Compliance', 'Sociality', 'Resilience']:
        if axis == 'Resilience':
            scores[axis] = quality_stress / quality_baseline
        elif axis == 'Reactivity':
            scores[axis] = pairwise_mean_delta(likert_scores)
        elif axis == 'Compliance':
            scores[axis] = flip_rate + nof_score
        elif axis == 'Sociality':
            scores[axis] = emotional_context_delta_ratio
    return scores

Common pitfalls

• Assuming higher model capability or size correlates with higher temperament scores; the paper explicitly shows temperament is size-independent and captures intrinsic dispositions rather than capability.
• Including the base model (llama3.1-base) in the primary instruction-tuned correlation analysis; it is a systematic outlier lacking the alignment Shell and is analyzed separately.
• Using LLM-as-judge or human raters for scoring; the protocol strictly uses fully automated deterministic scripts (keyword counting, heuristic stance detection, rule-based quality assessment).

Evidence (verbatim from paper)

All scoring was fully automated via deterministic scripts (keyword counting, heuristic stance detection, rule-based quality assessment), with no human raters or LLM-as-judge. Table 4 specifies the primary metric for Resilience as PM = quality_stress / quality_baseline.

Citation

@misc{jeong2026mti,
  title={MTI: A Behavior-Based Temperament Profiling System for AI Agents},
  author={Jeong (2026)},
  year={2026},
  note={arXiv:2604.02145}
}

• arXiv: 2604.02145