Mindbench Eval

mindbench-eval

MindBench: A Comprehensive Benchmark for Mind Map Structure Recognition and Analysis — Lei Chen et al. (arXiv:2407.02842, 2024)

What this evaluates

This benchmark evaluates multimodal large language models on structured document analysis, specifically focusing on mind map parsing and visual question answering. It probes text recognition, spatial awareness, hierarchical relationship discernment, and the ability to reconstruct complex graphical tree structures from high-resolution images.

Datasets

• MindBench — total ?; splits: test (-1)

Metrics

• TED-based accuracy (primary) — range: [0, 1]
  • Uses the Zhang-Shasha algorithm to compute the normalized Tree Edit Distance (nTED) between the predicted and ground-truth tree structures. Accuracy is calculated as max(1 - nTED, 0).
• field-level F1 score — range: [0, 1]
  • Flattens the nested JSON prediction and gold standard into a non-nested format, then computes the F1 score for each extracted field.

Input / output format

Input: High-resolution mind map images (real-world or synthetic) paired with task-specific prompts (e.g., full/partial parsing, position-related parsing, structured/position-related VQA).

Output: For parsing tasks: a JSON string representing the hierarchical tree structure of the mind map. For VQA tasks: natural language text answers.

Scoring recipe

def compute_ted_accuracy(pred_str, gold_str):
    pred_tree = parse_json_to_tree(pred_str)
    gold_tree = parse_json_to_tree(gold_str)
    nted = zhang_shasha_normalized_edit_distance(pred_tree, gold_tree)
    return max(1.0 - nted, 0.0)

def compute_field_f1(pred_str, gold_str):
    pred_flat = flatten_json_fields(pred_str)
    gold_flat = flatten_json_fields(gold_str)
    return calculate_f1_score(pred_flat, gold_flat)

Common pitfalls

• Models frequently misassign parent-child relationships by relying on visual layout proximity rather than actual graphical connectors.
• Performance degrades sharply on mind maps with over 60 nodes due to limitations in processing high-resolution images and long-range structural dependencies.
• Converting raw token sequences to valid, parseable JSON is error-prone and can cause complete failure in tree structure recovery.

Evidence (verbatim from paper)

For parsing task, following Donut, we evaluate the models using two metrics: field-level F1 score and Tree Edit Distance (TED) based accuracy. We first convert the predicted token sequence to JSON format to recover the tree structure of the graph. The F1 metric flattens the nested JSON into a non-nested format, and then calculates F1 score at each field. F1 can efficiently evaluate the extracted field information, but it cannot exactly measure the structure of the tree. The TED-based metric is appropriate for evaluating tree-structured documents. Specifically, it uses the Zhang-Shasha (ZSS) algorithm*[[63]]* to calculate the nTED between the prediction tree and the answer tree, where $n$ represents the size of the answer tree. The accuracy based on nTED is then computed using the formula $max(1-nTED,0)$.

Citation

@misc{chen2024mindbench,
  title={MindBench: A Comprehensive Benchmark for Mind Map Structure Recognition and Analysis},
  author={Lei Chen et al.},
  year={2024},
  note={arXiv:2407.02842}
}

• arXiv: 2407.02842