Starflow Eval

starflow-eval

StarFlow: Generating Structured Workflow Outputs From Sketch Images — Bechard et al. (2025) (arXiv:2503.21889, 2025)

What this evaluates

Evaluates vision-language models' ability to parse free-form workflow sketch images and generate structured JSON workflow definitions. It probes structural fidelity, trigger and component recognition, and hierarchical consistency in diagram-to-code translation.

Datasets

• StarFlow Dataset — total ?; splits: test (-1)

Metrics

• FlowSim (primary) — range: [0, 1]
  • 1 - TED(F, Fr) / (|F| + |Fr|), where TED is tree edit distance and |F|, |Fr| are node counts. Normalized to [0, 1].
• TreeBLEU — range: [0, 1]
  • |S(F) ∩ S(Fr)| / |S(F)|, where S(.) is the set of 1-height subtrees. Excludes Flow→Trigger and Flow→Components edges for fairness.
• Trigger Match — range: [0, 1]
  • 1 if predicted trigger exactly matches reference trigger, else 0. Averaged over instances.
• Component Match — range: [0, 1]
  • |CF ∩ CFr| / |CF ∪ CFr|, computing the intersection-over-union of predicted and reference component sets in an order-agnostic manner.

Input / output format

Input: Sketch image of a workflow diagram, optionally accompanied by input conditions/context.

Output: Structured JSON representing the workflow, including triggers, components, and execution logic.

Scoring recipe

def compute_metrics(pred_json, ref_json):
    pred_tree = decompose_to_tree(pred_json)
    ref_tree = decompose_to_tree(ref_json)
    ted = tree_edit_distance(pred_tree, ref_tree)
    flowsim = 1.0 - ted / (len(pred_tree) + len(ref_tree))
    
    pred_subtrees = get_1_height_subtrees(pred_json)
    ref_subtrees = get_1_height_subtrees(ref_json)
    treebleu = len(pred_subtrees & ref_subtrees) / len(pred_subtrees) if pred_subtrees else 0.0
    
    tm = 1.0 if pred_json['trigger'] == ref_json['trigger'] else 0.0
    
    pred_comps = set(pred_json['components'])
    ref_comps = set(ref_json['components'])
    cm = len(pred_comps & ref_comps) / len(pred_comps | ref_comps) if (pred_comps | ref_comps) else 0.0
    return flowsim, treebleu, tm, cm

Common pitfalls

• TreeBLEU requires explicitly excluding Flow→Trigger and Flow→Components edges; otherwise, empty flows incorrectly receive non-zero scores.
• Cross-platform evaluation is inherently limited because different workflow applications have unique logical patterns, meaning multiple valid JSON outputs may exist for a single sketch.
• Handwritten or manual sketches introduce high ambiguity, causing significant performance drops compared to synthetic or UI screenshots due to the need to read handwritten text.

Evidence (verbatim from paper)

Assessing the quality of generated flows presents challenges similar to those in evaluating generated code. In this work, we report four types of metrics that provide a comprehensive evaluation by capturing different aspects of flow generation. The metrics we report are Flow Similarity (FlowSim), Tree BLEU (TreeBLEU), Trigger Match (TM), and Component Match (CM). For Flow Similarity, we follow the methodology used in Ayala and Béchard (2024): we decompose generated workflows into trees and compute the tree edit distance using the algorithm from Zhang and Shasha (1989). We normalize the obtained tree edit distance by the number of nodes in each tree to obtain a score between 0 and 1.

Citation

@misc{bechard2025starflow,
  title={StarFlow: Generating Structured Workflow Outputs From Sketch Images},
  author={Bechard et al. (2025)},
  year={2025},
  note={arXiv:2503.21889}
}

• arXiv: 2503.21889