Muse Eval

muse-eval

From An LLM Swarm To A PDDL-Empowered HIVE: Planning Self-Executed Instructions In A Multi-Modal Jungle — Vyas et al. (2024) (arXiv:2412.12839, 2024)

What this evaluates

Evaluates an LLM-based planning framework's ability to decompose natural language queries into correct task selections, logical execution flows, and valid final multimodal outputs. It probes constraint-aware model orchestration and multi-modal task routing across heterogeneous AI services.

Datasets

• MuSE — total 100; splits: test (100); repo https://github.com/dgraux/Hive-ICLR-2025

Metrics

• Task Selection (TS) (primary) — range: [0, 1]
  • Binary score (0 or 1) averaged across queries. Assigned 1 if the system selects exactly the required tasks, 0 otherwise.
• Flow of Thought (FoT) — range: [0, 1]
  • Binary score (0 or 1) averaged across queries. Assigned 1 if the system establishes the correct logical sequence and respects task dependencies, 0 otherwise.
• Final Output (O) — range: [0, 1]
  • Binary score (0 or 1) averaged across queries. Assigned 1 if the output fulfills the user's requirements, 0 otherwise. Does not evaluate output quality or factual accuracy.

Input / output format

Input: Natural language user queries specifying 1–3 tasks across text, image, and audio modalities.

Output: Structured execution plan containing selected tasks, their dependency order, and the final multimodal output generated by the orchestrated models.

Scoring recipe

def compute_metrics(predictions, gold):
    ts_scores, fot_scores, o_scores = [], [], []
    for pred, gold in zip(predictions, gold):
        ts_scores.append(1 if pred.selected_tasks == gold.required_tasks else 0)
        fot_scores.append(1 if pred.task_order == gold.required_order else 0)
        o_scores.append(1 if pred.output_fulfills_intent else 0)
    return {
        'TS': sum(ts_scores) / len(ts_scores),
        'FoT': sum(fot_scores) / len(fot_scores),
        'O': sum(o_scores) / len(o_scores)
    }

Common pitfalls

• Evaluation focuses strictly on whether the expected task was performed, not on the factual accuracy or quality of the generated output.
• Baseline systems like HuggingGPT may over-select tasks to ensure coverage, artificially inflating Final Output scores while penalizing Task Selection and Flow of Thought metrics.
• ControlLLM relies on a fine-tuned task decomposer, causing significant performance drops on multi-task queries that deviate from its training distribution.

Evidence (verbatim from paper)

To assess our framework against state-of-the-art methods, we evaluate performance on three fronts, using binary metrics for simplicity and clarity: Task Selection (TS): Determines whether the system accurately identifies the required tasks from the user’s query. We assign a binary score of 1 if the system selects all the tasks correctly, and 0 if it does not or if it selected irrelevant tasks. Flow of Thought (FoT): We evaluate the logical sequence and integration of the selected tasks. A binary score is given based on whether the system establishes the correct flow—1 for a proper flow that respects task dependencies and order, and 0 for an incorrect sequence. Final Output (O): Assesses the correctness of the system’s final response to the user’s query. We adopt a binary evaluation—1 if the output fulfills the user’s requirements, and 0 if it falls short.

Citation

@misc{vyas2024hive,
  title={From An LLM Swarm To A PDDL-Empowered HIVE: Planning Self-Executed Instructions In A Multi-Modal Jungle},
  author={Vyas et al. (2024)},
  year={2024},
  note={arXiv:2412.12839}
}

• arXiv: 2412.12839