Llava Le Eval

llava-le-eval

LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration — Inal et al. (2026) (arXiv:2603.24696, 2026)

What this evaluates

Evaluates multimodal instruction-following and complex geological reasoning on lunar surface imagery. It probes the model's ability to interpret crater morphology, degradation states, and inferred geological processes beyond simple visual description.

Datasets

• LUCID (held-out eval set) — total 190; splits: test (190)

Metrics

• average_overall_score (primary) — range: [1, 10]
  • Each of two independent LLM judges (ChatGPT and Gemini) assigns a score from 1 to 10 based on relevance, clarity, and accuracy. The final metric is the arithmetic mean of the two judges' scores.

Input / output format

Input: Raw lunar image and a text question.

Output: Free-form textual response.

Scoring recipe

def compute_metric(predictions, references, captions):
    scores = []
    for pred, ref, cap in zip(predictions, references, captions):
        s1 = judge_chatgpt(cap, ref, pred) # returns 1-10
        s2 = judge_gemini(cap, ref, pred) # returns 1-10
        scores.append((s1 + s2) / 2)
    return sum(scores) / len(scores)

Common pitfalls

• Judges do not access the raw image during scoring; evaluation is grounded solely in the scientific caption.
• Reference answers are synthetically generated by GPT-5.1 and Gemini-2.5 rather than human-annotated ground truth.
• The 1-10 scale is averaged across two different LLM judges, which may introduce model-specific bias or calibration differences.

Evidence (verbatim from paper)

Each judge assigns a score from 1 to 10 based on relevance, clarity, and accuracy. ... LLaVA-LE Stage 2 achieves an average overall score of 0.921 (averaged across GPT and Gemini judges), representing a 3.3× improvement over Base LLaVA (0.278) and a 2.1× improvement over LLaVA-LE Stage 1 (0.443).

Citation

@misc{inal2026llavale,
  title={LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration},
  author={Inal et al. (2026)},
  year={2026},
  note={arXiv:2603.24696}
}

• arXiv: 2603.24696