Stair Captions Eval

stair-captions-eval

STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset — Yoshikawa et al. (2017) (arXiv:1705.00823, 2017)

What this evaluates

Evaluates a model's ability to generate fluent, contextually accurate Japanese image captions directly from visual input. It specifically probes whether native-language training data yields better captioning quality compared to a pipeline of English generation followed by machine translation.

Datasets

• STAIR Captions — total 164062; splits: train (113287), val (5000), test (5000)

Metrics

• BLEU-4 — range: [0, 1]
  • 4-gram precision with brevity penalty, measuring exact word overlap between generated and reference captions.
• ROUGE_L — range: [0, 1]
  • Longest Common Subsequence (LCS) based recall/precision, capturing sentence-level fluency and word order.
• CIDEr (primary) — range: [0, 1]
  • TF-IDF weighted n-gram similarity between generated and reference captions, emphasizing rare but informative words.

Input / output format

Input: RGB image (processed through a fixed VGG-16 CNN to extract features)

Output: Japanese text caption

Scoring recipe

def evaluate(predictions, references):
    # Preprocess Japanese text with MeCab morphological analysis
    preds = [mecab_tokenize(p) for p in predictions]
    refs = [mecab_tokenize(r) for r in references]
    bleu4 = nltk.bleu(refs, preds, weights=(0,0,0,1))
    rouge_l = rouge_score(refs, preds, rouge_types=['rougeL'])
    cider = compute_cider(refs, preds)
    return {'BLEU-4': bleu4, 'ROUGE_L': rouge_l, 'CIDEr': cider}

Common pitfalls

• Using English-to-Japanese machine translation pipelines instead of native Japanese training data produces unnatural, word-by-word translations.
• Failing to apply MeCab morphological analysis to Japanese captions before training or evaluation will break tokenization and metric computation.
• The CNN backbone (VGG-16) is fixed during training; only LSTM parameters are optimized, which must be replicated for fair comparison.

Evidence (verbatim from paper)

Following the literature (Chen et al., 2015; Karpathy and Fei-Fei, 2015), we use BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and CIDEr (Vedantam et al., 2015) as evaluation measures. Although BLEU and ROUGE were developed originally for evaluating machine translation and text summarization, we use them here because they are often used for measuring the quality of caption generation.

Citation

@misc{yoshikawa2017staircaptions,
  title={STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset},
  author={Yoshikawa et al. (2017)},
  year={2017},
  note={arXiv:1705.00823}
}

• arXiv: 1705.00823