Style Transfer Eval

style-transfer-eval

Reformulating Unsupervised Style Transfer as Paraphrase Generation — Krishna et al. (2020) (arXiv:2010.05700, 2020)

What this evaluates

Evaluates unsupervised style transfer models on their ability to transform text between two styles (e.g., Shakespeare vs. modern English, formal vs. informal) while preserving semantic meaning and maintaining linguistic quality.

Datasets

• Shakespeare author imitation dataset (Xu et al., 2012) — total 37000; splits: test (-1)
• Formality transfer dataset (Rao and Tetrault, 2018) — total 105000; splits: test (-1)

Metrics

• J(A,S,F) (primary) — range: percent
  • Proposed joint metric that balances accuracy, semantic similarity, and fluency. Designed to capture the style-transfer trade-off better than standard metrics.
• ACC — range: percent
  • Accuracy of a style classifier predicting the target style from the generated text.
• SIM — range: percent
  • Semantic similarity score measuring how well the generated text preserves the meaning of the source text.
• FL — range: percent
  • Fluency or linguistic quality score, evaluated automatically or via human judgment.
• GM(A,S,F) — range: percent
  • Geometric mean of ACC, SIM, and FL scores.

Input / output format

Input: Source sentence in a given style (e.g., modern English or Shakespearean text).

Output: Transformed sentence in the target style (e.g., Shakespearean or formal English).

Scoring recipe

acc = style_classifier.predict(gen_text) == target_style
sim = semantic_similarity(source_text, gen_text)
fl = fluency_score(gen_text)
gm = (acc * sim * fl) ** (1/3)
j = joint_metric(acc, sim, fl)  # Proposed metric combining ACC, SIM, FL
return {'ACC': acc, 'SIM': sim, 'FL': fl, 'GM(A,S,F)': gm, 'J(A,S,F)': j}

Common pitfalls

• Relying solely on ACC or SIM without considering the joint metric J, which the authors argue better captures the style-transfer trade-off.
• Using parallel data for model training instead of evaluation; the protocol explicitly restricts parallel data to automatic evaluation only, while training uses non-parallel splits.
• Confusing GM (geometric mean) with J (the proposed joint metric), as both combine ACC, SIM, and FL but are calculated differently.

Evidence (verbatim from paper)

STRAP significantly outperforms the prior state of the art (DLSM) on automatic metrics (Table 1) with a J(·) score of 45.5 (vs 18.6) on Formality and 34.7 (vs 16.3) on Shakespeare. The improvements are even larger when SIM and FL are measured through human evaluations (Table 2): in this setting, STRAP achieves 41.3 (vs 10.0) on Formality and 47.3 (vs 9.3) on Shakespeare.

Citation

@misc{krishna2020strap,
  title={Reformulating Unsupervised Style Transfer as Paraphrase Generation},
  author={Krishna et al. (2020)},
  year={2020},
  note={arXiv:2010.05700}
}

• arXiv: 2010.05700