Geppetto Eval

geppetto-eval

GePpeTto Carves Italian into a Language Model — De Mattei et al. (2020) (arXiv:2004.14253, 2020)

What this evaluates

Evaluates the quality and linguistic fidelity of an Italian generative language model (GePpeTto) by measuring its perplexity across in-domain and out-of-domain corpora, and profiling its lexical and syntactic complexity against human-written Italian text.

Datasets

• Wikipedia (Italian) — total ?; splits: test (-1)
• ItWac — total ?; splits: test (-1)
• EUR-Lex Italian Laws — total ?; splits: test (-1)
• la Repubblica & Il Giornale — total ?; splits: test (-1)
• Forum Comments — total ?; splits: test (-1)

Metrics

• Perplexity (primary) — range: other
  • Exponential of the negative average log-probability of each token in the sequence under the model: exp(-1/N * sum(log p(x_i))). Lower values indicate better language modeling performance.
• Linguistic Feature Means — range: other
  • Mean and standard deviation of extracted features: CPT (chars per token), TPS (tokens per sentence), TPC (tokens per clause), LLmax (max dependency link length), LLavg (avg dependency link length), and POS distribution. Computed using the Profiling-UD tool.

Input / output format

Input: For perplexity: raw text segments from target domains. For linguistic profiling: the first token of a gold human sentence used as a prompt to generate a continuation.

Output: For perplexity: model predicts next-token probabilities over the sequence. For linguistic profiling: generated text continuation starting from the prompt token.

Scoring recipe

def compute_perplexity(text, model):
    tokens = tokenize(text)
    log_probs = [log(model_prob(t, context)) for t in tokens]
    return exp(-mean(log_probs))

def compute_linguistic_features(texts, tool='Profiling-UD'):
    features = tool.extract(texts) # CPT, TPS, TPC, LLmax, LLavg, POS
    return {k: (mean(v), std(v)) for k, v in features.items()}

# Evaluation
pp = compute_perplexity(test_corpus, geppetto_model)
ling_gen = compute_linguistic_features(generated_texts)
ling_gold = compute_linguistic_features(gold_sentences)
compare(ling_gen, ling_gold)

Common pitfalls

• Perplexity is evaluated on a 1% hold-out of the training corpus for in-domain domains, not a fully independent test set, which may inflate performance estimates.
• Linguistic profiling uses a single-token prompt, which heavily biases sentence length and complexity compared to multi-token or zero-shot prompts.
• Profiling-UD relies on syntactic parsing; errors in Italian dependency parsing can skew LLmax/LLavg and clause-level metrics.

Evidence (verbatim from paper)

As a first evaluation, we are interested in understanding the quality of GePpeTto as a language model in its own training domain. As a second evaluation we want test its performance at zero-shot domain transfer (i.e. language modeling of a different domain). We use perplexity as a measure of language modelling performance. ... In our analysis we focus on two macro aspects of GePpeTto's output, namely lexical complexity and syntactic complexity, and compare them to human productions. To do so, we rely on a selection of Profiling-UD's features which we use as proxies for the macro-aspects that we consider.

Citation

@misc{demattei2020geppetto,
  title={GePpeTto Carves Italian into a Language Model},
  author={De Mattei et al. (2020)},
  year={2020},
  note={arXiv:2004.14253}
}

• arXiv: 2004.14253