Cpt Merging Eval

cpt-merging-eval

Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance — Ueda et al. (2025) (arXiv:2511.02451, 2025)

What this evaluates

Evaluates the effectiveness of merging Continual Pretraining (CPT) models to build domain-specialized financial LLMs. It probes the model's ability to recover lost general knowledge, exhibit cross-domain complementarity, and demonstrate emergent reasoning capabilities through weight-space integration.

Datasets

• Financial Benchmark — total 18; splits: test (-1)

Metrics

• Macro-Gain (primary) — range: percent
  • The average performance improvement of the merged model over the arithmetic mean of its two constituent models across all tasks.
• Macro-OG — range: percent
  • The average margin by which the merged model outperforms the best of its two constituent models across all tasks.

Input / output format

Input: Task prompts from a curated 18-task financial benchmark.

Output: Model-generated predictions or answers for each benchmark task.

Scoring recipe

def compute_metrics(merged_scores, c1_scores, c2_scores):
    avg_merged = np.mean(merged_scores)
    avg_c1 = np.mean(c1_scores)
    avg_c2 = np.mean(c2_scores)
    avg_const_avg = (avg_c1 + avg_c2) / 2
    best_const = max(avg_c1, avg_c2)
    macro_gain = avg_merged - avg_const_avg
    macro_og = avg_merged - best_const
    return macro_gain, macro_og

Common pitfalls

• Task Arithmetic (TA) is highly sensitive to the scaling hyperparameter λ, often degrading if poorly tuned, whereas TIES (TI) is more robust.
• Merging three CPT models frequently causes performance collapse and representational interference, contrary to additive expectations.
• Emergent capabilities (positive Macro-OG) do not correlate linearly with parameter-space similarity (L2/cosine), making them hard to predict a priori.

Evidence (verbatim from paper)

We first observe that Macro-Gain tends to be higher when models are more similar—i.e., when L2 distance is smaller or cosine similarity is larger.

Citation

@misc{ueda2025merging,
  title={Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance},
  author={Ueda et al. (2025)},
  year={2025},
  note={arXiv:2511.02451}
}

• arXiv: 2511.02451