# Gere Continual Learning > Prevents catastrophic forgetting in continual LLM learning using threshold-based margin loss with fixed general replay samples from pretraining data. - Skill: `adu2021/gere-continual-learning` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/gere-continual-learning` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/gere-continual-learning/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/gere-continual-learning --- ## GeRe: Anti-Forgetting in Continual Learning of LLMs ### Core Concept GeRe addresses catastrophic forgetting in continual LLM learning by employing a threshold-based margin (TM) loss function on fixed general replay samples derived from pretraining. This maintains activation state consistency during replay learning, effectively mitigating both forgetting of general capabilities and performance degradation on previously learned tasks. ### Architecture Overview - **General Sample Replay**: Small fixed set of pretraining samples - **Threshold-Based Margin Loss**: Constrains activation states during replay - **Task-Incremental Learning**: Learn new domains sequentially - **Activation State Consistency**: Maintain neural patterns from pretraining ### Implementation Steps **Step 1: Collect General Replay Samples** Curate pretraining samples for replay: ```python class GeneralSampleCollector: def __init__(self, pretraining_data, sample_size=1000): super().__init__() self.sample_size = sample_size def select_general_samples(self, pretraining_data): """Select representative pretraining samples.""" # Stratified sampling across topics samples = [] topics = self._identify_topics(pretraining_data) for topic in topics: topic_samples = [d for d in pretraining_data if d['topic'] == topic] num_select = self.sample_size // len(topics) selected = random.sample(topic_samples, min(num_select, len(topic_samples))) samples.extend(selected) return samples[:self.sample_size] def _identify_topics(self, data): """Identify topic distribution.""" topics = {} for sample in data[:100]: # Sample for efficiency topic = sample.get('topic', 'general') topics[topic] = topics.get(topic, 0) + 1 return topics.keys() ``` **Step 2: Implement Threshold-Based Margin Loss** Design loss for activation consistency: ```python class ThresholdBasedMarginLoss(nn.Module): def __init__(self, threshold=0.5, margin=0.1): super().__init__() self.threshold = threshold self.margin = margin def compute_loss(self, original_activations, replay_activations): """ Compute TM loss maintaining activation state consistency. Args: original_activations: Activations from original training replay_activations: Activations during replay Returns: loss: TM loss value """ # Normalize activations orig_norm = F.normalize(original_activations, dim=-1) replay_norm = F.normalize(replay_activations, dim=-1) # Compute similarity similarity = torch.mm(replay_norm, orig_norm.t()) # Threshold-based margin loss = 0 for i in range(similarity.shape[0]): for j in range(similarity.shape[1]): sim = similarity[i, j] if sim > self.threshold: # High similarity: maintain it loss += torch.relu(self.margin - sim) else: # Low similarity: don't force increase loss += torch.relu(sim - self.threshold) return loss / (similarity.shape[0] * similarity.shape[1]) ``` **Step 3: Implement Continual Learning Loop** Train on sequential tasks: ```python class ContinualLearningTrainer: def __init__(self, model, general_samples): super().__init__() self.model = model self.general_samples = general_samples self.activation_history = {} def learn_new_task(self, new_task_data, task_id): """Learn new task while replaying general samples.""" optimizer = AdamW(self.model.parameters(), lr=2e-5) tm_loss_fn = ThresholdBasedMarginLoss() for epoch in range(3): # Mix new task and replay batch_new = random.sample(new_task_data, min(32, len(new_task_data))) batch_replay = random.sample(self.general_samples, min(32, len(self.general_samples))) # Train on new task for example in batch_new: outputs = self.model(example['input_ids'], output_hidden_states=True) task_loss = F.cross_entropy(outputs.logits, example['labels']) # Get activations activations = outputs.hidden_states[-1] # Store if first task if task_id == 0: self.activation_history[example['id']] = activations.detach() optimizer.zero_grad() task_loss.backward() optimizer.step() # Replay general samples with TM loss for example in batch_replay: outputs = self.model(example['input_ids'], output_hidden_states=True) # Get current activations current_activations = outputs.hidden_states[-1] # Original activations if example['id'] in self.activation_history: original = self.activation_history[example['id']] else: original = current_activations.detach() # Compute TM loss tm_loss = tm_loss_fn.compute_loss(original, current_activations) optimizer.zero_grad() tm_loss.backward() optimizer.step() def evaluate_on_all_tasks(self, task_data_dict): """Evaluate performance on all learned tasks.""" results = {} for task_id, task_data in task_data_dict.items(): correct = 0 total = len(task_data) with torch.no_grad(): for example in task_data: outputs = self.model(example['input_ids']) pred = outputs.logits.argmax(dim=-1) if pred == example['labels']: correct += 1 results[f'task_{task_id}'] = correct / total return results ``` ### Practical Guidance **Hyperparameters and Configuration**: - General sample size: 1000-5000 samples - TM threshold: 0.5 - TM margin: 0.1-0.2 - Learning rate: 2e-5 to 5e-5 - Replay batch size: 32 **When to Use GeRe**: - Continual learning of LLMs on sequential domains - Systems where forgetting general knowledge is costly - Scenarios with limited task-specific data - Multi-domain adaptation requiring stability **When NOT to Use**: - Single-domain training (no continual aspect) - When task-specific performance is prioritized over general knowledge - Very large models (storage overhead) **Implementation Notes**: - Fixed replay samples prevent distribution shifts - TM loss more robust than standard replay approaches - Monitor forgetting vs learning tradeoff - Consider optimal buffer size vs memory constraints ### Reference Paper: GeRe: Anti-Forgetting in Continual Learning of LLM ArXiv: 2508.04676 Performance: Consistently improves upon label fitting and KL divergence replay baselines