adu2021 Qwen3 Embedding
Core Concept
Qwen3 Embedding builds state-of-the-art text embeddings and reranking models by leveraging the rich representation capacity of Qwen3 foundation models. Rather than training from scratch, the approach combines weak supervision at scale with targeted fine-tuning on curated datasets, achieving superior multilingual and code-understanding performance. The key innovation is using the Qwen3-32B model itself to synthesize diverse training data across multiple tasks and languages.
Architecture Overview
- Multi-Stage Pipeline: Two-phase training combining weakly-supervised pre-training (150M+ synthetic pairs) with supervised fine-tuning on curated datasets
- LLM-Driven Synthesis: Uses Qwen3-32B to generate synthetic text pairs across retrieval, bitext mining, classification, and semantic similarity tasks
- Model Variants: Offers three sizes (0.6B, 4B, 8B) for both embedding and reranking with flexible output dimensions
- Contrastive Learning: Modified InfoNCE loss with in-batch negatives and false-negative mitigation strategies
- Model Merging: Spherical linear interpolation across training checkpoints for improved robustness
Implementation
The following code demonstrates the synthetic data generation and training pipeline:
import torch
import torch.nn.functional as F
from torch.optim import AdamW
from typing import List, Tuple
class Qwen3EmbeddingTrainer:
def __init__(self, model_name: str = "qwen3-8b", embedding_dim: int = 1024):
self.model_name = model_name
self.embedding_dim = embedding_dim
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
def prepare_synthetic_pairs(self, task_type: str, num_pairs: int = 10000) -> List[Tuple[str, str]]:
"""
Generate synthetic training pairs using LLM synthesis.
task_type: 'retrieval', 'bitext', 'classification', or 'similarity'
"""
pairs = []
for i in range(num_pairs):
# In practice, use Qwen3-32B with task-specific prompts
query = f"Generate query for {task_type} task {i}"
document = f"Generate document for {task_type} task {i}"
pairs.append((query, document))
return pairs
def contrastive_loss(self, embeddings: torch.Tensor, labels: torch.Tensor,
temperature: float = 0.05, margin: float = 0.1) -> torch.Tensor:
"""
Modified InfoNCE loss with false-negative mitigation.
embeddings: batch of embeddings, shape (batch_size, embedding_dim)
labels: binary labels indicating positive pairs
"""
# Normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
# Compute similarity matrix
similarity = torch.mm(embeddings, embeddings.t()) / temperature
# Create positive/negative masks
mask_pos = labels.unsqueeze(1) == labels.unsqueeze(0)
mask_neg = ~mask_pos
# Remove self-similarity from positives
mask_pos.fill_diagonal_(False)
# InfoNCE: log(exp(sim_pos) / sum(exp(sim)))
exp_sim = torch.exp(similarity)
pos_sim = (similarity * mask_pos).sum(dim=1, keepdim=True)
# False-negative mitigation: reduce negative contribution
neg_weight = torch.ones_like(similarity)
neg_weight[mask_pos] = 0.0
weighted_exp = exp_sim * neg_weight
loss = -torch.log(
torch.exp(pos_sim) / (weighted_exp.sum(dim=1, keepdim=True) + 1e-8) + 1e-8
)
return loss.mean()
def merge_checkpoints(self, checkpoint_paths: List[str], output_path: str):
"""
Spherical linear interpolation (slerp) across checkpoints for robustness.
"""
import numpy as np
# Load and stack weights from all checkpoints
all_weights = []
for ckpt_path in checkpoint_paths:
weights = torch.load(ckpt_path, map_location=self.device)
all_weights.append(weights)
# Compute mean direction in weight space
merged_weights = {}
for key in all_weights[0].keys():
weight_vectors = torch.stack([w[key].flatten() for w in all_weights])
# Normalize and average (slerp approximation)
normalized = F.normalize(weight_vectors, p=2, dim=1)
merged = F.normalize(normalized.mean(dim=0), p=2, dim=0)
merged_weights[key] = merged.view(all_weights[0][key].shape)
torch.save(merged_weights, output_path)
return merged_weights
Practical Guidance
Data Synthesis Strategy: When generating synthetic pairs with Qwen3-32B, use diverse prompts covering different difficulty levels, domains (medical, code, general), and languages. Aim for 150M+ pairs for comprehensive pre-training.
Contrastive Temperature: The temperature parameter (typically 0.05-0.10) controls hardness of negatives. Lower values emphasize hard negatives; higher values smooth the distribution. Tune based on the diversity of your training corpus.
In-Batch Negatives: Ensure sufficient batch size (64-256) to provide enough negative examples for contrastive learning. Larger batches reduce variance in loss estimation.
Fine-tuning Curriculum: Start with general retrieval pairs, then progressively add domain-specific and language-specific examples. This curriculum approach improves final model robustness.
Model Size Selection: For resource-constrained applications, the 0.6B variant provides competitive performance on retrieval tasks. Use 4B for balanced speed/quality, and 8B when maximum accuracy is required.
Checkpoint Merging: Apply spherical linear interpolation across 3-5 checkpoints from the end of training to smooth out noisy updates and improve generalization.
Reference
Qwen3 Embedding achieves state-of-the-art results:
- MTEB Multilingual: 70.58 (8B model)
- MTEB Code: 80.68
- Multilingual Retrieval: Competitive performance across 216 diverse tasks
The model is particularly effective for code understanding and multilingual scenarios where foundation model pre-training provides strong inductive biases. The synthetic data generation approach enables efficient adaptation to new domains without requiring expensive manual annotation.