qwen3-embedding

name	qwen3-embedding
title	Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.05176
keywords	["text-embeddings","reranking","foundation-models","multilingual","retrieval"]
description	Leverages Qwen3 foundation models for text embedding and reranking via multi-stage training combining weakly-supervised pre-training on 150M synthetic pairs with supervised fine-tuning.

Qwen3 Embedding

Core Concept

Qwen3 Embedding builds state-of-the-art text embeddings and reranking models by leveraging the rich representation capacity of Qwen3 foundation models. Rather than training from scratch, the approach combines weak supervision at scale with targeted fine-tuning on curated datasets, achieving superior multilingual and code-understanding performance. The key innovation is using the Qwen3-32B model itself to synthesize diverse training data across multiple tasks and languages.

Architecture Overview

• Multi-Stage Pipeline: Two-phase training combining weakly-supervised pre-training (150M+ synthetic pairs) with supervised fine-tuning on curated datasets
• LLM-Driven Synthesis: Uses Qwen3-32B to generate synthetic text pairs across retrieval, bitext mining, classification, and semantic similarity tasks
• Model Variants: Offers three sizes (0.6B, 4B, 8B) for both embedding and reranking with flexible output dimensions
• Contrastive Learning: Modified InfoNCE loss with in-batch negatives and false-negative mitigation strategies
• Model Merging: Spherical linear interpolation across training checkpoints for improved robustness

Implementation

The following code demonstrates the synthetic data generation and training pipeline:

import torch
import torch.nn.functional as F
from torch.optim import AdamW
from typing import List, Tuple

class Qwen3EmbeddingTrainer:
    def __init__(self, model_name: str = "qwen3-8b", embedding_dim: int = 1024):
        self.model_name = model_name
        self.embedding_dim = embedding_dim
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    def prepare_synthetic_pairs(self, task_type: str, num_pairs: int = 10000) -> List[Tuple[str, str]]:
        """
        Generate synthetic training pairs using LLM synthesis.
        task_type: 'retrieval', 'bitext', 'classification', or 'similarity'
        """
        pairs = []
        for i in range(num_pairs):
            # In practice, use Qwen3-32B with task-specific prompts
            query = f"Generate query for {task_type} task {i}"
            document = f"Generate document for {task_type} task {i}"
            pairs.append((query, document))
         pairs

     () -> torch.Tensor:
        
        
        embeddings = F.normalize(embeddings, p=, dim=)

        
        similarity = torch.mm(embeddings, embeddings.t()) / temperature

        
        mask_pos = labels.unsqueeze() == labels.unsqueeze()
        mask_neg = ~mask_pos

        
        mask_pos.fill_diagonal_()

        
        exp_sim = torch.exp(similarity)
        pos_sim = (similarity * mask_pos).(dim=, keepdim=)

        
        neg_weight = torch.ones_like(similarity)
        neg_weight[mask_pos] = 
        weighted_exp = exp_sim * neg_weight

        loss = -torch.log(
            torch.exp(pos_sim) / (weighted_exp.(dim=, keepdim=) + ) + 
        )

         loss.mean()

     ():
        
         numpy  np

        
        all_weights = []
         ckpt_path  checkpoint_paths:
            weights = torch.load(ckpt_path, map_location=.device)
            all_weights.append(weights)

        
        merged_weights = {}
         key  all_weights[].keys():
            weight_vectors = torch.stack([w[key].flatten()  w  all_weights])
            
            normalized = F.normalize(weight_vectors, p=, dim=)
            merged = F.normalize(normalized.mean(dim=), p=, dim=)
            merged_weights[key] = merged.view(all_weights[][key].shape)

        torch.save(merged_weights, output_path)
         merged_weights

Practical Guidance

Data Synthesis Strategy: When generating synthetic pairs with Qwen3-32B, use diverse prompts covering different difficulty levels, domains (medical, code, general), and languages. Aim for 150M+ pairs for comprehensive pre-training.

Contrastive Temperature: The temperature parameter (typically 0.05-0.10) controls hardness of negatives. Lower values emphasize hard negatives; higher values smooth the distribution. Tune based on the diversity of your training corpus.

In-Batch Negatives: Ensure sufficient batch size (64-256) to provide enough negative examples for contrastive learning. Larger batches reduce variance in loss estimation.

Fine-tuning Curriculum: Start with general retrieval pairs, then progressively add domain-specific and language-specific examples. This curriculum approach improves final model robustness.

Model Size Selection: For resource-constrained applications, the 0.6B variant provides competitive performance on retrieval tasks. Use 4B for balanced speed/quality, and 8B when maximum accuracy is required.

Checkpoint Merging: Apply spherical linear interpolation across 3-5 checkpoints from the end of training to smooth out noisy updates and improve generalization.

Reference

Qwen3 Embedding achieves state-of-the-art results:

• MTEB Multilingual: 70.58 (8B model)
• MTEB Code: 80.68
• Multilingual Retrieval: Competitive performance across 216 diverse tasks

The model is particularly effective for code understanding and multilingual scenarios where foundation model pre-training provides strong inductive biases. The synthetic data generation approach enables efficient adaptation to new domains without requiring expensive manual annotation.