encoder-pretraining-strategy

name	encoder-pretraining-strategy
title	Should We Still Pretrain Encoders with Masked Language Modeling?
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.00994
keywords	["Pretraining Objectives","Masked Language Modeling","Causal Language Modeling","Bidirectional Attention","Transfer Learning"]
description	Choose optimal pretraining strategy for text encoders: pure MLM, pure CLM, or biphasic CLM-then-MLM training, with empirical guidance on performance across downstream tasks.

Encoder Pretraining: MLM vs CLM Trade-offs and Optimal Combinations

The question of how to pretrain text encoders has evolved since BERT's introduction of masked language modeling. As decoder-based models like GPT demonstrate strong transfer learning capabilities, researchers now question whether MLM remains optimal for encoders. This work conducts a large-scale controlled study (38 models, 15,000+ evaluation runs) isolating the effects of learning objectives while controlling for model size and data exposure. The findings reveal that neither pure MLM nor pure CLM dominates universally, but biphasic pretraining—sequential CLM followed by MLM—outperforms both under fixed compute budgets.

The practical implication is significant: practitioners building encoders can leverage pretrained decoder checkpoints from existing open-source models, fine-tune them with MLM, and achieve better results than training encoders from scratch. This reuses existing infrastructure while gaining the benefits of both causal and bidirectional training.

Core Concept

Bidirectional and causal pretraining optimize different aspects of representation learning. Causal language modeling (predicting next tokens sequentially) produces strong representations early in training with good data efficiency and fine-tuning stability. Masked language modeling (bidirectional context) converges to better ultimate performance on tasks requiring full contextual understanding. Sequential training that combines both objectives balances these trade-offs, with optimal splits varying by compute budget.

The insight is that the choice of pretraining objective should depend on downstream task requirements and compute constraints. For sequence classification and question-answering, bidirectional pretraining dominates. For early stopping or parameter-constrained settings, CLM's efficiency matters more. The biphasic strategy elegantly handles both regimes.

Architecture Overview

The experimental setup compares three pretraining configurations:

• Pure CLM: Causal decoder-style training from scratch or from existing models
• Pure MLM: Bidirectional masked training from scratch
• Biphasic CLM→MLM: Sequential phase 1 with CLM, phase 2 with MLM, with variable split ratios

The same model architecture (transformer encoders, 210M to 1B parameters) is used across conditions to isolate objective effects. Evaluation covers four task categories: sequence classification, token classification, question answering, and information retrieval. All models trained on 100B uniform tokens from FineWeb-Edu dataset.

Implementation

Set up the training framework with configurable objectives:

from typing import 
 torch
 torch  nn
 transformers  AutoConfig, AutoTokenizer
 numpy  np

 :
    

     ():
        
        .objective = objective
        .mlm_probability = mlm_probability
        .clm_weight = clm_weight
        .mlm_weight = mlm_weight
        .biphasic_split = biphasic_split  objective ==   

 (nn.Module):
    

     ():
        ().__init__()
        .config = config
        .vocab_size = vocab_size
        .lm_head = nn.Linear(, vocab_size)  
        .loss_fn = nn.CrossEntropyLoss()

     ():
        
        
        logits_flat = logits[..., :-, :].contiguous().view(-, .vocab_size)
        labels_flat = labels[..., :].contiguous().view(-)

         loss_mask   :
            loss_mask_flat = loss_mask[..., :].contiguous().view(-)
            loss = .loss_fn(logits_flat, labels_flat)
            loss = (loss * loss_mask_flat).() / loss_mask_flat.()
        :
            loss = .loss_fn(logits_flat, labels_flat)

         loss

     ():
        
        
        logits_masked = logits[mlm_mask]
        labels_masked = labels[mlm_mask]

        logits_flat = logits_masked.view(-, .vocab_size)
        labels_flat = labels_masked.view(-)

        loss = .loss_fn(logits_flat, labels_flat)
         loss

     ():
        
         .config.objective == :
             .forward_clm(logits, labels)

         .config.objective == :
             .forward_mlm(logits, labels, mlm_mask)

         .config.objective == :
            
             current_step     total_steps   :
                progress = current_step / total_steps
                clm_phase_end = .config.biphasic_split
            :
                clm_phase_end = .config.biphasic_split

             progress < clm_phase_end:
                
                 .forward_clm(logits, labels) * .config.clm_weight
            :
                
                 .forward_mlm(logits, labels, mlm_mask) * .config.mlm_weight

        :
             ValueError()