growing-transformers-modular-expansion

name	growing-transformers-modular-expansion
title	Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.07129
keywords	["Constructive Learning","Layer-wise Training","Parameter Efficiency","LLM Scaling"]
description	Build language models layer-by-layer on frozen embeddings, training new Transformer blocks sequentially while keeping lower layers frozen. Achieves 5% improvement over monolithic baselines on MMLU while fitting 740M trainable parameters per stage on single GPUs, enabling resource-efficient incremental scaling to 2.3B parameters.

Growing Transformers: Progressive Model Construction on Frozen Foundations

Traditional large language model training requires end-to-end gradient flow through billions of parameters, demanding massive compute clusters and suffering from optimization instabilities. Growing Transformers inverts this approach through constructive learning: build models incrementally by stacking Transformer layers on a fixed embedding foundation, training each new layer to convergence before freezing and adding the next. This strategy reduces memory requirements, enables single-GPU training of large models, and paradoxically improves performance by ~5% compared to training entire models from scratch.

The core insight is that semantic understanding emerges from architectural depth and composition rather than sophisticated input embeddings. By freezing random or minimally-trained embeddings, you force each subsequent layer to learn meaningful transformations from the layer below, creating a strong inductive bias toward hierarchical feature extraction.

Core Concept

Growing Transformers uses constructive training—a two-component strategy:

1. Layer-wise Sequential Addition: Start with a frozen embedding layer and trained lower layers. Add one Transformer block at a time, train it to convergence in isolation, then freeze before adding the next.
2. Holistic Fine-tuning via LoRA: After building the full stack, apply Low-Rank Adaptation (LoRA) across all layers for final performance gains without retraining from scratch.

Each new layer sees identical training conditions: fixed representations from below and a standard language modeling objective. The model grows incrementally while remaining trainable on single GPUs, then benefits from whole-model fine-tuning after construction completes.

Architecture Overview

• Frozen Embedding Layer: Random or pretrained embeddings (trainable dimension: 0 or 1B parameters) that serve as the non-learnable base
• Sequential Transformer Blocks: Standard 2-layer → 3-layer → 4-layer progression, each trained independently until convergence
• LoRA Fine-tuning Head: Low-rank weight matrices applied to attention and feed-forward layers across the full stack
• Training Scheduler: Per-layer convergence detection (loss plateau) with early stopping to determine when to add next layer
• Unified Tokenizer: Shared token vocabulary throughout training (no vocabulary shifts during growth)
• Checkpoint Management: Persistent storage of frozen layers during sequential training to enable resumption

Implementation

The following demonstrates layer-by-layer training with checkpoint management: