scaling-laws-code

name	scaling-laws-code
title	Scaling Laws for Code: Every Programming Language Matters
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.13472
keywords	["scaling-laws","multilingual-code","pre-training","allocation-strategy"]
description	Establish language-specific scaling laws for multilingual code pre-training across 7 programming languages (0.2B-14B models, 1T tokens). Show interpreted languages scale better than compiled, synergy gains depend on syntax similarity, and parallel pairing improves translation—enabling proportion-dependent token allocation outperforming uniform distribution.

Overview

This research establishes the first comprehensive scaling law framework for multilingual code pre-training. Unlike language-agnostic approaches, the work reveals that programming languages have fundamentally different scaling characteristics and transfer behaviors. These insights enable principled token allocation strategies.

Core Technique

The key insight is that different programming languages exhibit distinct scaling relationships and transfer properties.

Language-Specific Scaling Laws: Each language exhibits power-law relationships with distinct exponents.

# Scaling law analysis per programming language
import numpy as np

class LanguageScalingLaws:
    def __init__(self):
        self.languages = ['python', 'java', 'javascript', 'c', 'c++', 'rust', 'go']
        self.scaling_laws = {}

    def fit_scaling_law(self, language, model_sizes, perplexities):
        """
        Fit power law: L = a * (N + D)^(-b)
        where N = model size, D = data size, L = loss
        """
        # Log-linear regression
        log_sizes = np.log(model_sizes)
        log_losses = np.log(perplexities)

        coefficients = np.polyfit(log_sizes, log_losses, deg=1)
        # coefficients: [exponent_b, intercept_a]

        self.scaling_laws[language] = {
            'exponent': abs(coefficients[0]),  # b term
            'category': self.categorize_language(language)
        }

        return self.scaling_laws[language]

     ():
        
         language  [, ]:
             
        :
             

     ():
        
        interpreted = [
            law  lang, law  .scaling_laws.items()
             law[] == 
        ]
        compiled = [
            law  lang, law  .scaling_laws.items()
             law[] == 
        ]

        ()
        ()