菜单
Predict optimal data mixture proportions for multi-domain LLM training using scaling laws that require only 10-20 small experiments. Determine which domains should contribute how much data across model sizes (186M-7B), reducing computational waste in foundation model pretraining.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill scaling-laws-optimal-data-mixtures命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
基于 SOC 职业分类
| name | scaling-laws-optimal-data-mixtures |
| title | Scaling Laws for Optimal Data Mixtures |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2507.09404 |
| keywords | ["Scaling Laws","Data Mixtures","Multi-Domain Training","Compute Optimization","Foundation Models"] |
| description | Predict optimal data mixture proportions for multi-domain LLM training using scaling laws that require only 10-20 small experiments. Determine which domains should contribute how much data across model sizes (186M-7B), reducing computational waste in foundation model pretraining. |
Foundation models trained on diverse domains (code, math, text) must decide how much data from each domain to include. Current practice uses trial-and-error, wasting compute on suboptimal mixtures. This work proposes scaling laws that predict model loss as a function of model size (N), training tokens (D), and domain weights (h), enabling practitioners to determine optimal mixtures efficiently.
The key insight is that only 10-20 small-scale training runs are needed to fit accurate scaling laws that extrapolate to much larger models. Two law formulations are provided: additive (mixture-independent scale parameters, predicting scale-independent optimal mixtures) and joint (mixture-dependent scale parameters, predicting compute-budget-dependent optima). Both formulations have been validated across language models, multimodal models, and vision models.
Multi-domain training loss can be modeled as: E + 1/∑(C_i·h_i^γ_i) + A/N^α + B/D^β, where only the bias term E depends on domain weights h. This reveals a crucial property: for additive scaling laws, optimal domain weights are compute-independent—they don't change with model size or total training budget. Only the absolute performance scales.
For joint scaling laws where scale parameters A and B themselves depend on mixture composition, optimal weights become budget-dependent. The fitting procedure uses Basin-hopping with L-BFGS to navigate the high-dimensional parameter space efficiently.
Model loss where domain weights affect bias term but not scaling.
import numpy as np
from scipy.optimize import minimize, basinhopping
import torch
class AdditiveScalingLaw:
():
.num_domains = num_domains
.C = np.ones(num_domains)
.gamma = np.ones(num_domains)
.A =
.alpha =
.B =
.beta =
():
bias =
weighted_sum =
i (.num_domains):
weighted_sum += .C[i] * (mixture_weights[i] ** .gamma[i])
mixture_term = / (weighted_sum + )
bias += mixture_term
scale_loss = .A / (model_size ** .alpha) + .B / (tokens ** .beta)
bias + scale_loss
():
():
C = params[:.num_domains]
gamma = params[.num_domains:*.num_domains]
A = params[*.num_domains]
alpha = params[*.num_domains + ]
B = params[*.num_domains + ]
beta = params[*.num_domains + ]
.C = C
.gamma = gamma
.A = A
.alpha = alpha
.B = B
.beta = beta
mse =
exp experiments:
predicted = .predict_loss(
exp[],
exp[],
exp[]
)
mse += (predicted - exp[]) **
mse / (experiments)
x0 = np.concatenate([
.C,
.gamma,
[.A, .alpha, .B, .beta]
])
minimizer_kwargs = {
: ,
: [(, )] * (*.num_domains) + [(, ), (, ), (, ), (, )]
}
result = basinhopping(
objective,
x0,
minimizer_kwargs=minimizer_kwargs,
niter=,
seed=
)
result.fun
() -> np.ndarray:
():
np.isclose(weights.(), ):
weighted_sum =
i (.num_domains):
weighted_sum += .C[i] * (weights[i] ** .gamma[i])
loss = / (weighted_sum + )
loss
scipy.optimize LinearConstraint
A_constraint = np.ones((, .num_domains))
constraint = LinearConstraint(A_constraint, [], [])
x0 = np.ones(.num_domains) / .num_domains
result = minimize(
objective_mixture,
x0,
method=,
bounds=[(, ) _ (.num_domains)]
)
optimal = result.x / result.x.()
optimal
More complex formulation where scale parameters depend on mixture.
class JointScalingLaw:
"""
Joint formulation: Loss = E(h) + A^h/N^α^h + B^h/D^β^h
where A, B, α, β all depend on mixture weights h.
Captures scale-mixture interactions.
"""
def __init__(self, num_domains: int = 3):
self.num_domains = num_domains
# For A term
self.C_A = np.ones(num_domains)
self.gamma_A = np.ones(num_domains)
self.alpha = 0.5
# For B term
self.C_B = np.ones(num_domains)
self.gamma_B = np.ones(num_domains)
self.beta = 0.5
# Bias term
self.C_E = np.ones(num_domains)
self.gamma_E = np.ones(num_domains)
def predict_loss(self, model_size: int, tokens: int, mixture_weights: np.ndarray):
"""
Predict loss with mixture-dependent scale parameters.
Args:
model_size: N
tokens: D
mixture_weights: (num_domains,)
Returns:
Predicted loss
"""
# A term depends on mixture
A_weighted = 0.0
for i in range(self.num_domains):
A_weighted += self.C_A[i] * (mixture_weights[i] ** self.gamma_A[i])
# B term depends on mixture
B_weighted = 0.0
for i in range(self.num_domains):
B_weighted += self.C_B[i] * (mixture_weights[i] ** self.gamma_B[i])
# Bias depends on mixture
E_term = 0.0
for i in range(self.num_domains):
E_term += self.C_E[i] * (mixture_weights[i] ** self.gamma_E[i])
# Loss prediction
loss = E_term + A_weighted / (model_size ** self.alpha) + B_weighted / (tokens ** self.beta)
return loss
def fit_to_experiments(self, experiments: list) -> float:
"""Fit joint scaling law to experiments."""
def objective(params):
# Extract parameters
C_A = params[:self.num_domains]
gamma_A = params[self.num_domains:2*self.num_domains]
C_B = params[2*self.num_domains:3*self.num_domains]
gamma_B = params[3*self.num_domains:4*self.num_domains]
C_E = params[4*self.num_domains:5*self.num_domains]
gamma_E = params[5*self.num_domains:6*self.num_domains]
alpha = params[6*self.num_domains]
beta = params[6*self.num_domains + 1]
# Update
self.C_A = C_A
self.gamma_A = gamma_A
self.C_B = C_B
self.gamma_B = gamma_B
self.C_E = C_E
self.gamma_E = gamma_E
self.alpha = alpha
self.beta = beta
# MSE
mse = 0.0
for exp in experiments:
predicted = self.predict_loss(
exp['model_size'],
exp['tokens'],
exp['mixture_weights']
)
mse += (predicted - exp['loss']) ** 2
return mse / len(experiments)
x0 = np.concatenate([
self.C_A, self.gamma_A,
self.C_B, self.gamma_B,
self.C_E, self.gamma_E,
[self.alpha, self.beta]
])
minimizer_kwargs = {
'method': 'L-BFGS-B',
'bounds': [(0.1, 10)] * (6*self.num_domains) + [(0.1, 2), (0.1, 2)]
}
result = basinhopping(
objective,
x0,
minimizer_kwargs=minimizer_kwargs,
niter=20
)
return result.fun
def find_optimal_mixture(
self,
model_size: int,
tokens: int,
optimization_target: str = 'loss'
) -> np.ndarray:
"""
Find optimal mixture for specific compute budget (compute-dependent).
Args:
model_size: Target model size N
tokens: Target tokens D
optimization_target: 'loss' or 'efficiency'
Returns:
Optimal mixture weights for this budget
"""
def objective_mixture(weights):
if not np.isclose(weights.sum(), 1.0):
return 1e10
loss = self.predict_loss(model_size, tokens, weights)
return loss
x0 = np.ones(self.num_domains) / self.num_domains
from scipy.optimize import LinearConstraint
A_constraint = np.ones((1, self.num_domains))
constraint = LinearConstraint(A_constraint, [1], [1])
result = minimize(
objective_mixture,
x0,
method='SLSQP',
bounds=[(0, 1) for _ in range(self.num_domains)]
)
return result.x / result.x.sum()
Design efficient small experiments to fit scaling laws.
class ScalingLawExperimentDesigner:
"""Design efficient experiments for fitting scaling laws."""
@staticmethod
def generate_experiment_grid(
model_sizes: list,
token_counts: list,
mixture_points: list,
num_domains: int = 3
) -> list:
"""
Generate grid of experiments to fit scaling laws efficiently.
Args:
model_sizes: Model sizes to test (e.g., [186M, 370M, 750M, 1.5B])
token_counts: Token counts to test (e.g., [1B, 5B, 10B, 50B])
mixture_points: Mixture weight vectors (e.g., uniform, domain-heavy)
num_domains: Number of domains
Returns:
List of experiment configs
"""
experiments = []
for model_size in model_sizes:
for tokens in token_counts:
for mixture in mixture_points:
# Validate mixture
if not np.isclose(sum(mixture), 1.0):
mixture = np.array(mixture) / sum(mixture)
experiments.append({
'model_size': model_size,
'tokens': tokens,
'mixture_weights': mixture,
'loss': None # To be filled by actual training
})
return experiments
@staticmethod
def recommend_efficient_grid(num_domains: int = 3):
"""Recommend efficient experimental grid (10-20 runs only)."""
model_sizes = [186_000_000, 370_000_000, 750_000_000] # 3 sizes
token_counts = [1_000_000_000, 10_000_000_000] # 2 token budgets
# Mixture points: uniform + domain-heavy variants
mixtures = [
[1/num_domains] * num_domains # Uniform
]
# Add domain-heavy points
for i in range(num_domains):
mix = [0.1] * num_domains
mix[i] = 0.7
mix = [m / sum(mix) for m in mix] # Normalize
mixtures.append(mix)
return {
'model_sizes': model_sizes,
'token_counts': token_counts,
'mixtures': mixtures
}
Complete pipeline for determining optimal training mixture.
def determine_optimal_mixture_for_production(
train_losses: list,
target_model_size: int,
target_tokens: int,
num_domains: int = 3,
use_joint: bool = False
) -> dict:
"""
End-to-end: fit scaling law and find optimal mixture.
Args:
train_losses: Results from small-scale experiments
target_model_size: Production model size to optimize for
target_tokens: Production training budget
num_domains: Number of domains in training data
use_joint: Use joint (True) or additive (False) formulation
Returns:
{'optimal_mixture': weights, 'predicted_loss': loss, 'law_fit_error': mre}
"""
# Choose formulation
if use_joint:
law = JointScalingLaw(num_domains)
else:
law = AdditiveScalingLaw(num_domains)
# Fit to experimental data
fit_error = law.fit_to_experiments(train_losses)
print(f"Scaling law fit error (MSE): {fit_error:.6f}")
# Find optimal mixture for target compute
optimal_mixture = law.find_optimal_mixture(target_model_size, target_tokens)
# Predict loss with optimal mixture
predicted_loss = law.predict_loss(target_model_size, target_tokens, optimal_mixture)
print(f"\nOptimal mixture for {target_model_size/1e6:.0f}M model, {target_tokens/1e9:.0f}B tokens:")
for i, weight in enumerate(optimal_mixture):
print(f" Domain {i}: {weight:.1%}")
print(f"Predicted loss: {predicted_loss:.4f}")
# Extrapolation confidence
mean_relative_error = fit_error # Simplified; in practice compute MRE
return {
'optimal_mixture': optimal_mixture,
'predicted_loss': predicted_loss,
'law_fit_error': fit_error,
'extrapolation_confidence': 'high' if fit_error < 0.02 else 'medium'
}
# Example usage
sample_experiments = [
{'model_size': 186_000_000, 'tokens': 1_000_000_000, 'mixture_weights': [0.33, 0.33, 0.34], 'loss': 3.521},
{'model_size': 186_000_000, 'tokens': 10_000_000_000, 'mixture_weights': [0.33, 0.33, 0.34], 'loss': 3.201},
{'model_size': 750_000_000, 'tokens': 1_000_000_000, 'mixture_weights': [0.33, 0.33, 0.34], 'loss': 3.156},
{'model_size': 750_000_000, 'tokens': 10_000_000_000, 'mixture_weights': [0.33, 0.33, 0.34], 'loss': 2.834},
# ... more experiments
]
result = determine_optimal_mixture_for_production(
sample_experiments,
target_model_size=7_000_000_000,
target_tokens=1_000_000_000_000,
num_domains=3,
use_joint=False
)
| Parameter | Value | Notes |
|---|---|---|
| Minimum Experiments | 10-20 | Each requires full pretraining; 10 experiments sufficient for 3-domain problems |
| Model Sizes | 186M, 370M, 750M (baseline), 1.7B | Doubling strategy reveals exponential trends |
| Token Budgets | 1B, 5B, 10B, 50B | Test different scales of compute |
| Domain Mixtures | Uniform + domain-heavy | 3-5 mixture points sufficient |
| Basin-hopping Iterations | 20 | Balance between optimization quality and speed |
| Optimizer | L-BFGS-B | Robust for parameter fitting |
| Constraint Type | Simplex (weights sum to 1) | Ensures valid probability distribution |
Chen, Y., Wang, X., Liu, S., et al. (2024). Scaling Laws for Optimal Data Mixtures. arXiv preprint arXiv:2507.09404.