Merge multiple fine-tuned models using mergekit to combine capabilities without retraining. Use when creating specialized models by blending domain-specific expertise (math + coding + chat), improving performance beyond single models, or experimenting rapidly with model variants. Covers SLERP, TIES-Merging, DARE, Task Arithmetic, linear merging, and production deployment strategies.
Merge multiple fine-tuned models using mergekit to combine capabilities without retraining. Use when creating specialized models by blending domain-specific expertise (math + coding + chat), improving performance beyond single models, or experimenting rapidly with model variants. Covers SLERP, TIES-Merging, DARE, Task Arithmetic, linear merging, and production deployment strategies.
Improve performance beyond single models (often +5-10% on benchmarks)
Reduce training costs - no GPUs needed, merges run on CPU
Experiment rapidly - create new model variants in minutes, not days
Preserve multiple skills - merge without catastrophic forgetting
Success Stories: Marcoro14-7B-slerp (best on Open LLM Leaderboard 02/2024), many top HuggingFace models use merging
Tools: mergekit (Arcee AI), LazyMergekit, Model Soup
Installation
# Install mergekit
git clone https://github.com/arcee-ai/mergekit.git
cd mergekit
pip install -e .
# Or via pip
pip install mergekit
# Optional: Transformer library
pip install transformers torch
Quick Start
Simple Linear Merge
# config.yml - Merge two models with equal weightsmerge_method:linearmodels:-model:mistralai/Mistral-7B-v0.1parameters:weight:0.5-model:teknium/OpenHermes-2.5-Mistral-7Bparameters:weight:0.5dtype:bfloat16
# Run merge
mergekit-yaml config.yml ./merged-model --cuda
# Use merged model
python -m transformers.models.auto --model_name_or_path ./merged-model
merge_method:tiesbase_model:mistralai/Mistral-7B-v0.1models:-model:WizardLM/WizardMath-7B-V1.1parameters:density:0.5# Keep top 50% of parametersweight:1.0-model:teknium/OpenHermes-2.5-Mistral-7Bparameters:density:0.5weight:1.0-model:NousResearch/Nous-Hermes-2-Mistral-7B-DPOparameters:density:0.5weight:1.0parameters:normalize:truedtype:bfloat16
DARE Merge
Best for: Reducing redundancy
merge_method:dare_tiesbase_model:mistralai/Mistral-7B-v0.1models:-model:WizardLM/WizardMath-7B-V1.1parameters:density:0.5# Drop 50% of deltasweight:0.6-model:teknium/OpenHermes-2.5-Mistral-7Bparameters:density:0.5weight:0.4parameters:int8_mask:true# Use int8 for masks (saves memory)dtype:bfloat16
Advanced Patterns
Layer-wise Merging
# Different models for different layersmerge_method:passthroughslices:-sources:-model:mistralai/Mistral-7B-v0.1layer_range: [0, 16] # First half-sources:-model:teknium/OpenHermes-2.5-Mistral-7Blayer_range: [16, 32] # Second halfdtype:bfloat16
MoE from Merged Models
# Create Mixture of Expertsmerge_method:moebase_model:mistralai/Mistral-7B-v0.1experts:-source_model:WizardLM/WizardMath-7B-V1.1positive_prompts:-"math"-"calculate"-source_model:teknium/OpenHermes-2.5-Mistral-7Bpositive_prompts:-"chat"-"conversation"-source_model:ajibawa-2023/Code-Mistral-7Bpositive_prompts:-"code"-"python"dtype:bfloat16
Tokenizer Merging
merge_method:linearmodels:-model:mistralai/Mistral-7B-v0.1-model:custom/specialized-modeltokenizer:source:"union"# Combine vocabularies from both modelstokens:<|special_token|>:source:"custom/specialized-model"
Best Practices
1. Model Compatibility
# ✅ Good: Same architecture
models = [
"mistralai/Mistral-7B-v0.1",
"teknium/OpenHermes-2.5-Mistral-7B", # Both Mistral 7B
]
# ❌ Bad: Different architectures
models = [
"meta-llama/Llama-2-7b-hf", # Llama"mistralai/Mistral-7B-v0.1", # Mistral (incompatible!)
]
2. Weight Selection
# ✅ Good: Weights sum to 1.0models:-model:model_aparameters:weight:0.6-model:model_bparameters:weight:0.4# 0.6 + 0.4 = 1.0# ⚠️ Acceptable: Weights don't sum to 1 (for task arithmetic)models:-model:model_aparameters:weight:0.8-model:model_bparameters:weight:0.8# May boost performance
3. Method Selection
# Choose merge method based on use case:# 2 models, smooth blend → SLERP
merge_method = "slerp"# 3+ models, simple average → Linear
merge_method = "linear"# Multiple task-specific models → Task Arithmetic or TIES
merge_method = "ties"# Want to reduce redundancy → DARE
merge_method = "dare_ties"
4. Density Tuning (TIES/DARE)
# Start conservative (keep more parameters)parameters:density:0.8# Keep 80%# If performance good, increase sparsityparameters:density:0.5# Keep 50%# If performance degrades, reduce sparsityparameters:density:0.9# Keep 90%
5. Layer-specific Merging
# Preserve base model's beginning and endmerge_method:passthroughslices:-sources:-model:base_modellayer_range: [0, 2] # Keep first layers-sources:-model:merged_middle# Merge middle layerslayer_range: [2, 30]
-sources:-model:base_modellayer_range: [30, 32] # Keep last layers
Evaluation & Testing
Benchmark Merged Models
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load merged model
model = AutoModelForCausalLM.from_pretrained("./merged-model")
tokenizer = AutoTokenizer.from_pretrained("./merged-model")
# Test on various tasks
test_prompts = {
"math": "Calculate: 25 * 17 =",
"code": "Write a Python function to reverse a string:",
"chat": "What is the capital of France?",
}
for task, prompt in test_prompts.items():
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
print(f"{task}: {tokenizer.decode(outputs[0])}")
Common Benchmarks
Open LLM Leaderboard: General capabilities
MT-Bench: Multi-turn conversation
MMLU: Multitask accuracy
HumanEval: Code generation
GSM8K: Math reasoning
Production Deployment
Save and Upload
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load merged model
model = AutoModelForCausalLM.from_pretrained("./merged-model")
tokenizer = AutoTokenizer.from_pretrained("./merged-model")
# Upload to HuggingFace Hub
model.push_to_hub("username/my-merged-model")
tokenizer.push_to_hub("username/my-merged-model")
