model-merging-dual-anchors

name	model-merging-dual-anchors
title	Model Merging with Functional Dual Anchors
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.21223
keywords	["Model Merging","Multi-task Learning","Task Vectors","Parameter Space"]
description	Merges multiple fine-tuned models by operating in input-representation space rather than parameter space. Creates synthetic inputs whose gradients align with task vectors, bridging joint training and post-hoc merging for robust multi-task model combination.

Model Merging via Functional Dual Anchors: Beyond Parameter Space

Parameter-space merging methods suffer from conflicts when task vectors diverge. Functional Dual Anchors (FDAs) capture task shifts in representation space through synthetic inputs, enabling more robust model combination.

FDAs bridge multi-task training and post-hoc merging, offering flexibility to merge models trained independently while maintaining task-specific capabilities.

Core Concept

Rather than averaging or interpolating model weights directly, FDAs operate by:

• Creating synthetic inputs in input-representation space
• Designing these inputs so their gradients match task-specific vectors
• Using gradient alignment to represent how models diverge from base
• Merging in this aligned gradient space rather than parameter space

This approach mitigates parameter inconsistencies that plague traditional merging.

Architecture Overview

• Synthetic input generation that produces task-aligned gradients
• Principled initialization scheme for anchor points
• Gradient-based representation of task shifts
• Complementary to parameter-space merging methods

Implementation Steps

Define synthetic inputs as learnable parameters whose gradients capture task shifts. Rather than working with weight parameters directly, optimize inputs to produce task-relevant gradients:

class FunctionalDualAnchors:
    def __init__(self, base_model, num_tasks, input_dim=768):
        self.base_model = base_model
        self.num_tasks = num_tasks
        # Create synthetic inputs per task
        self.synthetic_inputs = nn.ParameterList([
            nn.Parameter(torch.randn(1, input_dim) * 0.01)
            for _ in range(num_tasks)
        ])

    def compute_task_gradient(self, task_id, target_vector):
        """Compute gradient of synthetic input to match task vector."""
        synthetic = .synthetic_inputs[task_id]
        synthetic.requires_grad_()

        
        output = .base_model(synthetic)

        
        loss = -torch.dot(output.squeeze(), target_vector)
        loss.backward()

         synthetic.grad