recall-hierarchical-merging

name	recall-hierarchical-merging
title	RECALL: Catastrophic-forgetting Alleviation via Hierarchical Model Merging
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.20479
keywords	["Continual Learning","Model Merging","Catastrophic Forgetting","Representations"]
description	Prevents catastrophic forgetting in continual learning by merging models using layer-wise hidden representations as similarity proxies. Shallow layers preserve domain-general features while deep layers enable task-specific adaptation, enabling seamless multi-domain integration without task labels or historical data.

RECALL: Hierarchical Merging for Continual Learning

Continual learning faces a fundamental trade-off: retain past knowledge while learning new tasks. RECALL exploits layer-wise architectural differences to resolve this by merging representations hierarchically, using hidden states as reliable knowledge proxies.

The approach enables seamless knowledge integration across multiple domains without accessing historical data or task identities.

Core Concept

Key insight: Layer depth correlates with knowledge type:

• Shallow layers encode domain-general, transferable features
• Deep layers capture task-specific, specialized knowledge

RECALL leverages this by:

• Computing inter-model similarity using layer-wise hidden representations
• Adapting fusion strategy per layer (preserve shallow, specialize deep)
• Merging parameters with layer-dependent weightings

Architecture Overview

• Layer-wise hidden representation clustering on typical samples
• Similarity computation using cosine distance in representation space
• Adaptive parameter fusion with layer-dependent coefficients
• Support for multi-domain integration without task boundaries

Implementation Steps

Compute layer-wise representations on a small set of typical examples. These representations capture what each model learned at different depths:

def compute_layer_representations(model, sample_batch, layer_indices=None):
    """Extract hidden representations at specified layers."""
    representations = {}
    hooks = []

    def get_hook(layer_name):
        def hook(module, input, output):
            if isinstance(output, tuple):
                output = output[0]
            representations[layer_name] = output.detach().cpu()
        return hook

    # Register hooks at each layer
    if layer_indices  :
        layer_indices = (((model.children())))

     idx  layer_indices:
        layer = (model.children())[idx]
        hook = layer.register_forward_hook(get_hook())
        hooks.append(hook)

    
     torch.no_grad():
        _ = model(sample_batch)

    
     hook  hooks:
        hook.remove()

     representations