| name | recall-hierarchical-merging |
| title | RECALL: Catastrophic-forgetting Alleviation via Hierarchical Model Merging |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2510.20479 |
| keywords | ["Continual Learning","Model Merging","Catastrophic Forgetting","Representations"] |
| description | Prevents catastrophic forgetting in continual learning by merging models using layer-wise hidden representations as similarity proxies. Shallow layers preserve domain-general features while deep layers enable task-specific adaptation, enabling seamless multi-domain integration without task labels or historical data. |
RECALL: Hierarchical Merging for Continual Learning
Continual learning faces a fundamental trade-off: retain past knowledge while learning new tasks. RECALL exploits layer-wise architectural differences to resolve this by merging representations hierarchically, using hidden states as reliable knowledge proxies.
The approach enables seamless knowledge integration across multiple domains without accessing historical data or task identities.
Core Concept
Key insight: Layer depth correlates with knowledge type:
- Shallow layers encode domain-general, transferable features
- Deep layers capture task-specific, specialized knowledge
RECALL leverages this by:
- Computing inter-model similarity using layer-wise hidden representations
- Adapting fusion strategy per layer (preserve shallow, specialize deep)
- Merging parameters with layer-dependent weightings
Architecture Overview
- Layer-wise hidden representation clustering on typical samples
- Similarity computation using cosine distance in representation space
- Adaptive parameter fusion with layer-dependent coefficients
- Support for multi-domain integration without task boundaries
Implementation Steps
Compute layer-wise representations on a small set of typical examples. These representations capture what each model learned at different depths:
def compute_layer_representations(model, sample_batch, layer_indices=None):
"""Extract hidden representations at specified layers."""
representations = {}
hooks = []
def get_hook(layer_name):
def hook(module, input, output):
if isinstance(output, tuple):
output = output[0]
representations[layer_name] = output.detach().cpu()
return hook
if layer_indices :
layer_indices = (((model.children())))
idx layer_indices:
layer = (model.children())[idx]
hook = layer.register_forward_hook(get_hook())
hooks.append(hook)
torch.no_grad():
_ = model(sample_batch)
hook hooks:
hook.remove()
representations
