rl-neural-model-editing

name	rl-neural-model-editing
description	Reinforcement learning framework for neural model editing where agents learn to modify models via reward feedback instead of manually engineered algorithms
trigger_words	["neural model editing","model editing","bias mitigation","machine unlearning","RL model editing","MaskWorld","ShiftWorld"]
version	1.0.0
last_updated	2026-06-13T00:00:00.000Z
source	arXiv:2606.13461v1
authors	Shaivi Malik

Reinforcement Learning for Neural Model Editing

Problem Addressed

Editing pretrained neural networks traditionally requires:

• Specialized algorithms per editing objective
• Manual engineering effort
• Task-specific design decisions

This framework automates editing policy learning via RL.

Core Methodology

RL Formulation

1. State: Current model weights
2. Action: Weight modifications (multiplicative or additive)
3. Reward: Utility preservation + editing objective

Two Environments

MaskWorld (Multiplicative Editing)

• Actions: Scale weights by multiplicative factors
• Agent learns which weights to suppress/enhance
• Suitable for selective forgetting

ShiftWorld (Additive Editing)

• Actions: Add weight deltas
• Agent learns directional weight adjustments
• Suitable for bias correction, knowledge update

Reward Function Design

reward = utility_preservation + editing_objective

utility_preservation:
  - Retain set accuracy (on unmodified data)
  - General model performance

editing_objective:
  - Forget set accuracy → 0 (for unlearning)
  - Bias reduction (for bias mitigation)

Implementation Steps

1. Define editing objective

   • Examples: unlearn specific data, reduce bias

2. Create datasets

   • Retain set: performance should be preserved
   • Forget set: performance should be reduced
   • Validation set: overall utility

3. Choose environment

   • MaskWorld for multiplicative scaling
   • ShiftWorld for additive updates

4. Train RL agent

   • Policy network selects weight modifications
   • Episodes: multiple editing attempts
   • Reward computed from edited model performance

5. Apply learned policy

   • Use trained agent to edit target model

Experimental Results

Machine Unlearning (Image Classification)

• Forget set accuracy: ~0%
• Retain set accuracy: >90%

Bias Mitigation (Text Classification)

• Bias-related improvement: >5%
• General utility preserved

Advantages

• No manual algorithm design - Policy learned from reward feedback
• General framework - Works across different editing objectives
• Learned policies - Potentially discover novel editing strategies

Use Cases

1. Machine unlearning - Remove specific data influence
2. Bias mitigation - Correct learned biases
3. Knowledge editing - Update specific facts
4. Model repair - Fix targeted errors

Practical Considerations

• RL training can be computationally expensive
• Need clear retain/forget split
• Reward function critical to success
• May need task-specific environment tuning

Limitations

• Requires well-defined editing objective
• RL convergence depends on reward design
• Editing quality varies by task
• Limited evaluation on large models

Related Methods

• Influence functions for data attribution
• Fine-tuning for model editing
• Specialized unlearning algorithms