huggingface-trl

name	huggingface-trl
description	Using Hugging Face TRL library for building custom trainers for Preference Optimization methods like DPO, SimPO, etc.

Hugging Face TRL (Transformer Reinforcement Learning)

TRL is a library by Hugging Face that provides tools to train language models using Reinforcement Learning and Preference Optimization (like PPO, DPO, etc.).

Custom Trainers

To implement custom loss functions like SimPO, you can subclass the DPOTrainer and override its specific methods (like dpo_loss or in this case, simpo_loss).

from trl import DPOTrainer
import torch
import torch.nn.functional as F

class CustomTrainer(DPOTrainer):
    def custom_loss(self, policy_chosen_logps, policy_rejected_logps):
        pi_logratios = policy_chosen_logps - policy_rejected_logps
        logits = pi_logratios - self.margin
        losses = -F.logsigmoid(self.beta * logits)
        return losses

Setup & Dependencies

Commonly requires transformers, torch, trl, and accelerate. Make sure you have the matching versions as defined in the project's requirement file (e.g. environment.yml or requirements.txt).