trl-setup

name	trl-setup
description	Setup and installation of TRL (Transformer Reinforcement Learning) library with compatible torch/transformers versions. Use when setting up preference optimization training environments.

TRL Setup

Installation

pip install torch transformers trl accelerate peft datasets --break-system-packages

Compatibility Notes

• TRL imports from trl.trainer.utils: DPODataCollatorWithPadding, pad_to_length, etc.
• Some imports may change across TRL versions (e.g., trl_sanitze_kwargs_for_tagging vs trl_sanitize_kwargs_for_tagging)
• If running in externally-managed Python (Debian), use --break-system-packages flag

Common Import Errors

• trl_sanitze_kwargs_for_tagging not found → check trl version, may be renamed or removed
• Missing CPOTrainer → upgrade trl: pip install --upgrade trl

Running Tests from Project Root

cd /root/SimPO && python -m pytest unit_test/unit_test_1.py -v
# or
cd /root/SimPO && python unit_test/unit_test_1.py

Key TRL Trainer Utilities

• DPODataCollatorWithPadding: handles padding for preference pairs
• pad_to_length: pads tensor to specified length
• disable_dropout_in_model: disables dropout during eval