| name | trl-setup |
| description | Setup and installation of TRL (Transformer Reinforcement Learning) library with compatible torch/transformers versions. Use when setting up preference optimization training environments. |
TRL Setup
Installation
pip install torch transformers trl accelerate peft datasets --break-system-packages
Compatibility Notes
- TRL imports from
trl.trainer.utils: DPODataCollatorWithPadding, pad_to_length, etc.
- Some imports may change across TRL versions (e.g.,
trl_sanitze_kwargs_for_tagging vs trl_sanitize_kwargs_for_tagging)
- If running in externally-managed Python (Debian), use
--break-system-packages flag
Common Import Errors
trl_sanitze_kwargs_for_tagging not found → check trl version, may be renamed or removed
- Missing
CPOTrainer → upgrade trl: pip install --upgrade trl
Running Tests from Project Root
cd /root/SimPO && python -m pytest unit_test/unit_test_1.py -v
cd /root/SimPO && python unit_test/unit_test_1.py
Key TRL Trainer Utilities
DPODataCollatorWithPadding: handles padding for preference pairs
pad_to_length: pads tensor to specified length
disable_dropout_in_model: disables dropout during eval