trlx

Why we included this project

trlX takes care of the heavy lifting when you want to fine-tune a language model with reinforcement learning but would rather not build the RL machinery yourself. It trains models with algorithms like PPO and ILQL, and you can supply either a hand-written reward function or a reward-labeled dataset to drive the learning. Hugging Face models up to about 20B parameters run on Accelerate-backed trainers, and if you need to go bigger, NeMo-backed trainers take the same algorithms to larger scale using parallelism. So a preference-tuning experiment that starts on a research notebook can move to a multi-GPU run without switching frameworks. The repo also ships Colab notebooks for small GPT-2 tasks like sentiment control, which makes it a reasonable place to learn the approach.