safe-rlhf

Why we included this project

Beaver is an RLHF training framework from the PKU-Alignment group at Peking University, built for researchers who want to treat safety as an explicit constraint rather than a post-training fix. It covers the usual alignment stages, from supervised fine-tuning through reward and cost model training to standard RLHF and the constrained Safe RLHF variant, and it runs on LLaMA, OPT, and Baichuan. The project also ships a large human-labeled preference dataset, up to about a million pairs scored for both helpfulness and harmfulness, along with pre-trained Beaver-7B checkpoints, so you can reproduce or extend published results without building the pipeline from scratch. Safety evaluation hooks for BIG-bench and GPT-4 help you verify that constrained training actually behaves as intended. If your team already uses DeepSpeed, the backend will feel familiar, though this is aimed at alignment research and custom preference datasets rather than one-click production deployment.