GRPO-Zero

Why we included this project

GRPO-Zero implements DeepSeek's GRPO reinforcement-learning algorithm from scratch, built on just PyTorch and the tokenizers library. Skipping the transformers and vLLM dependencies that most RL training stacks assume keeps the code compact enough to follow the group sampling, advantage normalization, and per-token policy gradient updates in a few hundred lines. The repo also has practical memory tricks: CPU optimizer offloading lets a 24GB RTX 4090 handle training, and the default config targets a single A40. It folds in DAPO-style improvements like token-level loss weighting and optional KL removal, so researchers and engineers experimenting with RL fine-tuning of small models have a useful starting point for reproducing CountDown-style reasoning tasks or adapting the loop to their own reward functions.