gpt-neox

Why we included this project

EleutherAI built GPT-NeoX to pretrain large autoregressive language models from scratch, and it is the library behind its own GPT-NeoX and GPT-J releases. It fuses Megatron's tensor parallelism with DeepSpeed's ZeRO plus a few of EleutherAI's own optimizations, so a research group or small team can train in the tens of billions of parameters without assembling the pieces by hand. The library has run on AWS and CoreWeave as well as leadership-class machines such as Oak Ridge's Frontier and Summit, and it supports launching through Slurm, MPI, or the IBM Job Step Manager, which matters for anyone already living in an HPC or multi-node cluster. The maintainers are also honest about scope: this is a training tool, not an inference server, and for running GPT-NeoX-style models in production they point users to Hugging Face transformers instead.