BMTrain

Why we included this project

BMTrain is for teams that want to pre-train or fine-tune models with tens of billions of parameters but would rather not rebuild their code as a distributed harness. It brings ZeRO-2/3-style sharding and tensor parallelism to PyTorch, so you keep writing ordinary nn.Module code and the toolkit handles the communication and the overlap of compute with data transfer underneath. Getting started means swapping a few base classes and wrapping transformer blocks, then launching with the torchrun command you already know. That small migration path is what makes it attractive to research groups and small teams who want big-model training without adopting a heavyweight framework. A GPT-2 example ships with the project, and it plugs into OpenPrompt and OpenDelta, so fine-tuning workflows are covered alongside from-scratch pre-training.