PatrickStar

Why we included this project

Training large language models usually hits a wall when the model stops fitting in GPU memory, and the standard fix is to buy more GPUs. PatrickStar takes a different route: it is a PyTorch training system that moves model data between GPU and CPU memory on the fly, using a chunk-based memory manager instead of the static split most heterogeneous trainers rely on. That dynamic scheduling is what lets it fit much larger models on fewer cards; the authors report training an 18B-parameter model on eight V100s, a 68B model on eight A100s with 1TB of host memory, and a 175B GPT-3 on 32 GPUs. Teams coming from DeepSpeed will recognize the configuration style, and the API wraps standard PyTorch training loops, so porting an existing project is fairly painless. It is research software with a paper behind it, so plan to read the docs and tune chunk sizes rather than expect a turnkey product.