hivemind

Why we included this project

Hivemind is a PyTorch library for training one large model across many internet-connected machines without a central orchestrator. It uses a distributed hash table to form a peer-to-peer network, so you can pool together ordinary volunteer computers instead of renting a GPU cluster. The library tolerates slow or unresponsive nodes during forward and backward passes, which matters when you're working with heterogeneous hardware that might drop out at any moment. It can also spread individual layers across participants via decentralized mixture-of-experts, so model size isn't limited by one machine's memory. Projects like Petals and a NeurIPS 2021 demo have built on it for large-scale collaborative training.