petals

Why we included this project

Most GPUs in ordinary homes can't hold a 176B model, so Petals takes a different route: it spreads the layers across a volunteer swarm and lets you reach that shared capacity from a desktop or a Colab notebook. You keep only part of the model on your machine and the library shuttles the rest over the network, exposing everything through a Hugging Face Transformers-style API that makes a distant 405B Llama 3.1 behave as if it were local. That makes it practical for teams who want to fine-tune or experiment with frontier-scale open weights like Mixtral without renting serious hardware, and Petals reports inference and fine-tuning up to 10x faster than offloading. It's not just a canned generation endpoint either; you can run custom fine-tuning, inspect hidden states, or route requests through the model, which suits interactive apps and research rather than one-off outputs. The trade-off is privacy: traffic in the public swarm is processed with other people's help, so sensitive work means standing up your own private swarm among people you trust.