What is Petals (petals.dev)?

Updated

Petals is a system for running and fine-tuning very large language models across multiple machines instead of on a single GPU. You load one part of a model on your own consumer-grade GPU or a Google Colab instance, then join a network of other people who are serving the remaining parts. The result is BitTorrent-style distributed inference: no single participant needs enough memory to hold the whole model. It is aimed at people who want to generate text with or fine-tune models like Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+), or BLOOM (176B) without owning datacenter hardware.

How the BitTorrent-style approach works

A large model is split into blocks. Each participant loads and serves a subset of those blocks. When you send a request, it passes through the blocks hosted by you and by other peers in the network, and the output comes back as if you were running the model locally.

The practical consequence is that your hardware requirement scales with the part of the model you host, not the whole model. The trade-off is that inference depends on other peers being online and responsive.

What you can do with it

  • Generate text with Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+), and BLOOM (176B).
  • Fine-tune those models for your own tasks.
  • Go beyond a standard LLM API: use any fine-tuning and sampling methods, execute custom paths through the model, or inspect its hidden states.
  • Keep PyTorch and ๐Ÿค— Transformers flexibility while getting API-like conveniences.

Performance expectations

Single-batch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B). That is slow compared with a local small model, but the project positions it as enough for chatbots and interactive apps.

When Petals fits โ€” and when it doesn't

Situation Petals is a good fit Petals is a poor fit
You want to try a 70Bโ€“405B model Yes โ€” you only host part of it โ€”
You have one consumer GPU or Colab Yes โ€”
You need low, predictable latency โ€” Throughput depends on peers; 4โ€“6 tokens/sec is the stated range
You need guaranteed availability โ€” The network is made of volunteers
You want full control of the model internals Yes โ€” custom paths, hidden states, custom sampling โ€”

How to start

  1. Open the Colab notebook linked from the site to try it without local setup.
  2. For a persistent setup, follow the Docs on GitHub to install the client and load your share of a model.
  3. Once connected, you join the network serving the model's other parts and can run inference or fine-tuning.
  4. If you want to give back capacity, use Contribute my GPU.

The site also lists top contributors and network status, and offers a Discord and an email list (updates sent once every few months) for following development. Petals is part of the BigScience research workshop.