Which Large Language Models Does Petals Support and How Fast Are They?

Petals supports Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+), and BLOOM (176B), and it can run them on consumer-grade GPUs or Google Colab by splitting the model across a peer-to-peer network. For single-batch inference, the project reports up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B) — speeds it describes as enough for chatbots and interactive apps.

Supported models

According to the Petals site, you can generate text with:

• Llama 3.1 — up to 405B parameters
• Mixtral — 8x22B
• Falcon — 40B and above
• BLOOM — 176B

You can also fine-tune these models for your own tasks. The same page lists Llama 2 (70B) and Falcon (180B) in its performance examples, so those sizes are part of the demonstrated range.

How fast is inference?

Petals reports single-batch inference speeds of:

These figures are for single-batch inference. The site frames them as sufficient for chatbots and interactive applications — not for high-throughput serving.

What you can do beyond standard APIs

Petals isn't limited to a fixed generate-text call. The site says you can:

• employ any fine-tuning and sampling methods
• execute custom paths through the model
• see its hidden states

That means you get API-like convenience while keeping the flexibility of PyTorch and 🤗 Transformers.

How the distributed setup works

You load a part of the model, then join a network of people serving its other parts. This BitTorrent-style approach is what lets a consumer-grade GPU or a Colab notebook participate in running models far larger than the local hardware could hold alone.

Practical takeaway

If your goal is interactive chat or a demo with a very large open model, Petals' reported 4–6 tokens/sec range is workable. If you need high-throughput batch inference, the single-batch numbers suggest this isn't the target use case. To try it, the site points to a Colab notebook and GitHub docs.