petals.dev

Petals' official entry points are its website (petals.dev), a "Try now in Colab" link for a hands-on demo, "Docs on GitHub" for documentation and source code, and a Discord server or email subscription for development updates. The site also displays a live list of top contributors and network status, so you can check who is currently serving model parts before you join.

Official entry points

What each channel is for

Trying it first: Colab

If you want to see Petals work before installing anything, the Colab link is the fastest path. The homepage describes generating text with Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+), or BLOOM (176B) using a consumer-grade GPU or Google Colab. The intended flow is: you load part of the model, then join a network of people serving its other parts.

Reading and building: GitHub

The "Docs on GitHub" link is where you go for the actual documentation and source. This matters if you want more than the standard API surface — the site states you can use any fine-tuning and sampling methods, execute custom paths through the model, or inspect its hidden states, with the flexibility of PyTorch and 🤗 Transformers.

Following development: Discord and email

Discord is the live community channel. The email list is explicitly low-volume: the site says updates go out once every few months and that it only emails for "really exciting updates." Choose Discord if you want ongoing discussion, email if you only want occasional announcements.

Checking network health before you commit

The homepage includes a "Top contributors right now" section and a network status display. On the page as captured, the contributor list was still loading and the network status failed to load ("Oops, can't load the network status..."), so treat these as live widgets that may not always render — if they appear empty, that reflects the widget state, not necessarily the network.

Speed expectations from the site, useful when deciding whether the network suits your use case: single-batch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B), which the project describes as enough for chatbots and interactive apps.

Practical notes

• Petals is part of the BigScience research workshop, per the homepage.
• The site does not state pricing or login requirements for these channels, so don't assume free access or no sign-up — check each destination directly.
• If you plan to contribute compute rather than just consume it, the homepage offers a "Contribute my GPU" action alongside the contributor list.

Petals lets you run large language models such as Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+), or BLOOM (176B) without owning enough GPU memory for the whole model. Instead of loading every weight locally, you load a part of the model and join a network of other people who are serving the remaining parts. A single inference request is then completed collaboratively across those machines, in a way that resembles BitTorrent's piece-by-piece sharing. This is a good fit if you want to generate text or fine-tune a very large model on a consumer-grade GPU or Google Colab, and can accept the latency of a distributed network.

The core mechanism

The model is split into blocks, and each participant is responsible for holding and serving some of those blocks. When you run a forward pass:

1. Your machine holds the blocks assigned to you (or the ones you choose to load).
2. For the blocks you don't hold, your request is routed to peers in the network who do hold them.
3. Activations pass from block to block across machines until the full forward pass is complete.

From your side this feels like calling an API, but under the hood you still have the flexibility of PyTorch and 🤗 Transformers. That means you are not limited to a fixed text-in/text-out interface: you can use custom fine-tuning and sampling methods, execute custom paths through the model, or inspect its hidden states.

What performance to expect

The published reference numbers are for single-batch inference:

Those rates are described as enough for chatbots and interactive apps. The important qualifier is single-batch: this design targets low-concurrency, interactive use, not high-throughput batch serving. If your workload is many parallel requests, the peer-to-peer routing becomes the bottleneck rather than a benefit.

When this approach makes sense

Choose Petals-style serving when:

• The model is too large to fit on your own GPU, but you still want to run it.
• You are doing interactive generation or experimentation where a few tokens per second is acceptable.
• You want fine-grained control (custom sampling, custom forward paths, hidden-state access) rather than a locked API.
• You are willing to depend on other participants being online for the parts you don't hold.

Avoid it when:

• You need high request throughput or strict, predictable latency.
• You need guaranteed availability independent of a peer network.
• Your task requires the entire model to be resident locally at all times.

Getting started

The project points to two entry points: a Colab notebook for trying it immediately, and documentation on GitHub. The practical path is to start in Colab, load a portion of a supported model, and let the network serve the rest. If you want to give back, you can also contribute your own GPU to host blocks for others.

One thing to note from the site itself: the live "top contributors" and network status panel can fail to load ("Oops, can't load the network status..."), so don't treat that widget as a reliable indicator of current network health.

Petals supports Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+), and BLOOM (176B), and it can run them on consumer-grade GPUs or Google Colab by splitting the model across a peer-to-peer network. For single-batch inference, the project reports up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B) — speeds it describes as enough for chatbots and interactive apps.

Supported models

According to the Petals site, you can generate text with:

• Llama 3.1 — up to 405B parameters
• Mixtral — 8x22B
• Falcon — 40B and above
• BLOOM — 176B

You can also fine-tune these models for your own tasks. The same page lists Llama 2 (70B) and Falcon (180B) in its performance examples, so those sizes are part of the demonstrated range.

How fast is inference?

Petals reports single-batch inference speeds of:

These figures are for single-batch inference. The site frames them as sufficient for chatbots and interactive applications — not for high-throughput serving.

What you can do beyond standard APIs

Petals isn't limited to a fixed generate-text call. The site says you can:

• employ any fine-tuning and sampling methods
• execute custom paths through the model
• see its hidden states

That means you get API-like convenience while keeping the flexibility of PyTorch and 🤗 Transformers.

How the distributed setup works

You load a part of the model, then join a network of people serving its other parts. This BitTorrent-style approach is what lets a consumer-grade GPU or a Colab notebook participate in running models far larger than the local hardware could hold alone.

Practical takeaway

If your goal is interactive chat or a demo with a very large open model, Petals' reported 4–6 tokens/sec range is workable. If you need high-throughput batch inference, the single-batch numbers suggest this isn't the target use case. To try it, the site points to a Colab notebook and GitHub docs.

You can start with Petals in two ways: open the Colab notebook for a zero-setup trial, or install the library locally and load a slice of a model onto your own consumer-grade GPU. Both paths connect you to a peer network that serves the rest of the model, so you never need to host the full weights yourself. Petals supports Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+), and BLOOM (176B), and it exposes an API-like interface backed by PyTorch and Hugging Face Transformers.

What Petals actually does

Petals splits a large language model across many machines. Each participant loads a contiguous block of layers, and the network routes your request through those blocks in order. From your side it looks like calling a normal model: you generate text, fine-tune, or inspect hidden states, while the heavy layers live on other people's GPUs.

The published throughput figures give a realistic sense of speed: single-batch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B). That is enough for chatbots and interactive apps, but not for high-volume batch generation.

Path 1: Try it in Colab (no local hardware)

1. Open the Colab notebook linked from the Petals site.
2. Run the setup cells, which install the library and connect you to the public swarm.
3. Load a supported model and call it like a Transformers model to generate text.

Expected result: you get generated output without downloading full model weights or owning a capable GPU. This is the fastest way to confirm the workflow fits your use case before committing local resources.

Path 2: Run it locally on a consumer GPU

1. Install the Petals library and its dependencies (see the GitHub docs for the current install command).
2. Load only part of the model onto your GPU, then join the network so your block serves other users while the network serves the rest to you.
3. Generate text, fine-tune, or run custom sampling methods through the same interface.

Expected result: your machine contributes a model block and you get full-model behavior in return. The trade-off is that latency depends on network health and which peers are online.

What you can do beyond a standard LLM API

• Apply any fine-tuning and sampling methods you want.
• Execute custom paths through the model rather than only the default forward pass.
• Inspect the model's hidden states.
• Keep the ergonomics of an API while retaining PyTorch and Transformers flexibility.

Choosing between the two paths

Common sticking points

• Speed is bounded by the network, so expect the token rates above rather than local-GPU speeds.
• Model availability depends on which blocks peers are currently serving; the site's live network status can fail to load, so retry if it appears empty.
• Fine-tuning and custom paths require more setup than plain generation, so start with inference to validate your environment first.

For current install commands and API details, use the GitHub docs and the Colab notebook linked from petals.dev.

Petals is a system for running and fine-tuning very large language models across multiple machines instead of on a single GPU. You load one part of a model on your own consumer-grade GPU or a Google Colab instance, then join a network of other people who are serving the remaining parts. The result is BitTorrent-style distributed inference: no single participant needs enough memory to hold the whole model. It is aimed at people who want to generate text with or fine-tune models like Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+), or BLOOM (176B) without owning datacenter hardware.

How the BitTorrent-style approach works

A large model is split into blocks. Each participant loads and serves a subset of those blocks. When you send a request, it passes through the blocks hosted by you and by other peers in the network, and the output comes back as if you were running the model locally.

The practical consequence is that your hardware requirement scales with the part of the model you host, not the whole model. The trade-off is that inference depends on other peers being online and responsive.

What you can do with it

• Generate text with Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+), and BLOOM (176B).
• Fine-tune those models for your own tasks.
• Go beyond a standard LLM API: use any fine-tuning and sampling methods, execute custom paths through the model, or inspect its hidden states.
• Keep PyTorch and 🤗 Transformers flexibility while getting API-like conveniences.

Performance expectations

Single-batch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B). That is slow compared with a local small model, but the project positions it as enough for chatbots and interactive apps.

When Petals fits — and when it doesn't

How to start

1. Open the Colab notebook linked from the site to try it without local setup.
2. For a persistent setup, follow the Docs on GitHub to install the client and load your share of a model.
3. Once connected, you join the network serving the model's other parts and can run inference or fine-tuning.
4. If you want to give back capacity, use Contribute my GPU.

The site also lists top contributors and network status, and offers a Discord and an email list (updates sent once every few months) for following development. Petals is part of the BigScience research workshop.