How to Start Using Petals to Run or Fine-Tune LLMs

Updated

You can start with Petals in two ways: open the Colab notebook for a zero-setup trial, or install the library locally and load a slice of a model onto your own consumer-grade GPU. Both paths connect you to a peer network that serves the rest of the model, so you never need to host the full weights yourself. Petals supports Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+), and BLOOM (176B), and it exposes an API-like interface backed by PyTorch and Hugging Face Transformers.

What Petals actually does

Petals splits a large language model across many machines. Each participant loads a contiguous block of layers, and the network routes your request through those blocks in order. From your side it looks like calling a normal model: you generate text, fine-tune, or inspect hidden states, while the heavy layers live on other people's GPUs.

The published throughput figures give a realistic sense of speed: single-batch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B). That is enough for chatbots and interactive apps, but not for high-volume batch generation.

Path 1: Try it in Colab (no local hardware)

  1. Open the Colab notebook linked from the Petals site.
  2. Run the setup cells, which install the library and connect you to the public swarm.
  3. Load a supported model and call it like a Transformers model to generate text.

Expected result: you get generated output without downloading full model weights or owning a capable GPU. This is the fastest way to confirm the workflow fits your use case before committing local resources.

Path 2: Run it locally on a consumer GPU

  1. Install the Petals library and its dependencies (see the GitHub docs for the current install command).
  2. Load only part of the model onto your GPU, then join the network so your block serves other users while the network serves the rest to you.
  3. Generate text, fine-tune, or run custom sampling methods through the same interface.

Expected result: your machine contributes a model block and you get full-model behavior in return. The trade-off is that latency depends on network health and which peers are online.

What you can do beyond a standard LLM API

  • Apply any fine-tuning and sampling methods you want.
  • Execute custom paths through the model rather than only the default forward pass.
  • Inspect the model's hidden states.
  • Keep the ergonomics of an API while retaining PyTorch and Transformers flexibility.

Choosing between the two paths

Dimension Colab trial Local consumer GPU
Setup effort Minimal, browser only Install library, manage GPU
Hardware needed None locally Consumer-grade GPU
Control over model internals Limited to notebook Full PyTorch/Transformers access
Best for First test, demos Fine-tuning, custom methods, hidden-state work

Common sticking points

  • Speed is bounded by the network, so expect the token rates above rather than local-GPU speeds.
  • Model availability depends on which blocks peers are currently serving; the site's live network status can fail to load, so retry if it appears empty.
  • Fine-tuning and custom paths require more setup than plain generation, so start with inference to validate your environment first.

For current install commands and API details, use the GitHub docs and the Colab notebook linked from petals.dev.