Skip to content
Running giant LLMs on a peer-to-peer GPU swarm with Petals
Article

Running giant LLMs on a peer-to-peer GPU swarm with Petals

Jordan BlakeHomelab & infra

The VRAM paywall for frontier LLMs

If you want to run the latest open-weights heavyweight, like Llama 3.1 405B, on your own hardware, you will run into an expensive math problem.

At standard 16-bit precision, a 405-billion parameter model requires over 800 GB of VRAM just to load the weights. Even if you compress it aggressively down to 4-bit precision, you are still looking at more than 200 GB of VRAM. On consumer hardware, that means chaining together nine or ten RTX 3090 or 4090 cards. Unless you have a custom server rack in your garage and a dedicated cooling loop, you are locked out of running these frontier-class models locally.

This VRAM wall has historically forced developers to make a choice. You either rent enterprise cloud instances with multiple A100 or H100 GPUs at high hourly rates, or you use a hosted API. The API route is convenient, but it means sending your data to a third-party server, accepting their privacy policies, and getting locked into their pricing tiers. For independent developers, researchers, and hobbyists, the cost of entry is simply too high.

What is Petals?

Petals is an open-source, decentralized project designed to break down this VRAM barrier. Developed by researchers from the BigScience workshop, Yandex Research, and Hugging Face, it operates like a BitTorrent network for running large language models.

Instead of trying to fit a massive model like Llama 3.1 405B or Mixtral 8x22B onto one machine, Petals allows a network of volunteer nodes to host different parts of the model. If you run the Petals client on your local computer, you can load a small fraction of the model’s layers onto your own GPU, then connect to other volunteers over the internet who are holding the remaining pieces.

Together, this public swarm forms a virtual machine capable of running models that would otherwise require a data center. Anyone can join the network to contribute their spare GPU cycles, or connect to the public swarm as a client to run inference and fine-tune models using standard PyTorch code.

Under the hood: Pipeline parallelism and activation streaming

The engineering behind Petals relies on three core techniques to run massive models across consumer hardware over the internet.

First, the system uses pipeline parallelism to split the model’s layers. A transformer model like Llama 3.1 405B is a stack of repetitive blocks (layers). Petals divides these blocks among the available servers in the swarm. For example, if the model has 126 layers, Host A might load layers 1 to 20, Host B loads 21 to 40, and Host C takes the rest. No single node needs the whole model in memory; they only need enough VRAM to hold their assigned block.

Second, Petals streams activations sequentially. When you send a prompt to the network, your client processes the initial input and generates a hidden state (a tensor representing the data). The client sends this tensor to the host holding the first block of layers. That node runs its portion of the model, calculates the updated tensor, and streams it to the next node in line over a standard TCP/IP connection. This assembly line process continues until the final node generates the next token and sends it back to your client.

Third, the swarm uses dynamic routing for fault tolerance. Because home internet connections and consumer GPUs are notoriously unreliable, nodes can disconnect or lag at any moment. Petals handles this by keeping multiple redundant nodes for each block of layers. If a node drops out mid-generation, your client immediately detects the failure and reroutes the intermediate activations to another available peer hosting those same layers. This keeps the generation process moving without failing the entire request.

The benefits: Collaborative fine-tuning and adapter hosting

For developers and researchers, the primary benefit of Petals is not just access to a free public API; it is the freedom to modify the model’s behavior.

Unlike closed model APIs that restrict you to a simple text interface, Petals gives you raw access to the model’s internal states. Because the client runs on PyTorch and interfaces directly with the remote layers, you can read attention maps, extract intermediate activations, and apply custom sampling techniques. This level of access is necessary for research teams who want to audit model behaviors or build custom agents that rely on low-level logit access.

This openness also enables collaborative fine-tuning. Instead of needing an enterprise cluster to train a custom model, a small group of researchers can pool their consumer GPUs using Petals to train a LoRA (Low-Rank Adaptation) adapter. Once trained, these adapters are lightweight. Petals allows users to host their custom LoRA adapters on the public swarm. Clients can then load different adapters on the fly, running customized versions of the model without requiring separate physical swarms for every single task.

The performance tax: Latency and trust boundaries

Despite the clever distributed engineering, Petals is not a drop-in replacement for a dedicated local server or a commercial API. The trade-offs start with network latency.

When you run a standard model on a single machine, activation tensors pass between the GPU core and VRAM over high-bandwidth internal buses (PCIe or NVLink) at hundreds of gigabytes per second. In Petals, those same high-dimensional tensors must travel over home internet connections via TCP/IP. Even with the project’s built-in 8-bit quantization, passing these activations back and forth across dozens of residential connections introduces a massive latency penalty. Instead of the lightning-fast token generation you expect from a local GPU, the public swarm typically crawls at one to five tokens per second. It is a slow, sequential process that makes it unsuitable for interactive applications or high-throughput production work.

Then there is the issue of trust. When you send a prompt to the public Petals swarm, your text and intermediate states stream through random volunteer GPUs connected to the internet. Anyone running a node on that chain can inspect the incoming activations and reconstruct the original text. If you are handling proprietary code, customer data, or sensitive personal information, running it on the public swarm is a major security risk. While you can deploy a private, closed swarm inside your own VPC, doing so requires you to supply all the hardware yourself, which defeats the cost-saving purpose of the public network.

Finally, verification is a constant battle. In a public peer-to-peer network, you cannot guarantee that every node is running honest hardware. A malicious peer could return corrupted activations to save compute power or poison the model’s output. While Petals uses verification checks to catch bad actors, these security protocols add even more computational overhead to an already slow pipeline.

The local alternative: Quantization and unified memory

When Petals launched in 2022, it felt like the only hope for developers who wanted to run large models without paying enterprise prices. But local hardware options and software tooling have shifted rapidly since then.

The biggest change is the rise of aggressive, high-quality local quantization formats like GGUF and EXL2. These formats allow developers to run highly capable 8B, 30B, and 70B models on modest local setups. A quantized 70B model can fit into the VRAM of two consumer RTX 3090 GPUs and run at usable speeds. At the same time, hardware with unified memory (like Apple Silicon Mac Studios configured with 128 GB or 192 GB of RAM) can host 70B and 120B models completely locally.

Because local performance has improved so much, the practical need for Petals is now restricted to a narrow niche. You only need it if you absolutely must run an ultra-large model like Llama 3.1 405B, and you refuse to use a hosted API. For daily development, the complexity of setting up and running a distributed peer-to-peer pipeline is rarely worth the performance hit and the privacy risks.

Petals is a brilliant proof of concept for decentralized AI, and it remains a valuable tool for open-source research groups. But for most developers, running a quantized model locally or hosting it on a single rented node is the path that actually gets work done.

Sources

If you have a stack of dusty GPUs in a closet, it is worth joining the swarm just to see the pipeline parallelism in action. But if you need to query a model for a real-world task, stick to a local quantized setup or keep paying the API tax.

Related