Mesh LLM pools your own GPUs into one OpenAI-compatible endpoint over iroh

Mesh LLM is a peer-to-peer system that pools GPUs across machines you already own—laptops, office desktops, closet servers—into one inference cluster exposed as an OpenAI-compatible API at localhost:9337/v1. It’s pitched as a direct answer to the cost and control tradeoffs of hosted models: rather than paying a large provider whose pricing, model versions, and data policies can shift under you, you run models on your own hardware. Any request can be served locally, routed to a peer that already has the model loaded, or split across multiple nodes. A pipeline mode (‘Skippy’) partitions a model by layer ranges—layers 0-15 on one node, 16-31 on the next—so a 235B mixture-of-experts model can run across several modest machines that couldn’t each hold it alone, with activations flowing stage to stage while the client only ever sees localhost. The catalog ships 40+ models, from laptop-scale half-billion-parameter models up to the giants.

The substrate is iroh. Each node boots an iroh endpoint identified by a public key, with no central server; iroh handles NAT traversal, hole-punching, and relay fallback to open direct, authenticated QUIC connections between any two peers, and Mesh LLM runs relays in multiple regions as a backstop. One QUIC connection multiplexes gossip, inference, route queries, and peer events, demultiplexed by a leading byte per stream. Because iroh collapses ‘route to a peer’ and ‘stream to the next pipeline stage’ into the same primitive as talking to localhost, the networking mostly recedes as a concern, while Mesh LLM’s own gossip layer governs who joins the mesh and which versions are compatible.

The significance is less about raw performance than about the deployment model it normalizes: roughly 18 MB to install, a public mesh or private deployments, and a plugin architecture exposing capabilities over MCP, HTTP, and mesh events. A mobile app built on iroh’s Swift SDK and support for the emerging ACP agent standard are planned. It’s an open, lock-in-averse alternative to centralized inference—compelling for teams that want data locality and cost control, though trust, reliability, and latency across heterogeneous consumer hardware remain the open questions.