Last week I stumbled on a post on HN about Mesh LLM. 18 points — not huge, but the title caught my eye: "distributed AI computing on iroh." TL;DR: connect all your GPUs together, pool them into a "super graphics card," and run models that won't fit on a single card.
I figured it was yet another "great concept, terrible execution" project. But after tinkering with it for two days, turns out it actually has some legs. Writing this up so others who want to try it have a reference.
What the Hell Is This Thing Even Doing?
Anyone who's run large language models has hit this wall: the model is too big for a single GPU. An 80GB H100 is nice, but not everyone has one. So someone had a clever workaround: slice the model into pieces, distribute them across multiple machines, each one handles its portion, then chain the inference together.
That's exactly what Mesh LLM does. It does two things:
- Pools GPUs across machines and exposes them as a single OpenAI-compatible API. You connect to
localhost:9337/v1and internally it decides which machine handles the request. - Supports model splitting — a big model gets sliced by layers, each layer runs on a different node. Layers 0-15 on machine A, 16-31 on machine B, activations flowing through the pipeline.
The coolest part is that it uses iroh, a P2P networking library. What does that mean? You don't need a central server to coordinate. Each machine establishes a direct QUIC connection through iroh, with NAT traversal and relay fallback built in. Two machines in different companies, different countries — they can communicate directly to run a model.
Sounds cool, but how does it work in practice? Read on.
Architecture Breakdown: How It Works Under the Hood
Let's get into the internals first, otherwise the experience section won't make sense.
Network Layer: iroh Handles Connectivity
iroh is a P2P networking library built on QUIC. Its core capabilities:
- NAT traversal: Two machines behind different routers can connect directly without you manually opening ports
- Identity authentication: Each node has a public key as its identifier, connections are encrypted
- Relay fallback: When direct connection fails, traffic routes through community relay servers
Mesh LLM runs two iroh relays in different regions so nodes always have a path to each other.
Transport Layer: QUIC Multiplexing
Three ALPN protocols run on top of the QUIC connections iroh establishes:
mesh-llm/1— Main protocol: gossip, routing, HTTP tunnels, plugin channelsmesh-llm-control/1— Admin control plane: config sync, ownership attestationskippy-stage/2— Latency-sensitive activation transport, dedicated to model splitting
Inside the main connection, all data flows through bidirectional QUIC streams, distinguished by the first byte:
| Byte | Type | Description | |------|------|-------------| | 0x01 | GOSSIP | Node broadcasts (models, GPU, RTT, capabilities) | | 0x04 | TUNNEL_HTTP | Inference requests forwarded to peers | | 0x05 | ROUTE_REQUEST | "What models do you host?" | | 0x06 | PEER_DOWN | Node went down | | 0x07 | PEER_LEAVING | Graceful shutdown | | 0x08 | PLUGIN_CHANNEL | Plugin RPC | | 0x0e | DIRECT_PATH_REQUEST | Share direct addresses for NAT traversal |
Inference Layer: Three Request Routing Strategies
When a request comes in, Mesh LLM has three ways to handle it:
- Local execution — this machine has a GPU and the model, just run it
- Route to peer — the model is already loaded on another node, forward it
- Split mode (Skippy) — model too big for one machine, slice by layers across multiple machines in a pipeline
The third option is the interesting one. Say you have a 235B parameter MoE model that won't fit on any single machine. Mesh LLM slices it into stages: layers 0-15 on node A, 16-31 on node B, activations flowing from A to B. The OpenAI client has no idea any of this is happening — it just sees localhost:9337/v1.
Why It's Called "Skippy"
The internal name for the split mode is "Skippy," borrowed from the cartoon character whose superpower is jumping. Because that's exactly what the activations do — they jump between nodes.
Here's how it works: the model gets divided into stages by layer range. Each stage handles a subset of layers. Input enters stage 0, stage 0 processes its layer, passes the intermediate activations over the network to stage 1, which continues processing, and so on.
This differs from traditional tensor parallelism. In conventional setups, all GPUs live in the same machine connected via NVLink with 400GB/s bandwidth. Mesh LLM nodes can be anywhere — gigabit LAN, home broadband, cellular. Bandwidth ranges from 1GB/s on LAN to 10Mbps over the public internet. That's why cross-node inference speed tanks — it's not the model computation that's slow, it's data moving over the network.
Understanding this is key: Mesh LLM's split mode is for scenarios where the model is too big to fit, not scenarios where you want maximum speed. If you have two 4090s and want to run a 70B model, vLLM tensor parallelism on a LAN will be faster. But if you have one 4090 and want to run a 70B model, Mesh LLM is one of the few options that actually lets you do it.
Installation: Surprisingly Easy
The Mesh LLM binary is small — about 18MB. Installation is like any CLI tool:
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
Once started, your machine becomes an OpenAI API-compatible server at http://localhost:9337/v1.
Joining the Public Mesh
If you want to join the public mesh and use other people's GPUs (or lend yours):
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 | |
| 7 | |
| 8 | |
Private Deployment
For enterprise use, you can run your own relay and control who joins:
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
The control plane uses the mesh-llm-control/1 protocol, supporting config sync and ownership attestation. You can precisely control which nodes and software versions join your mesh.
Real Experience: Running a 70B Model
I tested with two machines:
- Laptop: MacBook Pro M3 Max, 128GB RAM, no discrete GPU
- Desktop: Ubuntu 22.04, RTX 4090 24GB, 128GB RAM
Running Llama-3.1-70B on the RTX 4090 alone is impossible — FP16 70B params need roughly 140GB of VRAM. With Mesh LLM, though, things change.
Test 1: Single Machine, 8B Model
Started small:
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 | |
| 7 | |
| 8 | |
| 9 | |
Result: worked fine. 8B on M3 Max gives about 30-40 tokens/sec. Not bad for a laptop.
Test 2: Cross-Machine 70B
The real test — connecting desktop (4090) and laptop (M3 Max) into a mesh to run 70B:
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
Same Python code, connect to localhost:9337/v1.
Result: the model ran. But honestly, the speed was slower than I expected. Single-node 70B inference gave about 5-8 tokens/sec. Cross-node split mode dropped to 2-3 tokens/sec.
Why so slow? A few reasons:
- Network latency: activations need to travel between nodes, even on LAN that's 0.5-1ms
- Serialization overhead: tensors passing between machines need serialize/deserialize
- CPU mode inefficiency: M3 Max running models via Metal, which is fine but not a dedicated GPU
Test 3: Public Mesh Mode
I also tried joining the public mesh directly to use strangers' GPUs:
| 1 | |
| 2 | |
The public mesh has 40+ preset models, from half-billion parameter models to 235B MoE behemoths. Once connected, localhost:9337/v1 gives you access to all of them.
The problem: public mesh stability is shaky. Nodes come and go. Sometimes a model suddenly returns 503. This is P2P after all — nodes can disconnect at any time.
The Good and the Bad
What Works Well
1. Cost savings are real
If you have a GPU and don't want monthly API bills, Mesh LLM is a solid alternative. The public mesh has free models you can use, and you can share your idle GPU with others.
2. OpenAI-compatible API lowers the barrier
This matters a lot. No new SDK, no new protocol. Existing OpenAI Python clients, LangChain, LlamaIndex — they all connect directly. For developers, migration cost is basically zero.
3. P2P architecture is imaginative
Traditional distributed inference needs central schedulers, load balancers, health checks. Mesh LLM delegates all that to iroh. Automatic node discovery, routing, fallback. For small teams or individual devs, this is way simpler than spinning up a K8s cluster with vLLM.
4. Model splitting is a genuine feature
Running a big model across multiple machines is something that traditionally required specialized distributed inference frameworks (DeepSpeed, Megatron). Mesh LLM wraps this in a turnkey feature, transparent to the client.
The Pain Points
1. Public mesh instability
This was my biggest headache. Nodes come and go. No SLA on model availability. Fine for experimentation. For production, you need a private mesh or a fallback strategy.
2. Cross-node latency is a hard limitation
In split mode, activations travel between nodes. Even on gigabit LAN, the latency is noticeable. Over the public internet, it's worse — sometimes inference takes ten+ seconds per request.
3. Documentation is thin
The official docs are mostly blog posts. Detailed API references and troubleshooting guides are sparse. When things go wrong, you're reading source code or asking in Discord.
4. Mobile app isn't ready
They're building an iOS/Android app on iroh's Swift SDK with ACP support. But it's not out yet.
5. Plugin system is nascent
Mesh LLM supports plugins that declare node capabilities. But the plugin ecosystem is basically empty — only the basic model inference plugin exists.
Comparison with Alternatives
vs vLLM + Ray
vLLM is the king of single-machine inference. Ray handles multi-node distributed inference. But Ray is heavy — you need to configure clusters, manage nodes, handle failures. Mesh LLM's advantage is zero-config: install and run, no infrastructure needed.
The downside: Mesh LLM's performance doesn't match vLLM + Ray. Ray clusters live in the same datacenter with low network latency. Mesh LLM runs on P2P networks where latency is unpredictable.
vs Ollama
Ollama is a single-machine model management tool — simple, effective, but no distributed inference. Mesh LLM is basically Ollama's distributed cousin. If you have multiple machines you want to use together, this is the path.
vs Replicate / Modal
Replicate and Modal are managed inference platforms. Upload a model, they run it. Convenient, expensive. Mesh LLM's advantage is cost control — use your own GPU or free compute from the public mesh.
Quick Comparison
| Approach | Difficulty | Cost | Performance | Best For | |----------|-----------|------|-------------|----------| | Mesh LLM | Low | Low | Medium | Personal/small team, low-cost inference | | vLLM + Ray | High | Medium | High | Production, high-performance needs | | Ollama | Low | Low | Medium | Single-machine inference | | Replicate | Low | High | High | No infra management | | Modal | Medium | High | High | Elastic inference needs |
What the HN Community Is Saying
After Mesh LLM hit HN, the comments weren't massive, but a few viewpoints stood out.
First debate: is P2P inference actually better than centralized?
Some argued that P2P network latency is unpredictable, and for inference (which is latency-sensitive), centralized deployment (like a vLLM cluster) is clearly more reliable. I partially agree — if you're chasing peak performance, P2P isn't the answer. But the value of P2P isn't performance, it's accessibility. Not everyone has an 8x H100 cluster, but a lot of people have a few computers with GPUs lying around. Mesh LLM's point is to aggregate this "leftover" compute into something usable.
Second discussion: security of the public mesh
People worried about untrusted nodes in the public mesh — what if a node returns tampered inference results? This is a real concern. Mesh LLM currently uses iroh's identity authentication to verify node identity, but there's no integrity verification for inference results. The official roadmap mentions it, but no timeline.
Honestly, for chatbot use cases, the chance of tampered output is low — model outputs are probabilistic sampling, hard to subtly alter without breaking overall quality. But for code generation or data analysis where accuracy matters, you need to be careful.
Third topic: relationship to DePIN
Some classified Mesh LLM under the DePIN (Decentralized Physical Infrastructure Networks)赛道. Fair point — Mesh LLM is doing "decentralized AI infrastructure." But it differs from Render Network and Akash Network:
- Render Network is a rendering farm, billed per frame
- Akash Network is decentralized cloud, billed per VM
- Mesh LLM is decentralized inference, billed per API call
Three different layers. Mesh LLM is closer to Fugai and Gensyn — the "decentralized inference network" camp.
Cost Analysis: How Much Can You Actually Save?
Let's do the math. Assume you're running Llama-3.1-70B-Instruct, 1000 requests/day, ~500 tokens per request:
Option 1: OpenAI API (via Anthropic or other compatible endpoint)
At ~$3.0/million tokens (typical for 70B-class models):
- 1000 requests × 500 tokens = 500,000 tokens/day
- 500,000 × $3.0 / 1,000,000 = $1.50/day
- ~$45/month
Option 2: Own RTX 4090 + Mesh LLM Public Mesh
- RTX 4090 electricity (24/7): ~$15/month
- Public Mesh free usage: $0
- Total: $15/month
Option 3: Rent an A100 Cloud Instance
- AWS p5.48xlarge: $32.44/hour
- Monthly (730 hours): $23,681/month
- But you probably don't use it 24/7. Say 4 hours/day: ~$130/month
Option 4: Private Mesh (3 machines)
- Electricity for 3 machines: ~$45/month
- Relay VPS: ~$5/month
- Total: ~$50/month
| Option | Monthly Cost | Performance | Stability | |--------|-------------|-------------|-----------| | OpenAI API | $45 | High | High | | Public Mesh | $15 (electricity) | Medium | Medium | | A100 Cloud | $130 | Very High | High | | Private Mesh | $50 | Medium-High | High |
Bottom line: if your daily request volume is low (<1000), the public mesh is the cheapest option. If your business needs stability, private mesh or cloud is more reliable.
Common Misconceptions
Misconception 1: Mesh LLM Can Replace APIs
It can't. APIs offer SLA guarantees, high availability, auto-scaling. Mesh LLM's public mesh has none of these. For production, APIs are still the way to go. Mesh LLM is better suited as a backup or experimental environment.
Misconception 2: Model Splitting = Double the Speed
Split mode lets you run bigger models by spreading them across machines, but it does not double performance. In fact, due to inter-node communication overhead, split mode is typically slower than single-node inference. The value is "can run" not "runs fast."
Misconception 3: The Public Mesh Is Anonymous
It's not. Each node is identified by an iroh public key. While a public key isn't a real identity, the same key can be tracked. If your organization runs Mesh LLM nodes, consider data privacy — routing information for inference requests is visible.
Misconception 4: You Need a GPU to Use It
Not strictly. Mesh LLM supports CPU inference (via ONNX Runtime or llama.cpp). It's slow, but for small models (<7B), CPU is workable. I ran Llama-3.1-8B on the M3 Max in CPU mode at ~5-8 tokens/sec. Slow, but usable.
Recommendations by Scenario
Individual developer, wants to play with large models
Go with: Public Mesh + your own small GPU
Install Mesh LLM, join the public mesh, use the OpenAI-compatible API. Low cost, easy to start. When the public mesh is unstable, fall back to your own small model.
Small team (3-5 people), has budget but doesn't want to spend too much
Go with: Private Mesh + 2-3 GPU-equipped machines
Build a small private mesh. Cost is controllable, stability is much better than public mesh. LAN latency is ~0.1ms, inference speed barely affected.
Enterprise, needs high stability
Go with: vLLM + Ray cluster or managed API
Mesh LLM isn't ready for enterprise production yet. Node management, network stability, security auditing — all open problems. Stick with proven solutions.
Education / research
Go with: Mesh LLM Public Mesh
Teaching demos and research experiments are perfect for the public mesh. Students don't need their own GPU — just connect and start calling models. Plus, the P2P architecture itself is a great teaching case — students learn AI inference and network protocols simultaneously.
What's Next
After writing this, I plan to:
- Test private mesh deployment — set up a private mesh on two VPS instances, measure cross-region latency and stability
- Compare split effects across models — 7B, 13B, 70B with different node counts
- Watch the ACP protocol — once it ships, see if other AI Agents can connect to Mesh LLM nodes directly
Will write more when I have results.
- Written July 12, 2026, based on the latest public information and hands-on testing. Project: https://meshllm.cloud/*