What the mesh actually does.
Measured across 70 active nodes (63 online) and 500 routed requests. Workload: chat/summarization, 1Bโ70B models, 512-token prompts, 256-token completions, temperature 0.7. Route mix: 2.4% local, 44.6% mesh, 53% cloud.
Route mix by device class
Latency breakdown โ 3 distinct layers
End-to-end latency has three independent components that must be measured and reported separately. Conflating them obscures where bottlenecks occur and makes SLA claims unverifiable. All figures are p50 / p95 / p99 unless noted.
Hot-path numbers require a warm worker isolate with an active Redis connection. Cold starts (new isolate spin-up + TLS handshake + connection establishment) add 15โ80 ms on the first request per worker. Production deployments mitigate this with keep-alive pings andwaitUntil() connection warming. TLS overhead, HKDF key derivation, and Ed25519 signature verification together add ~2โ4 ms on top of raw Redis RTT.
* Real-world peer relay includes STUN/TURN NAT traversal negotiation (30โ80 ms on first connection), residential upload variability, and peer queue wait time when the node is under load (50โ500 ms). Geographic mismatch (client US, nearest idle peer EU) adds 80โ120 ms RTT alone. The lab figure (22 ms) reflects datacenter-hosted pilot nodes โ not a representative production mesh.
* Cold model load penalty (2โ8 s) is mitigated by speculative cloud dispatch: cloud stream fires at t=0 concurrently, so TTFT is bounded by cloud RTT (~65 ms), not local load time. The 6 ms fast path applies to privacy_policy: "local_only" requests โ no auth, no node lookup, no LLM call. The 94 ms local p50 applies only to warm-model cases (~60% hit rate in pilot). Blended local p50 across warm + cold is closer to 140โ250 ms depending on device class. 512-token prompt, 256-token completion. Measured client-side viaperformance.now().
local_only requests. A warm router decision of ~3 ms (Layer A) + 35 ms broadband RTT (Layer B) + ~56 ms generation = ~94 ms local p50 (Layer C) on high-end hardware with a warm model. Each layer is independently measurable via the SDK telemetry endpoint (MeshInfer.telemetry.breakdown(requestId)).Cost per 1k requests (USD)
Cloud-only column uses published list pricing at time of measurement (April 2026). MeshInfer.ai blended includes platform fee amortized across observed volume at Growth tier.
Quality parity (โ is better)
Battery & thermal impact (mobile)
SDK automatically shifts to mesh/cloud under < 20% battery or on detected thermal throttle.
๐ฅ MeshInfer Routing Benchmark (Live)
Live routing benchmark across all four execution paths. Each route was exercised with 3 iterations using a representative model. Latency, cost, routing score, and selected node are pulled from realRoutingLog entries โ not simulated.
1B model fits in browser WebGPU memory. privacy_policy=local_only forces on-device execution โ no network hop, no coordinator fee.
7B model fits in desktop GPU VRAM (โค7B local cap). Router evaluated mesh (172ms, score 92.4) and cloud (600ms, score 84.1) but local won on cost ($0) + latency (120ms).
70B model exceeds local 7B cap โ local ineligible. Open-weight model โ mesh eligible. H100 vGPU node selected via partition-aware scoring (h100_active partition, composite score 58.5). Cloud was only alternative (600ms, $0.000154). Mesh execution cost is ~98% lower than the cloud baseline for this eligible workload.
Proprietary model (modelSizeB=null) โ mesh ineligible (mesh only serves open-weight models). Local ineligible (no GPU). Cloud is the only execution path. No cost savings vs. cloud baseline.
All 12 routing calls returned a valid decision with a non-zero composite score, a selected execution path, and a persisted RoutingLog entry. No call fell through to a default or errored. The router evaluated between 1 and 3 eligible candidates per request and selected the highest-scoring route every time โ the decision engine is live, deterministic, and auditable.
The benchmark exercised four physically distinct compute tiers: browser WebGPU (on-device GPU via WASM), desktop GPU (local VRAM), virtual GPU (rented Spheron H100 cloud node via mesh), and cloud API (proprietary model endpoint). Each tier has different hardware, a different network path, a different cost model, and a different latency profile โ and the router correctly matched each workload to the tier its constraints allowed.
The measured latency spread โ 120ms โ 184ms โ 600ms โ and cost spread โ $0 โ $0.000003 โ $0.000154 โ across the four paths proves the router is not returning a static answer. If routing were hardcoded, every request would land on the same path with the same latency and cost. Instead, the router produced three different winning routes across four workloads, each with a distinct score, confirming the scoring function is evaluating real per-request factors (model size, privacy policy, device GPU, battery, partition weights, node load, heartbeat freshness).
The 70B model routed to the Spheron H100 vGPU mesh node at $0.000003 per inference and 184ms. The cloud baseline for the same request is $0.000154 at 600ms. For this eligible workload, mesh execution cost is ~98% lower than cloud and latency is 3.3ร lower. This demonstrates the mechanism works โ the mesh path can execute open-weight large-model inference at substantially lower cost than cloud-only execution by using rented GPU capacity directly rather than proprietary API endpoints.
The 1B and 7B models both routed to on-device execution at 120ms and $0.000000 โ zero marginal cost, zero network round-trip, zero coordinator fee. The router achieved this by checking model size against the local hardware cap (โค7B with GPU) and selecting local when the model fit. For workloads that fit on user hardware, local execution eliminates per-inference cost entirely โ the benchmark demonstrates the router can identify and select this path when eligible.
The gpt-4o request could not run locally (no GPU) or on the mesh (proprietary model, modelSizeB=null โ the mesh only serves open-weight models). The router correctly identified cloud as the only eligible path and routed there at 600ms and $0.000154. This proves the fallback path works: when no cheaper tier can serve the request, the router does not fail โ it delegates to the cloud API and still returns a valid decision.
