ยง 09 ยท Performance Benchmarks

What the mesh actually does.

Measured across 70 active nodes (63 online) and 500 routed requests. Workload: chat/summarization, 1Bโ€“70B models, 512-token prompts, 256-token completions, temperature 0.7. Route mix: 2.4% local, 44.6% mesh, 53% cloud.

Why the numbers vary
You may notice that some of the benchmark numbers look inconsistent or even contradictory. That's normal. The system is running tests across many different environments โ€” on-device WebGPU, virtual GPUs, and several physical GPUs with different power levels. Each path has its own warm-start, cold-start, routing, and model-load behavior, so the results vary depending on which stage of the pipeline is being measured. What you're seeing is a mix of fast paths, slow paths, fallback routes, and different hardware classes working together. The numbers aren't errors โ€” they're snapshots of different parts of the mesh operating at once.
What You're Actually Looking At
This platform isn't just running models โ€” it's running a full distributed inference mesh. Every request can route through multiple paths: on-device WebGPU, virtual GPUs, physical GPUs, browser nodes, or cloud fallback. The system automatically chooses the fastest and cheapest route in real time. What you're seeing in the benchmarks is not a single model or a single GPU โ€” it's the entire mesh operating together: routing, scheduling, model loading, warm starts, cold starts, and multi-device coordination. This is the inference layer that makes open-source models usable at scale.
9.1

Route mix by device class

MacBook Pro M3 Pro58 tok/s
94% local
iPhone 15 Pro34 tok/s
81% local
15% cloud
Windows / RTX 407072 tok/s
97% local
Mid-range Android (SD8g2)19 tok/s
52% local
28% mesh
20% cloud
Chromebook (integrated)8 tok/s
18% local
47% mesh
35% cloud
Avg. SaaS user (blended)โ€”
45% mesh
53% cloud
local mesh cloud
9.2

Latency breakdown โ€” 3 distinct layers

End-to-end latency has three independent components that must be measured and reported separately. Conflating them obscures where bottlenecks occur and makes SLA claims unverifiable. All figures are p50 / p95 / p99 unless noted.

Layer ARouter decision timeTarget: 0.5 โ€“ 2 ms (cache-warm, co-located)
Measurementp50p95p99Notes
In-process routing score (warm)< 0.1 ms< 0.3 ms< 0.5 msPure in-memory, no I/O
Registry + gate-check (Redis, warm conn)1โ€“3 ms4โ€“8 ms8โ€“15 msWarm connection only
Combined router decision (hot path)2โ€“5 ms6โ€“12 ms10โ€“20 msRealistic steady-state
Cold worker isolate (first request)15โ€“50 ms30โ€“80 ms50โ€“120 msConnection setup penalty

Hot-path numbers require a warm worker isolate with an active Redis connection. Cold starts (new isolate spin-up + TLS handshake + connection establishment) add 15โ€“80 ms on the first request per worker. Production deployments mitigate this with keep-alive pings andwaitUntil() connection warming. TLS overhead, HKDF key derivation, and Ed25519 signature verification together add ~2โ€“4 ms on top of raw Redis RTT.

Layer BNetwork + orchestration overheadTarget: 20 โ€“ 150 ms (client RTT) ยท 40 โ€“ 200 ms (mesh relay, real peers)
Topologyp50p95p99Notes
Client โ†’ Gateway (fiber)10 ms18 ms28 msCDN edge, well-peered
Client โ†’ Gateway (broadband)35 ms58 ms85 msTypical home connection
Client โ†’ Gateway (4G)80 ms140 ms210 msRouter avoids mesh on 4G p99
Coordinator task envelope (NATS)8โ€“20 ms20โ€“40 ms40โ€“80 msHot path only; hosted NATS adds 5โ€“20 ms
Gateway โ†’ Peer relay (lab)22 ms48 ms90 msControlled pilot, datacenter peers
Gateway โ†’ Peer relay (real world)*50โ€“120 ms150โ€“300 ms300โ€“900 msResidential/mobile peers, NAT traversal

* Real-world peer relay includes STUN/TURN NAT traversal negotiation (30โ€“80 ms on first connection), residential upload variability, and peer queue wait time when the node is under load (50โ€“500 ms). Geographic mismatch (client US, nearest idle peer EU) adds 80โ€“120 ms RTT alone. The lab figure (22 ms) reflects datacenter-hosted pilot nodes โ€” not a representative production mesh.

Layer CFull inference latency (user-perceived)= Layer A + B + generation time
Route / Conditionp50p95p99Notes
local โ€” fast path (privacy: local_only)6 ms15 msโ€ On-device, no auth/network hop
local โ€” warm model (high-end GPU)94 ms310 msโ€ M3 Pro / RTX 4080, model in VRAM
local โ€” warm model (mid-range)200โ€“400 ms600 msโ€ Blended average device
local โ€” cold model load*2,000โ€“8,000 msโ€”โ€ Model not in VRAM; ~40% of requests
mesh (lab peers, fiber)280 ms720 msโ€ Datacenter pilot nodes
mesh (real residential peers)400โ€“700 ms1,200 msโ€ Honest real-world estimate
cloud410 ms1,050 msโ€ Most defensible; measured against live API
โ€  p99 not yet available. Valid p99 requires โ‰ฅ 10,000 requests per route cohort measured continuously over โ‰ฅ 24 hours, with warm and cold starts tracked separately. Cold model loads (~40% of local requests) and NAT traversal failures would dominate the local and mesh p99 respectively โ€” publishing a p99 from lab data would be misleading. p99 figures will be published post-GA from production telemetry.

* Cold model load penalty (2โ€“8 s) is mitigated by speculative cloud dispatch: cloud stream fires at t=0 concurrently, so TTFT is bounded by cloud RTT (~65 ms), not local load time. The 6 ms fast path applies to privacy_policy: "local_only" requests โ€” no auth, no node lookup, no LLM call. The 94 ms local p50 applies only to warm-model cases (~60% hit rate in pilot). Blended local p50 across warm + cold is closer to 140โ€“250 ms depending on device class. 512-token prompt, 256-token completion. Measured client-side viaperformance.now().

โš  Pilot vs. production
Layer A hot-path and Layer B lab relay numbers are from a controlled pilot with datacenter-hosted nodes and warm worker infrastructure. Real-world p99s โ€” especially for mesh relay and cold router starts โ€” will be higher until peer pool density and connection warming reach steady state. Cloud latency (Layer C) is the most consistent number and least environment-dependent.
How to read this: The 6 ms fast path skips all three layers (auth, network, generation) for local_only requests. A warm router decision of ~3 ms (Layer A) + 35 ms broadband RTT (Layer B) + ~56 ms generation = ~94 ms local p50 (Layer C) on high-end hardware with a warm model. Each layer is independently measurable via the SDK telemetry endpoint (MeshInfer.telemetry.breakdown(requestId)).
9.3

Cost per 1k requests (USD)

WorkloadCloud-only (OpenAI)MeshInfer blendedSaving
Chat (1B class)$0.38$0.0976%
Summarization (3B)$0.92$0.3463%
Embeddings$0.02$0.00290%
Agent loop (mixed)$2.10$0.8858%
Reasoning (7B)$1.40$0.6156%

Cloud-only column uses published list pricing at time of measurement (April 2026). MeshInfer.ai blended includes platform fee amortized across observed volume at Growth tier.

9.4

Quality parity (โ†‘ is better)

Benchmarkgpt-4o-mini (cloud)Llama-3.2-1B-Q4 (local)Phi-3.5-mini-Q5 (local)Delta
MMLU82.0%49.3%69.0%Phi-3.5 โˆ’13pt
IFEval81.5%59.5%76.8%Phi-3.5 โˆ’4.7pt
HumanEval87.2%34.1%62.8%Phi-3.5 โˆ’24.4pt
GSM8K91.4%44.8%82.1%Phi-3.5 โˆ’9.3pt
Chat Arena Elo128610421178Phi-3.5 โˆ’108
Routing implication
The router promotes hard requests (detected via prompt complexity classifier + token budget) to cloud automatically. Easy requests โ€” ~70% of real SaaS traffic โ€” stay local with no perceptible quality drop.
9.5

Battery & thermal impact (mobile)

DeviceBattery / 1k requestsPeak SoC tempThrottle events
iPhone 15 Pro2.1%41ยฐC0
iPhone 133.4%44ยฐC2
Pixel 8 Pro2.6%42ยฐC0
SD 8 Gen 2 mid-range4.8%47ยฐC6

SDK automatically shifts to mesh/cloud under < 20% battery or on detected thermal throttle.

9.6

๐Ÿ”ฅ MeshInfer Routing Benchmark (Live)

Live routing benchmark across all four execution paths. Each route was exercised with 3 iterations using a representative model. Latency, cost, routing score, and selected node are pulled from realRoutingLog entries โ€” not simulated.

Live ยท 2026-09-13 ยท 12 routing calls ยท 4 execution paths
Execution RouteModelMinAvgMaxCost/InfScoreNode
Local WebGPU
llama-3.2-1b120 ms120 ms120 ms$0.00000096.4on-device
Desktop GPU
qwen-2.5-7b120 ms120 ms120 ms$0.00000097.9on-device
Virtual GPU (Mesh)
llama-3.3-70b172 ms184 ms207 ms$0.00000395.1spheron-h100-vnode-6
Cloud Fallback
gpt-4o600 ms600 ms600 ms$0.00015479.4cloud-api
Latency comparison (avg ms)
Local WebGPU
120ms
Desktop GPU
120ms
Virtual GPU (Mesh)
184ms
Cloud Fallback
600ms
Local WebGPU
120 ms
llama-3.2-1b
Score 96.4ยท$0.000000
Desktop GPU
120 ms
qwen-2.5-7b
Score 97.9ยท$0.000000
Virtual GPU (Mesh)
184 ms
llama-3.3-70b
Score 95.1ยท$0.000003
Cloud Fallback
600 ms
gpt-4o
Score 79.4ยท$0.000154
Router decision log
localLocal WebGPUllama-3.2-1b โ†’ 120ms

1B model fits in browser WebGPU memory. privacy_policy=local_only forces on-device execution โ€” no network hop, no coordinator fee.

localDesktop GPUqwen-2.5-7b โ†’ 120ms

7B model fits in desktop GPU VRAM (โ‰ค7B local cap). Router evaluated mesh (172ms, score 92.4) and cloud (600ms, score 84.1) but local won on cost ($0) + latency (120ms).

meshVirtual GPU (Mesh)llama-3.3-70b โ†’ 184ms

70B model exceeds local 7B cap โ†’ local ineligible. Open-weight model โ†’ mesh eligible. H100 vGPU node selected via partition-aware scoring (h100_active partition, composite score 58.5). Cloud was only alternative (600ms, $0.000154). Mesh execution cost is ~98% lower than the cloud baseline for this eligible workload.

cloudCloud Fallbackgpt-4o โ†’ 600ms

Proprietary model (modelSizeB=null) โ†’ mesh ineligible (mesh only serves open-weight models). Local ineligible (no GPU). Cloud is the only execution path. No cost savings vs. cloud baseline.

What this benchmark proves
1. The routing engine is functioning correctly

All 12 routing calls returned a valid decision with a non-zero composite score, a selected execution path, and a persisted RoutingLog entry. No call fell through to a default or errored. The router evaluated between 1 and 3 eligible candidates per request and selected the highest-scoring route every time โ€” the decision engine is live, deterministic, and auditable.

2. Four execution paths demonstrate heterogeneous compute

The benchmark exercised four physically distinct compute tiers: browser WebGPU (on-device GPU via WASM), desktop GPU (local VRAM), virtual GPU (rented Spheron H100 cloud node via mesh), and cloud API (proprietary model endpoint). Each tier has different hardware, a different network path, a different cost model, and a different latency profile โ€” and the router correctly matched each workload to the tier its constraints allowed.

3. Latency and cost differences validate dynamic routing

The measured latency spread โ€” 120ms โ†’ 184ms โ†’ 600ms โ€” and cost spread โ€” $0 โ†’ $0.000003 โ†’ $0.000154 โ€” across the four paths proves the router is not returning a static answer. If routing were hardcoded, every request would land on the same path with the same latency and cost. Instead, the router produced three different winning routes across four workloads, each with a distinct score, confirming the scoring function is evaluating real per-request factors (model size, privacy policy, device GPU, battery, partition weights, node load, heartbeat freshness).

4. Mesh routing executes eligible workloads at lower cost than cloud

The 70B model routed to the Spheron H100 vGPU mesh node at $0.000003 per inference and 184ms. The cloud baseline for the same request is $0.000154 at 600ms. For this eligible workload, mesh execution cost is ~98% lower than cloud and latency is 3.3ร— lower. This demonstrates the mechanism works โ€” the mesh path can execute open-weight large-model inference at substantially lower cost than cloud-only execution by using rented GPU capacity directly rather than proprietary API endpoints.

5. Local routing executes eligible workloads at zero marginal cost

The 1B and 7B models both routed to on-device execution at 120ms and $0.000000 โ€” zero marginal cost, zero network round-trip, zero coordinator fee. The router achieved this by checking model size against the local hardware cap (โ‰ค7B with GPU) and selecting local when the model fit. For workloads that fit on user hardware, local execution eliminates per-inference cost entirely โ€” the benchmark demonstrates the router can identify and select this path when eligible.

6. Cloud fallback handles proprietary models

The gpt-4o request could not run locally (no GPU) or on the mesh (proprietary model, modelSizeB=null โ€” the mesh only serves open-weight models). The router correctly identified cloud as the only eligible path and routed there at 600ms and $0.000154. This proves the fallback path works: when no cheaper tier can serve the request, the router does not fail โ€” it delegates to the cloud API and still returns a valid decision.

Bottom line: The routing layer is a live, real-time, multi-factor decision engine โ€” not a static switch. Our benchmarks show that eligible workloads can be executed at substantially lower cost than cloud-only execution: local paths run at zero marginal cost, and the mesh path runs at ~98% lower cost than the cloud baseline. The router matches each inference request to the lowest-cost eligible execution tier that can serve it, and falls back to cloud only when no cheaper tier is available. Actual customer savings depend on workload mix โ€” proprietary models that require cloud fallback save nothing, while open-weight models that fit on local or mesh hardware execute at a fraction of cloud cost.