Connecting 20 nodes to the distributed inference fabric.
A complete, production-ready reference for integrating every node with routing, scheduling, metering, DIU issuance, DIU spend, and settlement logging. Each section maps to an equivalent AWS multi-node compute onboarding step.
Guide overview & scope
This guide documents the authoritative process for connecting exactly 20 heterogeneous compute nodes to the MeshInfer.AI distributed inference fabric. Each node, regardless of platform, must complete a five-phase lifecycle before it is eligible to receive routed inference tasks:
- Phase IIdentity provisioning โ NodeID, DeviceID, OrgID chain-of-trust
- Phase IIRegistration โ capability declaration, DOSFI handshake, Coordinator enrollment
- Phase IIIRouting participation โ admission to the task routing pool
- Phase IVMetering & economics โ DIU spend accounting, inference-to-DIU issuance, settlement
- Phase VOngoing operations โ health checks, reputation scoring, removal / replacement
Reference topology โ 20-node mesh
โโโโโโโโโโโโโโโโโโโโโโโโ DOSFI IDENTITY SERVICE โโโโโโโโโโโโโโโโโโโโโโโโโโ
โ identity.dosfi.ai ยท OAuth 2.0 + OIDC ยท OrgID: org_mesh20_prod โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ mTLS chain-of-trust
โโโโโโโโโโโโโโโโโโโโโโโโ MESHINFER COORDINATOR โโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ coordinator.meshinfer.ai ยท Gateway (CF Edge) ยท Router Engine โ
โ Node Registry (Redis) ยท Task Queue (NATS JS) ยท Billing Meter โ
โโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Registered & active nodes (20 total)
โโโ DESKTOP TIER (8 nodes) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ node-d01 โฆ node-d08 ยท Linux/macOS/Windows ยท llama.cpp GGUF
โโโ BROWSER TIER (6 nodes) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ node-b01 โฆ node-b06 ยท Chrome/Edge ยท WebGPU / WASM fallback
โโโ MOBILE TIER (4 nodes) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ node-m01 โฆ node-m04 ยท iOS/Android ยท CoreML / NNAPI
โโโ SERVER TIER (2 nodes) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
node-s01, node-s02 ยท Linux bare-metal ยท CUDA 12+ / 70B models
Phase I โ Node Identity Model
Every node in the mesh is assigned a cryptographic identity before it is permitted to register. The identity model mirrors the chain-of-trust established by the DOSFI Identity Service at identity.dosfi.ai.
Identity hierarchy
| OrgID | Top-level organizational namespace. Created once per tenant. Format: org_{slug}_{env}. Controls billing, policy inheritance, and DIU settlement wallet. |
| UserID | Human operator identity. Bound to OrgID via OIDC claims. Required to issue API keys and approve node registrations. |
| DeviceID | Stable hardware fingerprint. Derived from TPM-attested UUID or, on non-TPM devices, a persisted Ed25519 keypair stored in secure enclave / IndexedDB. Never regenerated unless node is explicitly decommissioned. |
| NodeID | Runtime identity for one active mesh participant. Format: node_{uuid4}. Generated by the Coordinator on first successful registration. A DeviceID can hold at most one active NodeID at a time. |
| APIKeyID | Operator-issued bearer token. Bound to UserID + OrgID. Used for all Coordinator API calls. Rotatable without invalidating NodeID. |
Identity binding chain
OrgID
โโโ UserID (OIDC JWT, signed by identity.dosfi.ai)
โโโ APIKeyID (msk_live_..., scoped to OrgID)
โโโ DeviceID (Ed25519 pub, signed by UserID at registration)
โโโ NodeID (issued by Coordinator after DeviceID verification)
โโโ DIU Wallet Address (auto-created by MeshNativeExchange
on first NodeID activation)Key material lifecycle
| Ed25519 node keypair | Generated on-device at first install. Private key never leaves hardware secure enclave (Keychain/Keystore/TPM). Public key transmitted to Coordinator at registration. |
| Session key | AES-256-GCM derived via TLS exporter (RFC 5705) per WebSocket connection. Ephemeral โ not stored anywhere. |
| Capability signature | Every heartbeat payload is signed with the node Ed25519 private key. Coordinator verifies against registered public key. |
| Key rotation | Growth/Enterprise tier: automated rotation every N days via scheduleAPIKeyRotation function. NodeID persists across API key rotation. |
Phase II โ Node Registration Flow
Step-by-step: registering one node
Run the platform-specific installer from the Download page. The installer generates the Ed25519 keypair, creates the DeviceID, and writes the local config to ~/.meshinfer/node.toml.
# macOS / Linux
curl -fsSL https://get.meshinfer.ai/install.sh | sh
# Windows
iwr https://get.meshinfer.ai/install.ps1 | iexThe node daemon performs an OAuth 2.0 device-flow against identity.dosfi.ai. On success, a short-lived OIDC access token is returned and cached locally (15-min TTL, auto-refreshed).
meshinfer-node auth --org org_mesh20_prod
# Opens browser โ identity.dosfi.ai/device
# Displays one-time code: MESH-A1B2-C3D4
# User confirms in browser โ token issuedThe daemon queries local hardware (GPU, RAM, CPU, battery, network) and produces a signed capability vector. This takes โค 40 ms and is re-run on each Coordinator reconnect.
meshinfer-node probe
# Output:
# cpu: 8 cores ยท arm64 ยท neon+i8mm
# gpu: Apple M3 Pro ยท 18432 MB VRAM ยท WebGPU โ
# ram: 18432 MB total ยท 9112 MB free
# net: rtt=34ms ยท 118 Mbps ยท 4g
# sig: ed25519:abc123...The daemon sends the capability vector, DeviceID, and OIDC token to the Coordinator. The Coordinator verifies the OIDC signature against identity.dosfi.ai JWKS, issues a NodeID, and provisions the DIU wallet.
POST https://coordinator.meshinfer.ai/v1/nodes/register
Authorization: Bearer msk_live_YOUR_API_KEY
Content-Type: application/json
{
"device_id": "did:mesh:abc123...",
"oidc_token": "eyJhbGc...",
"capability": { ...signed_probe_output... },
"platform": "desktop",
"region": "us-east",
"mesh_profile":"standard"
}
# Response 201 Created:
{
"node_id": "node_f9a3e2b1...",
"diu_wallet": "wallet_7c2d...",
"registry_ttl_s": 1800,
"heartbeat_interval_s": 15
}The daemon opens a WebSocket to stream.meshinfer.ai/v1/nodes/stream using the NodeID as the session identifier. Heartbeats are sent every 15 s. Missing 3 consecutive heartbeats downgrades status to degraded; 6 missed โ eviction.
wss://stream.meshinfer.ai/v1/nodes/stream
Authorization: Bearer msk_live_...
X-Node-ID: node_f9a3e2b1...
โ { "type": "heartbeat",
"node_id": "node_f9a3e2b1...",
"ts": "2026-07-06T14:00:00Z",
"load": { "active_tasks": 0, "cpu_pct": 12 },
"battery": { "level": 0.94, "charging": true },
"thermal": "nominal",
"sig": "ed25519:..." }
โ { "type": "ack", "server_ts": "2026-07-06T14:00:00.031Z" }Batch registration โ all 20 nodes
For fleet deployments, use the fleet provisioning script. It reads a manifest file and registers all nodes in parallel, emitting a registration report on completion.
# nodes.yaml โ fleet manifest
nodes:
- id: node-d01
platform: linux
region: us-east
profile: standard
- id: node-d02
platform: linux
region: us-east
profile: standard
# ... 18 more ...
# Run fleet registration
meshinfer-fleet register --manifest nodes.yaml --api-key msk_live_... --org org_mesh20_prod --parallel 5
# Output: nodes_registration_report_2026-07-06.jsonPhase II-B โ Dual Handshake Protocol
DOSFI identity handshake
The DOSFI handshake establishes cross-platform identity binding. It occurs once per DeviceID lifetime and must succeed before the Coordinator registration in Phase II.
Node Daemon identity.dosfi.ai Coordinator
โ โ โ
โโโ /device/auth โโโโโโโโบโ โ
โโโโ device_code โโโโโโโโโ โ
โ โโโโ user_confirms โโโโโโ(browser)
โโโโ access_token โโโโโโโโ โ
โ (oidc_jwt, 15min) โ โ
โ โ โ
โโโ /v1/nodes/register โโโโโโโโโโโโโโโโโโโโโโโโโโโบโ
โ { oidc_jwt, device_id, capability_vec } โ
โ โโโJWKS verify โโโโโโโโโโบโ
โ โโโโpub key(s) โโโโโโโโโโโ
โโโโ { node_id, diu_wallet } โโโโโโโโโโโโโโโโโโโโโ
โ โ โMeshInfer Coordinator handshake
After DOSFI authentication, the Coordinator performs its own handshake to admit the node into the routing pool. This is an mTLS-authenticated exchange.
Node Daemon Coordinator
โ โ
โโโ TLS ClientHello โโโโโโโโโโโบโ
โโโโ TLS ServerHello โโโโโโโโโโโ (CF cert)
โโโ Client Certificate โโโโโโโโโบโ (Ed25519 node cert, signed by DOSFI CA)
โโโโ Certificate Verified โโโโโโโ
โ โ
โโโ { node_id, capability_vec, โ
โ mesh_profile, region, โ
โ sig: ed25519(...) } โโโโโบโ
โ โโโ verify sig against registered pubkey
โ โโโ capability eligibility check
โ โโโ assign routing pool slot
โโโโ { admitted: true, โ
โ pool_tier: "standard", โ
โ task_queue_subject: "tasks.us-east.standard",
โ heartbeat_interval: 15 โ
โ } โโโโโโHandshake failure modes & recovery
| OIDC token expired | Daemon auto-refreshes via refresh_token. If refresh fails โ re-run device-flow auth. |
| JWKS verification failure | Coordinator rejects registration. Check that identity.dosfi.ai is reachable from the Coordinator network. Retry with exponential backoff (max 5 attempts). |
| Capability ineligible | Coordinator returns 422 with ineligibility reason (e.g. "insufficient_vram"). Node cannot join pool for rejected model sizes; may still serve smaller models. |
| mTLS cert mismatch | Ed25519 cert in ClientHello does not match registered DeviceID. Re-run `meshinfer-node auth` to re-issue the DOSFI-signed certificate. |
| Region quota exceeded | Coordinator returns 429 with Retry-After header. Fleet manager queues and retries; does not count against reputation. |
Phase III โ Routing Participation Rules
Admission criteria
Traffic weight assignment
The Router assigns a weight to each eligible node in the pool based on reputation, uptime, model cache state, and latency adherence. Selection uses weighted random over the eligible pool. A small exploration factor routes a fraction of requests to random eligible nodes to maintain continuous learning and prevent any single node from monopolizing traffic.
The exact weight function, decay curves, and exploration constant are Coordinator-internal and tuned per release. Node operators influence their weight indirectly: higher reputation, better uptime, warm model caches, and lower latency all increase routing weight.
Pool tiers & task assignment
| standard | Eligible for all req_class โ {tiny, small}. Assigned to the standard task queue for its region. |
| capable | Eligible for req_class โ {tiny, small, medium}. Reputation โฅ 70 required. |
| elite | Eligible for all req_class including hard (70B+ sharded). Reputation โฅ 85, uptime โฅ 95%, โฅ 3 models cached. |
| restricted | Reputation 20โ39: tiny tasks only, 10% traffic share max. Under monitoring. |
| suspended | Reputation < 20 or verification pass rate < 90%: zero tasks. Human review required. |
Phase III-B โ Inference Scheduling Participation Rules
Task lifecycle on a node
NATS JetStream Node Daemon Coordinator
โ โ โ
โโ tasks.region.tier โโโโโโโบโ โ
โ { task_id, model, โ โ
โ prompt_enc, โ โ
โ deadline_ms, โ โ
โ cost_slo_usd } โ โ
โ โโโ preflight check โ
โ โ (thermal, battery, โ
โ โ model cached?) โ
โ โโโ ACK (accept) โโโโโโโโบโ
โ โโโ start inference โ
โ โโโ stream tokens โโโโโโโบโ
โ โโโ task_complete โโโโโโโโบโ
โ โ { result_hash, โ
โ โ tokens_out, โ
โ โ latency_ms } โ
โ โโโโ meter_event โโโโโโโโโ
โ โ (DIU issued to node โ
โ โ wallet on settlement)โScheduling priority classes
| interactive | Deadline โค 200 ms to first token. Only local or top-10 reputation mesh nodes eligible. Never cold-start. Preempts background tasks on the node. |
| standard | Deadline โค 2 s to first token. Full eligible pool. Cold-start allowed if model loads within 800 ms. |
| background | No deadline. Lowest cost mesh nodes. Batching enabled (up to 8 concurrent tasks per node). Used for embeddings, summarization, fine-tune prep. |
| verification | A sample of all tasks re-dispatched here for deterministic-decode hash comparison. Must match original result hash. Mismatch โ reputation penalty + investigation. |
Node-side scheduling rules
| Max concurrent tasks | Configurable per node. Default: 2 (standard), 8 (background). Hard cap enforced by daemon โ excess task offers are NACKed to Coordinator. |
| Task acceptance deadline | Node must ACK or NACK within 500 ms of receiving task. No response โ Coordinator re-dispatches with exponential backoff. |
| Preemption policy | Background tasks may be preempted by interactive tasks if the node has reached its concurrency cap. Preempted background task is re-queued; node reputation is not penalized. |
| Model cold-start window | Coordinator allocates up to 800 ms for model load before counting against latency SLO. Node must report model_loading: true in its NACK to trigger this window. |
Phase IV โ Compute Metering โ DIU Spend
Every inference request that consumes mesh compute generates a metering event. Metering quantifies how much DIU the requester spent on each task. Settlement (ยง G.7) then determines how much DIU the serving node earned.
Metering event structure
{
"event_type": "inference_metered",
"request_id": "req_a1b2c3...",
"node_id": "node_f9a3e2b1...",
"user_email": "operator@org.com",
"org_id": "org_mesh20_prod",
"model": "llama-3.2-3b-q4",
"req_class": "small",
"route_decision": "mesh",
"tokens_in": 128,
"tokens_out": 512,
"latency_ms": 284,
"success": true,
"hypothetical_cloud_cost_usd": 0.00034,
"actual_cost_usd": 0.00008,
"diu_spend": 0.08,
"diu_savings": 0.26,
"timestamp": "2026-07-06T14:02:33.120Z"
}DIU spend calculation
# DIU spend rate per 1,000 tokens (read from DOSFI pricing oracle):
RATES = {
"tiny": { local: 0.0, mesh: 0.02, cloud: 0.08 },
"small": { local: 0.0, mesh: 0.08, cloud: 0.34 },
"medium": { local: 0.0, mesh: 0.24, cloud: 1.20 },
"hard": { local: 0.0, mesh: 0.80, cloud: 4.80 },
}
diu_spend(event) =
(event.tokens_out / 1000) * RATES[event.req_class][event.route_decision]
# Example:
# 512 tokens_out, req_class="small", route="mesh"
# diu_spend = (512 / 1000) * 0.08 = 0.04096 DIUMetering pipeline
| Coordinator Billing Meter | Emits metering events to Kafka topic meshinfer.metering.events on task completion. |
| ClickHouse aggregator | Consumes Kafka stream, writes to UsageRecord entity. Aggregates into MetricSnapshot hourly/daily/monthly. |
| DOSFI pricing oracle | Consulted once per billing period to update DIU rate table. Rate changes take effect on the next billing period โ no retroactive adjustments. |
| Requester wallet debit | OrgID DIU wallet is debited atomically with the metering event write. No credit means no task dispatch. |
| Audit trail | Every metering event written to AuditLog entity with resource_type: "inference_charge" for SOC 2 compliance. |
Phase IV-B โ Inference Execution โ DIU Issuance
When a node successfully completes an inference task, it earns DIU. Issuance is calculated from the metering event and credited to the node's wallet by the MeshNativeExchange settlement layer.
DIU issuance calculation
When a node successfully completes a task, it earns a share of the requester's DIU spend. The issuance amount is determined by three factors:
- Node share: A fixed portion of the DIU spend is allocated to the serving node; the remainder covers platform infrastructure and protocol reserve.
- Reputation multiplier: Higher-reputation nodes earn an elevated multiplier. Elite-tier nodes receive the highest bonus; low-reputation nodes receive a reduced rate.
- Quality multiplier: Nodes that meet latency SLOs earn the full multiplier. Tasks completed beyond a multiple of SLO, or failed tasks, yield zero DIU โ no payment for poor service.
The exact share ratios, multiplier thresholds, and decay curves are Coordinator-internal economic parameters. They are tuned to ensure compute-backed sustainability and are not publicly replicable.
Issuance event structure
{
"event_type": "diu_issued",
"settlement_id": "stl_x9y8z7...",
"node_id": "node_f9a3e2b1...",
"diu_wallet": "wallet_7c2d...",
"source_request_id": "req_a1b2c3...",
"diu_gross": 0.04096,
"reputation_mult": 1.10,
"quality_mult": 1.00,
"diu_net": 0.04506,
"platform_fee": "<platform share of gross>",
"reserve": "<protocol reserve of gross>",
"settlement_epoch": "2026-07-06T15:00:00Z",
"status": "pending_settlement"
}Phase IV-C โ Settlement Logging โ MeshNativeExchange
Settlement lifecycle
Coordinator MeshNativeExchange Node Wallet
Billing Meter Settlement โ
โ โ โ
โโโ diu_issued events โโโโโโบโ โ
โ (batch, hourly epoch) โ โ
โ โโโ aggregate batch โ
โ โโโ integrity check โ
โ โ (hash chain verify) โ
โ โโโ commit credits โโโโโโบโ
โ โ { wallet, diu_net, โ
โ โ settlement_id } โ
โ โโโ write SettlementLog โ
โ โ โ AuditLog entity โ
โโโโ epoch_settled ACK โโโโโ โ
โ โโโ publish to DOSFI โ
โ โ accounting oracle โSettlement log record
{
"settlement_id": "stl_epoch_20260706T150000Z",
"epoch_start": "2026-07-06T14:00:00Z",
"epoch_end": "2026-07-06T15:00:00Z",
"org_id": "org_mesh20_prod",
"node_settlements": [
{
"node_id": "node_f9a3e2b1...",
"diu_wallet": "wallet_7c2d...",
"tasks_served": 147,
"tokens_out": 89234,
"diu_gross": 6.234,
"diu_net": 6.083,
"status": "settled"
}
// ... 19 more nodes ...
],
"total_diu_issued": 98.441,
"total_platform_fee": 17.720,
"total_reserve": 9.844,
"integrity_hash": "sha256:cafe...",
"committed_at": "2026-07-06T15:00:34Z"
}Settlement entity fields (AuditLog)
| resource_type | "settlement_epoch" โ queryable via audit log endpoint |
| action | "diu_settled" | "diu_disputed" | "diu_clawback" |
| status | "success" | "failure" | "disputed" |
| details.node_count | Number of nodes settled in this epoch |
| details.integrity_hash | SHA-256 hash of full settlement batch โ verifiable externally |
Phase V โ Node Health Checks
Health check types
Health check record schema
// NodeHealthCheck entity record:
{
"node_id": "node_f9a3e2b1...",
"check_type": "latency_test",
"status": "pass",
"latency_ms": 187,
"details": {
"model": "llama-3.2-3b-q4",
"req_class":"small",
"slo_ms": 300
},
"timestamp": "2026-07-06T14:05:00Z"
}Phase V-B โ Node Performance Scoring
Reputation score components
Score update frequency
| Real-time component | Reputation score updated immediately on verification mismatch, thermal eviction, or missed heartbeat. |
| Rolling component | Full score recalculated every 15 minutes using a 7-day weighted rolling window. Recent events weighted 2ร vs. older events. |
| Score floor | A minimum score floor protects against one-time anomalies destroying new nodes. Below a configured threshold โ auto-suspended. |
| Score ceiling boost | Nodes sustaining near-perfect metrics over a sustained period earn a permanent Elite bonus that stacks with the reputation multiplier in DIU issuance. |
Phase V-C โ Node Privacy Tier Enforcement
Privacy tiers
| local_only | Node serves only tasks from its own registered OrgID. No external prompts processed. Enforced at Coordinator โ tasks from other orgs are never dispatched. |
| mesh_ok | Node serves tasks from any OrgID in the mesh. Prompts are encrypted (AES-256-GCM) end-to-end โ node sees only ciphertext of the prompt, decrypts result for hash. |
| cloud_ok | Node may also be used as a relay for cloud fallback proxying. Only applies to server-tier nodes with explicit opt-in. |
Privacy enforcement stack
Privacy is enforced structurally at two layers:
- Coordinator-side (server): Before any task is dispatched, the Coordinator verifies that the node's privacy tier is compatible with the requester's privacy policy.
local_onlytasks are never sent to nodes outside the requester's OrgID.no_cloudtasks bypass cloud-relay nodes entirely. - Node-side (daemon): The daemon independently re-verifies the privacy tier before accepting any task. If a
local_onlynode receives a task from a different OrgID, it NACKs immediately. Prompts are decrypted only in memory, processed, and the result is re-encrypted before transmission โ plaintext never leaves the node.
This dual-layer enforcement means privacy is a structural property of the dispatch path, not a runtime policy check that could be bypassed.
Phase V-D โ Node Removal & Replacement Flow
Graceful removal
# Initiate graceful drain on one node:
meshinfer-node drain --node-id node_f9a3e2b1...
# Drain process:
# 1. Coordinator marks node as "draining" โ no new tasks dispatched
# 2. In-flight tasks complete normally (up to drain_timeout = 300 s)
# 3. After all tasks complete, daemon sends DEREGISTER to Coordinator
# 4. NodeID moved to "decommissioned" status in registry
# 5. DIU wallet settlement finalized for all pending epochs
# 6. DeviceID retained โ can re-register as a new NodeIDEmergency removal (reputation / security breach)
| Triggered by | Reputation score below the minimum floor, verification mismatch rate exceeding threshold, security policy violation, or admin manual override. |
| Immediate effect | Node removed from all routing pools. In-flight tasks cancelled and re-dispatched to next-best node. Requester is not charged for cancelled tasks. |
| Reputation impact | Score set to 0 and frozen. Node cannot re-register with same DeviceID for 72-hour cooling-off period. |
| DIU clawback | Pending unsettled DIU issuance events for the flagged node are held for 24 hours pending review. Settled DIU is final. |
Node replacement procedure
Appendix โ Per-Node Configuration Reference
# ~/.meshinfer/node.toml โ full configuration reference
[identity]
org_id = "org_mesh20_prod"
api_key = "msk_live_..." # rotate via scheduleAPIKeyRotation
node_id = "" # auto-populated on registration
device_id = "" # auto-populated on first probe
[coordinator]
endpoint = "https://coordinator.meshinfer.ai"
stream_ws = "wss://stream.meshinfer.ai/v1/nodes/stream"
region = "us-east" # us-east | us-west | eu-west | ap-south
heartbeat_s = 15
reconnect_backoff_ms = [500, 1000, 2000, 5000, 10000]
[mesh]
profile = "standard" # eco | standard | power | off
privacy_tier = "mesh_ok" # local_only | mesh_ok | cloud_ok
max_concurrent_tasks = 2
max_background_tasks = 8
[hardware]
gpu_util_cap_pct = 50 # max GPU utilization while serving tasks
cpu_util_cap_pct = 60
battery_floor = 0.20 # pause mesh work below this (mobile)
require_charging = true # mobile only
thermal_max = "fair" # nominal | fair | serious | critical
[models]
cache_dir = "~/.meshinfer/models"
auto_download = true
preferred = [
"llama-3.2-3b-q4",
"phi-3.5-mini-q4",
"qwen-2.5-7b-q5"
]
[logging]
level = "info" # debug | info | warn | error
audit_log = true # write local audit trail
telemetry = true # send telemetry to Coordinator