Cold Start Optimization
When a user's device loads a model for the first time, we don't make them wait. Speculative cloud dispatch fires at t=0 — before the gate-check even finishes.
The Cold Start Problem
First local inference on a device incurs a "cold start" penalty — the model must be decompressed and loaded into VRAM. On an M-series Mac with fast storage, that's 2–3 seconds. On mid-range Android with slower memory, it can be 5+ seconds. Pure local-first strategies fail here: the user waits in silence.
The original V1 approach dispatched cloud only after the gate-check confirmed the model was cold. That added 50–80ms of unnecessary silence before the first cloud token. The refactored router eliminates this entirely with speculative dispatch.
Speculative Dispatch (current implementation)
User Request arrives at Router
↓
[t=0ms] Step 0: Synchronous privacy check ← runs BEFORE any I/O, zero latency
policy == "local_only" → BLOCK speculative entirely — data never leaves device
policy == "no_cloud" → BLOCK speculative — local/mesh only
otherwise → speculative_ok = true
↓
[t=0ms] Step 1: Fire speculative cloud stream (AbortController created)
+ Run gate-check concurrently (50–80ms)
↓
┌────────────────────────────────┬──────────────────────────────┐
│ Cloud stream (speculative) │ Gate-check (concurrent) │
│ AbortController signal live │ registry: models_loaded? │
│ token 0 ← t+180ms │ capability: VRAM ≥ model? │
│ token 1 ← t+210ms │ history: used in last 2h? │
│ ... │ → resolves at t+50–80ms │
└────────────────────────────────┴──────────────────────────────┘
↓
[t+50-80ms] Step 2: Gate-check resolves
↓
gate_result.warm == true? (63% of requests — pilot data)
→ AbortController.abort() ← cloud stream cancelled immediately
→ local model takes over ← TTFT = ~12ms from warm model
→ 0–3 tokens billed to cloud ← cost: ~$0.000001
gate_result.warm == false? (37% of requests)
→ keep cloud stream ← already delivering tokens since t+180ms
→ background: warm local model ← 2–5s loading
→ seamless handoff when ready ← local takes over mid-streamKey improvement: TTFT drops from gate_check_ms + cloud_RTT (~230–260ms) to just cloud_RTT (~180ms). On warm-model hits (63%), the cloud stream is aborted before meaningful tokens are billed — saving ~$0.000001 per aborted request.
Privacy Gate (synchronous, pre-dispatch)
The privacy check is the first operation executed — synchronous, before any network call is issued. This is a structural guarantee, not a policy flag checked later.
The SDK router is the entry point for every inference request. Its first operation is the privacy gate — a synchronous check that runs before any network I/O is initiated. This is a structural guarantee, not a policy flag evaluated later in the request lifecycle.
| Step 0 · Privacy gate | Evaluated synchronously before any await. If the privacy policy forbids cloud, speculative dispatch is structurally impossible — no cloud request is created, no network call is issued, and no data leaves the device. |
| Step 1 · Speculative dispatch | Only reached if the privacy gate passed AND the device is a local candidate. A cloud stream is started concurrently with a local warm-up check. If the local model is already warm, the cloud stream is cancelled before any tokens are billed. If the model is cold, the cloud stream is kept and local warms in the background for future requests. |
| Step 2 · Standard scoring | Reached when the device is not a local candidate or the privacy policy blocked speculative dispatch. The request follows the standard routing path without any concurrent cloud stream. |
local_only andno_cloud policies, speculative dispatch is not merely discouraged — it is structurally impossible. The privacy gate resolves before the code path that creates a cloud request is ever entered.AbortController Mechanics
Why abort works here
- AbortController.abort() mid-stream: Closes the fetch ReadableStream and sends a connection reset to the provider. Billing stops at the last token the provider had already sent. If abort fires within the first RTT (~30ms), typically 0 tokens are billed.
- Gate-check target: <80ms: The gate-check must resolve fast enough that abort fires before the cloud provider bills a meaningful number of tokens. Target is sub-80ms; current measurement is 50–80ms, well inside the first cloud token window (~180ms RTT).
- Provider compatibility: All four primary cloud adapters (OpenAI, Anthropic, Groq, Together) honor the AbortController signal and stop streaming. Billing granularity varies — OpenAI bills per-token, Groq bills on completion.
- No partial output to client: Cloud tokens received before abort are buffered in the SDK, not streamed to the user. On abort, the buffer is discarded. Client sees a clean local stream starting from token 0.
Edge case: gate-check slower than first cloud token
If gate-check takes > 180ms (e.g., cold registry miss, slow device), the cloud stream has already delivered token 0 to the buffer before abort can fire. In this case, the SDK keeps cloud as primary and does not switch to local — aborting after token delivery would cause a visible stream discontinuity. The timeout heuristic:
if gate_check_ms > cloud_first_token_ms:
// Too late to abort cleanly — accept cloud as primary
// Local warmup continues in background for future requests
keep cloud_stream as primary
mark device as "warming" in registry (next request will be warm)Cold Start Detection Signals
Economics of Speculative Dispatch
Weighted average across warm/cold split: TTFT ≈ 75ms · cost ≈ $0.0000037 per request · savings ≈ 99.9% vs. pure cloud for warm-heavy workloads.