“Edge AI” can mean inference on a user's device or inference on a provider's infrastructure nearer to that user. These designs have different loading costs, data boundaries and operational responsibilities.
Neither design has zero latency. Choose the execution location from the task's requirements and measurements.
Distinguish the browser backends
ONNX Runtime's WebGPU guide documents the onnxruntime-web/webgpu import and the webgpu execution provider. WebGPU targets GPU computation; it should not be described as a universal browser interface to every device's NPU.
ONNX Runtime documents WebNN separately. Backend availability, supported operators and hardware behavior must be checked for the intended browser and device.
The model also defines tensor names, types and shapes. A generic array called “input” is not a valid language-model integration without matching that contract.
Budget the full model lifecycle
An idealized seven-billion-parameter model at four bits per weight needs 3.5 billion bytes for weights alone. The downloaded format and runtime add overhead, caches and working memory. That arithmetic does not establish smooth smartphone operation.
Measure download, initialization, preprocessing, inference and output handling separately. Repeat after a cold start and after assets are cached. Test interruption, memory pressure and GPU-device loss.
For an image classifier, confirm preprocessing against the model's expected dimensions, channel order and normalization. A fast inference on incorrectly prepared data is not a successful deployment.
Hosted inference has a different boundary
Cloudflare Workers AI is a hosted inference service. Sending input to it is different from keeping inference inside the browser. An application SDK that calls a provider should not be mistaken for the infrastructure hosting that provider's model.
Measure network time, queuing and generation together. Do not assume that every model is available at every network location or that proximity alone determines latency.
Design a truthful fallback
If the device cannot run the model, preserve a useful manual path. Offer a hosted alternative only when the user understands the changed data flow and the application is authorized to use it.
Review logging, analytics, remote tools and storage before claiming that data stays local. Legal and contractual requirements depend on the actual use and jurisdiction; there is no general rule that all medical, legal and financial inference must run on-device.
A reliable deployment has a tested model contract, a supported device set and a clear failure path. The location of inference is one design decision within that system.
Sardar Mukhshaf
Full-Stack & DevOps Engineer
I build scalable web applications, secure backend systems, and production-ready cloud infrastructure. My engineering focus spans interface design, API architecture, deployment pipelines, and observability. Follow for more deep dives into system architecture, performance engineering, and emerging technologies.
More from the practice
Keep the useful ideas close.
Follow new articles with your preferred RSS reader.
Open the RSS feed ↗
