CrocopliersCrocopliers

Architecture Guide

Edge AI for Connected Products: When to Process on Device vs. Cloud

Every connected product eventually faces the same question: where does the intelligence live? On the device, where data is fresh but compute is limited? Or in the cloud, where models are powerful but latency and bandwidth costs add up? The answer is almost never one or the other. Most production systems use both — the real engineering question is which decisions happen where, and how the two layers coordinate.

9 min read2026-11-13AI Engineering

Why the edge-vs-cloud question matters now

Three trends are converging. First, the models got smaller. Quantized neural networks that run on a $15 microcontroller can now do object detection, anomaly classification, and keyword spotting that required a server five years ago. Second, connectivity costs money and fails. A device that depends on a cloud API for every decision is a device that stops working in a tunnel, a factory with poor WiFi, or a country with expensive cellular data. Third, privacy regulations are tightening. Processing personal data on-device and sending only aggregates upstream avoids an entire category of compliance work.

For product companies building IoT device management platforms or connected device systems, the edge-vs-cloud decision shapes the entire product architecture — firmware, connectivity, backend, and update infrastructure.

The decision framework: four factors

The choice between edge and cloud processing comes down to four factors: latency requirements, connectivity reliability, data sensitivity, and model complexity. Each factor pushes in one direction, and the combination determines where the boundary sits.

Latency: if the device needs to respond in under 100 milliseconds — a safety shutoff, a real-time quality inspection, a user interaction — the processing must happen on-device. No cloud round-trip is fast enough. If the response can wait seconds or minutes — a daily report, a maintenance recommendation, a firmware decision — cloud processing is fine.

Connectivity: if the device operates in environments where network access is intermittent, unreliable, or expensive, edge processing is not optional — it is a requirement for basic functionality. The device needs to make decisions locally and sync results when connectivity returns. Industrial equipment, agricultural sensors, mobile assets, and consumer devices in emerging markets all face this constraint.

Data sensitivity: if the raw data contains personal information, protected health information, or proprietary industrial data, processing on-device and sending only derived results reduces the compliance surface. This is especially relevant under GDPR, HIPAA, and emerging AI regulations that care about where personal data is processed.

Model complexity: simple models (anomaly detection, classification, keyword spotting) run efficiently on edge hardware. Complex models (large language models, sophisticated computer vision, multi-modal reasoning) still need cloud compute. The boundary moves every year as hardware improves and models shrink, but today, anything requiring more than a few hundred million parameters is cloud territory.

  • Sub-100ms response → edge (safety, real-time control, user interaction)
  • Intermittent connectivity → edge with sync (industrial, mobile, remote)
  • Sensitive raw data → edge processing, aggregate uploads only
  • Large models or cross-device reasoning → cloud with edge fallback

The hybrid architecture most products need

In practice, the most effective architecture splits intelligence across three tiers. The first tier is on-device inference for time-critical decisions — the device runs a small model that processes sensor data in real time and acts immediately. A thermostat detecting occupancy, a camera classifying defects on an assembly line, a wearable detecting a fall.

The second tier is edge gateway aggregation. A local gateway — a Raspberry Pi-class device, an industrial PC, or a smartphone acting as a hub — collects data from multiple devices, runs slightly larger models, and handles decisions that need context across devices. A factory floor gateway that correlates readings from ten sensors to detect a process drift, or a home hub that optimizes energy across multiple smart devices.

The third tier is cloud analytics and retraining. The cloud receives aggregated data, runs complex models that need the full fleet's context, generates insights and reports, and retrains edge models based on new data. The cloud also manages the lifecycle: OTA updates push improved models to devices, A/B testing validates new versions, and monitoring tracks model performance across the fleet.

What edge AI looks like in firmware

On a microcontroller (Cortex-M class, ESP32, or similar), edge AI typically runs as a TinyML model using TensorFlow Lite Micro, Edge Impulse, or a vendor-specific runtime. The model is compiled into a C array and linked into the firmware — no operating system required. Inference takes milliseconds and uses kilobytes of RAM.

On a Linux-capable device (Raspberry Pi class, Jetson Nano, or an industrial SBC), you get more options: ONNX Runtime, TensorFlow Lite, PyTorch Mobile, or vendor-optimized runtimes like NVIDIA TensorRT. These handle larger models — full object detection, natural language classification, time-series forecasting — at the cost of higher power consumption and a full OS to manage.

The critical engineering challenge is not running the model — it is updating it. A model baked into firmware at factory time will drift as the real world changes. You need an OTA pipeline that can push model updates independently of firmware updates, validate the new model on-device before switching, and roll back if performance degrades. This is a solved problem architecturally, but getting it right requires disciplined versioning and testing infrastructure.

Cost comparison: edge vs. cloud at scale

At small scale (hundreds of devices), cloud processing is cheaper because there is no upfront hardware cost. You pay per API call, and the infrastructure scales automatically. At large scale (thousands to millions of devices), edge processing wins because the marginal cost of inference on a device you already ship is near zero, while cloud costs grow linearly with fleet size and data volume.

The crossover point depends on inference frequency. A device that runs AI once per day can afford cloud processing indefinitely — the data transfer and API costs are negligible. A device that runs inference every second generates enough data that cellular or even WiFi transfer costs alone exceed the cost of a more capable on-device chip.

A fleet of 10,000 devices running cloud inference 100 times per day at $0.001 per call costs $1,000 per day — $365,000 per year. The same fleet running edge inference costs $0 per day in API fees, with a one-time additional hardware cost of $2-5 per device for a more capable processor. The math is clear at scale, but only matters if the edge model is accurate enough for the task.

Getting started: the practical path

Start in the cloud, measure what matters, then push inference to the edge for the use cases where it makes economic or technical sense. This is not a philosophical choice — it is an engineering optimization.

First, build the model and the evaluation pipeline in the cloud. You need to know what good looks like before you can evaluate whether a smaller edge model is good enough. Run the full model on historical device data, establish accuracy baselines, and understand which features drive performance.

Second, distill or quantize the model for edge deployment. Test the smaller model against the same evaluation set. If accuracy drops below your threshold, the use case stays in the cloud. If it holds, ship the edge model and validate on real devices with a canary deployment.

Third, build the feedback loop. Edge devices send inference results (not raw data) to the cloud. The cloud monitors drift, identifies failure modes, retrains the model, and pushes updates via OTA. This loop is what turns a static edge model into a system that improves over time.

FAQ

Common questions

Can AI models run on microcontrollers?
Yes. TinyML frameworks like TensorFlow Lite Micro and Edge Impulse run inference on Cortex-M class microcontrollers with as little as 256KB of flash and 64KB of RAM. These handle classification, anomaly detection, keyword spotting, and simple object detection. More complex models (full object detection, NLP, image segmentation) need Linux-capable hardware like a Cortex-A processor or an NVIDIA Jetson.
How do you update AI models on deployed devices?
Through an OTA (over-the-air) update pipeline that can push model files independently of firmware updates. The device downloads the new model, validates it against test inputs, runs it in shadow mode alongside the current model to compare outputs, and switches over once validated. If the new model performs worse, it rolls back automatically. This requires versioned model storage and a reliable OTA infrastructure.
Is edge AI more secure than cloud AI?
Edge AI reduces the attack surface for data-in-transit because raw data stays on the device. But it increases the attack surface for the model itself — a physical device can be reverse-engineered. The security trade-off depends on your threat model. For privacy-sensitive data (health, personal, industrial IP), edge processing is generally preferred. For model IP protection, cloud processing keeps the model out of reach.
What hardware do we need for edge AI?
It depends on the model complexity. Simple classification and anomaly detection: any modern microcontroller (ESP32, STM32, nRF5340) with TinyML runtime. Object detection and time-series forecasting: a Linux SBC (Raspberry Pi 4+, BeagleBone AI) or an edge AI accelerator (Google Coral, NVIDIA Jetson Nano). Real-time video processing: NVIDIA Jetson Orin or equivalent with GPU acceleration. Start with the smallest hardware that meets your accuracy and latency requirements.

Next step

Building a connected product with on-device intelligence?

We build edge-to-cloud AI architectures for connected products — firmware inference, OTA model updates, fleet analytics. From microcontroller TinyML to full Jetson pipelines.

Book a project assessment