Info

  • IoT / Embedded Systems
  • Giantommaso Fogli

Table of contents

  1. Designing Production-Grade ESP32 IoT Fleets: MQTT, OTA, Failure Recovery and Real-World Constraints
  2. The Real Problem: Fleets, Not Devices
  3. Reference Architecture
  4. Layer 1 — Edge Nodes (ESP32)
  5. Layer 2 — Messaging (MQTT Broker)
  6. Layer 3 — Orchestration Backend (FastAPI)
  7. Layer 4 — Persistence (MariaDB)
  8. OTA Update Strategy
  9. Failure Modes in Real Deployments
  10. Network loss during state update
  11. ADC2 incompatibility with WiFi
  12. Relay logic inversion
  13. Water hammer (logic)
  14. Sensor drift (capacitive)
  15. Design Rules
  16. Common Anti-Patterns
  17. Conclusion
  18. Fonti
Designing Production-Grade ESP32 IoT Fleets: MQTT, OTA, Failure Recovery and Real-World ConstraintsDesigning Production-Grade ESP32 IoT Fleets: MQTT, OTA, Failure Recovery and Real-World Constraints

Designing Production-Grade ESP32 IoT Fleets: MQTT, OTA, Failure Recovery and Real-World Constraints

6/23/2026 / 6 minutes to read / Tags: ESP32, IoT, MQTT, NestGrow, Embedded

A single ESP32 is simple. A fleet is not.

This is the lesson we learned building NestGrow — an edge-distributed IoT irrigation control system deployed across greenhouse environments with dozens of ESP32 nodes per installation. The moment you scale beyond a handful of devices, the engineering problem stops being about hardware and becomes about fleet behavior under failure conditions.

This article documents the architecture, the failure modes we encountered, and the design principles that make a system genuinely production-grade — not just “it works on my bench”.

The Real Problem: Fleets, Not Devices

Once you scale beyond a few nodes, three problems emerge that single-device thinking cannot solve:

  • Message reliability under unstable networks — MQTT messages get lost, duplicated, or delayed
  • Firmware consistency across heterogeneous nodes — version drift is silent and dangerous
  • Partial failure states — a device that is neither fully online nor fully offline is the hardest case to handle

The key conceptual shift is this:

IoT is not about devices. It is about distributed state under uncertainty.

Reference Architecture

A production-grade ESP32 fleet consists of four layers working together.

Layer 1 — Edge Nodes (ESP32)

Each node is responsible for sensor acquisition, local actuation, MQTT communication, and OTA update reception. The critical constraint: each node must continue operating autonomously even when the network is unavailable. The backend is a policy engine, not a real-time controller.

Hardware stack used in NestGrow: ESP32 WROOM-32D, HW-316 4-channel relay (logic inverted: LOW=ON), HW-390 capacitive moisture sensors on ADC1 only (GPIO 32–35, 36 VP).

Layer 2 — Messaging (MQTT Broker)

Eclipse Mosquitto 2 handles pub/sub communication. One critical principle must be internalized early:

MQTT is a transport layer, not a system of record.

State must never be reconstructed from MQTT. The broker is ephemeral by design.

Layer 3 — Orchestration Backend (FastAPI)

The backend manages fleet state tracking, OTA coordination, rule-based policy evaluation, and health monitoring. It reconstructs authoritative state from the event stream — it does not trust device-reported state directly.

Layer 4 — Persistence (MariaDB)

All telemetry events, irrigation records, and device metadata are stored in MariaDB. The database is the only source of truth. The backend rebuilds its view of the fleet from it on startup.

OTA Update Strategy

OTA is where most IoT systems fail in production. The fundamental rule:

OTA must assume failure as the default state, not the exception.

A safe OTA flow for ESP32:

  1. Download firmware to the secondary partition (not the active one)
  2. Verify SHA-256 checksum before any partition switch
  3. Switch boot partition atomically
  4. Report firmware version to backend after successful reboot

If power is lost during step 1 or 2, the device boots from the unchanged primary partition. If lost during step 3, the ESP32 bootloader detects the incomplete switch and rolls back. No manual intervention required.

Failure Modes in Real Deployments

Network loss during state update

MQTT disconnect during telemetry send causes message loss. Mitigation: idempotent message design (every message carries a timestamp, duplicate processing is safe) and local FIFO buffer on ESP32 for reconnection sync.

ADC2 incompatibility with WiFi

This one is underdocumented and causes silent failures. On ESP32, ADC2 is hardware-multiplexed with the WiFi radio. When WiFi is active, ADC2 reads return garbage. All analog sensors must use ADC1 (GPIO 32–35, GPIO 36 VP).

Relay logic inversion

The HW-316 relay module uses inverted logic: LOW = relay ON, HIGH = relay OFF. Firmware that does not account for this will activate pumps on startup or at wrong times. Every relay command in NestGrow firmware is explicitly inverted.

Water hammer (logic)

When irrigation thresholds are set too aggressively, the backend emits commands too frequently, stressing actuators. Solution: enforce a per-zone cooldown window (10 minutes in NestGrow) at the rule engine level, regardless of sensor readings.

Sensor drift (capacitive)

Capacitive moisture sensors drift over weeks of use. Zones that read accurately at installation show +6–8% offset after 3–4 weeks. Solution: per-zone dynamic baseline calibration with weekly recalibration routines.

Design Rules

These are the architectural invariants of a stable IoT fleet. Violate them and you will eventually hit production incidents.

RuleRationale
Device state is always transientNever trust what a device reports as ground truth
Backend state is always authoritativeReconstruct from event stream, not from device
MQTT is transport, not truthBroker restart must never lose system state
Every message must be idempotentDuplicate delivery is guaranteed at scale
Failure is the normal operating modeDesign for failure, not for the happy path
OTA must survive power lossDual-partition + checksum is non-negotiable

Common Anti-Patterns

Using the device as source of truth. Leads to state drift and inconsistency at scale. The device knows what it measured — the backend decides what it means.

Assuming stable connectivity. Works on a bench. Breaks immediately in real environments with WiFi interference, power cycling, or firmware crashes.

Stateless OTA logic. Any OTA implementation that does not handle interrupted downloads as a first-class case will eventually brick a device in production.

No reconciliation layer. Without a backend process that periodically reconciles expected vs. actual device state, silent desynchronization accumulates invisibly.

Conclusion

A production ESP32 fleet is not defined by connectivity — it is defined by its ability to survive partial failure, recover deterministic state, maintain firmware consistency, and operate under unstable conditions.

The correct mental model is not “IoT devices connected to a backend”. It is:

A distributed system of unreliable nodes managed through deterministic reconstruction of state.

NestGrow is built on exactly this principle. The full source code for the ESP32 firmware is available as open source (MIT license) at github.com/jaffa2970/nestgrow-esp32. The Docker backend is available at lake8.dev/projects/nestgrow-docker.

Fonti

Espressif Systems — ESP32 Technical Reference Manual
https://www.espressif.com/en/support/documents/technical-documents
2023

OASIS Standard — MQTT Version 5.0
https://docs.oasis-open.org/mqtt/mqtt/v5.0/
2019

lake8.dev — NestGrow Project
https://lake8.dev/projects/nestgrow-docker/
2026

Diritti e attribuzioni

Immagini, loghi e fotografie citati o mostrati in questo articolo sono di proprietà dei legittimi titolari. Non si intende violare alcun diritto: i materiali sono utilizzati in relazione a fonti terze e con finalità di commento.

Rights and Attribution

Images, logos, and photographs cited or shown in this article are the property of their respective owners. No rights are intended to be infringed: materials are used in relation to third-party sources and for commentary purposes.

← Back to journal