A useful IoT demo does not stop when a sensor value appears in a terminal. A maintainable pipeline needs to identify each device, carry measurements across unreliable networks, reveal stale or missing data and preserve clear boundaries between broker acknowledgement and durable application storage. Keep the first version small, but make its failure states visible.
ESP-IDF's ESP-MQTT component implements an MQTT client; the broker distributes publications to subscribers. Neither component automatically gives you a complete data pipeline. The ingestion service should validate payloads and commit records to a database before downstream dashboards treat them as durable observations.
Give every device a stable, unique client ID and credentials scoped to that device. A simple topic family could be site/{siteId}/device/{deviceId}/telemetry, .../state, and .../command. Restrict publish and subscribe permissions separately: a sensor usually publishes telemetry and subscribes only to its own command path. Never put secrets or personally identifying information in topic names.
Choose a compact, versioned payload with a measurement timestamp, monotonically increasing sequence, firmware version, units and quality flags. Include a schema version so ingestion can evolve without guessing. Use server receipt time as a separate field; device clocks may be wrong, drift or reset.
MQTT QoS 0 is at-most-once; QoS 1 is at-least-once and can produce duplicates; QoS 2 adds a protocol exchange for exactly-once delivery between MQTT peers. QoS does not mean exactly-once database insertion or dashboard processing. For most telemetry, QoS 1 plus an idempotent ingestion key such as deviceId + sequence is a practical balance. Measure whether the added handshake or queueing of another level is worth its cost.
A successful client publish event or broker acknowledgement means the MQTT exchange reached its defined stage, not that your SQL transaction committed. If durable business processing matters, let the consumer acknowledge its own persisted event or expose an ingestion receipt. On reconnect, replay queued readings with their original IDs so deduplication remains possible.
A retained message is the broker's latest value for a topic, delivered to matching future subscribers. This suits a compact current-state document or availability flag, not an unbounded history stream. A Last Will can publish offline status after an unexpected disconnect; pair it with an online announcement and a timestamp/expiry policy so old state does not masquerade as current. MQTT 5 session and message expiry are distinct from retained-message lifecycle.
Wi-Fi loss is normal. Bound the device queue by bytes and age, choose which measurements may be coalesced, and define what happens when capacity is exhausted. Keep critical alarms separate from routine samples if they have different latency needs. Use exponential backoff with jitter and a maximum delay to avoid a whole fleet reconnecting at once after a broker outage. Persist only what the product's loss budget requires; flash writes and storage space are finite.
| Signal | What it reveals | Useful alert condition |
|---|---|---|
| Last sample age | Whether a node is reporting fresh data | Age exceeds its reporting contract |
| Sequence gaps | Loss between device creation and ingestion | Gap rate rises above baseline |
| Broker reconnects | Network or authentication instability | Reconnect bursts or repeated auth failure |
| Queue depth and oldest age | Backlog and recovery capacity | Queue approaches its configured limit |
| Ingest rejects | Schema, authorization or data-quality errors | Unexpected increase by firmware cohort |
Use TLS with certificate validation; encryption without authenticating the broker leaves a device vulnerable to connecting to the wrong endpoint. Protect credentials at rest, provision unique identities, rotate or revoke them, and avoid logging tokens. Broker ACLs should deny by default. Commands need authorization, freshness checks and safe device-side validation; a dashboard login alone does not secure the device channel.
Start with fleet inventory, last-seen time, firmware cohort, battery or signal quality where available, sequence gaps and ingest errors. A time-series chart should label units and show missing intervals instead of drawing a misleading continuous line. Add a per-device trace from sample ID through broker receipt to database row. These few views answer whether a sensor is quiet, a network is degraded or the backend is dropping valid messages.
The smallest useful MQTT observability pipeline has device identity, a versioned event, encrypted transport, bounded offline behavior, authorized topics, idempotent ingestion and freshness-focused dashboards. Pick QoS for the MQTT hop, then separately prove storage and processing. That distinction turns a sensor demo into an operable system.