The LoRaWAN Data Pipeline in Three Stages: Egress, Ingest, Dashboard

A familiar project milestone: devices are joined, decoding is configured, and real-time values show up on the platform. Then the customer asks one question — "Can you export the temperature curve for the last three months?" The project stalls right there.

Data landing on a platform is not the same as data being usable. This article covers the road after that moment: how data leaves the platform, which database it lands in, and how it becomes a dashboard someone actually looks at. The previous issue (FUOTA) was about "upgrading device firmware"; this one is about "not wasting the data your devices report."

Draw the boundary first. From device to dashboard, data passes through three stages:

Issue 25 covered the configuration details of egress (how to write forwardScript, which topics to subscribe to); this article connects all three stages, with the emphasis on the last two — the stages that non-developers rarely think about, and that everyone trips over the first time they build one.

1. Egress: know where responsibility transfers

ThinkLink has three data exits, mapped to three use cases.

Path one: built-in historical data. ThinkLink's historical data store is built on TimescaleDB — device telemetry lands as time series, and the web UI queries per-device history directly. Two details are easy to miss. First, the query range and the database retention policy are two independent conditions: moving the query start date earlier does not recover data already cleaned up by the retention policy. Second, historical data contains telemetry (business readings without the PARA prefix), while parameter echoes (configuration fields with PARA) rarely change — parameters only update after a configuration command is issued and answered. A flat parameter line is normal, not a broken feed.

Path two: API pull. Good for low-frequency reconciliation — a scheduled script pulls increments every hour into your own system. Fine at small scale; inefficient as polling at large scale.

Path three: the MQTT forwarder. This is the right path when volume matters. ThinkLink pushes decoded telemetry to a third-party MQTT broker, with three subscription topics matching three data shapes: decoded thing-model telemetry, decrypted raw LoRaWAN uplinks, and complete uplink messages after multi-gateway aggregation. The organization-account placeholder in the topic must be replaced with the real account — the most common first-day error in new projects.

Two forwarder rules belong in every project document:

• A forwardScript returning null drops the message. The script is a filter, not a replay buffer — null-returned messages are gone at the egress point, no cache, no replay.
• The forwarder does not guarantee delivery. Messages during a broker outage are not queued for you.

Together these mean: the egress point is the responsibility transfer point. Once a message leaves the platform, landing, buffering, and retrying are all your pipeline's job. Many data-loss incidents trace back to the assumption that "the platform will buffer for me" — when in fact nobody owns that segment.

One more note on why this is "the right path": since 2.00.066, ThinkLink removed its built-in third-party platform integrations. The architecture for external data consumption is broker plus forwarder — there is no third shortcut.

2. Ingest: calculate cardinality before choosing a database

Once data exits, where does it land? A general relational database holds up to a few hundred thousand rows, but device data has a distinctive access pattern — scan by time range, filter by device, almost never update — which is exactly the shape a time-series database optimizes for. Before choosing one, calculate a number — cardinality:

Cardinality = number of devices × metrics per device × tag combinations

100 devices × 5 metrics × 3 tag dimensions (line, zone, type) = 1,500 series. This number decides your write and query experience for the next two years, and it does so exponentially: every time series maintains its own metadata, so doubling cardinality doubles index and memory overhead.

Choose by three questions.

Question one: write throughput. LoRaWAN devices report slowly (typically one message every few minutes), but note that ingestion is never row-by-row — messages from the forwarder get batched in your pipeline. Community experience figures (hardware- and configuration-dependent; order of magnitude only): InfluxDB roughly 100k–300k points/s on one node, TimescaleDB roughly 100k–200k rows/s, TDengine roughly 500k–1M points/s. For almost any LoRaWAN project, all three exceed requirements by a wide margin — your bottleneck will almost never be database writes.

Question two: cardinality tolerance. This is the real fork among the three. InfluxDB 1.x/2.x's time-series index has a well-known weakness at high cardinality: at the million-series level, writes slow down and memory tightens; the documented ceiling sits near 30 million series, and series keys live in memory across all time — keys for devices retired years ago still occupy the index. InfluxDB 3.0 moved to a Parquet foundation and improves the situation, but the ecosystem (Flux scripts, Grafana plugins) is still migrating. TimescaleDB rides PostgreSQL's B-trees, handles high cardinality well, and lets you JOIN sensor data with business tables in plain SQL. TDengine uses a super-table plus sub-table model — one sub-table per device — that fits IoT naturally, with high compression as a bonus.

Question three: team and ecosystem. Team speaks SQL and already runs PostgreSQL — TimescaleDB is near-zero migration cost. Sovereign-stack requirements or extreme compression — TDengine. Just monitoring curves — InfluxDB works, but keep the device count in mind; past a few thousand devices, evaluate cardinality seriously.

The most expensive data-model mistake: putting mutable values into tags. The design rule is one sentence — dimensions you filter by go into tags (device ID, zone, line); values that change over time go into fields (temperature, battery, RSSI). People stuff humidity into tags to save a schema decision; a few thousand devices later, series counts explode past a million, queries degrade, and the fix is rebuilding the database. Tags are index keys; fields are the data itself. Swap them and you build your own cardinality bomb.

Two mandatory operational items: batched writes (batch size around 5,000, timeout 200 ms, gzip on — writing one point per request is the most common self-inflicted wound), and retention plus downsampling (keep raw data 90 days, then roll it up to hourly averages, minima and maxima — old data shrinks from one row per minute to one per hour, cutting storage cost by an order of magnitude. Skip downsampling and both query latency and the disk bill will teach you the lesson three months in).

3. Dashboard: Grafana is not the only answer, but the division of labor matters

Once data is in the database, the dashboard layer has three options in ascending cost.

The platform's own pages. Per-device history curves, live values, alert states — already built. For device debugging and ops views of single points, this is the fastest path; do not rebuild it.

Grafana. Mature data-source plugins for all three databases; connect and draw. The right tool for cross-device business views — average temperature along a production line, battery distribution across a campus, daily water-consumption trends for a group of meters. Grafana's value lies in template variables (a dropdown to switch devices) and alert panels, not in per-device curves.

A custom frontend. Only worth it when the dashboard goes to the end customer's operations staff and must embed into their business system. Budget from "one experienced frontend developer for two months"; most real requirements are covered by Grafana in a week.

One commonly missed division: do not duplicate alerting between the platform and the dashboard. Platform-side trigger rules (issue 22) own device-level real-time alerts — threshold crossed, push to email and enterprise messaging. Grafana alerts own business-level trend alerts — "line 3's weekly average temperature has been climbing for two weeks," the kind of judgment that needs an aggregate query. Build the same threshold alert in both, and ops will receive duplicate notifications and learn to ignore them.

4. Reliability: where data actually gets lost

Across the three stages, there are three gaps.

Gap one: the device side. Failed uplinks are retried by the device (confirmed uplinks), covered by the protocol. But Class A devices depend on receive windows, and loss on weak-signal devices belongs in the troubleshooting funnel from issue 28.

Gap two: the egress point. As established, the forwarder does not buffer. Mitigation: put a broker with persistent sessions in the egress path (EMQX with persistent sessions and QoS 1), so messages accumulate in the broker while the ingestion side is offline.

Gap three: the ingestion side. Write retries need an upper bound and dead-letter handling — one malformed record crashing the consumer in a loop hurts far more than a few lost points.

For small projects (a few dozen devices, dashboards for internal ops only), the whole chain can stay minimal: forwarder straight into a single-node InfluxDB, one Grafana board, no buffering, no dual writes — an hour of work. But the moment data is part of a customer deliverable or feeds reconciliation, gaps two and three must be closed — data lost there is data customers will ask about by contract.

5. Boundaries and decision checklist

Build the pipeline (egress → time-series DB → dashboard) when: data is viewed by customers or building owners; you need cross-device aggregation; compliance or audit requires data inside your own domain; the platform's views cannot express your business logic.

The platform's pages suffice when: you are debugging and operating single devices; device count is small with no external deliverable; nobody on the team can maintain a database — self-hosting a time-series DB is not free; backups, upgrades, and disk monitoring are ongoing costs.

Three don'ts: don't put mutable values in tags; don't write point-by-point; don't duplicate the same alert class on both platform and dashboard.

Back to the opening question: "Can you export the last three months of temperature?" — the answer was decided by how the retention policy was configured on day one, not by whatever tool you use to dig afterward. A data pipeline is a day-one design. Retrofitting one recovers nothing but lessons.

---

This is issue 35 of the LoRaWAN series. Egress configuration details: issue 25 (MQTT forwarder). Platform alerting: issue 22. Data-quality troubleshooting: issue 28. Third-party performance figures come from community benchmarks and vendor documentation; treat magnitudes as selection guidance and run a POC on your own data shape before production. This article involves no USD amounts and no currency conversion.