**AWS Dual-Path IoT Architecture: Real-Time and Batch Processing at Scale**
Amazon Web Services (AWS) has unveiled a detailed reference architecture designed to handle massive IoT fleets, combining NB-IoT and LoRaWAN ingestion with a dual-path approach to telemetry processing. The architecture, outlined in a recent AWS blog post, is built to support device fleets numbering in the millions by splitting data into two distinct pipelines: one optimized for immediate anomaly detection and the other tailored for historical batch analysis.
The design illustrates how organizations can manage competing priorities between real-time monitoring and long-term analytics, addressing common budget and resource tensions. Two example dashboards—a solar farm monitor and a smart building query tool—are provided as starting points. AWS emphasizes that these are illustrative samples, not customer case studies, signaling that the architecture may require adaptation before direct production deployment.
Underlying the proposal are efficiency claims drawn from McKinsey research, which AWS cites as evidence that digital maintenance and reliability transformations can increase asset availability by 5–15 percent and reduce maintenance costs by 18–25 percent. These figures underscore the financial incentive for enterprises to separate and optimize real-time and historical data workflows.
**Cellular and Radio Ingestion: Handling Diverse Operational Costs**
The ingestion layer of the architecture is designed to accommodate two types of devices with different connectivity needs and operational characteristics. NB-IoT devices typically connect via cellular carriers or connectivity partners, bridging radio networks to AWS IoT Core using lightweight protocols such as UDP or CoAP. A data broker handles protocol translation, making this pathway suitable for environments where cellular radio performance is superior, such as underground installations, dense urban cores, or remote areas with limited connectivity options.
In contrast, LoRaWAN devices rely on gateways running LoRa Basics Station packet forwarders to communicate with AWS IoT Core for LoRaWAN, AWS’s managed network server. Payloads arrive as base64-encoded binary and are routed through an IoT Rule, where a Lambda function triggers a decoder that converts the data into JSON format. While this approach broadens connectivity options, AWS notes a significant caveat: different device models often encode their payloads differently. As a result, fleets with diverse sensor types may require a corresponding number of custom decoder functions, increasing engineering overhead in proportion to device diversity rather than sheer device count.
**One Rule Engine, Two Destinations: Splitting Data Streams**
At the center of the architecture is the AWS IoT Core rules engine, which directs incoming telemetry toward two separate destinations. On one side, Amazon Data Firehose handles batch delivery for historical analysis. On the other, Amazon Kinesis Data Streams supports real-time processing. AWS illustrates the difference with a temperature sensor example: a reading that exceeds a threshold can immediately trigger a cooling system response through the real-time lane, while the same sensor’s aggregated data over time can inform long-term maintenance planning through the batch lane.
The streaming path channels data from Kinesis into Amazon Managed Service for Apache Flink, where stateful processing and anomaly detection occur. From there, data flows into Amazon Timestream, which AWS says delivers single-digit millisecond query response times for time-series workloads. However, AWS acknowledges a current limitation: Flink does not yet include a native sink connector for InfluxDB. As a result, organizations building this pipeline must develop and maintain a custom sink to interface with the InfluxDB v3 write API.
The batch path follows a three-tier structure within Amazon S3. Firehose delivers raw data to the first bucket unmodified. AWS Glue workflows then clean and standardize the data in a second tier, while Glue Crawlers catalog it in the AWS Glue Data Catalog to enable discovery by downstream tools. Finally, additional Glue ETL jobs generate business-level aggregations in a third bucket. This structure provides data lineage, improves query performance, and leverages serverless components to eliminate the need for capacity planning ahead of demand.
**Natural Language Queries Depend on Catalog Quality**
The architecture’s conversational interface relies on Amazon Bedrock AgentCore, which hosts Strands agents that translate plain-language questions into SQL queries for Amazon Athena. Example prompts, such as “Show me all devices with temperature above 80 degrees in the last hour,” are automatically converted into executable SQL statements.
Underlying this capability is Amazon Titan Text Embeddings, which semantically interpret Glue Data Catalog metadata, while Retrieval-Augmented Generation via Bedrock Knowledge Bases helps ensure query accuracy. A Lambda-based feedback loop, accessible through Bedrock AgentCore Gateway, allows failed queries to be refined and prompts users for clarification when necessary.
AWS does not disclose accuracy metrics for the query translation process. This is a relevant consideration, since IoT Data Catalogs in operational environments often lag behind the introduction of new device types. As a result, semantic interpretations may be based on incomplete or outdated information, potentially undermining query reliability. The architecture recommends using the AWS Well-Architected Framework and the IoT Lens to guide decisions around connectivity, processing, and scalability, and suggests adopting the design incrementally rather than all at once.
**See also**: Anthropic and Nozomi bring AI vulnerability research to OT security
—
### FAQ
**What is the purpose of AWS’s dual-path IoT architecture?**
The architecture is designed to process telemetry from large IoT fleets by separating data streams into real-time anomaly detection and historical batch analysis. This enables organizations to handle immediate operational needs while also supporting long-term maintenance and planning.
**Which communication protocols and technologies are involved?**
The architecture supports NB-IoT over cellular networks and LoRaWAN over gateway networks, both connecting to AWS IoT Core. It uses AWS services such as IoT Core rules, Lambda, Kinesis, Firehose, Glue, S3, Timestream, Flink, Athena, Bedrock AgentCore, and Titan Text Embeddings.
**Why does the architecture split data into two paths?**
Splitting data allows instant responses to critical conditions, such as overheating equipment, while simultaneously capturing trends and patterns over time for scheduling maintenance and capacity planning. This separation prevents conflicts between real-time and analytical priorities.
**What are the challenges related to device diversity?**
Different sensor models often encode payloads differently. For LoRaWAN and similar formats, each device type may require a custom decoder function, increasing engineering effort as fleet diversity grows rather than simply scaling with device count.
**What limitation exists with Apache Flink in this design?**
Flink currently lacks a native sink connector for InfluxDB, meaning organizations must create and maintain a custom integration to write data into InfluxDB v3.
**How are natural language queries processed?**
Queries entered in plain English are translated into SQL by Strands agents running on Bedrock AgentCore. Embeddings and knowledge bases help interpret catalog metadata and improve query accuracy, while a feedback loop allows refinement of unsuccessful queries.
**What risks are associated with relying on the Data Catalog for semantic queries?**
Because IoT Data Catalogs often lag behind the addition of new devices, the semantic understanding used by query agents may be based on outdated or incomplete information, potentially leading to inaccurate results.
—
### Conclusion
AWS’s dual-path IoT reference architecture offers a structured approach for handling telemetry at scale by clearly distinguishing between real-time responsiveness and historical insight. While the design highlights significant benefits in availability and cost savings, it also exposes practical challenges around device heterogeneity, custom integration needs, and catalog freshness. Organizations considering this architecture should weigh its strengths against implementation complexity and are encouraged to adopt it incrementally, guided by AWS best-practice frameworks. With careful planning and adaptation, the architecture can serve as a robust foundation for large-scale IoT deployments requiring both immediate action and long-term analysis.
