|
| 1 | +--- |
| 2 | +title: Azure IoT reference architecture |
| 3 | +description: Recommended architecture for IoT applications on Azure using PaaS (platform-as-a-service) components |
| 4 | +titleSuffix: Azure Reference Architectures |
| 5 | +author: MikeWasson |
| 6 | +ms.date: 01/09/2019 |
| 7 | +--- |
| 8 | + |
| 9 | +# Azure IoT reference architecture |
| 10 | + |
| 11 | +This reference architecture shows a recommended architecture for IoT applications on Azure using PaaS (platform-as-a-service) components. |
| 12 | + |
| 13 | + |
| 14 | + |
| 15 | +IoT applications can be described as **things** (devices) sending data that generates **insights**. These insights generate **actions** to improve a business or process. An example is an engine (the thing) sending temperature data. This data is used to evaluate whether the engine is performing as expected (the insight). The insight is used to proactively prioritize the maintenance schedule for the engine (the action). |
| 16 | + |
| 17 | +This reference architecture uses Azure PaaS (platform-as-a-service) components. Other options for building IoT solutions on Azure include: |
| 18 | + |
| 19 | +- [Azure IoT Central](/azure/iot-central/). IoT Central is a fully managed SaaS (software-as-a-service) solution. It abstracts the technical choices and lets you focus on your solution exclusively. This simplicity comes with a tradeoff in being less customizable than a PaaS-based solution. |
| 20 | +- Using OSS components such as the SMACK stack (Spark, Mesos, Akka, Cassandra, Kafka) deployed on Azure VMs. This approach offers a great deal of control but is more complex. |
| 21 | + |
| 22 | +At a high level, there are two ways to process telemetry data, hot path and cold path. The difference has to do with requirements for latency and data access. |
| 23 | + |
| 24 | +- The **hot path** analyzes data in near-real-time, as it arrives. In the hot path, telemetry must be processed with very low latency. The hot path is typically implemented using a stream processing engine. The output may trigger an alert, or be written to a structured format that can be queried using analytical tools. |
| 25 | +- The **cold path** performs batch processing at longer intervals (hourly or daily). The cold path typically operates over large volumes of data, but the results don't need to be as timely as the hot path. In the cold path, raw telemetry is captured and then fed into a batch process. |
| 26 | + |
| 27 | +## Architecture |
| 28 | + |
| 29 | +This architecture consists of the following components. Some applications may not require every component listed here. |
| 30 | + |
| 31 | +**IoT devices**. Devices can securely register with the cloud, and can connect to the cloud to send and receive data. Some devices may be **edge devices** that perform some data processing on the device itself or in a field gateway. We recommend [Azure IoT Edge](/azure/iot-edge/) for edge processing. |
| 32 | + |
| 33 | +**Cloud gateway**. A cloud gateway provides a cloud hub for devices to connect securely to the cloud and send data. It also provides device management, capabilities, including command and control of devices. For the cloud gateway, we recommend [IoT Hub](/azure/iot-hub/). IoT Hub is a hosted cloud service that ingests events from devices, acting as a message broken between devices and backend services. IoT Hub provides secure connectivity, event ingestion, bidirectional communication, and device management. |
| 34 | + |
| 35 | +**Device provisioning.** For registering and connecting large sets of devices, we recommend using the [IoT Hub Device Provisioning Service](/azure/iot-dps/) (DPS). DPS lets you assign and register devices to specific Azure IoT Hub endpoints at scale. |
| 36 | + |
| 37 | +**Stream processing**. Stream processing analyzes large streams of data records and evaluates rules for those streams. For stream processing, we recommend [Azure Stream Analytics](/azure/stream-analytics/). Stream Analytics can execute complex analysis at scale, using time windowing functions, stream aggregations, and external data source joins. Another option is Apache Spark on [Azure Databricks](/azure/azure-databricks/). |
| 38 | + |
| 39 | +Machine learning allows predictive algorithms to be executed over historical telemetry data, enabling scenarios such as predictive maintenance. For machine learning, we recommend [Azure Machine Learning Service](/azure/machine-learning/service/). |
| 40 | + |
| 41 | +**Warm path storage** holds data that must be available immediately from device for reporting and visualization. For warm path storage, we recommend [Cosmos DB](/azure/cosmos-db/introduction). Cosmos DB is a globally distributed, multi-model database. |
| 42 | + |
| 43 | +**Cold path storage** holds data that is kept longer term and is used for batch processing. For cold path storage, we recommend [Azure Blob Storage](/azure/storage/blobs/storage-blobs-introduction). Data can be archived in Blob storage indefinitely at low cost, and is easily accessible for batch processing. |
| 44 | + |
| 45 | +**Data transformation** manipulates or aggregates the telemetry stream. Examples include protocol transformation, such as converting binary data to JSON, or combining data points. If the data must be transformed before reaching IoT Hub, we recommend using a [protocol gateway](/azure/iot-hub/iot-hub-protocol-gateway) (not shown). Otherwise, data can be transformed after it reaches IoT Hub. In that case, we recommend using [Azure Functions](/azure/azure-functions/). Functions has built-in integration with IoT Hub, Cosmos DB, and Blob Storage. |
| 46 | + |
| 47 | +**Business process integration** performs actions based on insights from the device data. This could include storing informational messages, raising alarms, sending email or SMS messages, or integrating with CRM. We recommend using [Azure Logic Apps](/azure/logic-apps/logic-apps-overview) for business process integration. |
| 48 | + |
| 49 | +**User management** restricts which users or groups can perform actions on devices, such as upgrading firmware. It also defines capabilities for users in applications. We recommend using [Azure Active Directory](/azure/active-directory/) to authenticate and authorize users. |
| 50 | + |
| 51 | +## Scalability considerations |
| 52 | + |
| 53 | +An IoT application should be built as discrete services that can scale independently. Consider the following scalability points: |
| 54 | + |
| 55 | +**IoTHub**. For IoT Hub, consider the following scale factors: |
| 56 | + |
| 57 | +- The maximum [daily quota](/azure/iot-hub/iot-hub-devguide-quotas-throttling) of messages into IoT Hub. |
| 58 | +- The quota of connected devices in an IoT Hub instance. |
| 59 | +- Ingestion throughput — how quickly IoT Hub can ingest messages. |
| 60 | +- Processing throughput — how quickly the incoming messages are processed. |
| 61 | + |
| 62 | +Each IoT hub is provisioned with a certain number of units in a specific tier. The tier and number of units determine the maximum daily quota of messages that devices can send to the hub. For more information, see IoT Hub quotas and throttling. You can scale up a hub without interrupting existing operations. |
| 63 | + |
| 64 | +**Stream Analytics**. Stream Analytics jobs scale best if they are parallel at all points in the Stream Analytics pipeline, from input to query to output. A fully parallel job allows Stream Analytics to split the work across multiple compute nodes. Otherwise, Stream Analytics has to combine the stream data into one place. For more information, see [Leverage query parallelization in Azure Stream Analytics](/azure/stream-analytics/stream-analytics-parallelization). |
| 65 | + |
| 66 | +IoT Hub automatically partitions device messages based on the device ID. All of the messages from a particular device will always arrive on the same partition, but a single partition will have messages from multiple devices. Therefore, the unit of parallelization is the partition ID. |
| 67 | + |
| 68 | +**Functions**. When reading from the Event Hubs endpoint, there is a maximum of function instance per event hub partition. The maximum processing rate is determined by how fast one function instance can process the events from a single partition. The function should process messages in batches. |
| 69 | + |
| 70 | +**Cosmos DB**. To scale out a Cosmos DB collection, create the collection with a partition key and include the partition key in each document that you write. For more information, see [Best practices when choosing a partition key](/azure/cosmos-db/partition-data#best-practices-when-choosing-a-partition-key). |
| 71 | + |
| 72 | +- If you store and update a single document per device, the device ID is a good partition key. Writes are evenly distributed across the keys. The size of each partition is strictly bounded, because there is a single document for each key value. |
| 73 | +- If you store a separate document for every device message, using the device ID as a partition key would quickly exceed the 10-GB limit per partition. Message ID is a better partition key in that case. Typically you would still include device ID in the document for indexing and querying. |
| 74 | + |
| 75 | +## Security considerations |
| 76 | + |
| 77 | +### Trustworthy and secure communication |
| 78 | + |
| 79 | +All information received from and sent to a device must be trustworthy. Unless a device can support the following cryptographic capabilities, it should be constrained to local networks and all internetwork communication should go through a field gateway: |
| 80 | + |
| 81 | +- Data encryption with a provably secure, publicly analyzed, and broadly implemented symmetric-key encryption algorithm. |
| 82 | +- Digital signature with a provably secure, publicly analyzed, and broadly implemented symmetric-key signature algorithm. |
| 83 | +- Support for either TLS 1.2 for TCP or other stream-based communication paths or DTLS 1.2 for datagram-based communication paths. Support of X.509 certificate handling is optional and can be replaced by the more compute-efficient and wire-efficient pre-shared key mode for TLS, which can be implemented with support for the AES and SHA-2 algorithms. |
| 84 | +- Updateable key-store and per-device keys. Each device must have unique key material or tokens that identify it toward the system. The devices should store the key securely on the device (for example, using a secure key-store). The device should be able to update the keys or tokens periodically, or reactively in emergency situations such as a system breach. |
| 85 | +- The firmware and application software on the device must allow for updates to enable the repair of discovered security vulnerabilities. |
| 86 | + |
| 87 | +However, many devices are too constrained to support these requirements. In that case, a field gateway should be used. Devices connect securely to the field gateway through a local area network, and the gateway enables secure communication to the cloud. |
| 88 | + |
| 89 | +### Physical tamper proofing |
| 90 | + |
| 91 | +It is strongly recommended that device design incorporates features that defend against physical manipulation attempts, to help ensure the security integrity and trustworthiness of the overall system. |
| 92 | + |
| 93 | +For example: |
| 94 | + |
| 95 | +- Choose microcontrollers/microprocessors or auxiliary hardware that provide secure storage and use of cryptographic key material, such as trusted platform module (TPM) integration. |
| 96 | +- Secure boot loader and secure software loading, anchored in the TPM. |
| 97 | +- Use sensors to detect intrusion attempts and attempts to manipulate the device environment with alerting and potentially “digital self-destruction” of the device. |
| 98 | + |
| 99 | +For additional security considerations, see [Internet of Things (IoT) security architecture](/azure/iot-fundamentals/iot-security-architecture). |
| 100 | + |
| 101 | +### Monitoring and logging |
| 102 | + |
| 103 | +Logging and monitoring systems are used to determine whether the solution is functioning and to help troubleshoot problems. Monitoring and logging systems help answer the following operational questions: |
| 104 | + |
| 105 | +- Are devices or systems in an error condition? |
| 106 | +- Are devices or systems correctly configured? |
| 107 | +- Are devices or systems generating accurate data? |
| 108 | +- Are systems meeting the expectations of both the business and end customers? |
| 109 | + |
| 110 | +Logging and monitoring tools are typically comprised of the following four components: |
| 111 | + |
| 112 | +- System performance and timeline visualization tools to monitor the system and for basic troubleshooting. |
| 113 | +- Buffered data ingestion, to buffer log data. |
| 114 | +- Persistence store to store log data. |
| 115 | +- Search and query capabilities, to view log data for use in detailed troubleshooting. |
| 116 | + |
| 117 | +Monitoring systems provide insights into the health, security, and stability, and performance of an IoT solution. These systems can also provide a more detailed view, recording component configuration changes and providing extracted logging data that can surface potential security vulnerabilities, enhance the incident management process, and help the owner of the system troubleshoot problems. Comprehensive monitoring solutions include the ability to query information for specific subsystems or aggregating across multiple subsystems. |
| 118 | + |
| 119 | +Monitoring system development should begin by defining healthy operation, regulatory compliance, and audit requirements. Metrics collected may include: |
| 120 | + |
| 121 | +- Physical devices, edge devices, and infrastructure components reporting configuration changes. |
| 122 | +- Applications reporting configuration changes, security audit logs, request rates, response times, error rates, and garbage collection statistics for managed languages. |
| 123 | +- Databases, persistence stores, and caches reporting query and write performance, schema changes, security audit log, locks or deadlocks, index performance, CPU, memory, and disk usage. |
| 124 | +- Managed services (IaaS, PaaS, SaaS, and FaaS) reporting health metrics and configuration changes that impact dependent system health and performance. |
| 125 | + |
| 126 | +Visualization of monitoring metrics alert operators to system instabilities and facilitate incident response. |
| 127 | + |
| 128 | +### Tracing telemetry |
| 129 | + |
| 130 | +Tracing telemetry allows an operator to follow the journey of a piece of telemetry from creation through the system. Tracing is important for debugging and troubleshooting. For IoT solutions that use Azure IoT Hub and the [IoT Hub Device SDKs](/azure/iot-hub/iot-hub-devguide-sdks), tracing datagrams can be originated as Cloud-to-Device messages and included in the telemetry stream. |
| 131 | + |
| 132 | +### Logging |
| 133 | + |
| 134 | +Logging systems are integral in understanding what actions or activities a solution has performed, failures that have occurred, and can provide help in fixing those failures. Logs can be analyzed to help understand and remedy error conditions, enhance performance characteristics, and ensure compliance with governing rule and regulations. |
| 135 | + |
| 136 | +Though plain-text logging is lower impact on upfront development costs, it is more challenging for a machine to parse/read. We recommend structured logging be used, as collected information is both machine parsable and human readable. Structured logging adds situational context and metadata to the log information. In structured logging, properties are first class citizens formatted as key/value pairs, or with a fixed schema, to enhance search and query capabilities. |
| 137 | + |
| 138 | +## Next steps |
| 139 | + |
| 140 | +- For a more detailed discussion of the recommended architecture and implementation choices, see [Microsoft Azure IoT Reference Architecture](http://aka.ms/iotrefarchitecture) (PDF). |
| 141 | + |
| 142 | +- For detailed documentation of the various Azure IoT services, see [Azure IoT Fundamentals](/azure/iot-fundamentals/). |
0 commit comments