Unlock the potential of the Internet of Things (IoT) with this cutting-edge Event Streaming Architecture. Leveraging the power of Apache Kafka and MQTT, this project is designed to handle IoT data at scale. 🚀
Ingest real-time data from IoT sensors across diverse locations, analyze metrics like temperature, humidity, pressure, and luminosity, and store them efficiently in a high-performance MongoDB database. 🌡️💧📈
The technology stack includes Spring Boot, Kafka Streams, Micrometer, and Grafana for real-time monitoring, making this architecture a powerhouse for IoT solutions. 💻📈🔍
With Docker containerization, deployment is a breeze. Explore this project, visualize IoT data, and gain insights into your sensor networks. 🐳🚀📊
Check out the detailed Medium article for a comprehensive overview. 📖
Thank you for visiting the Smart IoT Event Streaming GitHub repository! Empower your IoT endeavors and scale with confidence. 🌐📈💡
Project developed to practice what I have learned in the Udemy course Apache Kafka Series - Kafka Connect Hands-on Learning and Apache Kafka Series - Kafka Streams for Data Processing.
This project, QuantumFlow IoT, has been developed for educational purposes only. Its primary objective is to provide a hands-on learning experience for building scalable event streaming architectures and processing real-time IoT data using technologies like Apache Kafka, MQTT, MongoDB, Grafana, and Spring Boot.
This repository demonstrates how to ingest, process, and store data from IoT sensors, and it does not provide access to real-world IoT data or production-grade sensor networks. The project uses simulated data for demonstration purposes, allowing users to explore the design and architecture of IoT solutions without the need for live sensor deployments.
The name QuantumFlow IoT was generated during a brainstorming session with the help of AI, and any resemblance to existing brands or services is purely coincidental.
The logo of QuantumFlow IoT was generated by ChatGPT and is free of copyright restrictions. It is provided exclusively for demonstration and educational purposes and does not represent an actual brand or commercial entity.
The main goals of this architecture are the following:
- Ingest and store real-time data from IoT sensors located in various locations.
- Analyze and make aggregations through rotating time windows to know average temperature by sensor or place, in addition to humidity, pressure and luminosity.
- Store data after processing for subsequent monitoring in a NOSQL database with good performance for frequent writes for relatively homogeneous document sizes such as MongoDB.
- Visualization of the metrics of the sensors in real time and of the performance of the components of the architecture through Grafana and the consolidated documents in MongoDB through Mongo-Express.
- Spring Boot 2.3.3 / Apache Maven 3.6.3.
- Spring Boot Starter Actuator.
- Kafka Streams.
- Spring Kafka.
- Micrometer Registry Prometheus.
- Eclipse Paho MQTT Client.
- Kafka Connect.
- Kafka Rest Proxy
- lombok.
- Jackson.
- NodeExporter (Exporter for machine metrics).
- Prometheus.
- Grafana.
- Eclipse Mosquitto.
- MongoDB.
- Mongo DB Express (Web-based MongoDB admin interface, written with Node.js and express).
- Cadvisor (Analyzes resource usage and performance characteristics of running containers).
- kafka-exporter (Kafka exporter for Prometheus).
The available tasks are detailed below (rake --task)
| Task | Description |
|---|---|
| check_deployment_file_task | Check Deployment File |
| check_docker_task | Check Docker and Docker Compose Task |
| cleaning_environment_task | Cleaning Evironment Task |
| deploy | Deploys the IoT Event Streaming Architecture and laun... |
| login | Authenticating with existing credentials |
| start | Start Containers |
| status | Status Containers |
| stop | Stop Containers |
| undeploy | UnDeploy IoT Event Streaming Architecture |
To start the platform make sure you have Ruby installed, go to the root directory of the project and run the rake deploy task, this task will carry out a series of preliminary checks, discard images and volumes that are no longer necessary and also proceed to download all the images and the initialization of the containers.
| Container | Port |
|---|---|
| kafka-topics-ui | localhost:8081 |
| kafka-connect-ui | localhost:8082 |
| zoonavigator-web | localhost:8083 |
| mongo-express | localhost:8084 |
| grafana | localhost:8085 |
| prometheus | localhost:8086 |
| kafka-rest-proxy | localhost:9999 |
This project, QuantumFlow IoT, has been developed for educational purposes only. Its primary objective is to provide a hands-on learning experience for building scalable event streaming architectures and processing real-time IoT data using technologies like Apache Kafka, MQTT, MongoDB, Grafana, and Spring Boot.
This repository demonstrates how to ingest, process, and store data from IoT sensors, and it does not provide access to real-world IoT data or production-grade sensor networks. The project uses simulated data for demonstration purposes, allowing users to explore the design and architecture of IoT solutions without the need for live sensor deployments.
The name QuantumFlow IoT was generated during a brainstorming session with the help of AI, and any resemblance to existing brands or services is purely coincidental.
The logo of QuantumFlow IoT was generated by ChatGPT and is free of copyright restrictions. It is provided exclusively for demonstration and educational purposes and does not represent an actual brand or commercial entity.
