feat(microservices): add LiDAR, Weight, Peripheral Analytics, and Grafana integration

[naman-ranka]

Introduces a single common-service for managing both LiDAR and Weight sensors, configurable through environment variables for MQTT, Kafka, and HTTP publishing. Integrates Grafana (with a pre-loaded dashboard and data source) for real-time visualization over MQTT.
Closes #537

PR Checklist

• Added label to the Pull Request for easier discoverability and search
• Commit Message meets guidelines as indicated in the URL https://github.com/intel-retail/automated-self-checkout/blob/main/CONTRIBUTING.md
• Every commit is a single defect fix and does not mix feature addition or changes
• Unit Tests have been added for new changes
• Updated Documentation as relevant to the changes
• All commented code has been removed
• If you've added a dependency, you've ensured license is compatible with repository license and clearly outlined the added dependency.
• PR change contains code related to security
• PR introduces changes that breaks compatibility with other modules (If YES, please provide details below)

What are you changing?

This PR introduces the following features:

1. Overview

1. Common-Service

• LiDAR & Weight Sensors handled within one container (common-service).

• Each sensor has its own configuration (e.g., sensor ID, port, mock mode).

• Publishes data to one or more protocols:

  • MQTT (e.g., broker host, port, topic)

  • Kafka (e.g., bootstrap server, topic)

  • HTTP (e.g., URL endpoint)

• Uses a shared framework (publisher.py & config.py) plus two apps (lidar_app.py, weight_app.py).

2. Environment Variables & Configuration

• Fully controlled via Fully controlled via docker-compose.yml .

• Example snippet for LiDAR (2 sensors) & Weight (2 sensors):

environment:
  # LiDAR environment variables
  LIDAR_COUNT: 2
  LIDAR_SENSOR_ID_1: lidar-001
  LIDAR_SENSOR_ID_2: lidar-002
  LIDAR_MOCK_1: "true"
  LIDAR_MOCK_2: "true"
  LIDAR_PORT_1: /dev/ttyUSB0
  LIDAR_PORT_2: /dev/ttyUSB1
  LIDAR_MQTT_ENABLE: "true"
  LIDAR_MQTT_BROKER_HOST: mqtt-broker_1
  LIDAR_MQTT_BROKER_PORT: 1883
  LIDAR_KAFKA_ENABLE: "true"
  KAFKA_BOOTSTRAP_SERVERS: kafka:9093
  LIDAR_KAFKA_TOPIC: "lidar-data"
  LIDAR_MQTT_TOPIC: "lidar/data"
  LIDAR_HTTP_ENABLE: "true"
  LIDAR_HTTP_URL: "http://localhost:5000/api/lidar_data"
  LIDAR_PUBLISH_INTERVAL: 1.0
  LIDAR_LOG_LEVEL: INFO

  # Weight environment variables
  WEIGHT_COUNT: 2
  WEIGHT_SENSOR_ID_1: weight-001
  WEIGHT_SENSOR_ID_2: weight-002
  WEIGHT_MOCK_1: "true"
  WEIGHT_MOCK_2: "true"
  WEIGHT_PORT_1: /dev/ttyUSB2
  WEIGHT_PORT_2: /dev/ttyUSB3
  WEIGHT_MQTT_ENABLE: "true"
  WEIGHT_MQTT_BROKER_HOST: mqtt-broker_1
  WEIGHT_MQTT_BROKER_PORT: 1883
  WEIGHT_KAFKA_ENABLE: "false"
  WEIGHT_MQTT_TOPIC: "weight/data"
  WEIGHT_HTTP_ENABLE: "false"
  WEIGHT_PUBLISH_INTERVAL: 1.0
  WEIGHT_LOG_LEVEL: INFO

• Change true/false to enable or disable each protocol.

• Adjust intervals, logging levels, or sensor counts as needed.

3. Kafka & Zookeeper

• Included in the docker-compose.yml file.

• By default, Kafka listens on PLAINTEXT://:9093.

4. MQTT Broker

• An Eclipse Mosquitto container, usually accessible at mqtt-broker_1:1883.

5. Grafana Integration

• A Grafana container is provided with:

  • Preloaded Dashboard : Sensor-Analytics.

  • Datasource Provisioning : An MQTT data source is set up (may require manual URI update in Grafana’s Data Sources if the hostname differs).

---

Issue this PR will close

close: #537
(Build a microservice to get sensor details from a Lidar sensor).

Test Instructions :

Run the Demo

• Start all services using the following command:

  make run-demo
  

A. MQTT (via Grafana)

1. Access Grafana

• Go to http://localhost:3000

• Default Credentials : admin / admin

2. Preloaded Dashboard :

• Look for Sensor-Analytics in the Dashboards list.

• Panels should be configured for:

  • lidar/data

  • weight/data

• If you see no data:

  • Go to Configuration > Data Sources and confirm the MQTT data source URI is set to tcp://mqtt-broker_1:1883 or tcp://mqtt-broker:1883 (depending on your Docker network name).

  • Click Test to ensure it connects.

3. Real-Time Visualization :

• You should see sensor data updating in each panel.

• Adjust the refresh interval (top-right in Grafana) if needed.

---

B. Kafka Publishing

1. Enable Kafka

• In your docker-compose.yml, set:

LIDAR_KAFKA_ENABLE: "true"
WEIGHT_KAFKA_ENABLE: "true"

2. Test from inside the common-service container:

docker exec asc_common_service python kafka_publisher_test.py --topic lidar-data

• This script subscribes to the lidar-data topic.

• Verify it prints incoming LiDAR messages published to Kafka.

3. (Optional) Use other Kafka clients to confirm message flow.

---

C. HTTP Publishing

This feature can be verified in two ways :

1. Local HTTP Test (Inside Docker)

   1. In your docker-compose.yml, set:

   LIDAR_HTTP_ENABLE: "true"
   LIDAR_HTTP_URL: "http://localhost:5000/api/lidar_data"

   2. Start the containers:

   make run-demo

   3. Run the test script inside the common-service container:

   docker exec asc_common_service python http_publisher_test.py

   4. The Flask server within common-service will print the received LiDAR data, confirming that HTTP publishing is working.

2. Webhook.site for External Validation

   1. Go to Webhook.site and generate a unique URL.

   2. Set LIDAR_HTTP_URL in docker-compose.yml to that URL, for example:

   LIDAR_HTTP_ENABLE: "true"
   LIDAR_HTTP_URL: "https://webhook.site/abcdef12-3456-7890-..."

   3. Restart the containers:

   make run-demo

   4. Observe the Recent Requests section on Webhook.site to see incoming LiDAR data in real time.

   ---

4. Additional Notes

• Mock Mode : If you set LIDAR_MOCK_1 or WEIGHT_MOCK_1 to "true", the service publishes randomized demo data instead of reading from hardware.

• Multi-Sensor Support : LIDAR_COUNT and WEIGHT_COUNT define how many sensors to handle. Each has its own environment variables (e.g., LIDAR_SENSOR_ID_1, LIDAR_PORT_1, etc.).

• Topic & Interval Config : The publish interval can be tuned (e.g., LIDAR_PUBLISH_INTERVAL, WEIGHT_PUBLISH_INTERVAL), and each sensor’s MQTT/Kafka topic is customizable.

[ejlee3]

Can you please update the title of your PR with a conventional commit style message?

[NeethuES-intel]

@naman-ranka can you fix the security issues flagged by github-advanced-security bot. Also please follow the git conventional commit format for the PR title -
https://github.com/intel-retail/automated-self-checkout/blob/main/CONTRIBUTING.md

<type>(<scope>): <subject>
<BLANK LINE>
<body>
<BLANK LINE>
<footer>

type here should be feat.

[NeethuES-intel]

@naman-ranka From the Design, app.py, Dockerfile and requirements.txt look the same for all 3 services. What is the difference between app.py in each of these services ? Can you consolidate all into one service then ? Looks like only sensor data type is changing here. Are you adding any business logic to any of them to make them unique ? Why duplicate ?

[antoniomtz]

@naman-ranka I was able to run your app, but I had to make so many changes in the docker-compose specifically mqtt and all the networks so the services can communicate with each other. Also, you can create a dashboard template to show the data for demo purposes.

[NeethuES-intel]

Good job getting the PR comments closed.

[ejlee3]

LGTM - Nice work!