From a65f84891c9bad2b401d1b03e5dac4b2ba2f4d2a Mon Sep 17 00:00:00 2001 From: Anthony Casagrande Date: Wed, 1 Nov 2023 10:33:52 -0700 Subject: [PATCH 1/2] feat: add docs on secure remote deployment of multiple device services Signed-off-by: Anthony Casagrande --- .../device/remote/Ch-RemoteSecure.md | 344 +++++++++++++++++- 1 file changed, 342 insertions(+), 2 deletions(-) diff --git a/docs_src/microservices/device/remote/Ch-RemoteSecure.md b/docs_src/microservices/device/remote/Ch-RemoteSecure.md index c9eb45381c..04ce9ca5bc 100644 --- a/docs_src/microservices/device/remote/Ch-RemoteSecure.md +++ b/docs_src/microservices/device/remote/Ch-RemoteSecure.md @@ -1,2 +1,342 @@ -# Remote deployment of device services in secure mode -Coming Soon. +# Remote deployment of multiple device service instances in secure mode + +In some use cases, devices are connected to nodes where core EdgeX services are not running. In these cases the appropriate device service +needs to run on the remote nodes where it can connect to the device(s) and communicate to the host node where the rest of the EdgeX services are running. + +This page provides an example of configuring and deploying multiple instances of [device-usb-camera](../services/device-usb-camera/General.md) +service using multiple nodes in EdgeX secure mode. + +## Example +This example uses 3 nodes for remote deployment. One of the nodes (host) is used to run all EdgeX core services in Docker and the other nodes (remote) +for running the device-usb-camera service either in Docker. This example can be further extended to run multiple instances of device-usb-camera +service in multiple nodes. + +## Pre-requisites +- 3 machines running Ubuntu 20.04 or newer OS, with docker installed. + - 1 "local" system for deploying all edgex core services in secure mode with docker + - 2 "remote" nodes for deploying USB service in secure mode with docker +- All dependencies such as docker, go, git, curl etc. installed on all 3 systems to run + edgex and usb services +- Local and remote nodes needs to be on the same network + + !!! note + This document explains without any VPN and Proxy configuration, if there is + any proxy and/or VPN then that needs to be configured appropriately for secure + communication between local and remote nodes + +This example assumes the following values; replace them as needed with your own values: + +- Local System IP: 192.168.4.103 +- Remote Node 1 IP: 192.168.4.104 +- Remote Node 2 IP: 192.168.4.105 +- Instance numbers for the services are 1 and 2: `device-usb-camera_2` and `device-usb-camera_2` + +## Local system setup (EdgeX Core) +This section will go over setting up the local node to run all the EdgeX Core services + + !!! note + Refer to sample [device virtual service deployment in secure mode](../../../security/Ch-RemoteDeviceServices.md) for general understanding + + +1. Set up the "local" node to be ready for deployment. Refer to [USB Service Setup](../services/device-usb-camera/walkthrough/setup.md) + for system requirements and dependencies such as Git, Docker, Docker Compose, etc. + +2. Next step is to install [EdgeX compose](https://github.com/edgexfoundry/edgex-compose) in the host node which will be used to run all EdgeX core services. So clone the `edgex-compose` + repository: + + ```bash + git clone https://github.com/edgexfoundry/edgex-compose.git + ``` + +3. Checkout the required version: + + ```bash + git checkout {{edgexversion}} + ``` + +### Generate SSH key pair for SSH tunnel +1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh` +2. Run `generate_keys.sh` script +3. Copy `remote` folder located at `edgex-examples/security/remote_devices/spiffe_and_ssh` to remote nodes + +### Edit docker-compose file +1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/local` and edit + `docker-compose.yml` file + + !!! note + Below instructions explains to run 2 instances of usb service: instance 1 and instance 2) + +2. Change `host_ip` to ip address of local system for `consul`, `core-metadata`, `database` and `vault` + +```yaml +services: + consul: + ports: + - mode: ingress + host_ip: 192.168.4.103 # insert local machine's host ip here + target: 8500 + published: "8500" + protocol: tcp + + core-metadata: + ports: + - mode: ingress + host_ip: 192.168.4.103 # insert local machine's host ip here + target: 59881 + published: "59881" + protocol: tcp + database: + ports: + - mode: ingress + host_ip: 192.168.4.103 # insert local machine's host ip here + target: 6379 + published: "6379" + protocol: tcp + vault: + ports: + - mode: ingress + host_ip: 192.168.4.103 # insert local machine's host ip here + target: 8200 + published: "8200" + protocol: tcp +``` +1. Add/edit environment variables for `consul`, `security-proxy-setup` and `security-secretstore-setup` services + +```yaml + services: + consul: + environment: + EDGEX_ADD_REGISTRY_ACL_ROLES: "device-usb-camera_1,device-usb-camera_2" + + security-proxy-setup: + environment: + EDGEX_ADD_PROXY_ROUTE: "device-usb-camera_1.http://192.168.4.104:59983,device-usb-camera_2.http://192.168.4.105:59983" + ROUTES_DEVICE_USB_CAMERAL_HOST: "device-usb-camera_1,device-usb-camera_2" + + security-secretstore-setup: + environment: + EDGEX_ADD_KNOWN_SECRETS: "redisdb[app-rules-engine],redisdb[device-usb-camera_1],message-bus[device-usb-camera_1],redisdb[device-usb-camera_2],message-bus[device-usb-camera_2]" +``` + +1. Add multiple `device-ssh-proxy` service and configure with remote node ips + - copy `device-ssh-proxy` service and paste it again below itself. + - rename services to `device-ssh-proxy-1` and `device-ssh-proxy-2` respectively + - Modify the existing environment variables and other fields for `device-ssh-proxy-1` + + ```yaml + services: + device-ssh-proxy-1: + container_name: edgex-device-ssh-proxy-1 + environment: + SERVICE_HOST: edgex-device-usb-camera + SERVICE_PORT: 59983 + TUNNEL_HOST: 192.168.4.104 # use ip address of the first remote node! + hostname: edgex-device-ssh-proxy-1 + networks: + edgex-network: + aliases: + - edgex-device-usb-camera-1 + ports: + - 127.0.0.1:59983:59983/tcp + ``` + + - Edit environment variables and other fields for `device-ssh-proxy-2` + + ```yaml + services: + device-ssh-proxy-2: + container_name: edgex-device-ssh-proxy-2 + environment: + SERVICE_HOST: edgex-device-usb-camera + SERVICE_PORT: 59983 + TUNNEL_HOST: 192.168.4.105 # use ip address of the second remote node! + hostname: edgex-device-ssh-proxy-2 + networks: + edgex-network: + aliases: + - edgex-device-usb-camera-2 + ports: + - 127.0.0.1:59984:59983/tcp # notice the different port mapping to avoid conflicts + ``` + +2. Update to latest docker image for `security-spiffe-token-provider` + +```yaml + services: + security-spiffe-token-provider: + image: nexus3.edgexfoundry.org:10004/security-spiffe-token-provider:latest +``` + +### Build and run services +1. Navigate to edgex-examples/security/remote_devices/spiffe_and_ssh/lcoal +2. Build and start the services + ```bash + docker compose build + docker compose up -d + ``` +3. Run `docker ps -a` and ensure all containers started without crashing (Note: + `edgex-device-ssh-proxy-1` and `edgex-device-ssh-proxy-2` will keep restarting until services on remote node start.) + +### Update server entries +Edit and run `add-server-entry.sh` script for each usb service + +1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/lcoal` + and edit `add-server-entry.sh` script like below. + + !!! example - Example for first device service + ```bash + docker exec -ti edgex-security-spire-config spire-server entry create \ + -socketPath /tmp/edgex/secrets/spiffe/private/api.sock \ + -parentID spiffe://edgexfoundry.org/spire/agent/x509pop/cn/ \ + remote-agent -dns "edgex-device-usb-camera" \ + -spiffeID spiffe://edgexfoundry.org/service/device-usb-camera_1 \ + -selector "docker:label:com.docker.compose.service:device-usb-camera_1" + ``` + +2. Save and then run `./add-server-entry.sh` + +3. Repeat previous steps for each additional device service + + !!! example - Example for second device service + ```bash + docker exec -ti edgex-security-spire-config spire-server entry create \ + -socketPath /tmp/edgex/secrets/spiffe/private/api.sock \ + -parentID spiffe://edgexfoundry.org/spire/agent/x509pop/cn/ \ + remote-agent -dns "edgex-device-usb-camera" \ + -spiffeID spiffe://edgexfoundry.org/service/device-usb-camera_2 \ + -selector "docker:label:com.docker.compose.service:device-usb-camera_2" + ``` +4. Save and then run `./add-server-entry.sh` again + +## Remote Node Setup + +1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/remote` + and update `docker-compose.yml`. +2. Remove `device-virtual` service and add 2 `device-usb-service` like below, one for each instance. + +=== "First Remote Device Service" + + ```yaml + device-usb-camera_1: # device-usb-camera_ + # command: -cp -r -rsh ,,0.0.0.0 -i + command: -cp -r -rsh 192.168.4.104,192.168.4.103,0.0.0.0 -i 1 + container_name: edgex-device-usb-camera + device_cgroup_rules: + - c 81:* rw + depends_on: + - remote-spire-agent + environment: + EDGEX_SECURITY_SECRET_STORE: "true" + SECRETSTORE_HOST: 192.168.4.103 # + SECRETSTORE_PORT: "8200" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENABLED: "true" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENDPOINTSOCKET: /tmp/edgex/secrets/spiffe/public/api.sock + SECRETSTORE_RUNTIMETOKENPROVIDER_HOST: edgex-security-spiffe-token-provider + SECRETSTORE_RUNTIMETOKENPROVIDER_PORT: 59841 + SECRETSTORE_RUNTIMETOKENPROVIDER_PROTOCOL: https + SECRETSTORE_RUNTIMETOKENPROVIDER_REQUIREDSECRETS: redisdb + SECRETSTORE_RUNTIMETOKENPROVIDER_TRUSTDOMAIN: edgexfoundry.org + DEVICE_DISCOVERY_ENABLED: 'true' # enable or disable auto-discovery + DEVICE_DISCOVERY_INTERVAL: '5m' # configure the auto-discovery interval + hostname: edgex-device-usb-camera + image: nexus3.edgexfoundry.org:10004/device-usb-camera:latest + networks: + edgex-network: {} + ports: + - "192.168.4.104:59983:59983/tcp" # :59983:59983/tcp + - "8554:8554" + - "8000:8000" + read_only: true + restart: always + security_opt: + - no-new-privileges:true + user: 'root:root' + volumes: + # - /tmp/edgex/secrets/device-usb-camera_:/tmp/edgex/secrets/device-usb-camera_:ro,z + - /tmp/edgex/secrets/device-usb-camera_1:/tmp/edgex/secrets/device-usb-camera_1:ro,z + - /tmp/edgex/secrets/spiffe/public:/tmp/edgex/secrets/spiffe/public:ro,z + - /dev:/dev:ro + - /run/udev:/run/udev:ro + ``` + +=== "Second Remote Device Service" + + ```yaml + device-usb-camera_2: # device-usb-camera_ + # command: -cp -r -rsh ,,0.0.0.0 -i + command: -cp -r -rsh 192.168.4.105,192.168.4.103,0.0.0.0 -i 2 + container_name: edgex-device-usb-camera + device_cgroup_rules: + - c 81:* rw + depends_on: + - remote-spire-agent + environment: + EDGEX_SECURITY_SECRET_STORE: "true" + SECRETSTORE_HOST: 192.168.4.103 # + SECRETSTORE_PORT: "8200" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENABLED: "true" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENDPOINTSOCKET: /tmp/edgex/secrets/spiffe/public/api.sock + SECRETSTORE_RUNTIMETOKENPROVIDER_HOST: edgex-security-spiffe-token-provider + SECRETSTORE_RUNTIMETOKENPROVIDER_PORT: 59841 + SECRETSTORE_RUNTIMETOKENPROVIDER_PROTOCOL: https + SECRETSTORE_RUNTIMETOKENPROVIDER_REQUIREDSECRETS: redisdb + SECRETSTORE_RUNTIMETOKENPROVIDER_TRUSTDOMAIN: edgexfoundry.org + DEVICE_DISCOVERY_ENABLED: 'true' # enable or disable auto-discovery + DEVICE_DISCOVERY_INTERVAL: '5m' # configure the auto-discovery interval + hostname: edgex-device-usb-camera + image: nexus3.edgexfoundry.org:10004/device-usb-camera:latest + networks: + edgex-network: {} + ports: + - "192.168.4.105:59983:59983/tcp" # :59983:59983/tcp + - "8554:8554" + - "8000:8000" + read_only: true + restart: always + security_opt: + - no-new-privileges:true + user: 'root:root' + volumes: + # - /tmp/edgex/secrets/device-usb-camera_:/tmp/edgex/secrets/device-usb-camera_:ro,z + - /tmp/edgex/secrets/device-usb-camera_1:/tmp/edgex/secrets/device-usb-camera_1:ro,z + - /tmp/edgex/secrets/spiffe/public:/tmp/edgex/secrets/spiffe/public:ro,z + - /dev:/dev:ro + - /run/udev:/run/udev:ro + ``` + +3. Build and run service on remote node + - Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/remote` + - `docker compose build` + - `docker compose up -d` + - Run `docker ps- a` to ensure all services started successfully +4) Wait for about 1 minute to establish ssh tunnel and to establish communication + between edgex server system and remote node +5. Run `docker logs edgex-device-usb-camera` to check usb service status. Ensure + usb service started successfully and detected and added connected usb cameras (if + any) to `edgex-core-metadata` + + +Note: If edgex services are restarted then remote node services must also be +restarted + +## Testing and execution of REST APIs +1. Download [get-api-gateway-token.sh](https://github.com/edgexfoundry/edgex-compose/blob/main/compose-builder/get-api-gateway-token.sh) script +2. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/lcoal` and + copy over the downloaded `get-api-gateway-token.sh` script +3. run `./get-api-gateway-token.sh` to retrieve secure token to run rest apis +4. Run curl command to get all connected usb cameras on remote node + ```bash + curl --location --request GET 'http://:59881/api/v3/device/service/name/device-usb-camera_' \ + --header 'Authorization: Bearer ' \ + --data-raw '' + ``` +5. Run curl command to get specific camera info + ```bash + curl --location --request GET 'http://:59882/api/v3/device/name//CameraInfo' \ + --header 'Authorization: Bearer ' \ + --data-raw '' + ``` + + !!! note + You can download the usb postman collection from [device-usb-service repo](https://github.com/edgexfoundry/device-usb-camera/tree/main/docs) and run + other APIs From 8730553c47f29cd05d9abb177b32a92d2a6b80b9 Mon Sep 17 00:00:00 2001 From: Anthony Casagrande Date: Tue, 5 Dec 2023 14:00:50 -0800 Subject: [PATCH 2/2] fix: address code reviews and fix rendering Signed-off-by: Anthony Casagrande --- .../device/how-to/Ch-RemoteSecure.md | 600 ++++++++++-------- 1 file changed, 338 insertions(+), 262 deletions(-) diff --git a/docs_src/microservices/device/how-to/Ch-RemoteSecure.md b/docs_src/microservices/device/how-to/Ch-RemoteSecure.md index 3867af3766..6b4882a4ed 100644 --- a/docs_src/microservices/device/how-to/Ch-RemoteSecure.md +++ b/docs_src/microservices/device/how-to/Ch-RemoteSecure.md @@ -8,326 +8,394 @@ In some use cases, devices are connected to nodes where core EdgeX services are needs to run on the remote nodes where it can connect to the device(s) and communicate to the host node where the rest of the EdgeX services are running. This page provides an example of configuring and deploying multiple instances of [device-usb-camera](../services/device-usb-camera/General.md) -service using multiple nodes in EdgeX secure mode. +service using multiple nodes in EdgeX secure mode. This example is applicable to any device service. ## Example This example uses 3 nodes for remote deployment. One of the nodes (host) is used to run all EdgeX core services in Docker and the other nodes (remote) -for running the device-usb-camera service either in Docker. This example can be further extended to run multiple instances of device-usb-camera +for running the device-usb-camera service in Docker. This example can be further extended to run multiple instances of device-usb-camera service in multiple nodes. ## Pre-requisites - 3 machines running Ubuntu 20.04 or newer OS, with docker installed. - - 1 "local" system for deploying all edgex core services in secure mode with docker - - 2 "remote" nodes for deploying USB service in secure mode with docker -- All dependencies such as docker, go, git, curl etc. installed on all 3 systems to run + - 1 `local` system for deploying all edgex core services in secure mode with docker + - 2 `remote` nodes for deploying USB service in secure mode with docker +- All dependencies such as docker, git, curl etc. installed on all 3 systems to run edgex and usb services - Local and remote nodes needs to be on the same network - !!! note - This document explains without any VPN and Proxy configuration, if there is - any proxy and/or VPN then that needs to be configured appropriately for secure - communication between local and remote nodes +!!! Note + This document assumes a network configuration without any VPN and/or proxy configuration. If your network environment + requires any proxy and/or VPN, then that needs to be configured appropriately and the below steps adjusted for your environment. This example assumes the following values; replace them as needed with your own values: -- Local System IP: 192.168.4.103 -- Remote Node 1 IP: 192.168.4.104 -- Remote Node 2 IP: 192.168.4.105 -- Instance numbers for the services are 1 and 2: `device-usb-camera_2` and `device-usb-camera_2` +- `local` node IP: `192.168.4.103` +- `remote` node 1 IP: `192.168.4.104` +- `remote` node 2 IP: `192.168.4.105` +- Instance numbers for the services are `1` and `2`. (i.e. `device-usb-camera_1` and `device-usb-camera_2`) ## Local system setup (EdgeX Core) This section will go over setting up the local node to run all the EdgeX Core services - !!! note - Refer to sample [device virtual service deployment in secure mode](../../../security/Ch-RemoteDeviceServices.md) for general understanding +!!! Info + Refer to sample [device virtual service deployment in secure mode](../../../security/Ch-RemoteDeviceServices.md) for a general understanding of remote device services. -1. Set up the "local" node to be ready for deployment. Refer to [USB Service Setup](../services/device-usb-camera/walkthrough/setup.md) +1. Set up the `local` node to be ready for deployment. Refer to [USB Service Setup](../services/device-usb-camera/walkthrough/setup.md) for system requirements and dependencies such as Git, Docker, Docker Compose, etc. -2. Next step is to install [EdgeX compose](https://github.com/edgexfoundry/edgex-compose) in the host node which will be used to run all EdgeX core services. So clone the `edgex-compose` +2. Next step is to install [EdgeX compose][edgex-compose] in the host node which will be used to run all EdgeX core services. So clone the `edgex-compose` repository: ```bash git clone https://github.com/edgexfoundry/edgex-compose.git ``` -3. Checkout the required version: +3. Navigate to the `edgex-compose` directory: + ```bash + cd edgex-compose + ``` +4. Checkout the compatible release ({{edgexversion}}): ```bash git checkout {{edgexversion}} ``` ### Generate SSH key pair for SSH tunnel -1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh` -2. Run `generate_keys.sh` script -3. Copy `remote` folder located at `edgex-examples/security/remote_devices/spiffe_and_ssh` to remote nodes +1. Clone the EdgeX Examples repository: + ```bash + git clone https://github.com/edgexfoundry/edgex-examples.git + ``` + +2. Navigate to the `edgex-examples` directory: + ```bash + cd edgex-examples + ``` + +3. Checkout the compatible release ({{edgexversion}}): + ```shell + git checkout {{edgexversion}} + ``` + +4. Navigate to [`edgex-examples/security/remote_devices/spiffe_and_ssh`][spiffe_and_ssh] on your local machine. +5. Run `generate_keys.sh` script +6. (Optional) Copy the entire `remote` sub-folder to somewhere on each remote node's filesystem at this point. If you prefer, you can wait until [Remote Node Setup](#remote-node-setup) to perform this step. ### Edit docker-compose file -1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/local` and edit - `docker-compose.yml` file +!!! Note + Below instructions explain how to run 2 instances of the `device-usb-camera` service: instance 1 (`device-usb-camera_1`) and instance 2 (`device-usb-camera_2`). - !!! note - Below instructions explains to run 2 instances of usb service: instance 1 and instance 2) +1. Navigate to [`edgex-examples/security/remote_devices/spiffe_and_ssh/local`][spiffe_and_ssh_local] and edit + [`docker-compose.yml`][local_compose] file 2. Change `host_ip` to ip address of local system for `consul`, `core-metadata`, `database` and `vault` -```yaml -services: - consul: - ports: - - mode: ingress - host_ip: 192.168.4.103 # insert local machine's host ip here - target: 8500 - published: "8500" - protocol: tcp - - core-metadata: - ports: - - mode: ingress - host_ip: 192.168.4.103 # insert local machine's host ip here - target: 59881 - published: "59881" - protocol: tcp - database: - ports: - - mode: ingress - host_ip: 192.168.4.103 # insert local machine's host ip here - target: 6379 - published: "6379" - protocol: tcp - vault: - ports: - - mode: ingress - host_ip: 192.168.4.103 # insert local machine's host ip here - target: 8200 - published: "8200" - protocol: tcp -``` -1. Add/edit environment variables for `consul`, `security-proxy-setup` and `security-secretstore-setup` services - -```yaml - services: - consul: - environment: - EDGEX_ADD_REGISTRY_ACL_ROLES: "device-usb-camera_1,device-usb-camera_2" - - security-proxy-setup: - environment: - EDGEX_ADD_PROXY_ROUTE: "device-usb-camera_1.http://192.168.4.104:59983,device-usb-camera_2.http://192.168.4.105:59983" - ROUTES_DEVICE_USB_CAMERAL_HOST: "device-usb-camera_1,device-usb-camera_2" - - security-secretstore-setup: - environment: - EDGEX_ADD_KNOWN_SECRETS: "redisdb[app-rules-engine],redisdb[device-usb-camera_1],message-bus[device-usb-camera_1],redisdb[device-usb-camera_2],message-bus[device-usb-camera_2]" -``` - -1. Add multiple `device-ssh-proxy` service and configure with remote node ips - - copy `device-ssh-proxy` service and paste it again below itself. - - rename services to `device-ssh-proxy-1` and `device-ssh-proxy-2` respectively - - Modify the existing environment variables and other fields for `device-ssh-proxy-1` - - ```yaml - services: - device-ssh-proxy-1: - container_name: edgex-device-ssh-proxy-1 - environment: - SERVICE_HOST: edgex-device-usb-camera - SERVICE_PORT: 59983 - TUNNEL_HOST: 192.168.4.104 # use ip address of the first remote node! - hostname: edgex-device-ssh-proxy-1 - networks: - edgex-network: - aliases: - - edgex-device-usb-camera-1 - ports: - - 127.0.0.1:59983:59983/tcp - ``` - - - Edit environment variables and other fields for `device-ssh-proxy-2` - - ```yaml - services: - device-ssh-proxy-2: - container_name: edgex-device-ssh-proxy-2 - environment: - SERVICE_HOST: edgex-device-usb-camera - SERVICE_PORT: 59983 - TUNNEL_HOST: 192.168.4.105 # use ip address of the second remote node! - hostname: edgex-device-ssh-proxy-2 - networks: - edgex-network: - aliases: - - edgex-device-usb-camera-2 - ports: - - 127.0.0.1:59984:59983/tcp # notice the different port mapping to avoid conflicts - ``` - -2. Update to latest docker image for `security-spiffe-token-provider` - -```yaml - services: - security-spiffe-token-provider: - image: nexus3.edgexfoundry.org:10004/security-spiffe-token-provider:latest -``` + ```yaml + services: + consul: + ports: + - mode: ingress + host_ip: 192.168.4.103 # insert local machine's host ip here + target: 8500 + published: "8500" + protocol: tcp + + core-metadata: + ports: + - mode: ingress + host_ip: 192.168.4.103 # insert local machine's host ip here + target: 59881 + published: "59881" + protocol: tcp + database: + ports: + - mode: ingress + host_ip: 192.168.4.103 # insert local machine's host ip here + target: 6379 + published: "6379" + protocol: tcp + vault: + ports: + - mode: ingress + host_ip: 192.168.4.103 # insert local machine's host ip here + target: 8200 + published: "8200" + protocol: tcp + ``` +3. Add/edit environment variables for `consul`, `security-proxy-setup` and `security-secretstore-setup` services + + ```yaml + services: + consul: + environment: + EDGEX_ADD_REGISTRY_ACL_ROLES: "device-usb-camera_1,device-usb-camera_2" + + security-proxy-setup: + environment: + # insert the IP addresses of your remote nodes below! + EDGEX_ADD_PROXY_ROUTE: "device-usb-camera_1.http://192.168.4.104:59983,device-usb-camera_2.http://192.168.4.105:59983" + ROUTES_DEVICE_USB_CAMERAL_HOST: "device-usb-camera_1,device-usb-camera_2" + + security-secretstore-setup: + environment: + EDGEX_ADD_KNOWN_SECRETS: "redisdb[app-rules-engine],redisdb[device-usb-camera_1],message-bus[device-usb-camera_1],redisdb[device-usb-camera_2],message-bus[device-usb-camera_2]" + ``` + +4. Add multiple `device-ssh-proxy` service and configure with remote node ips + 1. Copy `device-ssh-proxy` service and paste it again below itself + 2. Rename services to `device-ssh-proxy-1` and `device-ssh-proxy-2` respectively + 3. Modify the existing environment variables and other fields for `device-ssh-proxy-1` + ```yaml + services: + device-ssh-proxy-1: + container_name: edgex-device-ssh-proxy-1 + environment: + SERVICE_HOST: edgex-device-usb-camera + SERVICE_PORT: 59983 + TUNNEL_HOST: 192.168.4.104 # use ip address of the first remote node! + hostname: edgex-device-ssh-proxy-1 + networks: + edgex-network: + aliases: + - edgex-device-usb-camera-1 + ports: + - 127.0.0.1:59983:59983/tcp + ``` + 4. Edit environment variables and other fields for `device-ssh-proxy-2` + ```yaml + services: + device-ssh-proxy-2: + container_name: edgex-device-ssh-proxy-2 + environment: + SERVICE_HOST: edgex-device-usb-camera + SERVICE_PORT: 59983 + TUNNEL_HOST: 192.168.4.105 # use ip address of the second remote node! + hostname: edgex-device-ssh-proxy-2 + networks: + edgex-network: + aliases: + - edgex-device-usb-camera-2 + ports: + - 127.0.0.1:59984:59983/tcp # notice the different port mapping to avoid conflicts + ``` + +5. Update to latest docker image for `security-spiffe-token-provider` + + ```yaml + services: + security-spiffe-token-provider: + image: nexus3.edgexfoundry.org:10004/security-spiffe-token-provider:latest + ``` ### Build and run services -1. Navigate to edgex-examples/security/remote_devices/spiffe_and_ssh/lcoal +1. Navigate to [`edgex-examples/security/remote_devices/spiffe_and_ssh/local`][spiffe_and_ssh_local] 2. Build and start the services ```bash docker compose build docker compose up -d ``` -3. Run `docker ps -a` and ensure all containers started without crashing (Note: - `edgex-device-ssh-proxy-1` and `edgex-device-ssh-proxy-2` will keep restarting until services on remote node start.) +3. Run `docker ps -a` and ensure all containers started without crashing + !!! Note + `edgex-device-ssh-proxy-1` and `edgex-device-ssh-proxy-2` will keep restarting until services on remote node start ### Update server entries Edit and run `add-server-entry.sh` script for each usb service -1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/lcoal` - and edit `add-server-entry.sh` script like below. +1. Navigate to [`edgex-examples/security/remote_devices/spiffe_and_ssh/local`][spiffe_and_ssh_local] and edit `add-server-entry.sh` script like below. - !!! example - Example for first device service - ```bash - docker exec -ti edgex-security-spire-config spire-server entry create \ - -socketPath /tmp/edgex/secrets/spiffe/private/api.sock \ - -parentID spiffe://edgexfoundry.org/spire/agent/x509pop/cn/ \ - remote-agent -dns "edgex-device-usb-camera" \ - -spiffeID spiffe://edgexfoundry.org/service/device-usb-camera_1 \ - -selector "docker:label:com.docker.compose.service:device-usb-camera_1" - ``` + !!! Example - "Example for first device service" + ```bash + docker exec -ti edgex-security-spire-config spire-server entry create \ + -socketPath /tmp/edgex/secrets/spiffe/private/api.sock \ + -parentID spiffe://edgexfoundry.org/spire/agent/x509pop/cn/ \ + remote-agent -dns "edgex-device-usb-camera" \ + -spiffeID spiffe://edgexfoundry.org/service/device-usb-camera_1 \ + -selector "docker:label:com.docker.compose.service:device-usb-camera_1" + ``` 2. Save and then run `./add-server-entry.sh` 3. Repeat previous steps for each additional device service - !!! example - Example for second device service - ```bash - docker exec -ti edgex-security-spire-config spire-server entry create \ - -socketPath /tmp/edgex/secrets/spiffe/private/api.sock \ - -parentID spiffe://edgexfoundry.org/spire/agent/x509pop/cn/ \ - remote-agent -dns "edgex-device-usb-camera" \ - -spiffeID spiffe://edgexfoundry.org/service/device-usb-camera_2 \ - -selector "docker:label:com.docker.compose.service:device-usb-camera_2" - ``` + !!! Example - "Example for second device service" + ```bash + docker exec -ti edgex-security-spire-config spire-server entry create \ + -socketPath /tmp/edgex/secrets/spiffe/private/api.sock \ + -parentID spiffe://edgexfoundry.org/spire/agent/x509pop/cn/ \ + remote-agent -dns "edgex-device-usb-camera" \ + -spiffeID spiffe://edgexfoundry.org/service/device-usb-camera_2 \ + -selector "docker:label:com.docker.compose.service:device-usb-camera_2" + ``` 4. Save and then run `./add-server-entry.sh` again +5. Repeat these steps as needed for each additional remote node ## Remote Node Setup -1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/remote` - and update `docker-compose.yml`. -2. Remove `device-virtual` service and add 2 `device-usb-service` like below, one for each instance. - -=== "First Remote Device Service" - - ```yaml - device-usb-camera_1: # device-usb-camera_ - # command: -cp -r -rsh ,,0.0.0.0 -i - command: -cp -r -rsh 192.168.4.104,192.168.4.103,0.0.0.0 -i 1 - container_name: edgex-device-usb-camera - device_cgroup_rules: - - c 81:* rw - depends_on: - - remote-spire-agent - environment: - EDGEX_SECURITY_SECRET_STORE: "true" - SECRETSTORE_HOST: 192.168.4.103 # - SECRETSTORE_PORT: "8200" - SECRETSTORE_RUNTIMETOKENPROVIDER_ENABLED: "true" - SECRETSTORE_RUNTIMETOKENPROVIDER_ENDPOINTSOCKET: /tmp/edgex/secrets/spiffe/public/api.sock - SECRETSTORE_RUNTIMETOKENPROVIDER_HOST: edgex-security-spiffe-token-provider - SECRETSTORE_RUNTIMETOKENPROVIDER_PORT: 59841 - SECRETSTORE_RUNTIMETOKENPROVIDER_PROTOCOL: https - SECRETSTORE_RUNTIMETOKENPROVIDER_REQUIREDSECRETS: redisdb - SECRETSTORE_RUNTIMETOKENPROVIDER_TRUSTDOMAIN: edgexfoundry.org - DEVICE_DISCOVERY_ENABLED: 'true' # enable or disable auto-discovery - DEVICE_DISCOVERY_INTERVAL: '5m' # configure the auto-discovery interval - hostname: edgex-device-usb-camera - image: nexus3.edgexfoundry.org:10004/device-usb-camera:latest - networks: - edgex-network: {} - ports: - - "192.168.4.104:59983:59983/tcp" # :59983:59983/tcp - - "8554:8554" - - "8000:8000" - read_only: true - restart: always - security_opt: - - no-new-privileges:true - user: 'root:root' - volumes: - # - /tmp/edgex/secrets/device-usb-camera_:/tmp/edgex/secrets/device-usb-camera_:ro,z - - /tmp/edgex/secrets/device-usb-camera_1:/tmp/edgex/secrets/device-usb-camera_1:ro,z - - /tmp/edgex/secrets/spiffe/public:/tmp/edgex/secrets/spiffe/public:ro,z - - /dev:/dev:ro - - /run/udev:/run/udev:ro - ``` - -=== "Second Remote Device Service" - - ```yaml - device-usb-camera_2: # device-usb-camera_ - # command: -cp -r -rsh ,,0.0.0.0 -i - command: -cp -r -rsh 192.168.4.105,192.168.4.103,0.0.0.0 -i 2 - container_name: edgex-device-usb-camera - device_cgroup_rules: - - c 81:* rw - depends_on: - - remote-spire-agent - environment: - EDGEX_SECURITY_SECRET_STORE: "true" - SECRETSTORE_HOST: 192.168.4.103 # - SECRETSTORE_PORT: "8200" - SECRETSTORE_RUNTIMETOKENPROVIDER_ENABLED: "true" - SECRETSTORE_RUNTIMETOKENPROVIDER_ENDPOINTSOCKET: /tmp/edgex/secrets/spiffe/public/api.sock - SECRETSTORE_RUNTIMETOKENPROVIDER_HOST: edgex-security-spiffe-token-provider - SECRETSTORE_RUNTIMETOKENPROVIDER_PORT: 59841 - SECRETSTORE_RUNTIMETOKENPROVIDER_PROTOCOL: https - SECRETSTORE_RUNTIMETOKENPROVIDER_REQUIREDSECRETS: redisdb - SECRETSTORE_RUNTIMETOKENPROVIDER_TRUSTDOMAIN: edgexfoundry.org - DEVICE_DISCOVERY_ENABLED: 'true' # enable or disable auto-discovery - DEVICE_DISCOVERY_INTERVAL: '5m' # configure the auto-discovery interval - hostname: edgex-device-usb-camera - image: nexus3.edgexfoundry.org:10004/device-usb-camera:latest - networks: - edgex-network: {} - ports: - - "192.168.4.105:59983:59983/tcp" # :59983:59983/tcp - - "8554:8554" - - "8000:8000" - read_only: true - restart: always - security_opt: - - no-new-privileges:true - user: 'root:root' - volumes: - # - /tmp/edgex/secrets/device-usb-camera_:/tmp/edgex/secrets/device-usb-camera_:ro,z - - /tmp/edgex/secrets/device-usb-camera_1:/tmp/edgex/secrets/device-usb-camera_1:ro,z - - /tmp/edgex/secrets/spiffe/public:/tmp/edgex/secrets/spiffe/public:ro,z - - /dev:/dev:ro - - /run/udev:/run/udev:ro - ``` - -3. Build and run service on remote node - - Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/remote` - - `docker compose build` - - `docker compose up -d` - - Run `docker ps- a` to ensure all services started successfully -4) Wait for about 1 minute to establish ssh tunnel and to establish communication - between edgex server system and remote node -5. Run `docker logs edgex-device-usb-camera` to check usb service status. Ensure - usb service started successfully and detected and added connected usb cameras (if - any) to `edgex-core-metadata` - - -Note: If edgex services are restarted then remote node services must also be -restarted +!!! note + These steps will need to be repeated for **every single remote node**! + +1. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh` on your local machine +2. Copy the entire `remote` sub-folder somewhere onto the remote node +3. Navigate into the `remote` folder on the remote node +4. Edit the [`docker-compose.yml`][remote_compose] file +5. Remove the entire `device-virtual` service +6. Add the `device-usb-service_X` service like below, updating it with the proper values for your configuration + + !!! tip + Choose `Remote Node 1` to configure remote node 1, `Remote Node 2` to configure remote node 2, or `Remote Node Template` for any additional nodes. + + !!! example + === "Remote Node 1" + + ```yaml + device-usb-camera_1: # device-usb-camera_ + # command: -cp -r -rsh ,,0.0.0.0 -i + command: -cp -r -rsh 192.168.4.104,192.168.4.103,0.0.0.0 -i 1 + container_name: edgex-device-usb-camera + device_cgroup_rules: + - c 81:* rw + depends_on: + - remote-spire-agent + environment: + EDGEX_SECURITY_SECRET_STORE: "true" + SECRETSTORE_HOST: 192.168.4.103 # + SECRETSTORE_PORT: "8200" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENABLED: "true" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENDPOINTSOCKET: /tmp/edgex/secrets/spiffe/public/api.sock + SECRETSTORE_RUNTIMETOKENPROVIDER_HOST: edgex-security-spiffe-token-provider + SECRETSTORE_RUNTIMETOKENPROVIDER_PORT: 59841 + SECRETSTORE_RUNTIMETOKENPROVIDER_PROTOCOL: https + SECRETSTORE_RUNTIMETOKENPROVIDER_REQUIREDSECRETS: redisdb + SECRETSTORE_RUNTIMETOKENPROVIDER_TRUSTDOMAIN: edgexfoundry.org + DEVICE_DISCOVERY_ENABLED: 'true' # enable or disable auto-discovery + DEVICE_DISCOVERY_INTERVAL: '5m' # configure the auto-discovery interval + hostname: edgex-device-usb-camera + image: nexus3.edgexfoundry.org:10004/device-usb-camera:latest + networks: + edgex-network: {} + ports: + - "192.168.4.104:59983:59983/tcp" # :59983:59983/tcp + - "8554:8554" + - "8000:8000" + read_only: true + restart: always + security_opt: + - no-new-privileges:true + user: 'root:root' + volumes: + # - /tmp/edgex/secrets/device-usb-camera_:/tmp/edgex/secrets/device-usb-camera_:ro,z + - /tmp/edgex/secrets/device-usb-camera_1:/tmp/edgex/secrets/device-usb-camera_1:ro,z + - /tmp/edgex/secrets/spiffe/public:/tmp/edgex/secrets/spiffe/public:ro,z + - /dev:/dev:ro + - /run/udev:/run/udev:ro + ``` + + === "Remote Node 2" + + ```yaml + device-usb-camera_2: # device-usb-camera_ + # command: -cp -r -rsh ,,0.0.0.0 -i + command: -cp -r -rsh 192.168.4.105,192.168.4.103,0.0.0.0 -i 2 + container_name: edgex-device-usb-camera + device_cgroup_rules: + - c 81:* rw + depends_on: + - remote-spire-agent + environment: + EDGEX_SECURITY_SECRET_STORE: "true" + SECRETSTORE_HOST: 192.168.4.103 # + SECRETSTORE_PORT: "8200" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENABLED: "true" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENDPOINTSOCKET: /tmp/edgex/secrets/spiffe/public/api.sock + SECRETSTORE_RUNTIMETOKENPROVIDER_HOST: edgex-security-spiffe-token-provider + SECRETSTORE_RUNTIMETOKENPROVIDER_PORT: 59841 + SECRETSTORE_RUNTIMETOKENPROVIDER_PROTOCOL: https + SECRETSTORE_RUNTIMETOKENPROVIDER_REQUIREDSECRETS: redisdb + SECRETSTORE_RUNTIMETOKENPROVIDER_TRUSTDOMAIN: edgexfoundry.org + DEVICE_DISCOVERY_ENABLED: 'true' # enable or disable auto-discovery + DEVICE_DISCOVERY_INTERVAL: '5m' # configure the auto-discovery interval + hostname: edgex-device-usb-camera + image: nexus3.edgexfoundry.org:10004/device-usb-camera:latest + networks: + edgex-network: {} + ports: + - "192.168.4.105:59983:59983/tcp" # :59983:59983/tcp + - "8554:8554" + - "8000:8000" + read_only: true + restart: always + security_opt: + - no-new-privileges:true + user: 'root:root' + volumes: + # - /tmp/edgex/secrets/device-usb-camera_:/tmp/edgex/secrets/device-usb-camera_:ro,z + - /tmp/edgex/secrets/device-usb-camera_1:/tmp/edgex/secrets/device-usb-camera_1:ro,z + - /tmp/edgex/secrets/spiffe/public:/tmp/edgex/secrets/spiffe/public:ro,z + - /dev:/dev:ro + - /run/udev:/run/udev:ro + ``` + + === "Remote Node Template" + + ```yaml + device-usb-camera_: # todo + command: -cp -r -rsh ,,0.0.0.0 -i # todo + container_name: edgex-device-usb-camera + device_cgroup_rules: + - c 81:* rw + depends_on: + - remote-spire-agent + environment: + EDGEX_SECURITY_SECRET_STORE: "true" + SECRETSTORE_HOST: # todo + SECRETSTORE_PORT: "8200" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENABLED: "true" + SECRETSTORE_RUNTIMETOKENPROVIDER_ENDPOINTSOCKET: /tmp/edgex/secrets/spiffe/public/api.sock + SECRETSTORE_RUNTIMETOKENPROVIDER_HOST: edgex-security-spiffe-token-provider + SECRETSTORE_RUNTIMETOKENPROVIDER_PORT: 59841 + SECRETSTORE_RUNTIMETOKENPROVIDER_PROTOCOL: https + SECRETSTORE_RUNTIMETOKENPROVIDER_REQUIREDSECRETS: redisdb + SECRETSTORE_RUNTIMETOKENPROVIDER_TRUSTDOMAIN: edgexfoundry.org + DEVICE_DISCOVERY_ENABLED: 'true' # enable or disable auto-discovery + DEVICE_DISCOVERY_INTERVAL: '5m' # configure the auto-discovery interval + hostname: edgex-device-usb-camera + image: nexus3.edgexfoundry.org:10004/device-usb-camera:latest + networks: + edgex-network: {} + ports: + - ":59983:59983/tcp" # todo + - "8554:8554" + - "8000:8000" + read_only: true + restart: always + security_opt: + - no-new-privileges:true + user: 'root:root' + volumes: + - /tmp/edgex/secrets/device-usb-camera_:/tmp/edgex/secrets/device-usb-camera_:ro,z # todo + - /tmp/edgex/secrets/spiffe/public:/tmp/edgex/secrets/spiffe/public:ro,z + - /dev:/dev:ro + - /run/udev:/run/udev:ro + ``` + +7. Build and run services on the remote node + 1. Navigate to `remote` folder + 2. `docker compose build` + 3. `docker compose up -d` + 4. Run `docker ps -a` to ensure all services started successfully +8. Wait for about 1 minute for ssh tunnel and communication between edgex server system and remote node to be established. +9. Run `docker logs edgex-device-usb-camera` to check usb service status. Ensure usb service started successfully and detected and added connected usb cameras (if any) to `edgex-core-metadata`. +10. Repeat all the [Remote Node Setup](#remote-node-setup) steps for each additional remote node you wish to deploy + + +!!! warning + If EdgeX core services are restarted, then services on each remote node must also be restarted! ## Testing and execution of REST APIs -1. Download [get-api-gateway-token.sh](https://github.com/edgexfoundry/edgex-compose/blob/main/compose-builder/get-api-gateway-token.sh) script -2. Navigate to `edgex-examples/security/remote_devices/spiffe_and_ssh/lcoal` and - copy over the downloaded `get-api-gateway-token.sh` script -3. run `./get-api-gateway-token.sh` to retrieve secure token to run rest apis +1. Download [get-api-gateway-token.sh][get-api-gateway-token] script +2. Navigate to [`edgex-examples/security/remote_devices/spiffe_and_ssh/local`][spiffe_and_ssh_local] and + copy over the downloaded [`get-api-gateway-token.sh`][get-api-gateway-token] script +3. Run `./get-api-gateway-token.sh` to retrieve secure token to run rest apis 4. Run curl command to get all connected usb cameras on remote node ```bash curl --location --request GET 'http://:59881/api/v3/device/service/name/device-usb-camera_' \ @@ -339,9 +407,17 @@ restarted curl --location --request GET 'http://:59882/api/v3/device/name//CameraInfo' \ --header 'Authorization: Bearer ' \ --data-raw '' - ``` + ``` + +!!! Info + You can download the full device-usb-camera Postman collection from [device-usb-service repo][ds-usb-docs] in order to explore and run other APIs - !!! note - You can download the usb postman collection from [device-usb-service repo](https://github.com/edgexfoundry/device-usb-camera/tree/main/docs) and run - other APIs +[ds-usb-docs]: https://github.com/edgexfoundry/device-usb-camera/tree/{{edgexversion}}/docs +[get-api-gateway-token]: https://github.com/edgexfoundry/edgex-compose/blob/{{edgexversion}}/compose-builder/get-api-gateway-token.sh +[edgex-compose]: https://github.com/edgexfoundry/edgex-compose +[spiffe_and_ssh]: https://github.com/edgexfoundry/edgex-examples/tree/{{edgexversion}}/security/remote_devices/spiffe_and_ssh +[spiffe_and_ssh_remote]: https://github.com/edgexfoundry/edgex-examples/tree/{{edgexversion}}/security/remote_devices/spiffe_and_ssh/remote +[spiffe_and_ssh_local]: https://github.com/edgexfoundry/edgex-examples/tree/{{edgexversion}}/security/remote_devices/spiffe_and_ssh/local +[local_compose]: https://github.com/edgexfoundry/edgex-examples/tree/{{edgexversion}}/security/remote_devices/spiffe_and_ssh/local/docker-compose.yml +[remote_compose]: https://github.com/edgexfoundry/edgex-examples/tree/{{edgexversion}}/security/remote_devices/spiffe_and_ssh/remote/docker-compose.yml \ No newline at end of file