Fixes #1826 - Add a cross-network dynamic address capability

docs/notes/cross-network.md

@@ -0,0 +1,177 @@
+<!-- Licensed to the Apache Software Foundation (ASF) under one -->
+<!-- or more contributor license agreements.  See the NOTICE file -->
+<!-- distributed with this work for additional information -->
+<!-- regarding copyright ownership.  The ASF licenses this file -->
+<!-- to you under the Apache License, Version 2.0 (the -->
+<!-- "License"); you may not use this file except in compliance -->
+<!-- with the License.  You may obtain a copy of the License at -->
+
+<!--   http://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!-- Unless required by applicable law or agreed to in writing, -->
+<!-- software distributed under the License is distributed on an -->
+<!-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
+<!-- KIND, either express or implied.  See the License for the -->
+<!-- specific language governing permissions and limitations -->
+<!-- under the License. -->
+
+# Cross (Inter) Network Communication
+
+Cross-network communication is message transfer between endpoints connected to disjoint Skupper/AMQP networks.  The connectivity between the networks is achieved using connections with the 'inter-network' role and auto-links established over those connections.
+
+The easy part is creating connectivity for a particular mobile address, the address of a service.  More challenging is the need to provide reply-to connectivity using a router-assigned dynamic address on the client-side (for request/response traffic patterns).
+
+## Use Cases
+
+There are a couple of use cases that drive the requirements for cross-network communication.
+
+### Central Management of Networks
+
+In the case where an enterprise maintains a large number of virtual application networks, a good way to provide connectivity for a management plane is to create a management network that maintains cross-network connectivity to each of the managed networks.
+
+In such a network, there is a inter-network connection between at least one router in the managed network and at lease one router in the management network.  Multiple connections may be desired for redundancy and availability.  All message connectivity is strictly between the management network and the individual managed network.  No communication will be possible between two managed networks via the management network, thus preserving the isolation provided by an application network.
+
+### Inter-Network Federation
+
+In the case where an application running on a virtual application network wishes to expose a service for use from within another VAN without creating a general-use ingress, cross-network connectivity can be used to create a secure tunnel between VANs for specifically configured services.
+
+## Setting up Cross-Network Connectivity
+
+To illustrate the configuration of inter-network communication, we'll use the inter-network federation use case with two networks, net-A and net-B.
+
+Both networks are assumed to be composed of multiple interior routers and possibly some edge routers.  The inter-network connection must be between two interior routers, one in each network.
+
+### Router Configuration
+
+All routers (interior and edge) must be configured with the name of the network using the `network` configuration:
+
+```
+network {
+    networkId: net-A
+}
+```
+### Listener and Connector Configuration
+
+An interior router in each network must be designated to be endpoints of the inter-network connection.  For example, routers A3 (in net-A) and B2 (in net-B) are designated.  A3 is configured with a `listener` and B2 is configured with a `connector`, both using role `inter-network`.  B2 is configured to connect to A3 using standard host/port addressing and security.  Both the listener and connector must be named because the configurations will include `autoLink` entities that refer to those connections.
+
+### Exposing Services Cross-Network
+
+Since the `inter-network` connection does not provide the network-joining functions of an `inter-router` connection, the two networks are not joined and do not share any topology or routing information.  Nothing will flow over the `inter-network` connection until an `autoLink` is created which establishes a unidirectional link between the networks.
+
+The full functionality of auto-links is available for `inter-network` connections.  They behave similarly to `route-container` connections.  However, it is recommended for inter-network use cases that all auto-links be configured as `direction: in` and "pull" from the network hosting a particular destination.
+
+For example, if net-A hosts a service using the address `service-45` anywhere in its network, an auto-link should be configured in the designated router (A3) as follows:
+
+```
+autoLink {
+    connection: <name of listener/connector>
+    address: service-45
+    direction: in
+}
+```
+
+This will cause the `service-45` address to be reachable from all routers in net-B.
+
+Each service that is to be reachable cross-network must have an auto-link created for it in the network in which the service is hosted.
+
+Important Note:  Destinations cannot be load-balanced across different networks.  In other words, there must not be cross-network services in different networks with the same address.  If auto-links are created in both directions for an address, a loop is created that will almost certainly forward deliveries to that address to the wrong places.
+
+### Using Dynamic Addresses
+
+Almost always, service traffic is not a one-way affair.  A user of a service expects to receive a response from the service when a request is sent.  Such a client will first create a receiving link using a dynamic terminus.  The dynamically-allocated address is then placed in the request's `reply-to` header for the server to use as the destination address of the reply or replies.
+
+If a network in a federation relationship is going to host clients using dynamic addresses, it must create an auto-link to carry those replies back from the serving network.  The auto-link looks like this:
+
+```
+autoLink {
+    connection: <name of listener/connector>
+    direction: in
+    externalAddress: _xnet/<local-network-id>
+}
+```
+
+There are a couple of notable characteristics of this auto-link.  It does not have an `address` attribute, making it "anonymous".  This is important because it must issue credit to the peer network regardless of the existance of a particular destination locally.
+
+The other important aspect of this auto-link is that its remote address is interpreted at the other end as a "remote-network" address which will match any dynamic address created on the local network.  This means that only one auto-link needs to be created regardless of how many explicit service addresses are exposed from the peer network.
+
+## A Complete Example
+
+The following example shows how to create a simple two-network federation on a single host where each network has one router.
+
+Router A:
+```
+router {
+    id: A
+    mode: interior
+}
+
+network {
+    networkId: net-A
+}
+
+listener {
+    port: 10001
+    role: normal
+}
+
+listener {
+    name: federation
+    port: 11001
+    role: inter-network
+}
+
+autoLink {
+    connection: federation
+    direction: in
+    externalAddress: _xnet/net-A
+}
+
+autoLink {
+    connection: federation
+    direction: in
+    address: service-A-to-B
+}
+```
+
+Router B:
+```
+router {
+    id: B
+    mode: interior
+}
+
+network {
+    networkId: net-B
+}
+
+listener {
+    port: 10002
+    role: normal
+}
+
+connector {
+    name: federation
+    port: 11001
+    role: inter-network
+}
+
+autoLink {
+    connection: federation
+    direction: in
+    externalAddress: _xnet/net-B
+}
+
+autoLink {
+    connection: federation
+    direction: in
+    address: service-B-to-A
+}
+```
+
+## Using Cross-Network Connectivity as an Endpoint
+
+## Backward Compatibility and Breaking Changes
+
+The `qd.cross-network` link capability is needed to provide backward compatibility with the new changes.  At the next major version (v4), it is recommended that the `_xnet` address format be deprecated and its functionality be rolled into an extension of the `_topo` format for use by all endpoints.  At this point, it would also be recommended to leave `_xnet` in place as an alias for `_topo` to protect applications written to use it.
+
+Changing the `_topo` format between major versions will break networks that are partially upgraded (i.e. routers with different formats of the topological address).

include/qpid/dispatch/amqp.h

@@ -140,11 +140,12 @@ extern const char * const QD_CAPABILITY_ROUTER_DATA;
 extern const char * const QD_CAPABILITY_EDGE_DOWNLINK;
 extern const char * const QD_CAPABILITY_STREAMING_DELIVERIES;
 extern const char * const QD_CAPABILITY_RESEND_RELEASED;
+extern const char * const QD_CAPABILITY_CROSS_NETWORK;
 /// @}
 
 /** @name Dynamic Node Properties */
 /// @{
-extern const char * const QD_DYNAMIC_NODE_PROPERTY_ADDRESS;  ///< Address for routing dynamic sources
+extern const char * const QD_DYNAMIC_NODE_PROPERTY_ADDRESS;       ///< Address for routing dynamic sources
 /// @}
 
 /** @name Connection Properties */

include/qpid/dispatch/iterator.h

@@ -49,6 +49,7 @@ typedef struct qd_iterator_t qd_iterator_t;
  */
 #define QD_ITER_HASH_PREFIX_TOPOLOGICAL           'T'
 #define QD_ITER_HASH_PREFIX_LOCAL                 'L'
+#define QD_ITER_HASH_PREFIX_NETWORK               'N'
 #define QD_ITER_HASH_PREFIX_AREA                  'A'
 #define QD_ITER_HASH_PREFIX_ROUTER                'R'
 #define QD_ITER_HASH_PREFIX_MOBILE                'M'
@@ -137,6 +138,11 @@ void qd_iterator_finalize(void);
  */
 void qd_iterator_set_address(bool edge_mode, const char *area, const char *router);
 
+/**
+ * Set the network ID for the local router.  This can be updated repeatedly during run-time.
+ */
+void qd_iterator_set_network(const char *network);
+
 /**
  * Add and delete peer-edge router identities.  When in edge mode, peer edge routers
  * result in different hash results than remote edge routers.  These functions are used

include/qpid/dispatch/protocol_adaptor.h

@@ -347,6 +347,7 @@ typedef enum {
     QDR_ROLE_EDGE_CONNECTION,
     QDR_ROLE_INTER_ROUTER_DATA,
     QDR_ROLE_INTER_EDGE,
+    QDR_ROLE_INTER_NETWORK,
 } qdr_connection_role_t;
 
 typedef void (*qdr_connection_bind_context_t) (qdr_connection_t *context, void *token);
@@ -424,6 +425,7 @@ void qdr_connection_set_tracing(qdr_connection_t *conn, bool enable_tracing);
 void qdr_core_close_connection(qdr_connection_t *conn);
 
 bool qdr_connection_route_container(qdr_connection_t *conn);
+bool qdr_connection_inter_network(qdr_connection_t *conn);
 
 /**
  * qdr_connection_set_context

include/qpid/dispatch/router_core.h

@@ -49,7 +49,12 @@ ENUM_DECLARE(qd_router_mode);
 /**
  * Allocate and start an instance of the router core module.
  */
-qdr_core_t *qdr_core(qd_dispatch_t *qd, qd_router_mode_t mode, const char *area, const char *id, const char *van_id);
+qdr_core_t *qdr_core(qd_dispatch_t *qd, qd_router_mode_t mode, const char *area, const char *id, const char *tenant_id);
+
+/**
+ * Set the network id
+ */
+void qdr_core_set_network_id(qdr_core_t *core, const char *network_id);
 
 /**
  * Stop and deallocate an instance of the router core.
@@ -75,12 +80,12 @@ qd_dispatch_t *qdr_core_dispatch(qdr_core_t *core);
 void qdr_process_tick(qdr_core_t *core);
 
 /**
- * @brief Return the text of the router's virtual application network ID, or 0.
+ * @brief Return the text of the router's virtual application network tenant ID, or 0.
  *
  * @param core Pointer to the core object returned by qdr_core()
- * @return const char* null-terminated text of the van-id or 0 if there's no id.
+ * @return const char* null-terminated text of the tenant-id or 0 if there's no id.
  */
-const char *qdr_core_van_id(const qdr_core_t *core);
+const char *qdr_core_tenant_id(const qdr_core_t *core);
 
 
 /**

python/skupper_router/management/skrouter.json

@@ -462,6 +462,16 @@
                     "description": "Number of deliveries that were sent to route container connections.",
                     "graph": true
                 },
+                "deliveriesIngressInterNetwork": {
+                    "type": "integer",
+                    "description": "Number of deliveries that were received from a connected network.",
+                    "graph": true
+                },
+                "deliveriesEgressInterNetwork": {
+                    "type": "integer",
+                    "description": "Number of deliveries that were sent to a connected network.",
+                    "graph": true
+                },
                 "residentMemoryUsage": {
                     "type": "integer",
                     "graph": true,
@@ -511,7 +521,7 @@
                     "create": true
                 },
                 "vanId": {
-                    "description":"Deprecated:  See the managedRouter entity definition.",
+                    "description":"Deprecated:  See the network tenantId attribute.",
                     "type": "string",
                     "required": false,
                     "create": true
@@ -633,13 +643,21 @@
             }
         },
 
-        "managedRouter": {
-            "description": "This optional entity holds configuration needed for the router to be managed by a central manager",
+        "network": {
+            "description": "This optional entity holds configuration needed for the router to participate in inter-network communication",
             "extends": "configurationEntity",
             "singleton": true,
+            "operations": ["UPDATE"],
             "attributes": {
-                "vanId": {
-                    "description":"The unique ID of the Virtual Application Network that this router is a member of.",
+                "networkId": {
+                    "description":"The unique ID of the AMQP Network that this router is a member of.",
+                    "type": "string",
+                    "required": false,
+                    "create": true,
+                    "update": true
+                },
+                "tenantId": {
+                    "description":"The unique VAN tenant ID for routers that are connected to a multi-tenant service backbone.",
                     "type": "string",
                     "required": false,
                     "create": true
@@ -648,7 +666,7 @@
         },
 
         "site": {
-            "description":"Optional.  If present, this entity causes the router to emit a SITE record for VanFLow in the case that there is not an external controller to emit this record.",
+            "description":"Optional.  If present, this entity causes the router to emit a SITE record for VanFlow in the case that there is not an external controller to emit this record.",
             "extends": "configurationEntity",
             "attributes": {
                 "location": {
@@ -784,7 +802,8 @@
                         "inter-router",
                         "route-container",
                         "edge",
-                        "inter-edge"
+                        "inter-edge",
+                        "inter-network"
                     ],
                     "default": "normal",
                     "description": "The role of an established connection. In the normal role, the connection is assumed to be used for AMQP clients that are doing normal message delivery over the connection.  In the inter-router role, the connection is assumed to be to another router in the network.  Inter-router discovery and routing protocols can only be used over inter-router connections. route-container role can be used for router-container connections, for example, a router-broker connection.  In the edge role, the connection is assumed to be between an edge router and an interior router.",
@@ -958,7 +977,8 @@
                         "inter-router",
                         "route-container",
                         "edge",
-                        "inter-edge"
+                        "inter-edge",
+                        "inter-network"
                     ],
                     "default": "normal",
                     "description": "The role of an established connection. In the normal role, the connection is assumed to be used for AMQP clients that are doing normal message delivery over the connection.  In the inter-router role, the connection is assumed to be to another router in the network.  Inter-router discovery and routing protocols can only be used over inter-router connections. route-container role can be used for router-container connections, for example, a router-broker connection.  In the edge role, the connection is assumed to be between and edge router and an interior router.",
@@ -1598,6 +1618,16 @@
                     "description": "Number of deliveries that were sent to a route-container address.",
                     "graph": true
                 },
+                "deliveriesIngressInterNetwork": {
+                    "type": "integer",
+                    "description": "Number of deliveries that were received from a connected network.",
+                    "graph": true
+                },
+                "deliveriesEgressInterNetwork": {
+                    "type": "integer",
+                    "description": "Number of deliveries that were sent to a connected network.",
+                    "graph": true
+                },
                 "key": {
                     "description": "Internal unique (to this router) key to identify the address",
                     "type": "string"

python/skupper_router_internal/dispatch.py

@@ -103,7 +103,8 @@ def __init__(self) -> None:
         self._prototype(self.qd_error_code, c_long, [], check=False)
         self._prototype(self.qd_error_message, c_char_p, [], check=False)
         self._prototype(self.qd_log_entity, c_long, [py_object])
-        self._prototype(self.qd_dispatch_configure_managed_router, None, [self.qd_dispatch_p, py_object])
+        self._prototype(self.qd_dispatch_configure_network, None, [self.qd_dispatch_p, py_object])
+        self._prototype(self.qd_dispatch_update_network, None, [self.qd_dispatch_p, py_object])
         self._prototype(self.qd_dispatch_configure_router, None, [self.qd_dispatch_p, py_object])
         self._prototype(self.qd_dispatch_configure_site, None, [self.qd_dispatch_p, py_object])
         self._prototype(self.qd_dispatch_prepare, None, [self.qd_dispatch_p])

python/skupper_router_internal/management/agent.py

@@ -280,15 +280,18 @@ def __str__(self):
         return super(RouterEntity, self).__str__().replace("Entity(", "RouterEntity(")
 
 
-class ManagedRouterEntity(EntityAdapter):
+class NetworkEntity(EntityAdapter):
     def __init__(self, agent, entity_type, attributes=None):
-        super(ManagedRouterEntity, self).__init__(agent, entity_type, attributes, validate=False)
+        super(NetworkEntity, self).__init__(agent, entity_type, attributes, validate=False)
 
     def create(self):
-        self._qd.qd_dispatch_configure_managed_router(self._dispatch, self)
+        self._qd.qd_dispatch_configure_network(self._dispatch, self)
+
+    def _update(self):
+        self._qd.qd_dispatch_update_network(self._dispatch, self)
 
     def __str__(self):
-        return super(ManagedRouterEntity, self).__str__().replace("Entity(", "ManagedRouterEntity(")
+        return super(NetworkEntity, self).__str__().replace("Entity(", "NetworkEntity(")
 
 
 class SiteEntity(EntityAdapter):

python/skupper_router_internal/management/config.py

@@ -304,7 +304,7 @@ def configure(attributes):
 
     # Configure and prepare the router before we can activate the agent.
     configure(config.by_type('router')[0])
-    configure(config.by_type('managedRouter')[0])
+    configure(config.by_type('network')[0])
     qd.qd_dispatch_prepare(dispatch)
     qd.qd_router_setup_late(dispatch)  # Actions requiring active management agent.
     agent.activate("$_management_internal")