Skip to content

Latest commit

 

History

History
196 lines (141 loc) · 8.11 KB

README.md

File metadata and controls

196 lines (141 loc) · 8.11 KB

Implement a simple "Consumer Pull" Http transfer flow

The purpose of this sample is to show a data exchange between two connectors, one representing the data provider and the other, the consumer. You can find more details on consumer pull and provider push transfers documentation.

This sample consists of the following steps:

  • Provide a simple Http Proxy data plane implementation
  • Perform a file transfer initiated by the consumer
  • The provider will send an EndpointDataReference to the consumer
  • The consumer will call the endpoint and fetch the data

The Http Proxy functionality is not provided by EDC because out there is plenty of FOSS proxy implementations that can be extended to being able to interact with the EDC, but for example purposes we will implement a really basic Http Proxy that, please note, it's not meant to be used in production.

Prerequisites

The following steps assume you already built the base connector in the Prerequisites, that will be used for the "consumer" instance.

Implement a simple Http Proxy data plane

In the provider-proxy-data-plane there's a connector based on the basic one but with the data-plane-api-v2 module excluded, because that one has been deprecated:

    runtimeOnly(project(":transfer:transfer-00-prerequisites:connector")) {
        exclude("org.eclipse.edc", "data-plane-public-api-v2")
    }

The proxy implementation provided in this runtime is really bare-bone and it supports only GET. Needless to say, it is only for sample purposes, for a production environment you should rely on a proper proxy implementation.

Here's the rundown of the implementation:

CustomProxyDataPlaneExtension

Is the extension that takes care to inject the needed services and:

  • registers a PortMapping on the PortMappingRegistry: this tells the underlying Web Server (jetty) to expose an endpoint under the configured port and path
  • adds an endpoint generator function on the generatorFunction, this will ensure that all the EndpointDataReferences generated by the connector will point to the data-plane proxy public endpoint. Note that this endpoint can be configured, because in a real world scenario it will need to point to a public endpoint exposed on the internet.
  • registers the ProxyController, that is the actual proxy implementation.

Please check out the code.

ProxyController

This is modeled as a classic Jersey controller, implementing only, as stated above, only the GET method. On every call the proxy will:

  • verify the Authorization token
  • extract the source DataAddress from the token using the authorizationService
  • create and send an http request to the source server
  • pipe the response body stream to the output, adding status code and content-type information.

Please check out the code.

Run the sample

Running this sample consists of multiple steps, that are executed one by one and following the same order.

Build the provider connector

./gradlew transfer:transfer-03-consumer-pull:provider-proxy-data-plane:build

Run the connectors

To run the provider, just run the following command

java -Dedc.fs.config=transfer/transfer-03-consumer-pull/resources/configuration/provider.properties \
    -jar transfer/transfer-03-consumer-pull/provider-proxy-data-plane/build/libs/connector.jar

To run the consumer, just run the following command (different terminal). Note that the consumer is the same that was built in the prerequisites sample

java -Dedc.fs.config=transfer/transfer-00-prerequisites/resources/configuration/consumer-configuration.properties \
  -jar transfer/transfer-00-prerequisites/connector/build/libs/connector.jar

Negotiate the contract

You can follow the instructions in the transfer-01-negotiation to set up a Contract Negotiation between the two parts.

Start the transfer

In the request body, we need to specify which asset we want transferred, the ID of the contract agreement, the address of the provider connector and where we want the file transferred. Before executing the request, insert the contractAgreementId from the previous chapter. Then run:

curl -X POST "http://localhost:29193/management/v3/transferprocesses" \
  -H "Content-Type: application/json" \
  -d @transfer/transfer-03-consumer-pull/resources/start-transfer.json \
  -s | jq

the "HttpData-PULL" transfer type is used for the consumer pull method, and it means that it will be up to the consumer to request the data to the provider and that the request will be a proxy for the datasource

Then, we will get a UUID in the response. This time, this is the ID of the TransferProcess (process id) created on the consumer side, because like the contract negotiation, the data transfer is handled in a state machine and performed asynchronously.

Sample output:

{
  ...
  "@id": "591bb609-1edb-4a6b-babe-50f1eca3e1e9",
  "createdAt": 1674078357807,
  ...
}

Check the transfer status

Due to the nature of the transfer, it will be very fast and most likely already done by the time you read the UUID.

curl http://localhost:29193/management/v3/transferprocesses/<transfer process id> | jq

You should see the Transfer Process in STARTED state:

{
  ...
  "@id": "591bb609-1edb-4a6b-babe-50f1eca3e1e9",
  "state": "STARTED",
  ...
}

Note that for the consumer pull scenario the TP will stay in STARTED state after the data has been transferred successfully. It might get eventually get shifted to TERMINATED or DEPROVISIONED by other components, but this is not scope of this sample.

Check the data

At this step, an EndpointDataReference would have been generated by the provider and sent to the consumer. The latter stored it in a cache, so we can obtain it using the transfer process id:

curl http://localhost:29193/management/v3/edrs/<transfer process id>/dataaddress | jq
{
  "@type": "DataAddress",
  "type": "https://w3id.org/idsa/v4.1/HTTP",
  "endpoint": "http://localhost:19291/public",
  "authType": "bearer",
  "endpointType": "https://w3id.org/idsa/v4.1/HTTP",
  "authorization": "eyJraWQiOiJwdWJsaWMta2V5IiwiYWxnIjoiUlMyNTYifQ.eyJpc3MiOiJwcm92aWRlciIsImF1ZCI6ImNvbnN1bWVyIiwic3ViIjoicHJvdmlkZXIiLCJpYXQiOjE3MTc3NjkyMzEyOTYsImp0aSI6IjM2M2RhMGU4LWZmOGItNDY1My05YjQwLWY4MjdlMWMzOGMzYyJ9.WOVPz6m7XzIrbiMTfLqOXacGYz8Xk_-iQu7gmxoIgDFYsgo0da2Iv51EsugIpqbodPsmB0kK7zkyrmsFOfAASAq7fjsy4gQF-u5egYwoGpcxjYaJJdQa5lkwjC0fRxdVFVwZwrOaT5Mg-vGA9HssTEnlA64q-O0ae_aTH5ToflmPDM3FhAgL55I3odM5ysM2POEJY6pgOxIV9XjuhZFl_i_iTiUCZy__oQUZiYk58wKoqfK758Sy1WzpH-eyZCDUi_Z3n6cJB80_0ZThoPhtiFH7Tl9DfStnjsCoaeqMLFnTXp0s8h4ZGFmjfBc-72aAdRQqqLDT8WXNg3Csv5B56Q",
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "odrl": "http://www.w3.org/ns/odrl/2/"
  }
}

Once this json is read, use a tool like postman or curl to execute the following query, to read the data

curl --location --request GET 'http://localhost:19291/public/' --header 'Authorization: <edr authorization attribute>'

At the end, and to be sure that you correctly achieved the pull, you can check if the data you get is the same as the one you can get at https://jsonplaceholder.typicode.com/users

Since we configured the HttpData with proxyPath, we could also ask for a specific user with:

curl --location --request GET 'http://localhost:19291/public/1' --header 'Authorization: <auth code>'

And the data returned will be the same as in https://jsonplaceholder.typicode.com/users/1

Your first data transfer has been completed successfully.

Next Chapter