notifications on deletion of entities

[pdmangel2]

I found following telefonicaid#3561 (comment) regarding notifications on deletion of entities.
Is there any documentation on how to subscribe for such notifications ?

[kzangeli]

Yes, it is documented in the NGSI-.LD API spec v1.6.1.
Only, Orion-LD doesn't support it yet.
Just ... the issue you mention is about telefonicas Orion. You are now in the github of FIWARE Foundation's Orion-LD.
Not the same ...
That said. We have "notifications on deleted entities" quite high on our backlog.
Might move it even higher if you ask me to :)

[pdmangel2]

Thx for the clarification Ken!
My mistake ... I interpreted 'implementing ... in NGSI-LD API' as 'implementing in Orion-LD'.

We need this feature in Orion-LD, so yes please move its priority higher :-)

[kzangeli]

ok! :)

[kzangeli]

#280 (Look at the very first line of the ToDo)

[fgalan]

... the issue you mention is about telefonicas Orion

Orion is a full fledged FIWARE GE component. Who is the owner is not actually relevant as far it is open source. The main difference is that Orion implements NGSIv2 with a strong production scenarios commitment while Orion-LD implements NGSI-LD.

[kzangeli]

Implemented notifications for "delete entity" in PR #1309

Just one thing, you need to use the "new implementation" for it to work.
The CLI option for that is called -experimental. Don't let its name frighten you (the name is old and should be called -mongoc), I have +600 functests in place for it and the only tests missing for making it the default behaviour is a good concurrency and performance test (I expect it to be MUCH faster).
I have people starting to work on this and, hopefully soon, the "experimental behaviour" will be the default behaviour. FYI, "experimental_" is about a rewrite of the DB layer, using the new mongoc driver for MongoDB instead of the old deprecated legacy driver.

[kzangeli]

So, please test and let me know (by closing the issue if all is A-OK).

The "data" of the notification of a deleted entity looks like this:

{
  "id": "whatever URI the entity had",
  "type": "whatever type the entity had "
  "deletedAt": "timestamp of when the deletion occurred"
}

[pdmangel2]

I can certainly do some testing. And also looking forward to test the performance improvements.
Since we are only working with released docker versions (docker hub), which tag (version) should I use to test ?

[kzangeli]

Just use the lat4est you can find (nota bene, don't ever use the tag "latest", unless it's pure testing)

[pdmangel2]

I switched from 1.1.1 to 1.2.0-PRE-1247 and added -experimental and I'm facing issues with subscriptions. Need to dig some deeper, but after removing all subscriptions manually from the DB and restarting everything, I don't get any updates anymore.
Did something fundamentally change about subscriptions ?

[kzangeli]

I heard about subscription problems elsewhere, just a few hours ago.
No, nothing has changed really, and all functests pass ...
So, a bit weird. Need more info; I need to be able to reproduce the problem in order to fix it

[pdmangel2]

I created new issues for the subscription problems:

• Suscription not found by id when witching to new verions of orion-ld #1316
• Creating subscription results in error in orion-ld log #1317

[PietroGreco]

Hi @kzangeli,

I was trying out the delete notification on the latest version of the orion-ld (docker container), and I noticed that the notification sent by the context broker does not forward the fiware-correlator.

I use the fiware-correlator to implement a mechanism that prevents a context producer from receiving (through subscription) the updates that itself has generated. Basically it prevents self notification loops. I use this solution since, as far as I know, orion-ld does not provide such function out-of-the-box, correct?

Could you add forwarding for the fiware-correlator in delete notifications? It is very useful to me.

Thank you very much,
Pietro

P.S: I have also seen that in the Planning you have the "fiware-correlator" in the priority discussions list where you are evaluating whether you need it or not. As I have mentioned, I need it for the purpose I have illustrated above. Will you keep the fiware-correlator? If not, how could I implement a self-notification loop protection? Thank you very much again

[kzangeli]

ok, sure, I can do that.
The correlator is an NGSIv2 thing, it's not part of NGSI-LD, that's why it's not there.
Actually ... think we have a mechanism for this (for any header to be included in a forwarded message).
I'm just not sure I've implemented it yet.
I'll find out and let you know.

[PietroGreco]

ok, sure, I can do that.

Ok, thank you very much, appreciated.

I'll find out and let you know.

Ok, thank you again!

[kzangeli]

So, I looked into this and the mechanism I mentioned (for any header to be included in a forwarded message) is the receiverInfo key-value array in the Subscription.
However, you can only define static values for the headers in receiverInfo.

We implemented a special string ("urn:ngsi-ld:request") so that the broker would instead of blindly copy the value from the subscription, instead take the value from the initial request (the creation/update that triggered the notification), and thus have "real forwarding" of HTTP headers. BUT, we did that only for distributed opertations.
For some reason we didn't think of notifications.

Anyway, I contacted my colleagues in ETSI ISG CIM (where we define the NGSI-LD API) and we agreed to add this feature also to notifications.

So, I went ahead and implemented it. The PR should be in by tonight.

This is how to use it (tailor-made for FIWARE-Correlator, but works for any header):

In your subscription, use the field "receiverInfo".
Here's an example payload body for a Subscription-creation (the one I just used to implement this):

POST /ngsi-ld/v1/subscriptions
{
  "id": "urn:ngsi-ld:subs:S1",
  "type": "Subscription",
  "name": "Sub 01",
  "entities": [
    {
      "id": "urn:ngsi-ld:entities:T:E1",
      "type": "T"
    }
  ],
  "notification": {
    "format": "concise",
    "endpoint": {
      "uri": "http://127.0.0.1:'${LISTENER_PORT}'/notify",
      "receiverInfo": [
        {
          "key": "H1",
          "value": "123"
        },
        {
          "key": "H2",
          "value": "456"
        },
        {
          "key": "H3",
          "value": "789"
        },
        {
          "key": "FIWARE-Correlator",
          "value": "urn:ngsi-ld:request"
        }
      ]
    }
  }
}

The part you need to use is the "notification::endpoint::receiverInfo::FIWARE-Correlator"

[PietroGreco]

Great! I look forward to trying it out
Thank you very much @kzangeli!

[PietroGreco]

Hi @kzangeli,

I was testing the latest version. Whilst I can see the FIWARE-Correlator forwarded in the notifications for creation and update requests I still cannot see it for the deletion requests.

[kzangeli]

ok, never tested that.
I assumed I'd "get it for free".
Common functions are doing this.
Weird ...
I'll look into it.

Just to clarify, you mean you don't get the header in the notification triggered by the delete request, right?

[PietroGreco]

Exactly, I set the header in the deletion request but I don't see it arriving in the notification triggered by the delete request.

[kzangeli]

So, I extended the functest I had for notifications on deletion of entities to include a FIWARE-Correlator header
and it works just fine.
This is how the notification looks like:

POST http://xps/notify?subscriptionId=urn:S1
Content-Length: 233
User-Agent: orionld/post-v1.1.0
Host: xps
Accept: application/json
Content-Type: application/json
Fiware-Correlator: abc
Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"
Ngsild-Attribute-Format: Normalized

{
    "data": [
        {
            "deletedAt": "2023-04-28T09:30:58.780Z",
            "id": "urn:E1",
            "type": "T"
        }
    ],
    "id": "urn:ngsi-ld:Notification:622dca89-e5a7-11ed-83c3-6dd2ef25ba7d",
    "notifiedAt": "2023-04-28T09:30:58.780Z",
    "subscriptionId": "urn:S1",
    "type": "Notification"
}

So, not able to reproduce.
I'm gonna need more info for your case.
If I can't reproduce, I can't fix ...

Paste your complete payload for creation of the subscription and deletion of the entity.
I'll copy that info into a new functest and perhaps I can reproduce.

[PietroGreco]

Hi @kzangeli,

I've tested it again but I found a strange behaviour.

Here are the requests that I make with curl and a description of what happens

Subscription

payload='{
  "id": "urn:ngsi-ld:subs:S1",
  "type": "Subscription",
  "name": "Sub 01",
  "entities": [
    {
      "type": "Room"
    }
  ],
  "notification": {
    "format": "concise",
    "endpoint": {
      "uri": "http://192.168.1.5:1028/accumulate",
      "receiverInfo": [
        {
          "key": "H1",
          "value": "123"
        },
        {
          "key": "H2",
          "value": "456"
        },
        {
          "key": "H3",
          "value": "789"
        },
        {
          "key": "FIWARE-Correlator",
          "value": "urn:ngsi-ld:request"
        }
      ]
    }
  }
}'
curl -X POST -H "Content-Type: application/json" 'http://192.168.1.5:1026/ngsi-ld/v1/subscriptions/' -d "$payload"

This is the payload you used. I've only removed 'id' from the 'entities' object and changed the type to 'Room'

Entity Creation

payload='{
  "id": "urn:entities:E1",
  "type": "Room",
  "P1": {
    "type": "Property",
    "value": 23
  }
}'
curl -X POST -H "Content-Type: application/json" -H "FIWARE-Correlator: 66e71193-76b9-4c58-8e11-887766162699" 'http://192.168.1.5:1026/ngsi-ld/v1/entities' -d "$payload"

Entity Deletion
curl -X DELETE -H "FIWARE-Correlator: c846fdbb-b8ad-41f7-ba21-848a6a8ecfda" 'http://192.168.1.5:1026/ngsi-ld/v1/entities/urn:entities:E1'

Here is what happens.
I subscribe to Orion and then I create the entity. I can correctly see the notification including the FIWARE-Correlator and the other three headers (I'm using the FIWARE accumulator server for visualizing the notifications. I've only added H1, H2, H3 to the array used to sort the headers)

However, when I send the deletion request this is what I get:

POST http://orion/accumulate?subscriptionId=urn:ngsi-ld:subs:S1
Content-Length: 258
User-Agent: orionld/post-v1.1.0
Host: orion
Accept: application/json
Content-Type: application/json
**Fiware-Correlator: c846fdbb-b8ad-41f7-ba21-84H1: 123**
Ngsild-Attribute-Format: Concise
Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"

{
    "data": [
        {
            "deletedAt": "2023-04-28T11:49:19.944Z",
            "id": "urn:entities:E1",
            "type": "Room"
        }
    ],
    "id": "urn:ngsi-ld:Notification:b610449c-e5ba-11ed-9b10-0242c0a8d003",
    "notifiedAt": "2023-04-28T11:49:19.944Z",
    "subscriptionId": "urn:ngsi-ld:subs:S1",
    "type": "Notification"
}
=======================================

<ipaddr> - - [28/Apr/2023 13:49:19] "POST /accumulate?subscriptionId=urn:ngsi-ld:subs:S1 HTTP/1.1" 200 -

This time I see the FIWARE-Correlator but there's also "H1: 123" appended to that header and I don't see the other two

I've also tried starting again from scratch and if I remove H1, H2 and H3 from the subscription payload and try to delete the entity again, this time I don't receive the notification at all.

Thank you again

[kzangeli]

Ok, that clearly looks like a bug.
I'll have a look after lunch!

[PietroGreco]

Another thing I'm noticing in the Orion-LD logs for the deletion notification is:

notificationSend returned fd 26 (-2 is OK if HTTPS)

while in the other cases it says "notification sent successfully"

[kzangeli]

Yeah, that last thing is nothing to worry about.
For HTTP notifications that function returns a file descriptor, while for HTTPS, as libcurl is used, it can't return a file descriptor.
-1 means error while -2 means "all is OK I just don't have a file descriptor to give you"

[kzangeli]

Found the other thing. It's a bug. IDIOT bug !
First I calculate the number of bytes to hold key and value for the header.
Then, as the value is "urn:ngsi-ld:request", I change the value.
Thus, the total size I've already calculated isn't valid anymore.
Silly me! :)

I was just "lucky" that in my functest I used an example "FIWARE-Correlator: abc", the string is shorter than is was (with "urn:ngsi-ld:request"). Your example uses a longer value and of course, it provokes a memory corruption.

Stupid mistake, easy fix (recalculate the total length), ready in an hour or so.

[kzangeli]

The PR has been merged. Please test and let me know

[PietroGreco]

Ok, tested it and it works!

I tried both with only the FIWARE-Correlator header in receiverInfo and also with the other three H1, H2 and H3. The behaviour is as expected.

Thank you very much again @kzangeli!

[kzangeli]

ok, happy to hear it.
I'll then go ahead and close the issue

[pdmangel2]

I'm doing some tests with 1.3.0 (coming from 1.1.1)

• I'm not getting entity delete notifications
• with -experimental I'm not getting any notification anymore

Any idea what I'm doing wrong ?