Upserting Entities with special characters leads to inconsistancies

[hibke]

Hello,

We found that there is a bug in Orion-LD related to special characters. When using entityOperations/upsert to create or update a list of entities, if an attribute of one of the upserted entites contains a special character (e.g.: '), then Orion does not create any temporal data for the whole list of given entities, and no error message is returned.

Then, performing a GET /ngsi-ld/v1/entities/ on any of the upserted entities returns the entity. However, a Get request using Mintaka on “temporal/entities/” returns a 404 Not Found Error (even if the entity did not have any special character in its attributes).

We did check Timescaledb, we found out that no entity was created by the creation request (entities with or without special characters). The creation appears successful, but the entities are not available in the temporal database. Which means inconsistancy between context and temporal data.

Steps to reproduce:

1. Create entities using /ngsi-ld/v1/entityOperations/upsert?options=update where urn:Site:SpecialCharactersBugTest1 entity has an attribute name with the value NICE D'R ST BARTHELEMY

Request Body: 

[
  {
    "@context": "https://live-buildings.github.io/data-model/context-ngsi.jsonld",
    "id": "urn:Site:SpecialCharactersBugTest1",
    "type": "Site",   
    "name": {
      "type": "Property",
      "observedAt": "2023-02-02T12:42:00Z",
      "value": "NICE D'R ST BARTHELEMY"
    }
  },
  {
    "@context": "https://live-buildings.github.io/data-model/context-ngsi.jsonld",
    "id": "urn:Site:SpecialCharactersBugTest2",
    "type": "Site",
    "name": {
      "type": "Property",
      "observedAt": "2023-02-02T12:42:00Z",
      "value": "NICE GROSSO BAUMETTES"
    }
  }
]

2. Verify Context entities with:

Get /ngsi-ld/v1/entities/urn:Site:SpecialCharactersBugTest1
Get /ngsi-ld/v1/entities/urn:Site:SpecialCharactersBugTest2

These requests return the entities with status code 200 OK.

3. Temporal Entities

   1. Using Mintaka

   Get /temporal /entities/urn:Site:SpecialCharactersBugTest1
Get /temporal /entities/urn:Site:SpecialCharactersBugTest2

   These requests return a 404 not found error

   2. TimescaleDB verification

   Run the SELECT query:

   SELECT COUNT(*) FROM entities WHERE id IN ('urn:Site:SpecialCharactersBugTest1','urn:Site:SpecialCharactersBugTest2');

   Result: 0 rows

count
-------
    0
(1 row)

[kzangeli]

ok, that seems like a bug indeed. Important one if you ask me!
Thank you very much for reporting.
I will look into this as soon as I can, after a few fires we're experiencing right now

[kzangeli]

Just a quick comment.
If you're in a hurry and want to fix the problem right now, just urlencode the values.
I'm almost 100% sure that will do the trick.
That's what I will implement (should have implemented that, long time ago) once I have a few hours to decidate to this, hopefully tomorrow.

[kzangeli]

I just found this issue ...
So sorry, completely forgot about it!
I hope you followed my advice to URL-Encode your attribute values.
Anyway, the fix is on its way.
Better soon than never ...
Sorry again.