SubscribeForNotifications() usage and documentation

[jrlaff]

Hi,

Trying to use Realm in a dotnet application, I need to be notified about changes in a Realm query to monitor new/updated/deleted objects. I need to be able, among other things, to determine which objects have changed, and which changes have occurred in these objects.

To that extend, I try to properly use the Realms.CollectionExtensions.SubscribeForNotifications() method.

My first issue is when looking at the documentation: I find two conflicting descriptions for the Realms.NotificationCallbackDelegate's changes parameter:
https://docs.mongodb.com/realm-sdks/dotnet/latest/reference/Realms.IRealmCollection-1.html

The changes parameter will be null the first time the callback is invoked with the initial results. For each call after that, it will contain information about which rows in the results were added, removed or modified.

https://docs.mongodb.com/realm-sdks/dotnet/latest/reference/Realms.NotificationCallbackDelegate-1.html

changes
The ChangeSet describing the changes to a IRealmCollection, or null if an has error occurred.

Having debugged my application, I find that the first description seems to be the correct one.

The second issue I have is with the last remark for Realms.NotificationCallbackDelegate:
https://docs.mongodb.com/realm-sdks/dotnet/latest/reference/Realms.IRealmCollection-1.html

Notifications are delivered via the standard event loop, and so can't be delivered while the event loop is blocked by other activity. When notifications can't be delivered instantly, multiple notifications may be coalesced into a single notification. This can include the notification with the initial collection.

How can the notification with the initial collection be coalesced with subsequent changes notifications, as the two types of notifications rely on the changes parameter being null or not? How can I then distinguish between the two? Am I missing something?

Thanks.

[papafe]

Hi @jrlaff, you're right that the documentation could be clearer on how to interpret the different variables of the notification callback. The ChangeSet should be null both in the first notification and if an error has occurred.
Regarding on your second question, what would happen in this case is that you will see just one notification with changes parameter equal to null. It'll practically have the same format as if it was the first notification.

[LaPeste]

Hi @jrlaff,

Just to add a little more info on the subject.
About your first question you can find an answer in the remarks of SubscribeForNotifications(NotificationCallbackDelegate). Just a little after the sentence you quoted. It says:

If an error occurs the callback will be invoked with null for the sender parameter and a non-null error.

So there's no conflicting description. In NotificationCallbackDelegate<T>, changes is always null the first time you receive the notification. Afterwards it contains the changes that happened. At the same time, error is null if no errors have occurred, otherwise it holds info about the exception. So it would be like the following:

1. First notification, changes is null, but if error isn't null then an error occurred
2. Any following notification, changes isn't null, unless an error occurred. In which case, sender is null and you should check against error

Concluding, you should always check for error first. And if null, continue. Otherwise take care of the issue.

About the second issue, as written in the reference, changes is always null at the first notification. What that sentence that you quoted means is the following:
If, for example, you subscribe for changes in a collection and then immediately perform some writes to that collection, the initial notification may reflect the collection state after those writes. This simply means that changes is still null, but the collection may reflect the aforementioned writes you made. Effectively, this means that you lost a notification explaining what changes were made. Nothing more than this. After the first notification, any subsequent one may contain changes not executed in only 1 write but in multiple ones.

I hope this helps to get even more insight on the matter.

Andrea

[jrlaff]

Thanks @LaPeste; it's clearer now.

[A9G-Data-Droid]

I don't see error on these objects or in the documentation for this feature. How does one check for error in the latest version?

[peppy]

@A9G-Data-Droid See https://github.com/realm/realm-dotnet/releases/tag/11.0.0, it was removed as unused (so you no longer need to check for errors).

[A9G-Data-Droid]

Awesome! Thanks

[peppy]

Might be worthy of creating a separate issue, but there are two more uncertain areas regarding this function's usage:

• The current documentation also doesn't explain how to work with deleted indices. It refers to the "previous version" but does not say how you would go about accessing that previous version. Is it safe to assume that the sender is always the previous version in cases of deleted items?
• According to the xmldocs, moves also appear as insertions and deletions. If we were to care about deletions but not moves, is it safe to early-abort if changeSet.Moves.Any()? ie. is the callback called once per discrete operation type?

[peppy]

That does sound fair for the deletion case if one is only considering complete deletion (where a reference to the object likely won't exist in realm any more). The case I'm using it for here is where a custom LINQ IQueryable filter has been applied to the collection and I am watching for items entering or leaving the collection. The hope would be that I don't need to maintain a copy of the previous state just to get a reference to an object leaving the criteria I am monitoring.

[jrlaff]

Indeed I didn't think about this use-case. Have you been able to experiment it?

[nirinchev]

Unfortunately, there's no way to access the deleted objects, even if they are still physically present in the Realm file. This is something we'd like to eventually support, but don't right now. If this is a fast-changing collection, you can probably freeze the collection every time the callback is invoked and then look up the removed objects in the frozen collection. You have to be careful with that approach though as freezing pins the version of the Realm, so if you have a lot of changes in the Realm that don't trigger your callback and cause re-freezing the collection, you can end up with massive filesize growth.

Regarding moves - no, it's not safe to early-exit if the Moves collection contains elements - the callback will be invoked with a changeset containing all changes - so for example, if you did:

list.Move(from: 2, to: 5);
list.RemoveAt(0);

You'll end up with a changeset looking like:

{
  DeletedIndices: [0, 2],
  InsertedIndices: [4], // Because 0 is now removed
  Moves: [{
    From: 2,
    To: 4
  }]
}

[peppy]

Thanks for the answers.

That's quite unfortunate for our particular use case, but I can understand from a "we are going to display all results in a list view" perspective why it is done this way.

An example of when we would require such behaviour is subscribing to a large collection for any changes to a bool SoftDeleted flag, to react to and potentially add or remove an item from an externally filtered list. The way realm exposes changes works fine if the displayed content is sourced 100% via realm's filter engine (at which point the indices match what you have displayed) but this may not always be feasible for more advanced usages. When it's not, without the freezing workaround, it seems to be impossible to know what the deleted (aka removed) indices refer to.

[A9G-Data-Droid]

I would never assume that my copy of the database is safe to grab an object from based on index. There needs to be a way to get the primary key of the deleted object so it can be positively identified. It sounds like the only safe thing to do is reload the full table each time a delete operation comes in. I would rather do an expensive operation than an unsafe one.

[peppy]

I have one more general question regarding the usage of SubscribeForNotifications: it seems in very simple usage (guaranteed single thread) the callbacks for notifications can sometimes happen immediately (inline on the transaction.Commit) but other times they can be delayed somewhat.

Is it safe to assume that there is no guarantee of when the notifications are delayed by realm?

---

From another angle, here's what I'm working with and why I bring up this question: I am looking at a real-world example where we have a file lookup component that is caching realm results to a dictionary (because realm lookups are too slow for our purposes in this specific use case, with hundreds or thousands of lookup on EmbeddedObjects which cannot use indices). This is the flow:

string filename = "user_import.jpg";

realmObject.Files.Add(new File(filename)); // a new file is imported by a user interaction

// the file lookup cache is watching for changes to `realmObject` in the background and invalidated its cache on seeing any change

syncContext.Post(() => 
{
    // internally, Sprite uses DI to access the file lookup cache and query for filename.
    // so we need to be sure that the realm notification has arrived before running this call.
    // posting to the active syncContext isn't enough, as realm may not have updated yet.
    placeObject(new Sprite(filename));
});

Curious if there is a recommended way to handle this kind of flow. Basically, we want to know when realm has finished synchronising and propagating the callbacks. Calling realm.Refresh() between the two example lines above will work, but it'll also be a blocking operation.

[nirinchev]

There's no guarantee when the callback will fire - the only way to force it is using Refresh or Write which will force all change calculations to run. We do have a task to be able to do an async refresh which might work for your use case, but we don't have a timeframe for when the Core team will implement it. This is the ticket: realm/realm-core#5246.

[peppy]

One more question: If a transaction is doing both a modification and a deletion, both of which are going to fire a callback from a SubscribeForNotifications subscription, is it guaranteed that both changes will arrive in a single notification callback?
I presumed this was the case, but in practice it would seem that the results differ for other users running the same code.

[nirinchev]

Spoke with the folks in Core and they confirmed that notifications will never be sent out for parts of a transaction. A single notification may combine multiple transactions though.

[peppy]

Thanks for the confirmation, I'll circle back and check what we were seeing from our end, potentially something else going on.