Support secondary instance with BlobDB

[yunxiao3]

In the RocksDB wiki (https://github.com/facebook/rocksdb/wiki/BlobDB), it is stated that BlobDB does not support secondary instances. However, the specific reason for this lack of support isn't clearly explained.

Based on my understanding, due to the multi-version design of RocksDB, the primary instance cannot perceive references to some data (such as SST files and keys) that might be actively used by the secondary instance during compaction operations. This can lead to situations where data referenced by the secondary instance is inadvertently released by the primary instance. Blob files, which are also referenced by SST files, might be subject to the same issue.

I would like to know if there are other limitations, aside from the ones mentioned above, that prevent BlobDB from supporting secondary instances.

[yunxiao3]

I would like to understand the specific reasons why BlobDB does not support secondary instances and determine whether simple modifications could enable the simultaneous use of BlobDB and secondary instances.

[ltamasi]

This feature combination is indeed not supported at the moment. The main issue (there might be others) is that the secondary instance functionality relies on the secondary having open file descriptors for all data files, which on POSIX systems allows the secondary to keep using files deleted by the primary. Currently, max_open_files==-1 guarantees this for SSTs but not blob files.

[theolivenbaum]

@ltamasi I saw in the docs that this limitation is planned to be dropped at some point. Do you've an idea if this is still the case and +- when that might happen?

[ltamasi]

We don't have any near-term plans to support this at the moment.