Timeout Rocksdb Snapshots

[garyschulte]

Description

In rare instances, bonsai might fail to release a rocksdb snapshot. This prevents compaction of old SST files. This will result in larger size on disk until besu is restarted and the snapshot is released. In addition read performance suffers when old SST files cannot be compacted.

Bonsai worldstate / rocksdb snapshots should implement a timeout to force the worldstate to close and the snapshots to be released to prevent this condition. Careful consideration of the timeout will be necessary to not impact nominal operations of besu and the bonsai worldstate archive. The number of worldstates kept in memory should especially be considered.

Acceptance Criteria

• timeouts for snapshots
• no impact to nominal operation of besu