Deadlock when reading corrupted WAL file

[morguldir]

Hi, recently I had some out of space issues, which (probably due to other bugs? maybe similar to #12467?) resulted in WAL corruption. Usually we just switch to pointintime or alternatively do a rocksdb repair, recently this started deadlocking however.

This was definitely a newer issue, since it's worked fine before, so i started bisecting and narrowed it down to #12643

We don't use WAL compression currently, so maybe that's why the patch doesn't work in our case?

Anyway reverting that patch makes it successfully read the uncorrupted parts of the WAL and put it into lost, and we're sailing smoothly again

When i run ./ldb dump_wal --db=$HOME/test/out-of-space/ --ignore_unknown_options --walfile=$HOME/test/out-of-space/2268638.log with the patch, it it gets to the same point as normally, but gets stuck spinning (100% cpu usage) at the end

gdb shows that it's happening inside https://github.com/facebook/rocksdb/blob/v9.9.3/db/log_reader.cc#L463

Let me know if there's any additional information i can provide

[zaidoon1]

@hx235 thoughts on this?