memiavl state-synced node cannot be used as JSON-RPC node

[samuel-tay]

Describe the bug
memiavl state-synced node cannot be used as JSON-RPC node. Same result for node using enable-indexer = true

Another node that was traditionally state-synced with memiavl enabled provided the snapshot.

To Reproduce

• Traditional statesync with memiavl and versiondb enabled by adding following to app.toml
• Should get a snapshot when synced

[memiavl]

# Enable defines if the memiavl should be enabled.
enable = true

# ZeroCopy defines if the memiavl should return slices pointing to mmap-ed buffers directly (zero-copy),
# the zero-copied slices must not be retained beyond current block's execution.
zero-copy = false

# AsyncCommitBuffer defines the size of asynchronous commit queue, this greatly improve block catching-up
# performance, -1 means synchronous commit.
async-commit-buffer = 0

# SnapshotKeepRecent defines what many old snapshots (excluding the latest one) to keep after new snapshots are taken.
snapshot-keep-recent = 2

# SnapshotInterval defines the block interval the memiavl snapshot is taken, default to 1000.
snapshot-interval = 1000

# CacheSize defines the size of the cache for each memiavl store, default to 1000.
cache-size = 1000

[store]
streamers = ["versiondb"]

• Export memiavl snapshot with cronosd snapshots dump 9911000 2
• Copied the 9911000-2.tar.gz file into another server
• Import snapshot with cronosd snapshots load 9911000-2.tar.gz
• Restore snapshot using cronosd snapshots restore 9911000 2
• Bootstrap with cronosd tendermint bootstrap bootstrap-state
• Start cronosd with cronosd start. Node starts syncing
• Send eth_call to JSON-RPC port curl localhost:18454 -X POST -H "Content-Type: application/json" --data '{"method":"eth_call","params":[{"from":null,"to":"0x6b175474e89094c44da98b954eedeac495271d0f","data":"0x70a082310000000000000000000000006E0d01A76C3Cf4288372a29124A26D4353EE51BE"}, "latest"],"id":1,"jsonrpc":"2.0"}'
• Get reply of {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"rpc error: code = Unknown desc = rpc error: code = Internal desc = failed to obtain coinbase address: failed to retrieve validator from block proposer address crcvalcons1gulrn5tyrjhrs39gxnnfmp52y8duentwdf4shu: validator does not exist: unknown request"}}

Expected behavior
On another node with almost exact same app.toml and config.toml except for the memiavl and versiondb, got reply of {"jsonrpc":"2.0","id":1,"result":"0x"}

Desktop (please complete the following information):

• Linux v1.0.12

[yihuang]

You probably just need to restore the initial versiondb from state sync snapshot:
https://github.com/crypto-org-chain/cronos/wiki/VersionDB-Migration#state-sync

[yihuang]

cronosd snapshots restore only restores the iavl or memiavl, versiondb need to restore manually.

[samuel-tay]

Thanks @yihuang
I followed the steps here but I don't see any mention of versiondb restoration. Could you advise please?
https://github.com/crypto-org-chain/cronos/wiki/Local-State-Sync

[yihuang]

Thanks @yihuang I followed the steps here but I don't see any mention of versiondb restoration. Could you advise please? https://github.com/crypto-org-chain/cronos/wiki/Local-State-Sync

sure, I just added a sub-step under the step 3:

3a. (Optional) If you want to enable versiondb together, you need to restore the versiondb from the snapshot as well:

$ cronosd changeset restore-versiondb <height> <format>

[samuel-tay]

Thanks for the clarification

I have done the steps all again including the changeset restore-versiondb ccommand, but now I get another response for JSON RPC.

{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"header not found"}}

ETA: Both times cronosd status shows "earliest_block_height":"0"

[yihuang]

message":"header not found

did you query for historical blocks that's before the snapshot height, or not synced yet?

[samuel-tay]

I tried when it was syncing and synced to latest. I believe the command asks for the latest block

curl localhost:18454 -X POST -H "Content-Type: application/json" --data '{"method":"eth_call","params":[{"from":null,"to":"0x6b175474e89094c44da98b954eedeac495271d0f","data":"0x70a082310000000000000000000000006E0d01A76C3Cf4288372a29124A26D4353EE51BE"}, "latest"],"id":1,"jsonrpc":"2.0"}'