🔧 Mitigation against unexpected error or panic in tally end blocker

[hacheigriega]

🔧 Chore

Currently, the tally flow halts and becomes indefinitely stuck in the event of an error or panic. Although we catch errors in the tally module’s end blocker to prevent a chain halt, the data request causing the error remains at the front of the tally-ready queue. This leads to repeated errors in subsequent blocks, effectively blocking other requests from being tallied.

A sensible solution would be to modify the defer function to do the following

• If we can retrieve the problematic data request's ID, notify the contract of an unexpected error with the request ID so that the contract can remove the problematic request. If it is difficult to retrieve the data request ID, the module could send a message without any parameters so that the contract just removes the first data request in the tally-ready queue.
• If we can retrieve the problematic data request's ID, store a new data result with preliminary data indicating that an unexpected error has occurred.
• Appropriate logging so that the problem can be examined.