Provide FSC/FTS node a resync process with the ledger after a restart or crash #1213
Description
Hello,
after some stop on an inopportune time, it seems that our FSC/FTS node has lost token synchronisation (token was locked) with the actual HLF tx (which indeed was committed). It seems to us it would be important for FSC/FTS node resiliency to provide a startup process to insure all played transactions within FSC/FTS node are in sync with HLF finality.
Summary of issue:
it seems to us that there was transfer in progress at the moment of the restart and so FTS node didn't reflected the HLF committed state within the database after the restart. That result into locked tokens we're not able reuse (lack of funds error).
Details of issue :
We try to redeem 10 tokens but we have this error:
[1;32m[ info | 2025-08-18 12:42:52.980 (UTC) ] [0m[1;37m[ AMS.ffhlflcidemix@BDF - settlement_engine_ff_zkat.go:85 ] [0mredeem ttr : &fabflow.TransferTokenRequest{
SenderWalletEPName: WFREURPBGSGGXXDLT-PBG_WALLET_001
Type: EUR
Quantity: 695195.24
...
[31m2025-08-18 12:42:59.785 UTC [token-sdk.tokens] parse -> ERRO 23f[0m could not unmarshal identity when storing token: failed to unmarshal to TypedIdentity: asn1: syntax error: sequence truncated
On the 19th august (1 day after a reboot):
[34m2025-08-19 06:05:13.555 UTC [token-sdk.sql] logStaleLocks -> INFO 037[0m Found following entries ready for deletion: [[Invalid token lock state: [[a91492e7de8802a04f2618b041c8e81499d57431acbf29a9198973443afbc397:0]] created at [2025-08-18 18:00:39.808978 +0000 UTC], expired [true], status: [0xc01739b418] Invalid token lock state: [[35c4a3e451333e18789835bf019c4c617ce0b68226cd054389842eb2b09ce8c6:0]] created at [2025-08-18 18:00:42.515104 +0000 UTC], expired [true], status: [0xc01739b420] Invalid token lock state: [[a9f5a04e72b58f455f572e496c5b42bc9834a94b94c150c5639cf654ac73a597:0]] created at [2025-08-18 18:00:45.377994 +0000 UTC], expired [true], status: [0xc01739b428] Expired lock created at [2025-08-18 18:00:48.213327 +0000 UTC] for token [[88a3172335499e50f1965bdb4cb42fe302484579eb9db2ac8f6a849b695bf26b:0]] consumed by [13909b55e56f1e97079b3ab98e15d5b075542487c2a5647859c86876d9c9cb22]]]
Then we regularly log tokens to unlock but it does nothing
[34m2025-08-19 06:10:13.550 UTC [token-sdk.sql] logStaleLocks -> INFO 038[0m Found following entries ready for deletion: [[Expired lock created at [2025-08-18 18:00:48.213327 +0000 UTC] for token [[88a3172335499e50f1965bdb4cb42fe302484579eb9db2ac8f6a849b695bf26b:0]] consumed by [13909b55e56f1e97079b3ab98e15d5b075542487c2a5647859c86876d9c9cb22]]]
When we try to do something with those tokens we have this:
[33m2025-08-19 12:27:44.656 UTC [token-sdk.selector.shared.selector-6b36005c6a7f02a80b5cba16cbc763d59c3293d7322c6e4dd414fcb8f4a2028f] Select -> WARN 089[0m Exceeded max number of immediate retries. Unlock tokens and abort...
And we have this entry in the token DB (see following screenshot):
