Do not update state-root if tx fails

[p4u]

See the logs, the tx fails but the state root is still updated.

2025-10-06T08:51:17.355Z DBG sequencer/onchain.go:142 > proof ready to submit to the contract abiInputs=11dc35332814651cebd4d44afd706e5e54824f7249fc92f5f2698b046848ae83176d9e01646d4f3bf69cbfd4116bb6675b706bd5890f15585d6be115bd9bd95a0000000000000000000000000000000000000000000000000000000000000005000000000000000000000000000000000000000000000000000000000000000000a3a95cd126a283caf9c6980cdb3fb09459f39753f7107fc095044e95de3e27000000000000000000000000000000000000000000000000f18d1b1e81e9dd520000000000000000000000000000000000000000000000001595d108d6a4d061000000000000000000000000000000000000000000000000e4b9959951955175000000000000000000000000000000000000000000000000238a0d000b645483 abiProof=244ca22f380cbc60970eba6194ad4e568de5c6bcccbb5d3b46b751c3a88017ec1f9ba2f772c89e6ca175869a574fa3e7640dd83401779389f6ac3a00e294cde414c9c2e45896f5b8f6c3888fbd4b09e77b6ee65bf9f9cc4030092c07cba357aa19ed2da4101d7f0f450a92266fa1b892c54396c59443c38ee1e478d73f8b4704006b21f8db81a84d458bbc9b15b28157a5cb7b6bf00febb2d8c84516c8f549bd00b451e45ce53b5f4264150db3bb4edfe371085350cbb9e98b3ea86781f65112064fb6e15e5c55dba0432c493c369422ae40e6f2a88406cc17497806ec0a011e1a55292616a6c9603e7105ed8c7e379fcb84d8d8960462491519b9d5b1f32a500b0cff306973941a8466a2a08eb5fc0c076d8c2f0b2db1c2a2c3bcbfe6c0c51d0e8bad5ae6851bd12e5ae34151ba6e1314bf3f9ffc20aa1fe03572df6e77f79f23a74e4698e2215acff7b96dad6519a2687a09299b40ef1bef04e8d87a6db28401fec7cd8877b89727097b60a76e48b9df5d5d742164b1ac6ecd71b4642fed0d pid=a62e32147e9c1ea76da552be6e0636f1984143afea0fc2240000000000000006 strInputs="{\"rootHashBefore\":8078391951901314751230980218234560136528054186175268792278692498872633699971,\"rootHashAfter\":10596872360934906891284226032428437677662010655409224946913909977073818786138,\"numNewVotes\":5,\"numOverwritten\":0,\"blobEvaluationPointZ\":289164969032740032124465197822283783942163181064402528881212275229395402279,\"blobEvaluationPointY\":[17405597952699391314,1555379082217443425,16481368797070315893,2560873631955833987]}" strProof="{\"proof\":{\"Ar\":[16418662281849148175670808411012289083185520856055753799495484706639031834604,14296684355225045524378578967443819716508179192570243629278746147431453609444],\"Bs\":[[9402738326674483446024132779903010805488667278379638749659418217738886404010,11726878970648126535238234623461090235854964030450609832802013344334527284996],[189287102744234513250228371721583250963149104768371317351447986646616983997,318597669765070702048726579063970159449085638333859205911050316267923067154]],\"Krs\":[2854720203377546673791720220857380039738026250768529910544022227442152898846,11910600062134762985077417331311689838610670836552804945302429593025071229520]},\"commitments\":[4998404749694479375561217153903084473299272444604060122512046098849994949917,6579168074959704480808587089779941363223605408050034127625837791841158166431],\"commitment_pok\":[16126553399744123080730271586145662120473092701256222499082810328013907800708,902470991720776149066113524069657613920147123539070931324525964833275768077]}"
2025-10-06T08:51:18.686Z DBG sequencer/onchain.go:153 > local blob proof verification succeeded
2025-10-06T08:52:17.333Z INF process a62e32147e9c1ea76da552be6e0636f1984143afea0fc2240000000000000006 aggregatedVotes=5 currentBatchSize=0 lastBatchSize=5 lastTransitionTime=0001-01-01T00:00:00Z pendingVotes=0 settledTransitions=0 stateTransitions=1 verifiedVotes=5
2025-10-06T08:52:17.333Z INF active process statistics summary activeProcesses=2 aggregatedVotes=145 lastTransitionTime=2025-10-06T07:31:37Z pendingVotes=0 settledTransitions=4 stateTransitions=5 verifiedVotes=145
2025-10-06T08:52:20.955Z ERR sequencer/onchain.go:96 > failed to push to contract error="timeout waiting for tx 0x6acca0e94d6c772a3b0ff83181592c3fffd603f0af2253c7e819c5e031fd4950"
2025-10-06T08:53:07.952Z DBG service/process_monitor.go:131 > process state root changed pid=0xa62e32147e9c1ea76da552be6e0636f1984143afea0fc2240000000000000006 stateRoot=10596872360934906891284226032428437677662010655409224946913909977073818786138 voteCount=5 voteOverwrittenCount=0
2025-10-06T08:53:17.14Z DBG service/process_monitor.go:131 > process state root changed pid=0xa62e32147e9c1ea76da552be6e0636f1984143afea0fc2240000000000000006 stateRoot=10596872360934906891284226032428437677662010655409224946913909977073818786138 voteCount=5 voteOverwrittenCount=0

[lucasmenendez]

It seems that the timeout to wait for the tx is reached, but the transaction is confirmed anyway, so the onchain monitor catches the new stateroot when it occurs.

cc/ @p4u

[lucasmenendez]

Related with this:

2025-10-06T18:40:33.466Z INF sequencer/statetransition.go:141 > state transition proof generated blobHash=0x011a6c4005d5ffd571be1a73f63adbbc4fcaf78bc808b8a4c9984ba9ce96426d pid=a58b4a80de82b889ff40e487c58208a429c77f88ea0fc2240000000000000000 rootHashAfter=7788428835943794265352153223390247598330438373917331632995142784530154551644 rootHashBefore=15222329688298692992901312172657508059152703952469991708932768827612296421063 took=30.406958248s
2025-10-06T18:40:34.06Z INF sequencer/onchain.go:64 > state transition batch ready for on-chain upload batchID=a58b4a80de82b889ff40e487c58208a429c77f88ea0fc2240000000000000000e707c55fa9b7e79daaa8b7ff pid=a58b4a80de82b889ff40e487c58208a429c77f88ea0fc2240000000000000000
2025-10-06T18:40:34.087Z DBG sequencer/onchain.go:142 > proof ready to submit to the contract abiInputs=21a788851387e7c2c6bb44c3f26261e60d9fa542b18840adb60795f836009ec7113818316cbbc1eedb9e5b4363ad6bc4d0fe8441f5a450adb129f2fcb6f4795c0000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000100ea9e68ecf7b16c074f851b5723cc841ea0c5d65ccd1e098d6eda48b0d788100000000000000000000000000000000000000000000000004aeee32102e04c2b000000000000000000000000000000000000000000000000450efc20bf4bb0c5000000000000000000000000000000000000000000000000d7c88fca82153d1200000000000000000000000000000000000000000000000066255e8eb2281295 abiProof=14cc53db72916ffebfb4e2a8989fea50ee3c225c1e9aa45a34199d1c96baea391288d80cb150d048f086b1fecce95593f14b57db39103b274570b67d1ae50b9e226eaa0f8b349f3db415717fcccf2a403830380d04ca9ea86843db76c4fdb4e81b6f8742a528f7df52338f44f565a5a387f93d334e70a78749322ebb3443a2972a935bb0bbbaaee104cfaf25d9b6c006f5fa9a958ae1a79d01e3dd69593ce13a2488c25d3e6564961538b49722c1035d5ac1e2be6a2c5231ad355b8af004ccdb2745ab8cbb6a7472528ec9dda5ee61b7b1c1d45990d4e827cb1639349c91bd97064d7adaef5e0717eb4c33f4ad016f1252df74834dbd06cc3dffaacb60ddbd36157e05721dedfcbfa0f54c3ad719aeb60dfb987bc355e755dc9dc6d577823c8f1a2a0c0bfaf896edfc33aaef941f959424994c06bbaa6362dc35d7b1d830b2b20478375e294b957366ae56c57cff652783a7a2cbabfbefcbdb55e240632738ca01250fb3ba08ee193a41576f3affb49fbc6c9a8d0a456eeca998c407d45feb4b pid=a58b4a80de82b889ff40e487c58208a429c77f88ea0fc2240000000000000000 strInputs="{\"rootHashBefore\":15222329688298692992901312172657508059152703952469991708932768827612296421063,\"rootHashAfter\":7788428835943794265352153223390247598330438373917331632995142784530154551644,\"numNewVotes\":4,\"numOverwritten\":1,\"blobEvaluationPointZ\":414535517870984072201602747297446522590602817609710306747797818420929988624,\"blobEvaluationPointY\":[5399502734185483307,4976191855869538501,15548835813425233170,7360393132974281365]}" strProof="{\"proof\":{\"Ar\":[9407272534120652653111746300259678106142613469180893569465755161382423816761,8383413594712442444572187894020965629744782889312986525828680543980627495838],\"Bs\":[[15574163744894889875693153690840328504139942599949458676080723789060262180072,12409500468445271391108367897775389832425602544344089211858103971146537083543],[19257498982657977691566813310824393627623443570436674512410789404562989834554,16524895202444840374060203511487032991127397463522757025677261956053874298075]],\"Krs\":[17763297534972123788014360695693112060578263270581022727877780375874193964439,2850772231018534275830748862375770047869651415618455315506785244664332926262]},\"commitments\":[9721230135728283344028884655106477203049335208531997221514492660970873306255,11834424783831529799228346007664925682107757376636583142248661123268632162994],\"commitment_pok\":[2021655176739429193225384454795658145939453801261656794006732655710425266378,517794561597481756066011434887255828377533822268340838629037894662210186059]}"
2025-10-06T18:40:34.088Z DBG sequencer/onchain.go:153 > local blob proof verification succeeded
2025-10-06T18:41:34.321Z ERR sequencer/onchain.go:96 > failed to push to contract error="timeout waiting for tx 0x61d2a8404540bae5bc065a38bf4465768c7fb32995041e4e9bd9efe3722c0213"

[lucasmenendez]

The stuck transactions prevent that following ones were confirmed, when a new one is generated and confirmed, the stuck tx are reverted because the state root was changed from the moment where they were sent:

2025-10-06T19:40:54.325Z INF sequencer/onchain.go:64 > state transition batch ready for on-chain upload batchID=a58b4a80de82b889ff40e487c58208a429c77f88ea0fc2240000000000000000db88b83dfe7297ba08e502b9 pid=a58b4a80de82b889ff40e487c58208a429c77f88ea0fc2240000000000000000
2025-10-06T19:40:54.341Z DBG sequencer/onchain.go:142 > proof ready to submit to the contract abiInputs=21a788851387e7c2c6bb44c3f26261e60d9fa542b18840adb60795f836009ec726785fa5e136fb9ae8652583f53ce562f05c27850ded0c5a6c48e1f8bb1dbb7b0000000000000000000000000000000000000000000000000000000000000003000000000000000000000000000000000000000000000000000000000000000002c7450abc12f8f7f31fda9f151e2179901e92556e7c5b4c2d576d7554dc0f8c0000000000000000000000000000000000000000000000007efd02168b737aca000000000000000000000000000000000000000000000000ba35b113e460324a00000000000000000000000000000000000000000000000026d4599652cdce1800000000000000000000000000000000000000000000000061f48c89890d1e96 abiProof=06d9067e90678d90aa5eb18ee8c00d01a5a4bbc719629e7ab2e0aa2256168cdc083dfc77eed78d97ad2f4e8a8a026165e7425d592ec6f9d7eef6a29ed7e304950e14b8056be893cea1c07b5563f00ac817342222430a4480cf749a3e1c66eacf0cd42e1a9032ca8cbbb8bd742cea48ba996a5de40faef963c1c36095ad5dd3a31440bb3a1214d13ab673c9d30acb2cebe6bb4fa4a63415f440d1d165aefe319f18d38223fbf6e2c5da1a9a23d52431cf322952e9af8647c27f8e898d2f9862692f738c49fa1279f84f7406ad8106bbe04813d8689f4f962bd4b24d9e799b4e7a1e4b83b505a409a81bd56aaa084dcf75caa1b4e5a0c9905c4b965ba07bfba1750def35f4a4baec79cf94140b6e4796478f66f653086a2e51476bc6b64197d240015688163baf152bebd68b7ad88ba832c15505d5ab1cf1808058c60d5f02866e1b507a9bbd259c8a2914a5f4286fc70a58c60b0bf4c804e999774dffaf381ef00ea1ed0188cb713306e8c745d1321e4f6fbfd89b95ca64694011eaca33a52617 pid=a58b4a80de82b889ff40e487c58208a429c77f88ea0fc2240000000000000000 strInputs="{\"rootHashBefore\":15222329688298692992901312172657508059152703952469991708932768827612296421063,\"rootHashAfter\":17400570031949558311921977115885493615762916535680147487077686975781685541755,\"numNewVotes\":3,\"numOverwritten\":0,\"blobEvaluationPointZ\":1256704772961342159978650533329333473814676314491367130834862783603234574220,\"blobEvaluationPointY\":[9150472313738853066,13417825363814265418,2797959770673106456,7058421038334156438]}" strProof="{\"proof\":{\"Ar\":[3097327727195437763090463323659975402348079257979420872890069447114919152860,3728022933083655981517450121314759149788618096745871054822206242596401120405],\"Bs\":[[6368986888952931913110148253762830950563535906774519885669051988980147874511,5802643957208644142323260122826428964986913466899288799264365399414541243299],[9160627375959085886142676144220941133387008674642326900327598347849234788767,11229211293824796687084659806981266786958251068301388297858513948238132306537]],\"Krs\":[21462859534763365518195275888195095028606024858097969621858559504142386810490,13702807996472189456911353491454771453849450288131118024103991295959642120565]},\"commitments\":[6302715868195975385816844243569855139969483175766702965303864767293309833792,605200933061647688931367741798724853865939154044363350005343600068308141678],\"commitment_pok\":[12354640888696000681672419451338911371042749161510851692789075922885767864048,6618478012845348272566144654309678861122146280642546577464382973741925475863]}"
2025-10-06T19:40:54.342Z DBG sequencer/onchain.go:153 > local blob proof verification succeeded
2025-10-06T19:40:54.461Z DBG sequencer/onchain.go:167 > failed to simulate state transition error="eth_simulateV1 RPC error: err: nonce too high: address 0x1823C6cb39B19c93407E3aD186401Ec6d3bed485, tx: 65 state: 49 (supplied gas 438163)" pid=a58b4a80de82b889ff40e487c58208a429c77f88ea0fc2240000000000000000
2025-10-06T19:40:55.309Z ERR sequencer/onchain.go:96 > failed to push to contract error="failed to submit state transition: failed to submit state transition (EIP-4844 Tx): error after 3 retries: account limit exceeded: pooled 16 txs"

We need a mechanism to avoid lose votes. For example, wait for a while to those txs that are confirmed in short term, and for those that reach that waiting time, enqueue they in a background process to ensure that they are confirmed or retry them if not.