op-rbuilder unknown payload -38001 error against get_payload_v4

[yashvardhan-kukreja]

Hi everyone,
We've been noticing rare occurrence of situations where for a certain block number, the op-rbuilder fails to build and produce flashblocks because it responds with a -38001 Unknown payload error against op-node's get_payload_v4 call. The same call is successfully responded by the op-geth (as the backup builder).

This situation is usually enough to cause a 206 via rollup-boost, which further, makes our op-conductor to switch leadership. So we'd really appreciate diving into this and see if you folks by any chance encountered this issue.

I am attaching logs of happy path (block successfully built with flashblocks) and sad path (the next block failing to build due to the above issue).

Logs of a happy block (just before the failing one).txt
Logs of failing block.txt

[SozinM]

what time of l1 settlement do you have?
Can we see the graphs for FCU without attributes number?

[yashvardhan-kukreja]

what time of l1 settlement do you have?

Can you elaborate on what does this mean? You mean when did the unsafe block, whose building faced this hiccup, became a safe block?

Can we see the graphs for FCU without attributes number?

Which metrics are you intending to see? Are op-node metrics show the following ones but nothing abnormal there

Note: we're running a multi-sequencer setup backed by op-conductor.

When the above issue occurred, a 206 propagated from rollup-boost to op-conductor, causing that op-conductor to transfer sequencing to some other (standby) sequencer with no flashblocks components i.e. only op-node<>op-geth.

Due to this, the affected block was built and produced within a few seconds/milliseconds despite the above issue though by a different sequencer.

[SozinM]

Do you have rbuilder answering to FCU with attributes SYNCING?
If so you will need this: #229

[yashvardhan-kukreja]

Yep

2025-08-06 22:32:12.005|ts=2025-08-07T02:32:12.005729173Z level=debug target=reth::cli message="Event: Handler(ForkchoiceUpdated(ForkchoiceState { head_block_hash: 0xece7c934392a99362bb31982986ba7d43090b0d3927de09dfc232257ee955e77, safe_block_hash: 0xf3a2137b992063ff2702b9f79d7d7d67280f49d3ed437b0e7f803943bc13c720, finalized_block_hash: 0xc10d4b50943b739d2b30d50ac552be029b0b203dfaddb056d26bf898a00691fd }, Syncing))"

Note: 0xece7c934392a99362bb31982986ba7d43090b0d3927de09dfc232257ee955e77 is the parent of the affected block, confirming the correctness of the above log

---

Also, on a separate note, it seems that at least rollup-boost was able to successfully send the FCU call to l2 as well as the builder as signified by the following log

{"message":"block building started","payload_id":"0x038c5f8a8013c163","builder_building":true}

This was accompanied by rollup-boost successfully storing the payload_id in its cache as well as signified by the following log

builder has payload, calling get_payload on builder

---

Also, while all of this was happening, what I noticed was that, there were no logs related to "Building Flashblock" by op-rbuilder was this specific affected block/payload, while for all the previously successfully processed blocks, it did build flashblocks and logged "Building Flashblock".

I'll continue to dig further into the op-rbuilder's logs but any chance it clarifies the picture?

[yashvardhan-kukreja]

Another notable thing,

The following plot shows the count of the log line Parent found in memory

All the blue bars represent one log line, except the spike (the longest blue bar) preceded by roughly a 5 second gap (relatively abnormal as well).
That spike represents 2-log lines, which are the following

2025-08-06 22:32:18.318 | ts=2025-08-07T02:32:18.31865506Z level=debug target=engine::tree message="Parent found in memory" parent_hash=0x979b6f257f8022d50e7ab903268e956a3d5f6c544632ef3b3cfdb0eaa34ecceb historical=31365696 blocks=2 |  


  |   | 2025-08-06 22:32:18.281 | ts=2025-08-07T02:32:18.281581227Z level=debug target=engine::tree message="Parent found in memory" parent_hash=0x7cc43c06d84d0a5afec6272d2ed38b2a305742bf8205a3f8dedaf98d7c1c1ac9 historical=0xc1c549d5c74a11e37a9ef01004d539892ffce504e80a3fd07792117f71e1514b blocks=2

Our affected block's hash is 0x7cc43c06d84d0a5afec6272d2ed38b2a305742bf8205a3f8dedaf98d7c1c1ac9 meaning that the block child to this block (harmlessly?) faced a small delay with having its parent hash (our affected block) in memory.

Though, I am still not sure though if the delay (associated with the affected block's presence in memory), is a cause or effect rollup-boost facing "Unknown payload" failures.

[SozinM]

@yashvardhan-kukreja have you tried our latest PR i sent you?
We have seen the same problem and fixed it, soon it will be merged