Skip to content

devnet-5 exception to investigate #10796

@rolfyone

Description

@rolfyone

was on devnet-5 just after startup on teku-erigon-1

2026-06-05 12:14:53.215 INFO  - The latest finalized checkpoint at epoch 38 fell outside of the weak subjectivity period after epoch 102, which is prior to the current epoch 217.
2026-06-05 12:14:53.216 INFO  - The latest finalized checkpoint at epoch 38 is outside of the weak subjectivity period.  Please supply a recent weak subjectivity checkpoint using --ws-checkpoint=<BLOCK_ROOT>:<EPOCH>.
2026-06-05 12:14:53.240 INFO  - Starting libp2p network...
2026-06-05 12:14:53.325 INFO  - Starting discovery server listening on /0.0.0.0:9000
2026-06-05 12:14:53.330 INFO  - Listening for connections on: /ip4/91.107.208.38/tcp/9000/p2p/16Uiu2HAm7LGLvfyrod4sQY1Bd39EvuKancV6KTfacxrygpwwAfuN
2026-06-05 12:14:53.337 INFO  - UDP discovery client started
2026-06-05 12:14:53.341 INFO  - Scheduler QuartzScheduler_$_NON_CLUSTERED started.
2026-06-05 12:14:53.363 INFO  - Local ENR: enr:-LC4QPmthFrmF2CHw2gAdkb39oHh36n9ESz6GSyx3Zc3-jm3aU70DZs2xAx8RjqYefYwhAo3jKF6M57wFzUH_SGxYicPg2NnY4GAhGV0aDKQpNo5TYB2ZXH__________4JpZIJ2NIJpcIRba9Amg25mZIQAAAAAiXNlY3AyNTZrMaECsOmdOdN3MbAjpzkOpmLWMZroNWmcrA60D6054PFiMP2DdGNwgiMog3VkcIIjKA
2026-06-05 12:14:53.389 INFO  - Syncing     *** Slot: 6974, Head slot: 1273, Waiting for execution layer sync, Connected peers: 0
2026-06-05 12:14:53.400 INFO  - Starting Javalin ...
2026-06-05 12:14:53.405 INFO  - jetty-12.1.8; built: 2026-04-01T01:09:18.991Z; git: c9cdc9aaa434a3665b8a53b4d1cc3684992da649; jvm 25.0.3+9-LTS
2026-06-05 12:14:53.454 INFO  - Session workerName=node0
2026-06-05 12:14:53.458 INFO  - Started oeje10s.ServletContextHandler@452893ac{ROOT,/,b=null,a=AVAILABLE,h=oeje10s.SessionHandler@4793be4d{STARTED}}
2026-06-05 12:14:53.461 INFO  - Started oeje10s.ServletContextHandler@452893ac{ROOT,/,b=null,a=AVAILABLE,h=oeje10s.SessionHandler@4793be4d{STARTED}}
2026-06-05 12:14:53.465 INFO  - Started oejs.ServerConnector@1f8c146e{HTTP/1.1, (http/1.1)}{0.0.0.0:5052}
2026-06-05 12:14:53.466 INFO  - Started oejs.Server@577d487c{STARTING}[12.1.8,sto=0] @65ms
2026-06-05 12:14:53.466 INFO  - Javalin started in 124ms \o/
2026-06-05 12:14:53.467 INFO  - Listening on http://0.0.0.0:5052/
2026-06-05 12:14:53.469 INFO  - You are running Javalin 7.2.0 (released April 19, 2026).
2026-06-05 12:14:53.469 INFO  - Listening on port 5052
2026-06-05 12:15:00.005 INFO  - Syncing     *** Slot: 6975, Head slot: 1273, Remaining slots: 5702, Connected peers: 0
2026-06-05 12:15:00.018 WARN  - Validator   *** Slashing protection not loaded for validators: a334ddb, 8605799, b44ad99, b6d2034, 8d862a7, a8d3846, ab9228e, 8407288, a2dbaf8, 8675e2c, ae44723, 92359dc, a9b531c, 868b3b7, aa66856, 9472cda, abcf188, a1f541d, a3a190e, 90383a5… (200 total)
2026-06-05 12:15:12.004 INFO  - Syncing     *** Slot: 6976, Head slot: 1273, Remaining slots: 5703, Connected peers: 0
2026-06-05 12:15:14.804 INFO  - Syncing started
2026-06-05 12:15:20.506 ERROR - Encountered error while processing response
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at io.netty.util.internal.RefCnt.throwIllegalRefCountOnRelease(RefCnt.java:239)
at io.netty.util.internal.RefCnt$UnsafeRefCnt.release0(RefCnt.java:451)
at io.netty.util.internal.RefCnt$UnsafeRefCnt.release(RefCnt.java:436)
at io.netty.util.internal.RefCnt.release(RefCnt.java:144)
at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:83)
at io.netty.buffer.AbstractDerivedByteBuf.release0(AbstractDerivedByteBuf.java:98)
at io.netty.buffer.AbstractDerivedByteBuf.release(AbstractDerivedByteBuf.java:94)
at tech.pegasys.teku.networking.eth2.rpc.core.encodings.LengthPrefixedPayloadDecoder.decodeOneMessage(LengthPrefixedPayloadDecoder.java:97)
at tech.pegasys.teku.networking.eth2.rpc.core.RpcResponseDecoder.decodeNextResponse(RpcResponseDecoder.java:112)
at tech.pegasys.teku.networking.eth2.rpc.core.RpcResponseDecoder.decodeNextResponses(RpcResponseDecoder.java:70)
at tech.pegasys.teku.networking.eth2.rpc.core.Eth2OutgoingRequestHandler.processData(Eth2OutgoingRequestHandler.java:117)
at tech.pegasys.teku.networking.p2p.libp2p.rpc.RpcHandler$Controller.lambda$channelRead0$0(RpcHandler.java:248)
at java.base/java.util.Optional.ifPresentOrElse(Unknown Source)
at tech.pegasys.teku.networking.p2p.libp2p.rpc.RpcHandler$Controller.runHandler(RpcHandler.java:327)
at tech.pegasys.teku.networking.p2p.libp2p.rpc.RpcHandler$Controller.channelRead0(RpcHandler.java:248)
at tech.pegasys.teku.networking.p2p.libp2p.rpc.RpcHandler$Controller.channelRead0(RpcHandler.java:217)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1429)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
at io.libp2p.etc.util.netty.mux.AbstractMuxHandler.childRead(AbstractMuxHandler.kt:71)
at io.libp2p.mux.mplex.MplexHandler.channelRead(MplexHandler.kt:32)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at tech.pegasys.teku.networking.p2p.libp2p.MuxFirewall$MuxFirewallHandler.channelRead(MuxFirewall.java:118)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:361)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:325)
at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:120)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:434)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:361)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:348)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:470)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:249)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1429)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:176)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.handle(AbstractNioChannel.java:445)
at io.netty.channel.nio.NioIoHandler$DefaultNioRegistration.handle(NioIoHandler.java:388)
at io.netty.channel.nio.NioIoHandler.processSelectedKey(NioIoHandler.java:596)
at io.netty.channel.nio.NioIoHandler.processSelectedKeysOptimized(NioIoHandler.java:571)
at io.netty.channel.nio.NioIoHandler.processSelectedKeys(NioIoHandler.java:512)
at io.netty.channel.nio.NioIoHandler.run(NioIoHandler.java:484)
at io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)
at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1195)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
2026-06-05 12:15:24.005 INFO  - Syncing     *** Slot: 6977, Head slot: 1275, Remaining slots: 5702, Target chain: 2426ca..c225 (6880) with 1 peers, Connected peers: 2
2026-06-05 12:15:24.005 INFO  - Sync Info   *** Range: 1274 - 1398 (5 batches), Downloading: 25 slots (1 batches), Ready: 100 slots (4 batches), Batch import: in progress
2026-06-05 12:15:24.006 WARN  - Unable to import blocks: DATA_NOT_AVAILABLE

There were also a bunch of 'invalid dataColumnSidecar which may be unrelated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions