[1.4.0] Datanode not start

[yazynin]

Datanode not start Last error is

2024-09-06 07:53:05,973 ERROR org.apache.hadoop.ozone.container.common.states.datanode.RunningDatanodeState: Error in executing end point task.
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.hadoop.ozone.container.common.states.datanode.RunningDatanodeState.computeNextContainerState(RunningDatanodeState.java:199)
	at org.apache.hadoop.ozone.container.common.states.datanode.RunningDatanodeState.await(RunningDatanodeState.java:239)
	at org.apache.hadoop.ozone.container.common.states.datanode.RunningDatanodeState.await(RunningDatanodeState.java:50)
	at org.apache.hadoop.ozone.container.common.statemachine.StateContext.execute(StateContext.java:682)
	at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.startStateMachineThread(DatanodeStateMachine.java:354)
	at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$1(DatanodeStateMachine.java:556)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at org.apache.hadoop.ozone.container.common.states.datanode.RunningDatanodeState.lambda$execute$0(RunningDatanodeState.java:157)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more

Any suggestion pleas

[errose28]

Is your SCM up and running, and are other datanodes able to connect to it? I think datanodes have a timeout of how long they will wait to see SCM before shutting down, but I don't recall for sure.

[yazynin]

Yes SCM is up and other nodes also, if remove Ratis.db DataNode up, but after sometimes if restart DataNode service it not register in SCM

[yazynin]

Also broken nodes giving many prometheus metrics around 120mb, in datanode logs

2024-09-06 17:20:11,633 INFO org.apache.ratis.server.impl.RoleInfo: 66ca580b-9ab9-4b18-ac98-ef1fc0d9a682: start 66ca580b-9ab9-4b18-ac98-ef1fc0d9a682@group-222424229325-LeaderElection75306
2024-09-06 17:20:11,633 INFO org.apache.ratis.server.impl.LeaderElection: 66ca580b-9ab9-4b18-ac98-ef1fc0d9a682@group-222424229325-LeaderElection75306 PRE_VOTE round 0: submit vote requests at term 1 for 0: peers:[66ca580b-9ab9-4b18-ac98-ef1fc0d9a682|172.16.6.33:9856, f3e20975-9166-45f0-a586-d63d8cb32d9a|172.16.6.37:9856, 7bfda539-e431-4af5-82f8-11c67cd856ba|172.16.6.25:9856]|listeners:[], old=null
2024-09-06 17:20:11,634 INFO org.apache.ratis.server.impl.LeaderElection: 66ca580b-9ab9-4b18-ac98-ef1fc0d9a682@group-222424229325-LeaderElection75306 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2024-09-06 17:20:11,635 INFO org.apache.ratis.server.impl.LeaderElection: 66ca580b-9ab9-4b18-ac98-ef1fc0d9a682@group-222424229325-LeaderElection75306 got exception when requesting votes: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: f3e20975-9166-45f0-a586-d63d8cb32d9a: group-222424229325 not found.
2024-09-06 17:20:11,635 INFO org.apache.ratis.server.impl.LeaderElection: 66ca580b-9ab9-4b18-ac98-ef1fc0d9a682@group-222424229325-LeaderElection75306: PRE_VOTE REJECTED received 0 response(s) and 2 exception(s):
2024-09-06 17:20:11,635 INFO org.apache.ratis.server.impl.LeaderElection:   Exception 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2024-09-06 17:20:11,635 INFO org.apache.ratis.server.impl.LeaderElection:   Exception 1: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: f3e20975-9166-45f0-a586-d63d8cb32d9a: group-222424229325 not found.
2024-09-06 17:20:11,635 INFO org.apache.ratis.server.impl.LeaderElection: 66ca580b-9ab9-4b18-ac98-ef1fc0d9a682@group-222424229325-LeaderElection75306 PRE_VOTE round 0: result REJECTED
2024-09-06 17:20:11,635 INFO org.apache.ratis.server.RaftServer$Division: 66ca580b-9ab9-4b18-ac98-ef1fc0d9a682@group-222424229325: changes role from CANDIDATE to FOLLOWER at term 1 for REJECTED
2024-09-06 17:20:11,635 INFO org.apache.ratis.server.impl.RoleInfo: 66ca580b-9ab9-4b18-ac98-ef1fc0d9a682: shutdown 66ca580b-9ab9-4b18-ac98-ef1fc0d9a682@group-222424229325-LeaderElection75306
2024-09-06 17:20:11,635 INFO org.apache.ratis.server.impl.RoleInfo: 66ca580b-9ab9-4b18-ac98-ef1fc0d9a682: start 66ca580b-9ab9-4b18-ac98-ef1fc0d9a682@group-222424229325-FollowerState

[yazynin]

look like this issues https://issues.apache.org/jira/browse/HDDS-10750, fixed by update to 1.4.1 RC1