Help me,BE crash at OlapMeta init,open rocksdb failed !

[lyh20093867]

version:doris-1.2.5-rc01(AVX2)
openjdk:1.8.0_332

start time: Wed Aug 7 15:13:39 CST 2024
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/be/lib/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
F0807 15:13:40.352217 102351 doris_main.cpp:408] fail to open StorageEngine, res=[INTERNAL_ERROR]init path failed, error=[IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage;[IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage;
*** Check failure stack trace: ***
@ 0x55b0a223ccad google::LogMessage::Fail()
@ 0x55b0a223f1e9 google::LogMessage::SendToLog()
@ 0x55b0a223c816 google::LogMessage::Flush()
@ 0x55b0a223f859 google::LogMessageFatal::~LogMessageFatal()
@ 0x55b09c453a8b main
@ 0x7f8760cfc555 __libc_start_main
@ 0x55b09c45202a _start
@ (nil) (unknown)
*** Query id: 0-0 ***
*** Aborted at 1723014820 (unix time) try "date -d @1723014820" if you are using GNU date ***
*** Current BE git commitID: Unknown ***
*** SIGABRT unkown detail explain (@0x3e800018fcf) received by PID 102351 (TID 0x7f87616dcb40) from PID 102351; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420
1# 0x00007F8760D10400 in /lib64/libc.so.6
2# gsignal in /lib64/libc.so.6
3# abort in /lib64/libc.so.6
4# 0x000055B0A2247699 in /opt/module/be/lib/doris_be
5# 0x000055B0A223CCAD in /opt/module/be/lib/doris_be
6# google::LogMessage::SendToLog() in /opt/module/be/lib/doris_be
7# google::LogMessage::Flush() in /opt/module/be/lib/doris_be
8# google::LogMessageFatal::~LogMessageFatal() in /opt/module/be/lib/doris_be
9# main at /root/doris/be/src/service/doris_main.cpp:428
10# __libc_start_main in /lib64/libc.so.6
11# _start in /opt/module/be/lib/doris_be

start time: Wed Aug 7 15:14:02 CST 2024
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/be/lib/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: fb1e59d3c354575-87ffc98a44c1f8b1 ***
*** Aborted at 1723018291 (unix time) try "date -d @1723018291" if you are using GNU date ***
*** Current BE git commitID: Unknown ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 104066 (TID 0x7efc645df700) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420
1# os::Linux::chained_handler(int, siginfo*, void*) in /opt/module//openjdk/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /opt/module//openjdk/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo*, void*) in /opt/module//openjdk/jre/lib/amd64/server/libjvm.so
4# 0x00007EFD7FF3E400 in /lib64/libc.so.6
5# je_arena_dalloc_promoted at ../src/arena.c:1604
6# __GI__dl_deallocate_tls in /lib64/ld-linux-x86-64.so.2
7# __free_stacks in /lib64/libpthread.so.0
8# __deallocate_stack in /lib64/libpthread.so.0
9# pthread_join in /lib64/libpthread.so.0
10# std::thread::join() at ../../../../../libstdc++-v3/src/c++11/thread.cc:114
11# doris::ThreadGroup::join_all() in /opt/module/be/lib/doris_be
12# doris::KafkaDataConsumerGroup::start_all(doris::StreamLoadContext*) at /root/doris/be/src/runtime/routine_load/data_consumer_group.cpp:141
13# doris::RoutineLoadTaskExecutor::exec_task(doris::StreamLoadContext*, doris::DataConsumerPool*, std::function<void (doris::StreamLoadContext*)>) at /root/doris/be/src/runtime/routine_load/routine_load_task_executor.cpp:306
14# std::_Function_handler<void (), std::_Bind_result<void, void (doris::RoutineLoadTaskExecutor::(doris::RoutineLoadTaskExecutor, doris::StreamLoadContext*, doris::DataConsumerPool*, doris::RoutineLoadTaskExecutor::submit_task(doris::TRoutineLoadTask const&)::$_0))(doris::StreamLoadContext*, doris::DataConsumerPool*, std::function<void (doris::StreamLoadContext*)>)> >::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
15# doris::PriorityThreadPool::work_thread(int) at /root/doris/be/src/util/priority_thread_pool.hpp:146
16# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
17# start_thread in /lib64/libpthread.so.0
18# clone in /lib64/libc.so.6

start time: Wed Aug 7 16:11:32 CST 2024
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/be/lib/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
F0807 16:11:35.432806 12745 doris_main.cpp:408] fail to open StorageEngine, res=[INTERNAL_ERROR]init path failed, error=[IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage;
*** Check failure stack trace: ***
@ 0x555669752cad google::LogMessage::Fail()
@ 0x5556697551e9 google::LogMessage::SendToLog()
@ 0x555669752816 google::LogMessage::Flush()
@ 0x555669755859 google::LogMessageFatal::~LogMessageFatal()
@ 0x555663969a8b main
@ 0x7fdaa7f5e555 __libc_start_main
@ 0x55566396802a _start
@ (nil) (unknown)
*** Query id: 0-0 ***
*** Aborted at 1723018295 (unix time) try "date -d @1723018295" if you are using GNU date ***
*** Current BE git commitID: Unknown ***
*** SIGABRT unkown detail explain (@0x3e8000031c9) received by PID 12745 (TID 0x7fdaa893eb40) from PID 12745; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420
1# 0x00007FDAA7F72400 in /lib64/libc.so.6
2# gsignal in /lib64/libc.so.6
3# abort in /lib64/libc.so.6
4# 0x000055566975D699 in /opt/module/be/lib/doris_be
5# 0x0000555669752CAD in /opt/module/be/lib/doris_be
6# google::LogMessage::SendToLog() in /opt/module/be/lib/doris_be
7# google::LogMessage::Flush() in /opt/module/be/lib/doris_be
8# google::LogMessageFatal::~LogMessageFatal() in /opt/module/be/lib/doris_be
9# main at /root/doris/be/src/service/doris_main.cpp:428
10# __libc_start_main in /lib64/libc.so.6
11# _start in /opt/module/be/lib/doris_be

logs in be.INFO

I0807 15:13:40.349098 103013 data_dir.cpp:745] path: /data1/doris/be/storage total capacity: 7999426224128, available capacity: 7905475198976
I0807 15:13:40.349145 103014 data_dir.cpp:745] path: /data2/doris/be/storage total capacity: 7999426224128, available capacity: 7907773960192
I0807 15:13:40.349195 103013 data_dir.cpp:192] path: /data1/doris/be/storage, hash: 8911238614356524183
I0807 15:13:40.349210 103014 data_dir.cpp:192] path: /data2/doris/be/storage, hash: 8825406747811533581
W0807 15:13:40.351935 103013 olap_meta.cpp:80] rocks db open failed, reason:IO error: While lock file: /data1/doris/be/storage/meta/LOCK: Resource temporarily unavailable
W0807 15:13:40.351981 103014 olap_meta.cpp:80] rocks db open failed, reason:IO error: While lock file: /data2/doris/be/storage/meta/LOCK: Resource temporarily unavailable
W0807 15:13:40.351994 103013 data_dir.cpp:204] init OlapMeta failed, error: [IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage
W0807 15:13:40.352031 103013 data_dir.cpp:115] _init_meta failed, error: [IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage
W0807 15:13:40.352030 103014 data_dir.cpp:204] init OlapMeta failed, error: [IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage
W0807 15:13:40.352039 103013 storage_engine.cpp:208] Store load failed, status=[IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage, path=/data1/doris/be/storage
W0807 15:13:40.352041 103014 data_dir.cpp:115] _init_meta failed, error: [IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage
W0807 15:13:40.352072 103014 storage_engine.cpp:208] Store load failed, status=[IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage, path=/data2/doris/be/storage
W0807 15:13:40.352162 102351 storage_engine.cpp:172] _init_store_map failed, error: [INTERNAL_ERROR]init path failed, error=[IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage;[IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage;
W0807 15:13:40.352175 102351 storage_engine.cpp:99] open engine failed, error: [INTERNAL_ERROR]init path failed, error=[IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage;[IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage;
F0807 15:13:40.352217 102351 doris_main.cpp:408] fail to open StorageEngine, res=[INTERNAL_ERROR]init path failed, error=[IO_ERROR]open rocksdb failed, path=/data1/doris/be/storage;[IO_ERROR]open rocksdb failed, path=/data2/doris/be/storage;

[maruyue]

W0807 15:13:40.351935 103013 olap_meta.cpp:80] rocks db open failed, reason:IO error: While lock file: /data1/doris/be/storage/meta/LOCK: Resource temporarily unavailable

Failed to obtain the lock file. Check whether the file path is readable or whether another BE has been started.

[lyh20093867]

thanks bro! The codes have some bug，I have fixed it !