[Bug] docker start skywalking-oap-server:9.7.0 fail : Out of Memory Error #12151

ibingdian · 2024-04-23T08:41:54Z

ibingdian
Apr 23, 2024

Search before asking

I had searched in the issues and found no similar issues.

Apache SkyWalking Component

OAP server (apache/skywalking)

What happened

i want to start skywalking-oap-server:9.7.0 by docker command as blew , but a Out of Memory Error occur.
docker run --name sw_oap --restart always -d -p 12800:12800 -p 11800:11800 apache/skywalking-oap-server:9.7.0

my Docker version 19.03.8.

the error as follow :

Memory: 4k page, physical 32765892k(5253432k free), swap 16777212k(13892920k free)
vm_info: OpenJDK 64-Bit Server VM (11.0.21+9) for linux-amd64 JRE (11.0.21+9), built on Oct 17 2023 21:44:42 by "" with gcc 7.5.0
END.
[Entrypoint] Apache SkyWalking Docker Image

exec java -Xms2G -classpath '/skywalking/ext-libs/*.jar:oap-libs/zstd-jni-1.4.9-1.jar:oap-libs/zookeeper-jute-3.5.7.jar:oap-libs/zookeeper-3.5.7.jar:oap-libs/zjsonpatch-0.3.0.jar:oap-libs/zipkin-receiver-plugin-9.7.0.jar:oap-libs/zipkin-query-plugin-9.7.0.jar:oap-libs/zipkin-2.24.1.jar:oap-libs/vavr-match-0.10.3.jar:oap-libs/vavr-0.10.3.jar:oap-libs/tool-profile-snapshot-server-mock-9.7.0.jar:oap-libs/tool-profile-snapshot-bootstrap-9.7.0.jar:oap-libs/telemetry-prometheus-9.7.0.jar:oap-libs/telemetry-api-9.7.0.jar:oap-libs/storage-jdbc-hikaricp-plugin-9.7.0.jar:oap-libs/storage-elasticsearch-plugin-9.7.0.jar:oap-libs/storage-banyandb-plugin-9.7.0.jar:oap-libs/snappy-java-1.1.8.1.jar:oap-libs/snakeyaml-engine-2.6.jar:oap-libs/snakeyaml-2.0.jar:oap-libs/slf4j-api-1.7.30.jar:oap-libs/skywalking-zabbix-receiver-plugin-9.7.0.jar:oap-libs/skywalking-trace-receiver-plugin-9.7.0.jar:oap-libs/skywalking-telegraf-receiver-plugin-9.7.0.jar:oap-libs/skywalking-sharing-server-plugin-9.7.0.jar:oap-libs/skywalking-profile-receiver-plugin-9.7.0.jar:oap-libs/skywalking-meter-receiver-plugin-9.7.0.jar:oap-libs/skywalking-mesh-receiver-plugin-9.7.0.jar:oap-libs/skywalking-management-receiver-plugin-9.7.0.jar:oap-libs/skywalking-log-receiver-plugin-9.7.0.jar:oap-libs/skywalking-jvm-receiver-plugin-9.7.0.jar:oap-libs/skywalking-event-receiver-plugin-9.7.0.jar:oap-libs/skywalking-ebpf-receiver-plugin-9.7.0.jar:oap-libs/skywalking-clr-receiver-plugin-9.7.0.jar:oap-libs/skywalking-browser-receiver-plugin-9.7.0.jar:oap-libs/simpleclient_hotspot-0.6.0.jar:oap-libs/simpleclient_common-0.6.0.jar:oap-libs/simpleclient-0.6.0.jar:oap-libs/service-1.12.0.jar:oap-libs/server-testing-9.7.0.jar:oap-libs/server-starter-9.7.0.jar:oap-libs/server-health-checker-9.7.0.jar:oap-libs/server-core-9.7.0.jar:oap-libs/server-alarm-plugin-9.7.0.jar:oap-libs/retrofit-2.9.0.jar:oap-libs/receiver-proto-9.7.0.jar:oap-libs/reactive-streams-1.0.3.jar:oap-libs/re2j-1.5.jar:oap-libs/query-graphql-plugin-9.7.0.jar:oap-libs/proto-google-common-protos-2.9.0.jar:oap-libs/protobuf-java-util-3.21.8.jar:oap-libs/protobuf-java-3.21.8.jar:oap-libs/promql-plugin-9.7.0.jar:oap-libs/postgresql-42.4.1.jar:oap-libs/pgv-java-stub-0.6.13.jar:oap-libs/perfmark-api-0.25.0.jar:oap-libs/otel-receiver-plugin-9.7.0.jar:oap-libs/okio-1.17.2.jar:oap-libs/okhttp-3.14.9.jar:oap-libs/oal-rt-9.7.0.jar:oap-libs/oal-grammar-9.7.0.jar:oap-libs/netty-transport-native-unix-common-4.1.100.Final-osx-x86_64.jar:oap-libs/netty-transport-native-unix-common-4.1.100.Final-osx-aarch_64.jar:oap-libs/netty-transport-native-unix-common-4.1.100.Final-linux-x86_64.jar:oap-libs/netty-transport-native-unix-common-4.1.100.Final-linux-aarch_64.jar:oap-libs/netty-transport-native-unix-common-4.1.100.Final.jar:oap-libs/netty-transport-native-kqueue-4.1.100.Final-osx-x86_64.jar:oap-libs/netty-transport-native-kqueue-4.1.100.Final-osx-aarch_64.jar:oap-libs/netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:oap-libs/netty-transport-native-epoll-4.1.100.Final-linux-aarch_64.jar:oap-libs/netty-transport-native-epoll-4.1.100.Final.jar:oap-libs/netty-transport-classes-kqueue-4.1.100.Final.jar:oap-libs/netty-transport-classes-epoll-4.1.100.Final.jar:oap-libs/netty-transport-4.1.100.Final.jar:oap-libs/netty-tcnative-classes-2.0.61.Final.jar:oap-libs/netty-tcnative-boringssl-static-2.0.61.Final-windows-x86_64.jar:oap-libs/netty-tcnative-boringssl-static-2.0.61.Final-osx-x86_64.jar:oap-libs/netty-tcnative-boringssl-static-2.0.61.Final-osx-aarch_64.jar:oap-libs/netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar:oap-libs/netty-tcnative-boringssl-static-2.0.61.Final-linux-aarch_64.jar:oap-libs/netty-tcnative-boringssl-static-2.0.52.Final.jar:oap-libs/netty-resolver-dns-native-macos-4.1.100.Final-osx-x86_64.jar:oap-libs/netty-resolver-dns-native-macos-4.1.100.Final-osx-aarch_64.jar:oap-libs/netty-resolver-dns-classes-macos-4.1.100.Final.jar:oap-libs/netty-resolver-dns-4.1.100.Final.jar:oap-libs/netty-resolver-4.1.100.Final.jar:oap-libs/netty-handler-proxy-4.1.100.Final.jar:oap-libs/netty-handler-4.1.100.Final.jar:oap-libs/netty-common-4.1.100.Final.jar:oap-libs/netty-codec-socks-4.1.100.Final.jar:oap-libs/netty-codec-http-4.1.100.Final.jar:oap-libs/netty-codec-http2-4.1.100.Final.jar:oap-libs/netty-codec-haproxy-4.1.100.Final.jar:oap-libs/netty-codec-dns-4.1.100.Final.jar:oap-libs/netty-codec-4.1.100.Final.jar:oap-libs/netty-buffer-4.1.100.Final.jar:oap-libs/nacos-common-1.4.2.jar:oap-libs/nacos-client-1.4.2.jar:oap-libs/nacos-api-1.4.2.jar:oap-libs/mqe-rt-9.7.0.jar:oap-libs/mqe-grammar-9.7.0.jar:oap-libs/micrometer-observation-1.11.3.jar:oap-libs/micrometer-core-1.11.3.jar:oap-libs/micrometer-commons-1.11.3.jar:oap-libs/meter-analyzer-9.7.0.jar:oap-libs/lz4-java-1.7.1.jar:oap-libs/logql-plugin-9.7.0.jar:oap-libs/log-analyzer-9.7.0.jar:oap-libs/log4j-slf4j-impl-2.17.1.jar:oap-libs/log4j-over-slf4j-1.7.30.jar:oap-libs/log4j-core-2.17.1.jar:oap-libs/log4j-api-2.17.1.jar:oap-libs/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:oap-libs/library-util-9.7.0.jar:oap-libs/library-server-9.7.0.jar:oap-libs/library-module-9.7.0.jar:oap-libs/library-kubernetes-support-9.7.0.jar:oap-libs/library-elasticsearch-client-9.7.0.jar:oap-libs/library-datacarrier-queue-9.7.0.jar:oap-libs/library-client-9.7.0.jar:oap-libs/LatencyUtils-2.0.3.jar:oap-libs/kubernetes-model-storageclass-6.7.1.jar:oap-libs/kubernetes-model-scheduling-6.7.1.jar:oap-libs/kubernetes-model-resource-6.7.1.jar:oap-libs/kubernetes-model-rbac-6.7.1.jar:oap-libs/kubernetes-model-policy-6.7.1.jar:oap-libs/kubernetes-model-node-6.7.1.jar:oap-libs/kubernetes-model-networking-6.7.1.jar:oap-libs/kubernetes-model-metrics-6.7.1.jar:oap-libs/kubernetes-model-gatewayapi-6.7.1.jar:oap-libs/kubernetes-model-flowcontrol-6.7.1.jar:oap-libs/kubernetes-model-extensions-6.7.1.jar:oap-libs/kubernetes-model-events-6.7.1.jar:oap-libs/kubernetes-model-discovery-6.7.1.jar:oap-libs/kubernetes-model-core-6.7.1.jar:oap-libs/kubernetes-model-coordination-6.7.1.jar:oap-libs/kubernetes-model-common-6.7.1.jar:oap-libs/kubernetes-model-certificates-6.7.1.jar:oap-libs/kubernetes-model-batch-6.7.1.jar:oap-libs/kubernetes-model-autoscaling-6.7.1.jar:oap-libs/kubernetes-model-apps-6.7.1.jar:oap-libs/kubernetes-model-apiextensions-6.7.1.jar:oap-libs/kubernetes-model-admissionregistration-6.7.1.jar:oap-libs/kubernetes-httpclient-jdk-6.7.1.jar:oap-libs/kubernetes-client-api-6.7.1.jar:oap-libs/kubernetes-client-6.7.1.jar:oap-libs/kotlinx-coroutines-reactive-1.6.4.jar:oap-libs/kotlinx-coroutines-jdk8-1.6.4.jar:oap-libs/kotlinx-coroutines-core-jvm-1.6.4.jar:oap-libs/kotlinx-coroutines-core-1.6.4.jar:oap-libs/kotlin-stdlib-jdk8-1.6.21.jar:oap-libs/kotlin-stdlib-jdk7-1.6.21.jar:oap-libs/kotlin-stdlib-common-1.7.10.jar:oap-libs/kotlin-stdlib-1.7.10.jar:oap-libs/kotlin-reflect-1.7.10.jar:oap-libs/kafka-fetcher-plugin-9.7.0.jar:oap-libs/kafka-clients-2.8.1.jar:oap-libs/jsr305-3.0.2.jar:oap-libs/joda-time-2.10.5.jar:oap-libs/jetcd-resolver-0.5.3.jar:oap-libs/jetcd-core-0.5.3.jar:oap-libs/jetcd-common-0.5.3.jar:oap-libs/jcl-over-slf4j-1.7.30.jar:oap-libs/javax.inject-1.jar:oap-libs/javassist-3.25.0-GA.jar:oap-libs/java-dataloader-3.2.0.jar:oap-libs/jackson-module-kotlin-2.13.4.jar:oap-libs/jackson-datatype-jsr310-2.15.2.jar:oap-libs/jackson-datatype-jdk8-2.12.0.jar:oap-libs/jackson-datatype-guava-2.12.0.jar:oap-libs/jackson-dataformat-yaml-2.15.2.jar:oap-libs/jackson-databind-2.15.2.jar:oap-libs/jackson-core-2.15.2.jar:oap-libs/jackson-annotations-2.15.2.jar:oap-libs/j2objc-annotations-1.3.jar:oap-libs/istio-model-v1beta1-6.7.1.jar:oap-libs/istio-model-v1alpha3-6.7.1.jar:oap-libs/istio-client-6.7.1.jar:oap-libs/httpcore-nio-4.4.13.jar:oap-libs/httpcore-4.4.13.jar:oap-libs/httpclient-4.5.13.jar:oap-libs/httpasyncclient-4.1.3.jar:oap-libs/HikariCP-3.1.0.jar:oap-libs/HdrHistogram-2.1.12.jar:oap-libs/h2-2.1.212.jar:oap-libs/guice-4.1.0.jar:oap-libs/guava-32.0.1-jre.jar:oap-libs/gson-2.9.0.jar:oap-libs/grpc-stub-1.53.0.jar:oap-libs/grpc-protobuf-lite-1.53.0.jar:oap-libs/grpc-protobuf-1.53.0.jar:oap-libs/grpc-netty-1.53.0.jar:oap-libs/grpc-grpclb-1.53.0.jar:oap-libs/grpc-core-1.53.0.jar:oap-libs/grpc-context-1.53.0.jar:oap-libs/grpc-configuration-sync-9.7.0.jar:oap-libs/grpc-api-1.53.0.jar:oap-libs/groovy-4.0.15.jar:oap-libs/graphql-java-tools-13.0.1.jar:oap-libs/graphql-java-extended-scalars-18.1.jar:oap-libs/graphql-java-21.0.jar:oap-libs/freemarker-2.3.31.jar:oap-libs/flatbuffers-java-1.12.0.jar:oap-libs/failureaccess-1.0.1.jar:oap-libs/failsafe-2.3.4.jar:oap-libs/exporter-9.7.0.jar:oap-libs/event-analyzer-9.7.0.jar:oap-libs/error_prone_annotations-2.11.0.jar:oap-libs/envoy-metrics-receiver-plugin-9.7.0.jar:oap-libs/debugging-query-plugin-9.7.0.jar:oap-libs/data-generator-9.7.0.jar:oap-libs/curator-x-discovery-4.3.0.jar:oap-libs/curator-recipes-4.3.0.jar:oap-libs/curator-framework-4.3.0.jar:oap-libs/curator-client-4.3.0.jar:oap-libs/converter-jackson-2.9.0.jar:oap-libs/consul-client-1.5.3.jar:oap-libs/configuration-zookeeper-9.7.0.jar:oap-libs/configuration-nacos-9.7.0.jar:oap-libs/configuration-k8s-configmap-9.7.0.jar:oap-libs/configuration-etcd-9.7.0.jar:oap-libs/configuration-discovery-receiver-plugin-9.7.0.jar:oap-libs/configuration-consul-9.7.0.jar:oap-libs/configuration-apollo-9.7.0.jar:oap-libs/configuration-api-9.7.0.jar:oap-libs/commons-validator-1.7.jar:oap-libs/commons-text-1.4.jar:oap-libs/commons-net-3.9.0.jar:oap-libs/commons-logging-1.2.jar:oap-libs/commons-lang3-3.12.0.jar:oap-libs/commons-io-2.7.jar:oap-libs/commons-codec-1.11.jar:oap-libs/commons-beanutils-1.9.4.jar:oap-libs/cluster-zookeeper-plugin-9.7.0.jar:oap-libs/cluster-standalone-plugin-9.7.0.jar:oap-libs/cluster-nacos-plugin-9.7.0.jar:oap-libs/cluster-kubernetes-plugin-9.7.0.jar:oap-libs/cluster-etcd-plugin-9.7.0.jar:oap-libs/cluster-consul-plugin-9.7.0.jar:oap-libs/classmate-1.5.1.jar:oap-libs/checker-qual-3.33.0.jar:oap-libs/brotli4j-1.12.0.jar:oap-libs/banyandb-java-client-0.5.0.jar:oap-libs/aws-firehose-receiver-9.7.0.jar:oap-libs/audience-annotations-0.5.0.jar:oap-libs/armeria-protobuf-1.25.0.jar:oap-libs/armeria-graphql-protocol-1.25.0.jar:oap-libs/armeria-graphql-1.25.0.jar:oap-libs/armeria-1.25.0.jar:oap-libs/apollo-core-1.8.0.jar:oap-libs/apollo-client-1.8.0.jar:oap-libs/apm-network-9.7.0.jar:oap-libs/aopalliance-1.0.jar:oap-libs/antlr4-runtime-4.11.1.jar:oap-libs/annotations-17.0.0.jar:oap-libs/animal-sniffer-annotations-1.21.jar:oap-libs/ai-pipeline-9.7.0.jar:oap-libs/agent-analyzer-9.7.0.jar:config:' org.apache.skywalking.oap.server.starter.OAPServerStartUp
[0.004s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.

There is insufficient memory for the Java Runtime Environment to continue.
Cannot create worker GC thread. Out of system resources.
Can not save log file, dump to screen..

There is insufficient memory for the Java Runtime Environment to continue.
Cannot create worker GC thread. Out of system resources.
Possible reasons:
The system is out of physical RAM or swap space
The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
JVM is running with Unscaled Compressed Oops mode in which the Java heap is
placed in the first 4GB address space. The Java Heap base address is the
maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
to set the Java Heap base and to place the Java Heap above 4GB virtual address.
This output file may be truncated or incomplete.

Out of Memory Error (workerManager.hpp:70), pid=1, tid=1

JRE version: (11.0.21+9) (build )
Java VM: OpenJDK 64-Bit Server VM (11.0.21+9, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
Core dump will be written. Default location: /skywalking/core.1

--------------- S U M M A R Y ------------

Command Line: -Xms2G org.apache.skywalking.oap.server.starter.OAPServerStartUp

Host: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz, 8 cores, 31G, Ubuntu 22.04.3 LTS
Time: Tue Apr 23 08:24:06 2024 UTC elapsed time: 0.005420 seconds (0d 0h 0m 0s)

--------------- T H R E A D ---------------

Current thread (0x00007f1eedb99800): JavaThread "Unknown thread" [_thread_in_vm, id=1, stack(0x00007fffe57fe000,0x00007fffe5ffb000)]

Stack: [0x00007fffe57fe000,0x00007fffe5ffb000], sp=0x00007fffe5ff56e0, free space=8157k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xf4e28a] VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x1ca
V [libjvm.so+0xf4f14b] VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x2b
V [libjvm.so+0x6d5f15] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xd5
V [libjvm.so+0xf7b87b] AbstractWorkGang::add_workers(unsigned int, bool)+0x14b
V [libjvm.so+0x7b40a5] G1CollectedHeap::G1CollectedHeap(G1CollectorPolicy*)+0x5b5
V [libjvm.so+0x7a5cb9] G1Arguments::create_heap()+0x69
V [libjvm.so+0xf098e1] Universe::initialize_heap()+0x31
V [libjvm.so+0xf09b96] universe_init()+0x46
V [libjvm.so+0x8ad5d7] init_globals()+0x57
V [libjvm.so+0xee0f27] Threads::create_vm(JavaVMInitArgs*, bool*)+0x357
V [libjvm.so+0x96b8a2] JNI_CreateJavaVM+0x52
C [libjli.so+0xa3af] JavaMain+0x8f
C [libjli.so+0xf794] CallJavaMainInNewThread+0x74
C [libjli.so+0xbd7d] ContinueInNewThread+0x4d
C [libjli.so+0xd400] JLI_Launch+0x1470
C [java+0xa63]

--------------- P R O C E S S ---------------

Threads class SMR info:
_java_thread_list=0x00007f1eedb5bec0, length=0, elements={
}

Java Threads: ( => current thread )

Other Threads:

=>0x00007f1eedb99800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=1, stack(0x00007fffe57fe000,0x00007fffe5ffb000)]

Threads with active compile tasks:

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

[error occurred during error reporting (printing compressed oops mode), id 0xb, SIGSEGV (0xb) at pc=0x00007f1eeaf08213]

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (2 events):
Event: 0.003 Protecting memory [0x00007fffe57fe000,0x00007fffe5802000] with protection modes 0
Event: 0.003 Loaded shared library /opt/java/openjdk/lib/libzip.so

Dynamic libraries:
7f1ed4247000-7f1ed44b8000 rw-p 00000000 00:00 0
7f1ed44b8000-7f1ed4728000 rwxp 00000000 00:00 0
7f1ed4728000-7f1ed4a49000 ---p 00000000 00:00 0
7f1ed4a49000-7f1ed4cb9000 rwxp 00000000 00:00 0
7f1ed4cb9000-7f1edbf80000 ---p 00000000 00:00 0
7f1edbf80000-7f1edc1f0000 rwxp 00000000 00:00 0
7f1edc1f0000-7f1ee34b8000 ---p 00000000 00:00 0
7f1ee34b8000-7f1ee9000000 r--s 00000000 08:11 18621844 7f1ee9000000-7f1ee901b000 r-xp 00000000 08:11 18621843 7f1ee901b000-7f1ee921a000 ---p 0001b000 08:11 18621843 7f1ee921a000-7f1ee921b000 r--p 0001a000 08:11 18621843 7f1ee921b000-7f1ee921c000 rw-p 0001b000 08:11 18621843 7f1ee9400000-7f1ee941b000 r-xp 00000000 08:11 18621826 7f1ee941b000-7f1ee961a000 ---p 0001b000 08:11 18621826 7f1ee961a000-7f1ee961c000 r--p 0001a000 08:11 18621826 7f1ee961c000-7f1ee961d000 rw-p 0001c000 08:11 18621826 7f1ee9800000-7f1ee9829000 r-xp 00000000 08:11 18621822 7f1ee9829000-7f1ee9a28000 ---p 00029000 08:11 18621822 7f1ee9a28000-7f1ee9a29000 r--p 00028000 08:11 18621822 7f1ee9a29000-7f1ee9a2a000 rw-p 00029000 08:11 18621822 7f1ee9a2a000-7f1ee9a2b000 rw-p 00000000 00:00 0
7f1ee9ab1000-7f1ee9c00000 rw-p 00000000 00:00 0
7f1ee9c00000-7f1ee9c0d000 r-xp 00000000 08:11 18621842 7f1ee9c0d000-7f1ee9e0c000 ---p 0000d000 08:11 18621842 7f1ee9e0c000-7f1ee9e0e000 r--p 0000c000 08:11 18621842 7f1ee9e0e000-7f1ee9e0f000 rw-p 0000e000 08:11 18621842 7f1ee9eff000-7f1ee9f00000 ---p 00000000 00:00 0
7f1ee9f00000-7f1eea000000 rw-p 00000000 00:00 0
7f1eea000000-7f1eeb280000 r-xp 00000000 08:11 18621856 7f1eeb280000-7f1eeb480000 ---p 01280000 08:11 18621856 7f1eeb480000-7f1eeb545000 r--p 01280000 08:11 18621856 7f1eeb545000-7f1eeb580000 rw-p 01345000 08:11 18621856 7f1eeb580000-7f1eeb5d8000 rw-p 00000000 00:00 0
7f1eeb5e9000-7f1eeb5ee000 rw-p 00000000 00:00 0
7f1eeb5ee000-7f1eeb6d4000 ---p 00000000 00:00 0
7f1eeb6d4000-7f1eeb6d9000 rw-p 00000000 00:00 0
7f1eeb6d9000-7f1eeb7bf000 ---p 00000000 00:00 0
7f1eeb7bf000-7f1eeb7c4000 rw-p 00000000 00:00 0
7f1eeb7c4000-7f1eeb7cb000 ---p 00000000 00:00 0
7f1eeb7cb000-7f1eeb7d3000 rw-s 00000000 08:11 18743325 7f1eeb7d3000-7f1eeb7d4000 r--p 00000000 08:11 18613868 7f1eeb7d4000-7f1eeb7d5000 r-xp 00001000 08:11 18613868 7f1eeb7d5000-7f1eeb7d6000 r--p 00002000 08:11 18613868 7f1eeb7d6000-7f1eeb7d7000 r--p 00002000 08:11 18613868 7f1eeb7d7000-7f1eeb7d8000 rw-p 00003000 08:11 18613868 7f1eeb7d8000-7f1eeb800000 r--p 00000000 08:11 18613771 7f1eeb800000-7f1eeb995000 r-xp 00028000 08:11 18613771 7f1eeb995000-7f1eeb9ed000 r--p 001bd000 08:11 18613771 7f1eeb9ed000-7f1eeb9f1000 r--p 00214000 08:11 18613771 7f1eeb9f1000-7f1eeb9f3000 rw-p 00218000 08:11 18613771 7f1eeb9f3000-7f1eeba00000 rw-p 00000000 00:00 0
7f1eeba00000-7f1eeba1a000 r-xp 00000000 08:11 18621807 7f1eeba1a000-7f1eebc1a000 ---p 0001a000 08:11 18621807 7f1eebc1a000-7f1eebc1b000 r--p 0001a000 08:11 18621807 7f1eebc1b000-7f1eebc1c000 rw-p 0001b000 08:11 18621807 7f1eebc20000-7f1eebd20000 rw-p 00000000 00:00 0
7f1eebd20000-7f1eebd2e000 r--p 00000000 08:11 18613823 7f1eebd2e000-7f1eebdaa000 r-xp 0000e000 08:11 18613823 7f1eebdaa000-7f1eebe05000 r--p 0008a000 08:11 18613823 7f1eebe05000-7f1eebe06000 r--p 000e4000 08:11 18613823 7f1eebe06000-7f1eebe07000 rw-p 000e5000 08:11 18613823 7f1eebe07000-7f1eebe09000 rw-p 00000000 00:00 0
7f1eebe09000-7f1eebe0a000 r--p 00000000 08:11 18613785 7f1eebe0a000-7f1eebe0b000 r-xp 00001000 08:11 18613785 7f1eebe0b000-7f1eebe0c000 r--p 00002000 08:11 18613785 7f1eebe0c000-7f1eebe0d000 r--p 00002000 08:11 18613785 7f1eebe0d000-7f1eebe0e000 rw-p 00003000 08:11 18613785 7f1eebe0e000-7f1eebe0f000 r--p 00000000 08:11 18613866 7f1eebe0f000-7f1eebe10000 r-xp 00001000 08:11 18613866 7f1eebe10000-7f1eebe11000 r--p 00002000 08:11 18613866 7f1eebe11000-7f1eebe12000 r--p 00002000 08:11 18613866 7f1eebe12000-7f1eebe13000 rw-p 00003000 08:11 18613866 7f1eebe14000-7f1eebe15000 ---p 00000000 00:00 0
7f1eebe15000-7f1eebe16000 r--p 00000000 00:00 0
7f1eebe16000-7f1eebe18000 rw-p 00000000 00:00 0
7f1eebe18000-7f1eebe1a000 r-xp 00000000 00:00 0 7f1eebe1a000-7f1eebe1c000 r--p 00000000 08:11 18613753 7f1eebe1c000-7f1eebe46000 r-xp 00002000 08:11 18613753 7f1eebe46000-7f1eebe51000 r--p 0002c000 08:11 18613753 7f1eebe52000-7f1eebe54000 r--p 00037000 08:11 18613753 7f1eebe54000-7f1eebe56000 rw-p 00039000 08:11 18613753 7f1eebe56000-7f1eebe57000 r-xp 00000000 08:11 18620567 7f1eec057000-7f1eec058000 r--p 00001000 08:11 18620567 7f1eec058000-7f1eec059000 rw-p 00002000 08:11 18620567 7f1eedb55000-7f1eedbd9000 rw-p 00000000 00:00 0 7fffe57fe000-7fffe5802000 ---p 00000000 00:00 0
7fffe5fdb000-7fffe5ffe000 rw-p 00000000 00:00 0 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 /opt/java/openjdk/lib/modules
/opt/java/openjdk/lib/libzip.so
/opt/java/openjdk/lib/libzip.so
/opt/java/openjdk/lib/libzip.so
/opt/java/openjdk/lib/libzip.so
/opt/java/openjdk/lib/libjimage.so
/opt/java/openjdk/lib/libjimage.so
/opt/java/openjdk/lib/libjimage.so
/opt/java/openjdk/lib/libjimage.so
/opt/java/openjdk/lib/libjava.so
/opt/java/openjdk/lib/libjava.so
/opt/java/openjdk/lib/libjava.so
/opt/java/openjdk/lib/libjava.so
/opt/java/openjdk/lib/libverify.so
/opt/java/openjdk/lib/libverify.so
/opt/java/openjdk/lib/libverify.so
/opt/java/openjdk/lib/libverify.so
/opt/java/openjdk/lib/server/libjvm.so
/opt/java/openjdk/lib/server/libjvm.so
/opt/java/openjdk/lib/server/libjvm.so
/opt/java/openjdk/lib/server/libjvm.so
/tmp/hsperfdata_root/1
/usr/lib/x86_64-linux-gnu/librt.so.1
/usr/lib/x86_64-linux-gnu/librt.so.1
/usr/lib/x86_64-linux-gnu/librt.so.1
/usr/lib/x86_64-linux-gnu/librt.so.1
/usr/lib/x86_64-linux-gnu/librt.so.1
/usr/lib/x86_64-linux-gnu/libc.so.6
/usr/lib/x86_64-linux-gnu/libc.so.6
/usr/lib/x86_64-linux-gnu/libc.so.6
/usr/lib/x86_64-linux-gnu/libc.so.6
/usr/lib/x86_64-linux-gnu/libc.so.6
/opt/java/openjdk/lib/jli/libjli.so
/opt/java/openjdk/lib/jli/libjli.so
/opt/java/openjdk/lib/jli/libjli.so
/opt/java/openjdk/lib/jli/libjli.so
/usr/lib/x86_64-linux-gnu/libm.so.6
/usr/lib/x86_64-linux-gnu/libm.so.6
/usr/lib/x86_64-linux-gnu/libm.so.6
/usr/lib/x86_64-linux-gnu/libm.so.6
/usr/lib/x86_64-linux-gnu/libm.so.6
/usr/lib/x86_64-linux-gnu/libdl.so.2
/usr/lib/x86_64-linux-gnu/libdl.so.2
/usr/lib/x86_64-linux-gnu/libdl.so.2
/usr/lib/x86_64-linux-gnu/libdl.so.2
/usr/lib/x86_64-linux-gnu/libdl.so.2
/usr/lib/x86_64-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/libpthread.so.0
[vdso]
/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
/opt/java/openjdk/bin/java
/opt/java/openjdk/bin/java
/opt/java/openjdk/bin/java
[heap]
[stack]
[vsyscall]

VM Arguments:
jvm_args: -Xms2G
java_command: org.apache.skywalking.oap.server.starter.OAPServerStartUp
java_class_path (initial): /skywalking/ext-libs/*.jar:oap-libs/zstd-jni-1.4.9-1.jar:oap-libs/zookeeper-jute-3.5.7.jar:oap-libs/zookeeper-3.5.7.jar:oap-libs/zjsonpatch-0.3.0.jar:oap-libs/zipkin-receiver-plugin-9.7.0.jar:oap-libs/zipkin-query-plugin-9.7.0.jar:oap-libs/zipkin-2.24.1.jar:oap-libs/vavr-match-0.10.3.jar:oap-libs/vavr-0.10.3.jar:oap-libs/tool-profile-snapshot-server-mock-9.7.0.jar:oap-libs/tool-profile-snapshot-bootstrap-9.7.0.jar:oap-libs/telemetry-prometheus-9.7.0.jar:oap-libs/telemetry-api-9.7.0.jar:oap-libs/storage-jdbc-hikaricp-plugin-9.7.0.jar:oap-libs/storage-elasticsearch-plugin-9.7.0.jar:oap-libs/storage-banyandb-plugin-9.7.0.jar:oap-libs/snappy-java-1.1.8.1.jar:oap-libs/snakeyaml-engine-2.6.jar:oap-libs/snakeyaml-2.0.jar:oap-libs/slf4j-api-1.7.30.jar:oap-libs/skywalking-zabbix-receiver-plugin-9.7.0.jar:oap-libs/skywalking-trace-receiver-plugin-9.7.0.jar:oap-libs/skywalking-telegraf-receiver-plugin-9.7.0.jar:oap-libs/skywalking-sharing-server-plugin-9.7.0.jar:oap-libs/skywalking-profile-receiver-plugin-9.7.0.jar:oap-libs/skywalking-meter-receiver-plugin-9.7.0.jar:oap-libs/skywalking-mesh-receiver-plugin-9.7.0.jar:oap-libs/skywalking-management-receiver-plugin-9.7.0.jar:oap-libs/skywalking-log-receiver-plugin-9.7.0.jar:oap-libs/skywalking-jvm-receiver-plugin-9.7.0.jar:oap-libs/skywalking-event-receiver-plugin-9.7.0.jar:oap-libs/skywalking-ebpf-receiver-plugin-9.7.0.jar:oap-libs/skywalking-clr-receiver-plugin-9.7.0.jar:oap-libs/skywalking-browser-receiver-plugin-9.7.0.jar:oap-libs/simpleclient_hotspot-0.6.0.jar:oap-libs/simpleclient_common-0.6.0.jar:oap-libs/simpleclient-0.6.0.jar:oap-libs/service-1.12.0.jar:oap-libs/server-testing-9.7.0.jar:oap-libs/server-starter-9.7.0.jar:oap-libs/server-health-checker-9.7.0.jar:oap-libs/server-core-9.7.0.jar:oap-libs/server-alarm-plugin-9.7.0.jar:oap-libs/retrofit-2.9.0.jar:oap-libs/receiver-proto-9.7.0.jar:oap-libs/reactive-streams-1.0.3.jar:oap-libs/re2j-1.5.jar:oap-libs/query-graphql-plugin-9.7.0.jar:oap-li
Launcher Type: SUN_STANDARD

[Global flags]
intx CICompilerCount = 4 {product} {ergonomic}
uint G1ConcRefinementThreads = 8 {product} {ergonomic}
size_t G1HeapRegionSize = 2097152 {product} {ergonomic}
uintx GCDrainStackTargetSize = 64 {product} {ergonomic}
size_t InitialHeapSize = 2147483648 {product} {command line}
size_t MaxHeapSize = 8388608000 {product} {ergonomic}
size_t MinHeapDeltaBytes = 2097152 {product} {ergonomic}
uintx NonNMethodCodeHeapSize = 5836300 {pd product} {ergonomic}
uintx NonProfiledCodeHeapSize = 122910970 {pd product} {ergonomic}
uintx ProfiledCodeHeapSize = 122910970 {pd product} {ergonomic}
uintx ReservedCodeCacheSize = 251658240 {pd product} {ergonomic}
bool SegmentedCodeCache = true {product} {ergonomic}
bool UseCompressedClassPointers = true {lp64_product} {ergonomic}
bool UseCompressedOops = true {lp64_product} {ergonomic}
bool UseG1GC = true {product} {ergonomic}

Logging:
Log output configuration:
#0: stdout all=warning uptime,level,tags
#1: stderr all=off uptime,level,tags

Environment Variables:
JAVA_HOME=/opt/java/openjdk
PATH=/skywalking/bin:/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8

Signal Handlers:
SIGSEGV: [libjvm.so+0xf4fea0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xf4fea0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0xf4fea0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0xc70220], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0xc70220], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0xf4fea0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR2: [libjvm.so+0xc700c0], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none

--------------- S Y S T E M ---------------

OS:DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
uname:Linux 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64
OS uptime: 65 days 2:28 hours
libc:glibc 2.35 NPTL 2.35
rlimit (soft/hard): STACK 8192k/infinity , CORE infinity/infinity , NPROC infinity/infinity , NOFILE 1048576/1048576 , AS infinity/infinity , CPU infinity/infinity , DATA infinity/infinity , FSIZE infinity/infinity , MEMLOCK 64k/64k
load average:1.23 1.32 2.02

/proc/meminfo:
MemTotal: 32765892 kB
MemFree: 5278012 kB
MemAvailable: 18130408 kB
Buffers: 418876 kB
Cached: 11779992 kB
SwapCached: 198756 kB
Active: 16205788 kB
Inactive: 8790176 kB
Active(anon): 11551512 kB
Inactive(anon): 2050096 kB
Active(file): 4654276 kB
Inactive(file): 6740080 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 16777212 kB
SwapFree: 13892836 kB
Dirty: 424 kB
Writeback: 0 kB
AnonPages: 12701908 kB
Mapped: 331716 kB
Shmem: 804512 kB
Slab: 2043752 kB
SReclaimable: 1860872 kB
SUnreclaim: 182880 kB
KernelStack: 36640 kB
PageTables: 56196 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 33160156 kB
Committed_AS: 27949568 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 234200 kB
VmallocChunk: 34359310332 kB
HardwareCorrupted: 0 kB
AnonHugePages: 9988096 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 128832 kB
DirectMap2M: 5113856 kB
DirectMap1G: 30408704 kB

/sys/kernel/mm/transparent_hugepage/enabled:
[always] madvise never
/sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter):
[always] madvise never

Process Memory:
Virtual Size: 384424K (peak: 384424K)
Resident Set Size: 11904K (peak: 11904K) (anon: 5572K, file: 6332K, shmem: 0K)
Swapped out: 0K
C-Heap outstanding allocations: 4266K, retained: 101K
glibc malloc tunables: (default)

/proc/sys/kernel/threads-max (system-wide limit on the number of threads):
255811
/proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):
655360
/proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):
131072

container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-7
cpu_memory_nodes: 0
active_processor_count: 8
cpu_quota: no quota
cpu_period: 100000
cpu_shares: no shares
memory_limit_in_bytes: unlimited
memory_and_swap_limit_in_bytes: unlimited
memory_soft_limit_in_bytes: unlimited
memory_usage_in_bytes: 5760 k
memory_max_usage_in_bytes: 8736 k
kernel_memory_usage_in_bytes: unlimited
kernel_memory_max_usage_in_bytes: unlimited
kernel_memory_limit_in_bytes: unlimited
maximum number of tasks: unlimited
current number of tasks: 1

VMWare virtualization detected
Steal ticks since vm start: 0
Steal ticks percentage since vm start: 0.000

CPU:total 8 (initial active 8) (4 cores per cpu, 1 threads per core) family 6 model 63 stepping 2 microcode 0x43, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, tscinvbit, bmi1, bmi2, fma
CPU Model and flags from /proc/cpuinfo:
model name : Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm arat fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt
Online cpus:
0-7
Offline cpus:
8-127
BIOS frequency limitation:

Frequency switch latency (ns):

Available cpu frequencies:

Current governor:

Core performance/turbo boost:

Memory: 4k page, physical 32765892k(5278012k free), swap 16777212k(13892836k free)

vm_info: OpenJDK 64-Bit Server VM (11.0.21+9) for linux-amd64 JRE (11.0.21+9), built on Oct 17 2023 21:44:42 by "" with gcc 7.5.0

END.

What you expected to happen

success start skywalking-oap-server:9.7.0

How to reproduce

start skywalking-oap-server:9.7.0 by docker

docker run --name sw_oap --restart always -d -p 12800:12800 -p 11800:11800 apache/skywalking-oap-server:9.7.0

Anything else

No response

Are you willing to submit a pull request to fix on your own?

Yes I am willing to submit a pull request on my own!

Code of Conduct

I agree to follow this project's Code of Conduct

Answered by wu-sheng

Apr 23, 2024

Then OOM must happen. H2 is holding data in memory for sure.

View full answer

wu-sheng · 2024-04-23T08:44:40Z

wu-sheng
Apr 23, 2024
Collaborator

Which database are you using?

8 replies

ibingdian Apr 23, 2024
Author

default database , should h2

wu-sheng Apr 23, 2024
Collaborator

Then OOM must happen. H2 is holding data in memory for sure.

Answer selected by ibingdian

wu-sheng Apr 23, 2024
Collaborator

H2 is for demo only, we write this very clearly in the docs.

https://skywalking.apache.org/docs/main/next/en/setup/backend/storages/h2/

ibingdian Apr 23, 2024
Author

wait a moment, i turn to mysql

ibingdian Apr 23, 2024
Author

when i turn to mysql ,i meet the same problem , the docker command is
docker run --name sw_oap --restart always -d -p 12800:12800 -p 11800:11800 -v /home/dev/LY_TEST/tools/sw_oap/oap-libs:/skywalking/oap-libs -e SW_STORAGE=mysql -e SW_JDBC_URL=jdbc:mysql://192.168.8.82:13306/sw_db?rewriteBatchedStatements=true -e SW_DATA_SOURCE_USER=root -e SW_DATA_SOURCE_PASSWORD=123456 apache/skywalking-oap-server:9.7.0

the error as follow :

Memory: 4k page, physical 32765892k(3804668k free), swap 16777212k(13892996k free)

vm_info: OpenJDK 64-Bit Server VM (11.0.21+9) for linux-amd64 JRE (11.0.21+9), built on Oct 17 2023 21:44:42 by "" with gcc 7.5.0

END.
[Entrypoint] Apache SkyWalking Docker Image

exec java -Xms2G -classpath '/skywalking/ext-libs/*.jar:oap-libs/zstd-jni-1.4.9-1.jar:oap-libs/zookeeper-jute-3.5.7.jar:oap-libs/zookeeper-3.5.7.jar:oap-libs/zjsonpatch-0.3.0.jar:oap-libs/zipkin-receiver-plugin-9.7.0.jar:oap-libs/zipkin-query-plugin-9.7.0.jar:oap-libs/zipkin-2.24.1.jar:oap-libs/vavr-match-0.10.3.jar:oap-libs/vavr-0.10.3.jar:oap-libs/tool-profile-snapshot-server-mock-9.7.0.jar:oap-libs/tool-profile-snapshot-bootstrap-9.7.0.jar:oap-libs/telemetry-prometheus-9.7.0.jar:oap-libs/telemetry-api-9.7.0.jar:oap-libs/storage-jdbc-hikaricp-plugin-9.7.0.jar:oap-libs/storage-elasticsearch-plugin-9.7.0.jar:oap-libs/storage-banyandb-plugin-9.7.0.jar:oap-libs/snappy-java-1.1.8.1.jar:oap-libs/snakeyaml-engine-2.6.jar:oap-libs/snakeyaml-2.0.jar:oap-libs/slf4j-api-1.7.30.jar:oap-libs/skywalking-zabbix-receiver-plugin-9.7.0.jar:oap-libs/skywalking-trace-receiver-plugin-9.7.0.jar:oap-libs/skywalking-telegraf-receiver-plugin-9.7.0.jar:oap-libs/skywalking-sharing-server-plugin-9.7.0.jar:oap-libs/skywalking-profile-receiver-plugin-9.7.0.jar:oap-libs/skywalking-meter-receiver-plugin-9.7.0.jar:oap-libs/skywalking-mesh-receiver-plugin-9.7.0.jar:oap-libs/skywalking-management-receiver-plugin-9.7.0.jar:oap-libs/skywalking-log-receiver-plugin-9.7.0.jar:oap-libs/skywalking-jvm-receiver-plugin-9.7.0.jar:oap-libs/skywalking-event-receiver-plugin-9.7.0.jar:oap-libs/skywalking-ebpf-receiver-plugin-9.7.0.jar:oap-libs/skywalking-clr-receiver-plugin-9.7.0.jar:oap-libs/skywalking-browser-receiver-plugin-9.7.0.jar:oap-libs/simpleclient_hotspot-0.6.0.jar:oap-libs/simpleclient_common-0.6.0.jar:oap-libs/simpleclient-0.6.0.jar:oap-libs/service-1.12.0.jar:oap-libs/server-testing-9.7.0.jar:oap-libs/server-starter-9.7.0.jar:oap-libs/server-health-checker-9.7.0.jar:oap-libs/server-core-9.7.0.jar:oap-libs/server-alarm-plugin-9.7.0.jar:oap-libs/retrofit-2.9.0.jar:oap-libs/receiver-proto-9.7.0.jar:oap-libs/reactive-streams-1.0.3.jar:oap-libs/re2j-1.5.jar:oap-libs/query-graphql-plugin-9.7.0.jar:oap-libs/proto-google-common-protos-2.9.0.jar:oap-libs/protobuf-java-util-3.21.8.jar:oap-libs/protobuf-java-3.21.8.jar:oap-libs/promql-plugin-9.7.0.jar:oap-libs/postgresql-42.4.1.jar:oap-libs/pgv-java-stub-0.6.13.jar:oap-libs/perfmark-api-0.25.0.jar:oap-libs/otel-receiver-plugin-9.7.0.jar:oap-libs/okio-1.17.2.jar:oap-libs/okhttp-3.14.9.jar:oap-libs/oal-rt-9.7.0.jar:oap-libs/oal-grammar-9.7.0.jar:oap-libs/netty-transport-native-unix-common-4.1.100.Final-osx-x86_64.jar:oap-libs/netty-transport-native-unix-common-4.1.100.Final-osx-aarch_64.jar:oap-libs/netty-transport-native-unix-common-4.1.100.Final-linux-x86_64.jar:oap-libs/netty-transport-native-unix-common-4.1.100.Final-linux-aarch_64.jar:oap-libs/netty-transport-native-unix-common-4.1.100.Final.jar:oap-libs/netty-transport-native-kqueue-4.1.100.Final-osx-x86_64.jar:oap-libs/netty-transport-native-kqueue-4.1.100.Final-osx-aarch_64.jar:oap-libs/netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:oap-libs/netty-transport-native-epoll-4.1.100.Final-linux-aarch_64.jar:oap-libs/netty-transport-native-epoll-4.1.100.Final.jar:oap-libs/netty-transport-classes-kqueue-4.1.100.Final.jar:oap-libs/netty-transport-classes-epoll-4.1.100.Final.jar:oap-libs/netty-transport-4.1.100.Final.jar:oap-libs/netty-tcnative-classes-2.0.61.Final.jar:oap-libs/netty-tcnative-boringssl-static-2.0.61.Final-windows-x86_64.jar:oap-libs/netty-tcnative-boringssl-static-2.0.61.Final-osx-x86_64.jar:oap-libs/netty-tcnative-boringssl-static-2.0.61.Final-osx-aarch_64.jar:oap-libs/netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar:oap-libs/netty-tcnative-boringssl-static-2.0.61.Final-linux-aarch_64.jar:oap-libs/netty-tcnative-boringssl-static-2.0.52.Final.jar:oap-libs/netty-resolver-dns-native-macos-4.1.100.Final-osx-x86_64.jar:oap-libs/netty-resolver-dns-native-macos-4.1.100.Final-osx-aarch_64.jar:oap-libs/netty-resolver-dns-classes-macos-4.1.100.Final.jar:oap-libs/netty-resolver-dns-4.1.100.Final.jar:oap-libs/netty-resolver-4.1.100.Final.jar:oap-libs/netty-handler-proxy-4.1.100.Final.jar:oap-libs/netty-handler-4.1.100.Final.jar:oap-libs/netty-common-4.1.100.Final.jar:oap-libs/netty-codec-socks-4.1.100.Final.jar:oap-libs/netty-codec-http-4.1.100.Final.jar:oap-libs/netty-codec-http2-4.1.100.Final.jar:oap-libs/netty-codec-haproxy-4.1.100.Final.jar:oap-libs/netty-codec-dns-4.1.100.Final.jar:oap-libs/netty-codec-4.1.100.Final.jar:oap-libs/netty-buffer-4.1.100.Final.jar:oap-libs/native-osx-aarch64-1.12.0.jar:oap-libs/nacos-common-1.4.2.jar:oap-libs/nacos-client-1.4.2.jar:oap-libs/nacos-api-1.4.2.jar:oap-libs/mysql-connector-java-8.0.30.jar:oap-libs/mqe-rt-9.7.0.jar:oap-libs/mqe-grammar-9.7.0.jar:oap-libs/micrometer-observation-1.11.3.jar:oap-libs/micrometer-core-1.11.3.jar:oap-libs/micrometer-commons-1.11.3.jar:oap-libs/meter-analyzer-9.7.0.jar:oap-libs/lz4-java-1.7.1.jar:oap-libs/logql-plugin-9.7.0.jar:oap-libs/log-analyzer-9.7.0.jar:oap-libs/log4j-slf4j-impl-2.17.1.jar:oap-libs/log4j-over-slf4j-1.7.30.jar:oap-libs/log4j-core-2.17.1.jar:oap-libs/log4j-api-2.17.1.jar:oap-libs/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:oap-libs/library-util-9.7.0.jar:oap-libs/library-server-9.7.0.jar:oap-libs/library-module-9.7.0.jar:oap-libs/library-kubernetes-support-9.7.0.jar:oap-libs/library-elasticsearch-client-9.7.0.jar:oap-libs/library-datacarrier-queue-9.7.0.jar:oap-libs/library-client-9.7.0.jar:oap-libs/LatencyUtils-2.0.3.jar:oap-libs/kubernetes-model-storageclass-6.7.1.jar:oap-libs/kubernetes-model-scheduling-6.7.1.jar:oap-libs/kubernetes-model-resource-6.7.1.jar:oap-libs/kubernetes-model-rbac-6.7.1.jar:oap-libs/kubernetes-model-policy-6.7.1.jar:oap-libs/kubernetes-model-node-6.7.1.jar:oap-libs/kubernetes-model-networking-6.7.1.jar:oap-libs/kubernetes-model-metrics-6.7.1.jar:oap-libs/kubernetes-model-gatewayapi-6.7.1.jar:oap-libs/kubernetes-model-flowcontrol-6.7.1.jar:oap-libs/kubernetes-model-extensions-6.7.1.jar:oap-libs/kubernetes-model-events-6.7.1.jar:oap-libs/kubernetes-model-discovery-6.7.1.jar:oap-libs/kubernetes-model-core-6.7.1.jar:oap-libs/kubernetes-model-coordination-6.7.1.jar:oap-libs/kubernetes-model-common-6.7.1.jar:oap-libs/kubernetes-model-certificates-6.7.1.jar:oap-libs/kubernetes-model-batch-6.7.1.jar:oap-libs/kubernetes-model-autoscaling-6.7.1.jar:oap-libs/kubernetes-model-apps-6.7.1.jar:oap-libs/kubernetes-model-apiextensions-6.7.1.jar:oap-libs/kubernetes-model-admissionregistration-6.7.1.jar:oap-libs/kubernetes-httpclient-jdk-6.7.1.jar:oap-libs/kubernetes-client-api-6.7.1.jar:oap-libs/kubernetes-client-6.7.1.jar:oap-libs/kotlinx-coroutines-reactive-1.6.4.jar:oap-libs/kotlinx-coroutines-jdk8-1.6.4.jar:oap-libs/kotlinx-coroutines-core-jvm-1.6.4.jar:oap-libs/kotlinx-coroutines-core-1.6.4.jar:oap-libs/kotlin-stdlib-jdk8-1.6.21.jar:oap-libs/kotlin-stdlib-jdk7-1.6.21.jar:oap-libs/kotlin-stdlib-common-1.7.10.jar:oap-libs/kotlin-stdlib-1.7.10.jar:oap-libs/kotlin-reflect-1.7.10.jar:oap-libs/kafka-fetcher-plugin-9.7.0.jar:oap-libs/kafka-clients-2.8.1.jar:oap-libs/jsr305-3.0.2.jar:oap-libs/joda-time-2.10.5.jar:oap-libs/jetcd-resolver-0.5.3.jar:oap-libs/jetcd-core-0.5.3.jar:oap-libs/jetcd-common-0.5.3.jar:oap-libs/jcl-over-slf4j-1.7.30.jar:oap-libs/javax.inject-1.jar:oap-libs/javassist-3.25.0-GA.jar:oap-libs/java-dataloader-3.2.0.jar:oap-libs/jackson-module-kotlin-2.13.4.jar:oap-libs/jackson-datatype-jsr310-2.15.2.jar:oap-libs/jackson-datatype-jdk8-2.12.0.jar:oap-libs/jackson-datatype-guava-2.12.0.jar:oap-libs/jackson-dataformat-yaml-2.15.2.jar:oap-libs/jackson-databind-2.15.2.jar:oap-libs/jackson-core-2.15.2.jar:oap-libs/jackson-annotations-2.15.2.jar:oap-libs/j2objc-annotations-1.3.jar:oap-libs/istio-model-v1beta1-6.7.1.jar:oap-libs/istio-model-v1alpha3-6.7.1.jar:oap-libs/istio-client-6.7.1.jar:oap-libs/httpcore-nio-4.4.13.jar:oap-libs/httpcore-4.4.13.jar:oap-libs/httpclient-4.5.13.jar:oap-libs/httpasyncclient-4.1.3.jar:oap-libs/HikariCP-3.1.0.jar:oap-libs/HdrHistogram-2.1.12.jar:oap-libs/h2-2.1.212.jar:oap-libs/guice-4.1.0.jar:oap-libs/guava-32.0.1-jre.jar:oap-libs/gson-2.9.0.jar:oap-libs/grpc-stub-1.53.0.jar:oap-libs/grpc-protobuf-lite-1.53.0.jar:oap-libs/grpc-protobuf-1.53.0.jar:oap-libs/grpc-netty-1.53.0.jar:oap-libs/grpc-grpclb-1.53.0.jar:oap-libs/grpc-core-1.53.0.jar:oap-libs/grpc-context-1.53.0.jar:oap-libs/grpc-configuration-sync-9.7.0.jar:oap-libs/grpc-api-1.53.0.jar:oap-libs/groovy-4.0.15.jar:oap-libs/graphql-java-tools-13.0.1.jar:oap-libs/graphql-java-extended-scalars-18.1.jar:oap-libs/graphql-java-21.0.jar:oap-libs/freemarker-2.3.31.jar:oap-libs/flatbuffers-java-1.12.0.jar:oap-libs/failureaccess-1.0.1.jar:oap-libs/failsafe-2.3.4.jar:oap-libs/exporter-9.7.0.jar:oap-libs/event-analyzer-9.7.0.jar:oap-libs/error_prone_annotations-2.11.0.jar:oap-libs/envoy-metrics-receiver-plugin-9.7.0.jar:oap-libs/debugging-query-plugin-9.7.0.jar:oap-libs/data-generator-9.7.0.jar:oap-libs/curator-x-discovery-4.3.0.jar:oap-libs/curator-recipes-4.3.0.jar:oap-libs/curator-framework-4.3.0.jar:oap-libs/curator-client-4.3.0.jar:oap-libs/converter-jackson-2.9.0.jar:oap-libs/consul-client-1.5.3.jar:oap-libs/configuration-zookeeper-9.7.0.jar:oap-libs/configuration-nacos-9.7.0.jar:oap-libs/configuration-k8s-configmap-9.7.0.jar:oap-libs/configuration-etcd-9.7.0.jar:oap-libs/configuration-discovery-receiver-plugin-9.7.0.jar:oap-libs/configuration-consul-9.7.0.jar:oap-libs/configuration-apollo-9.7.0.jar:oap-libs/configuration-api-9.7.0.jar:oap-libs/commons-validator-1.7.jar:oap-libs/commons-text-1.4.jar:oap-libs/commons-net-3.9.0.jar:oap-libs/commons-logging-1.2.jar:oap-libs/commons-lang3-3.12.0.jar:oap-libs/commons-io-2.7.jar:oap-libs/commons-codec-1.11.jar:oap-libs/commons-beanutils-1.9.4.jar:oap-libs/cluster-zookeeper-plugin-9.7.0.jar:oap-libs/cluster-standalone-plugin-9.7.0.jar:oap-libs/cluster-nacos-plugin-9.7.0.jar:oap-libs/cluster-kubernetes-plugin-9.7.0.jar:oap-libs/cluster-etcd-plugin-9.7.0.jar:oap-libs/cluster-consul-plugin-9.7.0.jar:oap-libs/classmate-1.5.1.jar:oap-libs/checker-qual-3.33.0.jar:oap-libs/brotli4j-1.12.0.jar:oap-libs/banyandb-java-client-0.5.0.jar:oap-libs/aws-firehose-receiver-9.7.0.jar:oap-libs/audience-annotations-0.5.0.jar:oap-libs/armeria-protobuf-1.25.0.jar:oap-libs/armeria-graphql-protocol-1.25.0.jar:oap-libs/armeria-graphql-1.25.0.jar:oap-libs/armeria-1.25.0.jar:oap-libs/apollo-core-1.8.0.jar:oap-libs/apollo-client-1.8.0.jar:oap-libs/apm-network-9.7.0.jar:oap-libs/aopalliance-1.0.jar:oap-libs/antlr4-runtime-4.11.1.jar:oap-libs/annotations-17.0.0.jar:oap-libs/animal-sniffer-annotations-1.21.jar:oap-libs/ai-pipeline-9.7.0.jar:oap-libs/agent-analyzer-9.7.0.jar:config:' org.apache.skywalking.oap.server.starter.OAPServerStartUp
[0.007s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.

There is insufficient memory for the Java Runtime Environment to continue.
Cannot create worker GC thread. Out of system resources.
Can not save log file, dump to screen..

There is insufficient memory for the Java Runtime Environment to continue.
Cannot create worker GC thread. Out of system resources.
Possible reasons:
The system is out of physical RAM or swap space
The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
JVM is running with Unscaled Compressed Oops mode in which the Java heap is
placed in the first 4GB address space. The Java Heap base address is the
maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
to set the Java Heap base and to place the Java Heap above 4GB virtual address.
This output file may be truncated or incomplete.

Out of Memory Error (workerManager.hpp:70), pid=1, tid=1

JRE version: (11.0.21+9) (build )
Java VM: OpenJDK 64-Bit Server VM (11.0.21+9, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
Core dump will be written. Default location: /skywalking/core.1

--------------- S U M M A R Y ------------

Command Line: -Xms2G org.apache.skywalking.oap.server.starter.OAPServerStartUp

Host: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz, 8 cores, 31G, Ubuntu 22.04.3 LTS
Time: Tue Apr 23 08:56:20 2024 UTC elapsed time: 0.008423 seconds (0d 0h 0m 0s)

--------------- T H R E A D ---------------

Current thread (0x00007f10df94d800): JavaThread "Unknown thread" [_thread_in_vm, id=1, stack(0x00007ffc9fea1000,0x00007ffca069f000)]

Stack: [0x00007ffc9fea1000,0x00007ffca069f000], sp=0x00007ffca0699430, free space=8161k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xf4e28a] VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x1ca
V [libjvm.so+0xf4f14b] VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x2b
V [libjvm.so+0x6d5f15] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xd5
V [libjvm.so+0xf7b87b] AbstractWorkGang::add_workers(unsigned int, bool)+0x14b
V [libjvm.so+0x7b40a5] G1CollectedHeap::G1CollectedHeap(G1CollectorPolicy*)+0x5b5
V [libjvm.so+0x7a5cb9] G1Arguments::create_heap()+0x69
V [libjvm.so+0xf098e1] Universe::initialize_heap()+0x31
V [libjvm.so+0xf09b96] universe_init()+0x46
V [libjvm.so+0x8ad5d7] init_globals()+0x57
V [libjvm.so+0xee0f27] Threads::create_vm(JavaVMInitArgs*, bool*)+0x357
V [libjvm.so+0x96b8a2] JNI_CreateJavaVM+0x52
C [libjli.so+0xa3af] JavaMain+0x8f
C [libjli.so+0xf794] CallJavaMainInNewThread+0x74
C [libjli.so+0xbd7d] ContinueInNewThread+0x4d
C [libjli.so+0xd400] JLI_Launch+0x1470
C [java+0xa63]

--------------- P R O C E S S ---------------

Threads class SMR info:
_java_thread_list=0x00007f10df90ff10, length=0, elements={
}

Java Threads: ( => current thread )

Other Threads:

=>0x00007f10df94d800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=1, stack(0x00007ffc9fea1000,0x00007ffca069f000)]

Threads with active compile tasks:

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

[error occurred during error reporting (printing compressed oops mode), id 0xb, SIGSEGV (0xb) at pc=0x00007f10ddf08213]

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (2 events):
Event: 0.004 Protecting memory [0x00007ffc9fea1000,0x00007ffc9fea5000] with protection modes 0
Event: 0.005 Loaded shared library /opt/java/openjdk/lib/libzip.so

Dynamic libraries:
7f10c7247000-7f10c74b8000 rw-p 00000000 00:00 0
7f10c74b8000-7f10c7728000 rwxp 00000000 00:00 0
7f10c7728000-7f10c7a49000 ---p 00000000 00:00 0
7f10c7a49000-7f10c7cb9000 rwxp 00000000 00:00 0
7f10c7cb9000-7f10cef80000 ---p 00000000 00:00 0
7f10cef80000-7f10cf1f0000 rwxp 00000000 00:00 0
7f10cf1f0000-7f10d64b8000 ---p 00000000 00:00 0
7f10d64b8000-7f10dc000000 r--s 00000000 08:11 18621844 /opt/java/openjdk/lib/modules
7f10dc000000-7f10dc01b000 r-xp 00000000 08:11 18621843 /opt/java/openjdk/lib/libzip.so
7f10dc01b000-7f10dc21a000 ---p 0001b000 08:11 18621843 /opt/java/openjdk/lib/libzip.so
7f10dc21a000-7f10dc21b000 r--p 0001a000 08:11 18621843 /opt/java/openjdk/lib/libzip.so
7f10dc21b000-7f10dc21c000 rw-p 0001b000 08:11 18621843 /opt/java/openjdk/lib/libzip.so
7f10dc400000-7f10dc41b000 r-xp 00000000 08:11 18621826 /opt/java/openjdk/lib/libjimage.so
7f10dc41b000-7f10dc61a000 ---p 0001b000 08:11 18621826 /opt/java/openjdk/lib/libjimage.so
7f10dc61a000-7f10dc61c000 r--p 0001a000 08:11 18621826 /opt/java/openjdk/lib/libjimage.so
7f10dc61c000-7f10dc61d000 rw-p 0001c000 08:11 18621826 /opt/java/openjdk/lib/libjimage.so
7f10dc6b1000-7f10dc800000 rw-p 00000000 00:00 0
7f10dc800000-7f10dc829000 r-xp 00000000 08:11 18621822 /opt/java/openjdk/lib/libjava.so
7f10dc829000-7f10dca28000 ---p 00029000 08:11 18621822 /opt/java/openjdk/lib/libjava.so
7f10dca28000-7f10dca29000 r--p 00028000 08:11 18621822 /opt/java/openjdk/lib/libjava.so
7f10dca29000-7f10dca2a000 rw-p 00029000 08:11 18621822 /opt/java/openjdk/lib/libjava.so
7f10dca2a000-7f10dca2b000 rw-p 00000000 00:00 0
7f10dcaff000-7f10dcb00000 ---p 00000000 00:00 0
7f10dcb00000-7f10dcc00000 rw-p 00000000 00:00 0
7f10dcc00000-7f10dcc0d000 r-xp 00000000 08:11 18621842 /opt/java/openjdk/lib/libverify.so
7f10dcc0d000-7f10dce0c000 ---p 0000d000 08:11 18621842 /opt/java/openjdk/lib/libverify.so
7f10dce0c000-7f10dce0e000 r--p 0000c000 08:11 18621842 /opt/java/openjdk/lib/libverify.so
7f10dce0e000-7f10dce0f000 rw-p 0000e000 08:11 18621842 /opt/java/openjdk/lib/libverify.so
7f10dce2a000-7f10dce2f000 rw-p 00000000 00:00 0
7f10dce2f000-7f10dcf15000 ---p 00000000 00:00 0
7f10dcf15000-7f10dcf1a000 rw-p 00000000 00:00 0
7f10dcf1a000-7f10dd000000 ---p 00000000 00:00 0
7f10dd000000-7f10de280000 r-xp 00000000 08:11 18621856 /opt/java/openjdk/lib/server/libjvm.so
7f10de280000-7f10de480000 ---p 01280000 08:11 18621856 /opt/java/openjdk/lib/server/libjvm.so
7f10de480000-7f10de545000 r--p 01280000 08:11 18621856 /opt/java/openjdk/lib/server/libjvm.so
7f10de545000-7f10de580000 rw-p 01345000 08:11 18621856 /opt/java/openjdk/lib/server/libjvm.so
7f10de580000-7f10de5d8000 rw-p 00000000 00:00 0
7f10de5f1000-7f10de6f1000 rw-p 00000000 00:00 0
7f10de6f1000-7f10de6ff000 r--p 00000000 08:11 18613823 /usr/lib/x86_64-linux-gnu/libm.so.6
7f10de6ff000-7f10de77b000 r-xp 0000e000 08:11 18613823 /usr/lib/x86_64-linux-gnu/libm.so.6
7f10de77b000-7f10de7d6000 r--p 0008a000 08:11 18613823 /usr/lib/x86_64-linux-gnu/libm.so.6
7f10de7d6000-7f10de7d7000 r--p 000e4000 08:11 18613823 /usr/lib/x86_64-linux-gnu/libm.so.6
7f10de7d7000-7f10de7d8000 rw-p 000e5000 08:11 18613823 /usr/lib/x86_64-linux-gnu/libm.so.6
7f10de7d8000-7f10de800000 r--p 00000000 08:11 18613771 /usr/lib/x86_64-linux-gnu/libc.so.6
7f10de800000-7f10de995000 r-xp 00028000 08:11 18613771 /usr/lib/x86_64-linux-gnu/libc.so.6
7f10de995000-7f10de9ed000 r--p 001bd000 08:11 18613771 /usr/lib/x86_64-linux-gnu/libc.so.6
7f10de9ed000-7f10de9f1000 r--p 00214000 08:11 18613771 /usr/lib/x86_64-linux-gnu/libc.so.6
7f10de9f1000-7f10de9f3000 rw-p 00218000 08:11 18613771 /usr/lib/x86_64-linux-gnu/libc.so.6
7f10de9f3000-7f10dea00000 rw-p 00000000 00:00 0
7f10dea00000-7f10dea1a000 r-xp 00000000 08:11 18621807 /opt/java/openjdk/lib/jli/libjli.so
7f10dea1a000-7f10dec1a000 ---p 0001a000 08:11 18621807 /opt/java/openjdk/lib/jli/libjli.so
7f10dec1a000-7f10dec1b000 r--p 0001a000 08:11 18621807 /opt/java/openjdk/lib/jli/libjli.so
7f10dec1b000-7f10dec1c000 rw-p 0001b000 08:11 18621807 /opt/java/openjdk/lib/jli/libjli.so
7f10dec72000-7f10dec77000 rw-p 00000000 00:00 0
7f10dec77000-7f10dec7e000 ---p 00000000 00:00 0
7f10dec7e000-7f10dec86000 rw-s 00000000 08:11 18743325 /tmp/hsperfdata_root/1
7f10dec86000-7f10dec87000 r--p 00000000 08:11 18613868 /usr/lib/x86_64-linux-gnu/librt.so.1
7f10dec87000-7f10dec88000 r-xp 00001000 08:11 18613868 /usr/lib/x86_64-linux-gnu/librt.so.1
7f10dec88000-7f10dec89000 r--p 00002000 08:11 18613868 /usr/lib/x86_64-linux-gnu/librt.so.1
7f10dec89000-7f10dec8a000 r--p 00002000 08:11 18613868 /usr/lib/x86_64-linux-gnu/librt.so.1
7f10dec8a000-7f10dec8b000 rw-p 00003000 08:11 18613868 /usr/lib/x86_64-linux-gnu/librt.so.1
7f10dec8b000-7f10dec8d000 rw-p 00000000 00:00 0
7f10dec8d000-7f10dec8e000 r--p 00000000 08:11 18613785 /usr/lib/x86_64-linux-gnu/libdl.so.2
7f10dec8e000-7f10dec8f000 r-xp 00001000 08:11 18613785 /usr/lib/x86_64-linux-gnu/libdl.so.2
7f10dec8f000-7f10dec90000 r--p 00002000 08:11 18613785 /usr/lib/x86_64-linux-gnu/libdl.so.2
7f10dec90000-7f10dec91000 r--p 00002000 08:11 18613785 /usr/lib/x86_64-linux-gnu/libdl.so.2
7f10dec91000-7f10dec92000 rw-p 00003000 08:11 18613785 /usr/lib/x86_64-linux-gnu/libdl.so.2
7f10dec92000-7f10dec93000 r--p 00000000 08:11 18613866 /usr/lib/x86_64-linux-gnu/libpthread.so.0
7f10dec93000-7f10dec94000 r-xp 00001000 08:11 18613866 /usr/lib/x86_64-linux-gnu/libpthread.so.0
7f10dec94000-7f10dec95000 r--p 00002000 08:11 18613866 /usr/lib/x86_64-linux-gnu/libpthread.so.0
7f10dec95000-7f10dec96000 r--p 00002000 08:11 18613866 /usr/lib/x86_64-linux-gnu/libpthread.so.0
7f10dec96000-7f10dec97000 rw-p 00003000 08:11 18613866 /usr/lib/x86_64-linux-gnu/libpthread.so.0
7f10dec98000-7f10dec99000 ---p 00000000 00:00 0
7f10dec99000-7f10dec9a000 r--p 00000000 00:00 0
7f10dec9a000-7f10dec9c000 rw-p 00000000 00:00 0
7f10dec9c000-7f10dec9e000 r--p 00000000 08:11 18613753 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f10dec9e000-7f10decc8000 r-xp 00002000 08:11 18613753 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f10decc8000-7f10decd3000 r--p 0002c000 08:11 18613753 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f10decd4000-7f10decd6000 r--p 00037000 08:11 18613753 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f10decd6000-7f10decd8000 rw-p 00039000 08:11 18613753 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f10decd8000-7f10decd9000 r-xp 00000000 08:11 18620567 /opt/java/openjdk/bin/java
7f10deed9000-7f10deeda000 r--p 00001000 08:11 18620567 /opt/java/openjdk/bin/java
7f10deeda000-7f10deedb000 rw-p 00002000 08:11 18620567 /opt/java/openjdk/bin/java
7f10df909000-7f10df98d000 rw-p 00000000 00:00 0 [heap]
7ffc9fea1000-7ffc9fea5000 ---p 00000000 00:00 0
7ffca067e000-7ffca06a1000 rw-p 00000000 00:00 0 [stack]
7ffca06b0000-7ffca06b2000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

VM Arguments:
jvm_args: -Xms2G
java_command: org.apache.skywalking.oap.server.starter.OAPServerStartUp
java_class_path (initial): /skywalking/ext-libs/*.jar:oap-libs/zstd-jni-1.4.9-1.jar:oap-libs/zookeeper-jute-3.5.7.jar:oap-libs/zookeeper-3.5.7.jar:oap-libs/zjsonpatch-0.3.0.jar:oap-libs/zipkin-receiver-plugin-9.7.0.jar:oap-libs/zipkin-query-plugin-9.7.0.jar:oap-libs/zipkin-2.24.1.jar:oap-libs/vavr-match-0.10.3.jar:oap-libs/vavr-0.10.3.jar:oap-libs/tool-profile-snapshot-server-mock-9.7.0.jar:oap-libs/tool-profile-snapshot-bootstrap-9.7.0.jar:oap-libs/telemetry-prometheus-9.7.0.jar:oap-libs/telemetry-api-9.7.0.jar:oap-libs/storage-jdbc-hikaricp-plugin-9.7.0.jar:oap-libs/storage-elasticsearch-plugin-9.7.0.jar:oap-libs/storage-banyandb-plugin-9.7.0.jar:oap-libs/snappy-java-1.1.8.1.jar:oap-libs/snakeyaml-engine-2.6.jar:oap-libs/snakeyaml-2.0.jar:oap-libs/slf4j-api-1.7.30.jar:oap-libs/skywalking-zabbix-receiver-plugin-9.7.0.jar:oap-libs/skywalking-trace-receiver-plugin-9.7.0.jar:oap-libs/skywalking-telegraf-receiver-plugin-9.7.0.jar:oap-libs/skywalking-sharing-server-plugin-9.7.0.jar:oap-libs/skywalking-profile-receiver-plugin-9.7.0.jar:oap-libs/skywalking-meter-receiver-plugin-9.7.0.jar:oap-libs/skywalking-mesh-receiver-plugin-9.7.0.jar:oap-libs/skywalking-management-receiver-plugin-9.7.0.jar:oap-libs/skywalking-log-receiver-plugin-9.7.0.jar:oap-libs/skywalking-jvm-receiver-plugin-9.7.0.jar:oap-libs/skywalking-event-receiver-plugin-9.7.0.jar:oap-libs/skywalking-ebpf-receiver-plugin-9.7.0.jar:oap-libs/skywalking-clr-receiver-plugin-9.7.0.jar:oap-libs/skywalking-browser-receiver-plugin-9.7.0.jar:oap-libs/simpleclient_hotspot-0.6.0.jar:oap-libs/simpleclient_common-0.6.0.jar:oap-libs/simpleclient-0.6.0.jar:oap-libs/service-1.12.0.jar:oap-libs/server-testing-9.7.0.jar:oap-libs/server-starter-9.7.0.jar:oap-libs/server-health-checker-9.7.0.jar:oap-libs/server-core-9.7.0.jar:oap-libs/server-alarm-plugin-9.7.0.jar:oap-libs/retrofit-2.9.0.jar:oap-libs/receiver-proto-9.7.0.jar:oap-libs/reactive-streams-1.0.3.jar:oap-libs/re2j-1.5.jar:oap-libs/query-graphql-plugin-9.7.0.jar:oap-li
Launcher Type: SUN_STANDARD

[Global flags]
intx CICompilerCount = 4 {product} {ergonomic}
uint G1ConcRefinementThreads = 8 {product} {ergonomic}
size_t G1HeapRegionSize = 2097152 {product} {ergonomic}
uintx GCDrainStackTargetSize = 64 {product} {ergonomic}
size_t InitialHeapSize = 2147483648 {product} {command line}
size_t MaxHeapSize = 8388608000 {product} {ergonomic}
size_t MinHeapDeltaBytes = 2097152 {product} {ergonomic}
uintx NonNMethodCodeHeapSize = 5836300 {pd product} {ergonomic}
uintx NonProfiledCodeHeapSize = 122910970 {pd product} {ergonomic}
uintx ProfiledCodeHeapSize = 122910970 {pd product} {ergonomic}
uintx ReservedCodeCacheSize = 251658240 {pd product} {ergonomic}
bool SegmentedCodeCache = true {product} {ergonomic}
bool UseCompressedClassPointers = true {lp64_product} {ergonomic}
bool UseCompressedOops = true {lp64_product} {ergonomic}
bool UseG1GC = true {product} {ergonomic}

Logging:
Log output configuration:
#0: stdout all=warning uptime,level,tags
#1: stderr all=off uptime,level,tags

Environment Variables:
JAVA_HOME=/opt/java/openjdk
PATH=/skywalking/bin:/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8

Signal Handlers:
SIGSEGV: [libjvm.so+0xf4fea0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xf4fea0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0xf4fea0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0xc70220], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0xc70220], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0xf4fea0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR2: [libjvm.so+0xc700c0], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none

--------------- S Y S T E M ---------------

OS:DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
uname:Linux 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64
OS uptime: 65 days 3:00 hours
libc:glibc 2.35 NPTL 2.35
rlimit (soft/hard): STACK 8192k/infinity , CORE infinity/infinity , NPROC infinity/infinity , NOFILE 1048576/1048576 , AS infinity/infinity , CPU infinity/infinity , DATA infinity/infinity , FSIZE infinity/infinity , MEMLOCK 64k/64k
load average:1.46 1.39 1.52

/proc/meminfo:
MemTotal: 32765892 kB
MemFree: 3801996 kB
MemAvailable: 18015876 kB
Buffers: 452564 kB
Cached: 13078364 kB
SwapCached: 198780 kB
Active: 17234460 kB
Inactive: 9195512 kB
Active(anon): 11655364 kB
Inactive(anon): 2048192 kB
Active(file): 5579096 kB
Inactive(file): 7147320 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 16777212 kB
SwapFree: 13892996 kB
Dirty: 704 kB
Writeback: 0 kB
AnonPages: 12803980 kB
Mapped: 335080 kB
Shmem: 804512 kB
Slab: 2082724 kB
SReclaimable: 1890296 kB
SUnreclaim: 192428 kB
KernelStack: 36848 kB
PageTables: 56996 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 33160156 kB
Committed_AS: 27957880 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 234200 kB
VmallocChunk: 34359310332 kB
HardwareCorrupted: 0 kB
AnonHugePages: 10057728 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 128832 kB
DirectMap2M: 5113856 kB
DirectMap1G: 30408704 kB

/sys/kernel/mm/transparent_hugepage/enabled:
[always] madvise never
/sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter):
[always] madvise never

Process Memory:
Virtual Size: 384424K (peak: 384424K)
Resident Set Size: 11896K (peak: 11896K) (anon: 5564K, file: 6332K, shmem: 0K)
Swapped out: 0K
C-Heap outstanding allocations: 4266K, retained: 101K
glibc malloc tunables: (default)

/proc/sys/kernel/threads-max (system-wide limit on the number of threads):
255811
/proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):
655360
/proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):
131072

container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-7
cpu_memory_nodes: 0
active_processor_count: 8
cpu_quota: no quota
cpu_period: 100000
cpu_shares: no shares
memory_limit_in_bytes: unlimited
memory_and_swap_limit_in_bytes: unlimited
memory_soft_limit_in_bytes: unlimited
memory_usage_in_bytes: 5844 k
memory_max_usage_in_bytes: 8696 k
kernel_memory_usage_in_bytes: unlimited
kernel_memory_max_usage_in_bytes: unlimited
kernel_memory_limit_in_bytes: unlimited
maximum number of tasks: unlimited
current number of tasks: 1

VMWare virtualization detected
Steal ticks since vm start: 0
Steal ticks percentage since vm start: 0.000

CPU:total 8 (initial active 8) (4 cores per cpu, 1 threads per core) family 6 model 63 stepping 2 microcode 0x43, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, tscinvbit, bmi1, bmi2, fma
CPU Model and flags from /proc/cpuinfo:
model name : Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm arat fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt
Online cpus:
0-7
Offline cpus:
8-127
BIOS frequency limitation:

Frequency switch latency (ns):

Available cpu frequencies:

Current governor:

Core performance/turbo boost:

Memory: 4k page, physical 32765892k(3801996k free), swap 16777212k(13892996k free)

vm_info: OpenJDK 64-Bit Server VM (11.0.21+9) for linux-amd64 JRE (11.0.21+9), built on Oct 17 2023 21:44:42 by "" with gcc 7.5.0

END.
[Entrypoint] Apache SkyWalking Docker Image

wu-sheng Apr 23, 2024
Collaborator

We don't have MySQL driver in the image, how your command could work property?

ibingdian · 2024-04-24T06:56:43Z

ibingdian
Apr 24, 2024
Author

Thank you for answering my question.

I increased the memory, but it didn't work.
i update docker, it work, and there's no Out of Memory Error left , but there is anther problem when start skywalking-ui : UnknownHostException: Failed to resolve 'oap' [A(1)] after 2 queries.

the skywalking-oap-server start log: skywalking-oap-server-log.txt

the skywalking-ui start log: skywalking-ui-log.txt

this my docker command :
run oap:
docker run --name oap --restart always -d -p 12800:12800 -p 11800:11800 -v /home/dev/LY_TEST/tools/sw_oap/oap-libs:/skywalking/oap-libs -e SW_STORAGE=mysql -e SW_JDBC_URL=jdbc:mysql://192.168.8.82:13306/sw_new_db?rewriteBatchedStatements=true -e SW_DATA_SOURCE_USER=root -e SW_DATA_SOURCE_PASSWORD=123456 apache/skywalking-oap-server:9.7.0
run ui:
docker run --name oap-ui --restart always -d -p 8080:8080 -e SW_OAP_ADDRESS=http://oap:12800 apache/skywalking-ui:9.7.0

2 replies

wu-sheng Apr 24, 2024
Collaborator

Go to use quick start https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-docker/#start-the-storage-oap-and-booster-ui-with-docker-compose

If you are using something, make sure you know what you are doing actually. You should understand architecture, network connectivity, data flow before running a command and expect it works as always.

The scripts are working on ours and many other people's end, but your env is unknown to us.

ibingdian Apr 25, 2024
Author

thank you，it is my fault

[Bug] docker start skywalking-oap-server:9.7.0 fail : Out of Memory Error #12151

Uh oh!

Uh oh!

ibingdian Apr 23, 2024

Search before asking

Apache SkyWalking Component

What happened

the error as follow :

What you expected to happen

How to reproduce

Anything else

Are you willing to submit a pull request to fix on your own?

Code of Conduct

Replies: 2 comments · 10 replies

Uh oh!

wu-sheng Apr 23, 2024 Collaborator

Uh oh!

ibingdian Apr 23, 2024 Author

Uh oh!

wu-sheng Apr 23, 2024 Collaborator

Uh oh!

wu-sheng Apr 23, 2024 Collaborator

Uh oh!

ibingdian Apr 23, 2024 Author

Uh oh!

ibingdian Apr 23, 2024 Author

the error as follow :

Uh oh!

wu-sheng Apr 23, 2024 Collaborator

Uh oh!

Uh oh!

ibingdian Apr 24, 2024 Author

Uh oh!

wu-sheng Apr 24, 2024 Collaborator

Uh oh!

ibingdian Apr 25, 2024 Author

ibingdian
Apr 23, 2024

Replies: 2 comments 10 replies

wu-sheng
Apr 23, 2024
Collaborator

ibingdian Apr 23, 2024
Author

wu-sheng Apr 23, 2024
Collaborator

wu-sheng Apr 23, 2024
Collaborator

ibingdian Apr 23, 2024
Author

ibingdian Apr 23, 2024
Author

wu-sheng Apr 23, 2024
Collaborator

ibingdian
Apr 24, 2024
Author

wu-sheng Apr 24, 2024
Collaborator

ibingdian Apr 25, 2024
Author