status=11/SEGV

[sanjesss]

Добрый день! Столкнулись с проблемой - ошибка сегментации памяти. Видел прошлые ишью, не нашёл решения для себя. Получаем ошибку на втором коннекте через SSL. Сертификаты верные, аналогичные настройки на другой машине. Самое странное, что на другой машине всё работает, конфиги аналогичные, виртуализатор такой же (nutanix), SSL там тоже используется.
Очень хотим использовать пулер, но эта проблема блочит его использование. Прикладываю конфиг и ошибку.

● odyssey-baas-pl.service - Advanced multi-threaded PostgreSQL connection pooler and request router
   Loaded: loaded (/etc/systemd/system/odyssey-baas-pl.service; enabled; vendor preset: disabled)
   Active: failed (Result: signal) since Mon 2023-05-29 13:43:25 UTC; 181ms ago
  Process: 233532 ExecStart=/opt/odyssey/bin/odyssey /opt/odyssey/odyssey.conf (code=killed, signal=SEGV)
 Main PID: 233532 (code=killed, signal=SEGV)

May 29 13:41:36 baas-pl-stolon1 odyssey-baas-pl[233532]: 233532 2023-05-29T13:41:36Z info [none none] (rules)   tls_opts->tls_cert_file                     /opt/odyssey/secure/stolon-pl.crt
May 29 13:41:36 baas-pl-stolon1 odyssey-baas-pl[233532]: 233532 2023-05-29T13:41:36Z info [none none] (rules)   log_debug                         no
May 29 13:41:36 baas-pl-stolon1 odyssey-baas-pl[233532]: 233532 2023-05-29T13:41:36Z info [none none] (rules)   log_query                         no
May 29 13:41:36 baas-pl-stolon1 odyssey-baas-pl[233532]: 233532 2023-05-29T13:41:36Z info [none none] (rules)   options:                         todo
May 29 13:41:36 baas-pl-stolon1 odyssey-baas-pl[233532]: 233532 2023-05-29T13:41:36Z info [none none] (rules)
May 29 13:41:36 baas-pl-stolon1 odyssey-baas-pl[233532]: 233532 2023-05-29T13:41:36Z info [none none] (server) listening on 0.0.0.0:54332
May 29 13:41:36 baas-pl-stolon1 odyssey-baas-pl[233532]: 233532 2023-05-29T13:41:36Z info [none none] (server) listening on [::]:54332
May 29 13:43:25 baas-pl-stolon1 systemd[1]: odyssey-baas-pl.service: main process exited, code=killed, status=11/SEGV
May 29 13:43:25 baas-pl-stolon1 systemd[1]: Unit odyssey-baas-pl.service entered failed state.
May 29 13:43:25 baas-pl-stolon1 systemd[1]: odyssey-baas-pl.service failed.

Конфиг:

log_format "%p %t %l [%i %s] (%c) %m\n"
log_file "/var/log/odyssey/odyssey.log"
log_to_stdout yes
log_syslog yes
log_syslog_ident "odyssey-baas-pl"
log_syslog_facility "daemon"
log_debug no
log_config yes
log_session no
log_query no
log_stats no
stats_interval 60
workers 2
resolvers 1
readahead 8192
cache_coroutine 5300
coroutine_stack_size 4
nodelay yes
keepalive 7200
listen {
host "*"
port 54332
backlog 128
tls "allow"
tls_ca_file "/opt/odyssey/secure/CA_root-pl.crt"
tls_key_file "/opt/odyssey/secure/stolon-pl-private.key"
tls_cert_file "/opt/odyssey/secure/stolon-pl.crt"
}
storage "postgres_server" {
type "remote"
host "baas-pl-stolon1"
port 6432
tls "allow"
tls_ca_file "/opt/odyssey/secure/CA_root-pl.crt"
tls_key_file "/opt/odyssey/secure/stolon-pl-private.key"
tls_cert_file "/opt/odyssey/secure/stolon-pl.crt"
server_max_routing 12
}
database "postgres" {
  user "postgres" {
  authentication "md5"
  password "password"
  pool_routing "internal"
  storage "postgres_server"
  pool "session"
  }
}
database default {
  user default {
  authentication "md5"
  auth_query "SELECT usename, passwd FROM pg_shadow WHERE usename=$1"
  auth_query_user "postgres"
  auth_query_db "postgres"
  storage "postgres_server"
  pool "transaction"
  pool_size 400
  }
}

Версия ядра:
Linux baas-pl-stolon1 5.4.17-2136.316.7.el7uek.x86_64

[x4m]

Hi! I think backtrace or coredump would help here.

Привет! Для диагностики нужен стек вызовов. Из логов не ясно что там навернулось...

[jayant07-yb]

@x4m I also got the same problem on small machines (2 cores, 4 GB ram, CentOS7). It's working fine on large machines.
Here's the backtrace that I got when I enabled the SSL connectivity between the server and Odyssey and the client and Odyssey.

#0  0x00005652886fbadb in tcmalloc::tcmalloc_internal::MmapAligned(unsigned long, unsigned long, tcmalloc::tcmalloc_internal::MemoryTag) ()
[Current thread is 1 (LWP 26525)]
(gdb) bt
#0  0x00005652886fbadb in tcmalloc::tcmalloc_internal::MmapAligned(unsigned long, unsigned long, tcmalloc::tcmalloc_internal::MemoryTag) ()
#1  0x00005652886fb566 in tcmalloc::tcmalloc_internal::SystemAlloc(unsigned long, unsigned long, tcmalloc::tcmalloc_internal::MemoryTag) ()
#2  0x00005652886bafdb in tcmalloc::tcmalloc_internal::AddressRange tcmalloc::tcmalloc_internal::HugePageAwareAllocator::AllocAndReport<(tcmalloc::tcmalloc_internal::MemoryTag)0>(unsigned long, unsigned long) ()
#3  0x00005652886de736 in tcmalloc::tcmalloc_internal::HugeAllocator::Get(tcmalloc::tcmalloc_internal::HugeLength) ()
#4  0x00005652886df560 in tcmalloc::tcmalloc_internal::HugeCache::Get(tcmalloc::tcmalloc_internal::HugeLength, bool*) ()
#5  0x00005652886f0a8f in tcmalloc::tcmalloc_internal::HugePageAwareAllocator::RefillFiller(tcmalloc::tcmalloc_internal::Length, unsigned long, bool*) ()
#6  0x00005652886e3b0f in tcmalloc::tcmalloc_internal::HugePageAwareAllocator::New(tcmalloc::tcmalloc_internal::Length, unsigned long) ()
#7  0x00005652886c73b2 in void* tcmalloc::tcmalloc_internal::SampleifyAllocation<tcmalloc::tcmalloc_internal::Static, tcmalloc::tcmalloc_internal::TCMallocPolicy<tcmalloc::tcmalloc_internal::MallocOomPolicy, tcmalloc::tcmalloc_internal::MallocAlignPolicy, tcmalloc::tcmalloc_internal::AllocationAccessHotPolicy, tcmalloc::tcmalloc_internal::InvokeHooksPolicy, tcmalloc::tcmalloc_internal::LocalNumaPartitionPolicy> >(tcmalloc::tcmalloc_internal::Static&, tcmalloc::tcmalloc_internal::TCMallocPolicy<tcmalloc::tcmalloc_internal::MallocOomPolicy, tcmalloc::tcmalloc_internal::MallocAlignPolicy, tcmalloc::tcmalloc_internal::AllocationAccessHotPolicy, tcmalloc::tcmalloc_internal::InvokeHooksPolicy, tcmalloc::tcmalloc_internal::LocalNumaPartitionPolicy>, unsigned long, unsigned long, unsigned long, void*, tcmalloc::tcmalloc_internal::Span*, unsigned long*) ()
#8  0x00005652886c6fbb in void* slow_alloc<tcmalloc::tcmalloc_internal::TCMallocPolicy<tcmalloc::tcmalloc_internal::MallocOomPolicy, tcmalloc::tcmalloc_internal::MallocAlignPolicy, tcmalloc::tcmalloc_internal::AllocationAccessHotPolicy, tcmalloc::tcmalloc_internal::InvokeHooksPolicy, tcmalloc::tcmalloc_internal::LocalNumaPartitionPolicy>, decltype(nullptr)>(tcmalloc::tcmalloc_internal::TCMallocPolicy<tcmalloc::tcmalloc_internal::MallocOomPolicy, tcmalloc::tcmalloc_internal::MallocAlignPolicy, tcmalloc::tcmalloc_internal::AllocationAccessHotPolicy, tcmalloc::tcmalloc_internal::InvokeHooksPolicy, tcmalloc::tcmalloc_internal::LocalNumaPartitionPolicy>, unsigned long, decltype(nullptr)) ()
#9  0x00005652886c3c66 in malloc ()
#10 0x00007fd44135edac in ssl3_setup_read_buffer () from /home/yugabyte/yb-software/yugabyte-2.19.3.0-b138-centos-x86_64/lib/yb-thirdparty/libssl.so.1.1
#11 0x00007fd44135ccd7 in ssl3_read_bytes () from /home/yugabyte/yb-software/yugabyte-2.19.3.0-b138-centos-x86_64/lib/yb-thirdparty/libssl.so.1.1
#12 0x00007fd44138d3f2 in state_machine () from /home/yugabyte/yb-software/yugabyte-2.19.3.0-b138-centos-x86_64/lib/yb-thirdparty/libssl.so.1.1
#13 0x00005652886c0b43 in mm_tls_handshake_cb ()
#14 0x00005652886bf23c in mm_epoll_step ()
#15 0x00005652886beb92 in machine_main ()
#16 0x00007fd441109694 in start_thread (arg=0x7fd440102700) at pthread_create.c:333
#17 0x00007fd44084641d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109