Stopping a MariaDB-Node: Watchdog crashes GLB

[megabert]

Environment

Debian Jessie
mariadb 10.1.10
galera-3 25.3.9
galera-load-balancer 1.0.1

Error description

I set up glb and it works basically. When I stop one of the cluster nodes, glb crashes:

Sequence of Commands

Starting glb

glbd -c 127.0.0.1:4444 -w exec:"/usr/local/bin/mysql.sh -unode_checker -p'secret'"  192.168.199.10:13306 192.168.199.11:3306:100 192.168.199.12:3306:1

glb shows that it is running correctly

Destinations: 2
   0:  192.168.199.11:3306 , w: 100.000
   1:  192.168.199.12:3306 , w: 1.000
Watchdog:
------------------------------------------------------------
        Address       : exp  setw     state    lat     curw
 192.168.199.11:3306  :  + 100.000    READY  0.01183 100.000
 192.168.199.12:3306  :  +   1.000    READY  0.01517   1.000
------------------------------------------------------------
Destinations: 2

Router:
------------------------------------------------------
        Address       :   weight   usage    map  conns
 192.168.199.12:3306  :    1.000   0.000    N/A      0
 192.168.199.11:3306  :  100.000   0.000    N/A      0
------------------------------------------------------
Destinations: 2, total connections: 0 of 32749 max

Stopping a DB-Node

systemctl stop mysql.service

glb crash

Pool: connections per thread:     0

Watchdog:
------------------------------------------------------------
        Address       : exp  setw     state    lat     curw
 192.168.199.11:3306  :  + 100.000    READY  0.00995 100.000
 192.168.199.12:3306  :  +   1.000    READY  0.00996   1.000
------------------------------------------------------------
Destinations: 2

Router:
------------------------------------------------------
        Address       :   weight   usage    map  conns
 192.168.199.12:3306  :    1.000   0.000    N/A      0
 192.168.199.11:3306  :  100.000   0.000    N/A      0
------------------------------------------------------
Destinations: 2, total connections: 0 of 32749 max

Pool: connections per thread:     0

Watchdog:
------------------------------------------------------------
        Address       : exp  setw     state    lat     curw
 192.168.199.11:3306  :  + 100.000    READY  0.00928 100.000
 192.168.199.12:3306  :  +   1.000    AVOID  0.01066   0.000
------------------------------------------------------------
Destinations: 2

Router:
------------------------------------------------------
        Address       :   weight   usage    map  conns
 192.168.199.12:3306  :    0.000    -nan    N/A      0
 192.168.199.11:3306  :  100.000   0.000    N/A      0
------------------------------------------------------
Destinations: 2, total connections: 0 of 32749 max

Pool: connections per thread:     0

ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 104 "Connection reset by peer"
*** Error in `glbd': realloc(): invalid pointer: 0x00007f6aec000920 ***
   INFO: glb_signal.c:42: Received signal 6. Terminating.
Abgebrochen

[megabert]

The crash is caused by the watchdog. When I do not use the watchdog, glb is not crashing.

[megabert]

Thanks for the effort. glb is still crashing.

Pool: connections per thread:     0

Watchdog:
------------------------------------------------------------
        Address       : exp  setw     state    lat     curw
 192.168.199.11:3306  :  + 100.000    READY  0.00951 100.000
 192.168.199.12:3306  :  +   1.000    AVOID  0.01047   0.000
------------------------------------------------------------
Destinations: 2

Router:
------------------------------------------------------
        Address       :   weight   usage    map  conns
 192.168.199.11:3306  :  100.000   0.000    N/A      0
 192.168.199.12:3306  :    0.000    -nan    N/A      0
------------------------------------------------------
Destinations: 2, total connections: 0 of 32749 max

Pool: connections per thread:     0

ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 104 "Connection reset by peer"
*** Error in `glbd': realloc(): invalid pointer: 0x00007f5c84000990 ***
   INFO: glb_signal.c:42: Received signal 6. Terminating.
Aborted

[ayurchen]

Ok, then my guess was wrong.
Could you supply a backtrace from the core?
(for that you'll need

• a debug build ($ ./configure --enable-debug)
• enable core dumps ($ ulimit -c unlimited)
• start glbd manually from the same shell and without -d option
• cause a crash
• open resulting core in gdb ($ gdb -c <path to core> <path to glbd>)
• or start glbd inside gdb:
  • $ gdb <path to glbd>
  • gdb> run <glbd options>
  • again cause a crash
• gdb> bt

)