I am running a test Vitess Cluster in EKS and have run into an issue when adding static authentication to the vtgate moving from --mysql_auth_server_impl=none to --mysql_auth_server_impl=static (+ the file and the reload interval), namely 1 of the 2 vttablets begins bootlooping.
This is the vtgate config https://gist.github.com/dan-frincu-udemy/2ab3797a01ce82438602789089da7ca4
Working vttablet config https://gist.github.com/dan-frincu-udemy/0767a9636030db953bb0e5685540f825
Failing vttablet config https://gist.github.com/dan-frincu-udemy/1a1736217ddcca1157e96063be89f1d4
Worth noting that
a) the config between vttablets is identical other than being deployed to different zones
b) if I remove the second vttablet and leave just the first, the same failure behaviour is now observed on the first, and when I add the 2nd tablet back, it moves the failure to the 2nd tablet.
The tablet that is working has the following in the describe output
The following is output from kubectl describe for
Because of the bootlooping of the tablet, we get 2 different kinds of logs, the 2nd one is potentially more relevant
The mysql log https://gist.github.com/dan-frincu-udemy/629c75973f9bebe7036b1eb31b2ef5f6
Other observations:
- there are logs mentioning that /vt/socket/mysql.sock is missing on the vttablet that has errors, that is correct, only
mysqlctl.sock is present in /vt/socket directory, but at the same time the mysqld server is running, makes a socket on 3306 which is reachable from outside the pod and the process on the pod is seen running
1 /vt/bin/mysqlctld --db_charset=utf8mb4 --db_dba_user=vt_dba --init_db_sql_file=/vt/secrets/db-init-script/init_db.sql --logtostderr=true --mysql_socket=/vt/socket/mysql.sock --socket_file=/vt/socket/mysqlctl.sock --tablet_uid=2169852485 --wait_time=2h0m0s
17 /usr/bin/sh /usr/bin/mysqld_safe --defaults-file=/vt/vtdataroot/vt_2169852485/my.cnf --basedir=/usr
539 /usr/sbin/mysqld --defaults-file=/vt/vtdataroot/vt_2169852485/my.cnf --basedir=/usr --datadir=/vt/vtdataroot/vt_2169852485/data --plugin-dir=/usr/lib64/mysql/plugin --log-error=/vt/config/stderr.symlink --pid-file=/vt/vtdataroot/vt_2169852485/mysql.pid --socket=/vt/socket/mysql.sock --port=3306
- the vitess user is not present on the mysqld container, so it shows UID 999 for the files in
/vt/socket (this may not be a problem per se, the UID is correct)
I am running a test Vitess Cluster in EKS and have run into an issue when adding static authentication to the vtgate moving from
--mysql_auth_server_impl=noneto--mysql_auth_server_impl=static(+ the file and the reload interval), namely 1 of the 2 vttablets begins bootlooping.This is the vtgate config https://gist.github.com/dan-frincu-udemy/2ab3797a01ce82438602789089da7ca4
Working vttablet config https://gist.github.com/dan-frincu-udemy/0767a9636030db953bb0e5685540f825
Failing vttablet config https://gist.github.com/dan-frincu-udemy/1a1736217ddcca1157e96063be89f1d4
Worth noting that
a) the config between vttablets is identical other than being deployed to different zones
b) if I remove the second vttablet and leave just the first, the same failure behaviour is now observed on the first, and when I add the 2nd tablet back, it moves the failure to the 2nd tablet.
The tablet that is working has the following in the describe output
The following is output from
kubectl describeforBecause of the bootlooping of the tablet, we get 2 different kinds of logs, the 2nd one is potentially more relevant
The mysql log https://gist.github.com/dan-frincu-udemy/629c75973f9bebe7036b1eb31b2ef5f6
Other observations:
mysqlctl.sockis present in/vt/socketdirectory, but at the same time the mysqld server is running, makes a socket on 3306 which is reachable from outside the pod and the process on the pod is seen running/vt/socket(this may not be a problem per se, the UID is correct)