frontend failed to add healthy backend in minikube k8s cluster

[huandzh]

Steps to reproduce the behavior (Required)

1. start a minikube cluster

2. deploy with demo.yaml provided in this issue report. It creates deployments and services and adds the be to the fe.

demo.zip

kubectl apply -f demo.yaml

stateDiagram
  state MiniKubeCluster {
     fe_deployment --> fe_service
     be_deployment --> be_service
     be_service --> fe_service
   }
 fe_service --> ExternalApp

Loading

3. try connecting to fe, or check error logs of fe and be

Expected behavior (Required)

except frontend to:

• Allow saved IP different than backend localhost as long as the backend is reachable and healthy
• or allow using hostnames to access backends
• or provide config to skip address validation, since there are many ips, such as Cluster IP\Node IP\External IP, pointing to the same workload in k8s

Real behavior (Required)

1. frontend correctly save Cluster IP when backend is added with hostname as expected
2. but validate the saved address with then localhost of backend, which is Endpoint Node IP (generally randomly assigned by k8s except create endpoints or endpointSlice Kind)
3. the backend can't be activated

be is healthy, but cannot be added :

FE LOG(from 8030 site):

2023-04-12 05:22:30,697 WARN (heartbeat mgr|31) [HeartbeatMgr.runAfterCatalogReady():165] get bad heartbeat response: type: BACKEND, status: BAD, msg: FE saved address not match backend address
2023-04-12 05:22:32,524 WARN (tablet stat mgr|25) [TabletStatMgr.updateLocalTabletStat():129] task exec error. backend[10002]
org.apache.thrift.transport.TTransportException: Invalid port -1
	at org.apache.thrift.transport.TSocket.open(TSocket.java:213) ~[libthrift-0.13.0.jar:0.13.0]
	at com.starrocks.common.GenericPool$ThriftClientFactory.create(GenericPool.java:131) ~[starrocks-fe.jar:?]
	at com.starrocks.common.GenericPool$ThriftClientFactory.create(GenericPool.java:116) ~[starrocks-fe.jar:?]
	at org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:62) ~[commons-pool2-2.3.jar:2.3]
	at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1036) ~[commons-pool2-2.3.jar:2.3]
	at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356) ~[commons-pool2-2.3.jar:2.3]
	at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:278) ~[commons-pool2-2.3.jar:2.3]
	at com.starrocks.common.GenericPool.borrowObject(GenericPool.java:88) ~[starrocks-fe.jar:?]
	at com.starrocks.catalog.TabletStatMgr.updateLocalTabletStat(TabletStatMgr.java:121) [starrocks-fe.jar:?]
	at com.starrocks.catalog.TabletStatMgr.runAfterCatalogReady(TabletStatMgr.java:71) [starrocks-fe.jar:?]
	at com.starrocks.common.util.LeaderDaemon.runOneCycle(LeaderDaemon.java:60) [starrocks-fe.jar:?]
	at com.starrocks.common.util.Daemon.run(Daemon.java:115) [starrocks-fe.jar:?]

BE_LOG(from be.WARNING in be pod):

W0412 03:15:05.854683  1184 heartbeat_server.cpp:148] 10.107.157.178 not equal to to backend localhost 10.244.0.73

connect error(show backends; in sql):

SQL Error (1064): Backend not found. Check if any backend is down or not. backend: [10.107.157.178 alive: false inBlacklist: false]

further details

• the localhost ip of the backend node is endpoint IP:

ifconfig info:

• the service before the node is cluster IP, and is also where the hostname pointing to:

codes may cause the issue:

starrocks/be/src/agent/heartbeat_server.cpp

Lines 159 to 164 in daa7b03

	if (master_info.__isset.backend_ip) {
	if (master_info.backend_ip != BackendOptions::get_localhost()) {
	LOG(WARNING) << master_info.backend_ip << " not equal to to backend localhost "
	<< BackendOptions::get_localhost();
	bool fe_saved_is_valid_ip = is_valid_ip(master_info.backend_ip);
	if (fe_saved_is_valid_ip && is_valid_ip(BackendOptions::get_localhost())) {

attempted workarounds

1. add an endpointslice kind to predefine a static endpoint ip to the service
2. add backend with the static ip instead of the hostname

Result: sql client can query fe now with backend added and activated. But stream load api still not works when other services in the same minikube cluster call it.(could be another issue)

StarRocks version (Required)

2.5.4

minikube version

minikube version: v1.30.1(windows default installation)

[kevincai]

1. please use FQDN mode to deploy in k8s env. refer to doc: https://docs.starrocks.io/en-us/latest/administration/enable_fqdn#enable-fqdn-access
2. we have provided a k8s operator to ease the deployment in k8s env. you can have a try: https://github.com/StarRocks/starrocks-kubernetes-operator
3. or you can try it with helm charts: https://artifacthub.io/packages/helm/kube-starrocks/kube-starrocks

[huandzh]

1. please use FQDN mode to deploy in k8s env. refer to doc: https://docs.starrocks.io/en-us/latest/administration/enable_fqdn#enable-fqdn-access
2. we have provided a k8s operator to ease the deployment in k8s env. you can have a try: https://github.com/StarRocks/starrocks-kubernetes-operator
3. or you can try it with helm charts: https://artifacthub.io/packages/helm/kube-starrocks/kube-starrocks

Thanks for your reply.

I have just tried FQDN but with no luck. It seems localhost IP of backend is still checked against host name.

A new cluster was started with:

    spec:
      containers:
        - args:
            - /opt/starrocks/fe/bin/start_fe.sh
            - --host_type
            - FQDN
          image: starrocks/fe-ubuntu:2.5.4

backend hostname was added but IP was still checked

Error log FE:

2023-04-12 06:44:07,475 WARN (heartbeat mgr|31) [HeartbeatMgr.runAfterCatalogReady():165] get bad heartbeat response: type: BACKEND, status: BAD, msg: Unmatched backend ip

Error log BE:

W0412 06:44:57.566421  1195 heartbeat_server.cpp:148] starrocks-be not equal to to backend localhost 10.244.0.86
W0412 06:44:57.567199  1195 heartbeat_server.cpp:74] Fail to handle heartbeat: Internal error: Unmatched backend ip cached master info: TMasterInfo(network_address=TNetworkAddress(hostname=, port=0), cluster_id=-1, epoch=0, token=<null>, backend_ip=<null>, http_port=<null>, heartbeat_flags=<null>, backend_id=<null>, min_active_txn_id=0) received master info: TMasterInfo(network_address=TNetworkAddress(hostname=starrocks-fe-deployment, port=9020), cluster_id=1371045861, epoch=1, token=6391d8c2-7c94-48f1-8448-430c2017c5bf, backend_ip=starrocks-be, http_port=8030, heartbeat_flags=0, backend_id=10003, min_active_txn_id=0)

[kevincai]

add each individual be fqdn name to fe, not the service fqdn, fe needs to be able to access be directly.

[huandzh]

add each individual be fqdn name to fe, not the service fqdn, fe needs to be able to access be directly.

Thx, but it doesn't work either. The be is alive but not counted as a valid resource.

Besides, the fe(host type of FQDN) doesn't use FQDN host name in its own network parameters or resolve short host name.

FE status:

Network interfaces:
 Name: eth0 (eth0)
   MAC Address: ca:0d:17:8f:45:92
   MTU: 1500, Speed: 10.5 Gbps
   IPv4: [10.244.0.107]
   IPv6: []
   Traffic: received 2366 packets/257.1 KiB (0 err); transmitted 2510 packets/1.3 MiB (0 err)
Network parameters:
 Host name: starrocks-fe-deployment
 Domain name: starrocks-fe-deployment
 DNS servers: [10.96.0.10]
 IPv4 Gateway: 10.244.0.1
 IPv6 Gateway: 

FE log:

2023-04-12 08:49:21,296 WARN (starrocks-mysql-nio-pool-4|129) [Coordinator.getNext():1545] get next fail, need cancel. status errorCode THRIFT_RPC_ERROR Couldn't open transport for starrocks-fe-deployment:9020 (Could not resolve host for client socket.), query id: 546f5be2-2cf9-4ec1-8c0d-8dbbe9d12e4c
2023-04-12 08:49:21,296 WARN (starrocks-mysql-nio-pool-4|129) [Coordinator.updateStatus():1525] one instance report fail throw updateStatus(), need cancel. job id: -1, query id: 546f5be2-2cf9-4ec1-8c0d-8dbbe9d12e4c, instance id: NaN
2023-04-12 08:49:21,297 WARN (starrocks-mysql-nio-pool-4|129) [StmtExecutor.execute():534] execute Exception, sql SELECT `DEFAULT_COLLATION_NAME` FROM `information_schema`.`SCHEMATA` WHERE `SCHEMA_NAME`='_statistics_'
com.starrocks.rpc.RpcException: Couldn't open transport for starrocks-fe-deployment:9020 (Could not resolve host for client socket.), host: unknown
	at com.starrocks.qe.Coordinator.getNext(Coordinator.java:1563) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:764) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:440) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:323) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:440) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:698) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:55) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2023-04-12 08:49:22,256 WARN (statistics meta manager|22) [StatisticsMetaManager.createSampleStatisticsTable():143] Failed to create tableFailed to find enough host in all backends. need: 1, Current alive backend is [10002]

[kevincai]

your BE doesn't report any storage capacity, please check the BE configuration in be.conf, why it doesn't report any available disk space?

[huandzh]

your BE doesn't report any storage capacity, please check the BE configuration in be.conf, why it doesn't report any available disk space?

no idea. I am using the official be image. The be.conf should be intact.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    starrocks: be
    status: testing
  name: starrocks-be
spec:
  replicas: 1
  selector:
    matchLabels:
      starrocks: be
  template:
    metadata:
      labels:
        starrocks: be
    spec:
      initContainers:
        - name: wait-for-fe
          image: curlimages/curl:8.00.1
          args:
            - /bin/sh
            - -c
            - |
              response=$(curl -s http://$FE_HOST:8030/api/bootstrap)
              status=$(echo "$response" | grep -Eo '"status":"[^"]+"' | cut -d '"' -f 4)
              [ "$status" = "OK" ]
          envFrom:
            - configMapRef:
                name: starrocks-deploy-envs
      containers:
        - args:
            - /bin/bash
            - -c
            - |
              sleep 15s; mysql --connect-timeout 2 -h $FE_HOST -P9030 -uroot -e "alter system add backend \"$BE_HOST-deployment.starrocks-be.default.svc.cluster.local:9050\";"
              /opt/starrocks/be/bin/start_be.sh
          image: starrocks/be-ubuntu:2.5.4
          name: starrocks-be-container
          ports:
            - containerPort: 8040
            - containerPort: 9050
            - containerPort: 9060
          resources: {}
          envFrom:
            - configMapRef:
                name: starrocks-deploy-envs
      hostname: starrocks-be-deployment
      subdomain: starrocks-be
      restartPolicy: Always

root@starrocks-be-deployment:/opt/starrocks# cat be/conf/be.conf 
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

# INFO, WARNING, ERROR, FATAL
sys_log_level = INFO

# ports for admin, web, heartbeat service 
be_port = 9060
webserver_port = 8040
heartbeat_service_port = 9050
brpc_port = 8060

# Enable jaeger tracing by setting jaeger_endpoint
# jaeger_endpoint = localhost:6831

# Choose one if there are more than one ip except loopback address. 
# Note that there should at most one ip match this list.
# If no ip match this rule, will choose one randomly.
# use CIDR format, e.g. 10.10.10.0/24
# Default value is empty.
# priority_networks = 10.10.10.0/24;192.168.0.0/16

# data root path, separate by ';'
# you can specify the storage medium of each root path, HDD or SSD, seperate by ','
# eg:
# storage_root_path = /data1,medium:HDD;/data2,medium:SSD;/data3
# /data1, HDD;
# /data2, SSD;
# /data3, HDD(default);
#
# Default value is ${STARROCKS_HOME}/storage, you should create it by hand.
# storage_root_path = ${STARROCKS_HOME}/storage

# Advanced configurations
# sys_log_dir = ${STARROCKS_HOME}/log
# sys_log_roll_mode = SIZE-MB-1024
# sys_log_roll_num = 10
# sys_log_verbose_modules = *
# log_buffer_level = -1

# When integrate with STAROS with local disk cache enabled, this is the absolute
# root dir for cache to write, multiple paths can be separated by colon (:).
# e.g. starlet_cache_dir = "/data/disk1/cache/:data/disk2/cache/"
#starlet_cache_dir = ""

# JVM options for be
# eg:
# JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"
# For jdk 9+, this JAVA_OPTS will be used as default JVM options
# JAVA_OPTS_FOR_JDK_9="-Djava.security.krb5.conf=/etc/krb5.conf"

[kevincai]

check be/log/be.INFO, if there is any clue.

[huandzh]

Docker Desktop is currently recommended for preview users of my developing app powered by starrocks.

It seems that the behaviors of starrocks with hostname deployment are not well-documented or tested, and the services do not behave as expected. I will further investigate the issue in more detail and explore other options if I have more time. However, the YAML files and logs that have already been provided should be sufficient for those concerned to reproduce the issue and conduct further investigations.

Using an operator solution may require too much effort for developers who are using Minikube on a desktop. They would appreciate a simple and direct way to start services with a single YAML file.

[kevincai]

operator is the preferred way to deploy starrocks in k8s env, even for minikube. starrocks is not a simple application, a single yaml file might be too naive and requires deeper understanding of sr deployment and application interaction.

[huandzh]

Fine.

I cannot speak to preview users in the same dismissive or condescending manner, as they likely do not wish to be viewed as "naive" or insulted while testing my product.

[huandzh]

A demo yaml file for deploying a pair of fe and be on minikube cluster using FQDN is avaliable per StarRocks/demo#26 (deploy tips included).

[huandzh]

docs added for deploying with minikube and kubectl per #21646

[github-actions]

We have marked this issue as stale because it has been inactive for 6 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to StarRocks!