Need help for TSaslTransportException when launching kyuubi

[zhichaoleo]

Hi Team,

We are using kyuubi 1.6.0 in k8s cluster. It works as expected, however, we keep geting error logs list below after kyuubi started. Although it doesn't affect our service, we still want to fix it, as it produce too many error logs.

2022-09-19 08:23:57.005 ERROR org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
	at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) ~[libthrift-0.9.3.jar:0.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:328) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[libthrift-0.9.3.jar:0.9.3]
	... 4 more

Our configure is like this:

kyuubi.engine.connection.url.use.hostname false
kyuubi.authentication           NONE
kyuubi.frontend.bind.port       10009
kyuubi.ha.zookeeper.quorum      kyuubi-zk-headless.argo.svc.cluster.local:2181

kyuubi.metrics.enabled            true
kyuubi.metrics.reporters          PROMETHEUS
kyuubi.metrics.prometheus.path    /metrics

kyuubi.engine.pool.name           kyuubi-engine-pool
kyuubi.engine.pool.size           1
kyuubi.engine.pool.size.threshold 3
kyuubi.metrics.prometheus.port  10019
kyuubi.authentication           LDAP
kyuubi.authentication.ldap.base.dn   ou=people,dc=hadoop,dc=apache,dc=org
kyuubi.authentication.ldap.url	 ldap://ldap-svc.argo.svc.cluster.local:3893
spark.master                    k8s://https://api.kube-playground
spark.submit.deployMode         cluster
spark.sql.extensions            io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog
spark.eventLog.enabled          true
spark.eventLog.dir              file:/logs/
spark.kubernetes.file.upload.path=/tmp
spark.kubernetes.driver.container.image=sf-docker-staging.common.repositories/dev/cloud-native-gdpr/spark33/spark-cng:latest
spark.kubernetes.executor.container.image=sf-docker-staging.common.repositories/dev/cloud-native-gdpr/spark33/spark-cng:latest
spark.kubernetes.container.image.pullSecrets=regcred
spark.kubernetes.container.image.pullPolicy=Always
spark.kubernetes.namespace=gdpr-argo
spark.kubernetes.authenticate.driver.serviceAccountName=spark
spark.dynamicAllocation.enabled=false
spark.shuffle.service.enabled=false
spark.databricks.delta.autoCompact.enabled=true
spark.kubernetes.driver.volumes.persistentVolumeClaim.log.options.claimName=spark-log-pvc
spark.kubernetes.driver.volumes.persistentVolumeClaim.log.mount.path=/logs/
spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.path.style.access=true

Could you help with this problem?

Thanks,
Alex

[pan3793]

Have you enabled the liveness probe? It occurs when using an HTTP client to call the thrift endpoint.

[zhichaoleo]

Hi @pan3793 ,
Thanks for your quick response!
We doesn't open liveness and readness probe.

[zhichaoleo]

Hi @pan3793 ,

You are right. And thanks for your remind.

We catch network packages and do some research to find where these HTTP request come from. We found there is something wrong with k8s cluster nodes. Some nodes will keep send HTTP request to kyuubi 10009 port. We will do more research for these nodes.

Reagrds,
Alex

[wjxiz1992]

Hi, I'm using tcpSocket for liveness probe, but still keep getting the ERROR logs mentioned in this topic. The occurance is 10 seconds, the same as the livenessProbe timewindow. Any more insights for it？ Like what kind of liveness check you would suggest?