[Question] Spark Configurations Via JDBC Connection URL #2937

hanna-liashchuk · 2022-06-22T18:43:04Z

hanna-liashchuk
Jun 22, 2022

Code of Conduct

I agree to follow this project's Code of Conduct

Search before asking

I have searched in the issues and found no similar issues.

Describe the bug

Seems that passing spark configurations has no effect.
For example,
I'm running

beeline -u jdbc:hive2://localhost:10009/default;#spark.sql.shuffle.partitions=2;spark.executor.memory=5g

but I can see from the logs that Spark engine is created without these configurations

2022-06-22 21:36:17.656 INFO state.ConnectionStateManager: State change: CONNECTED
2022-06-22 21:36:17.690 INFO engine.EngineRef: Launching engine:
/opt/spark-3.2.1-bin-hadoop3.2/bin/spark-submit \
	--class org.apache.kyuubi.engine.spark.SparkSQLEngine \
	--conf spark.driver.host=kyuubi-hs.kyuubi-test.svc \
	--conf spark.kubernetes.namespace=kyuubi-test \
	--conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
	--conf spark.kyuubi.ha.zookeeper.quorum=zookeeper-connect.kafka.svc:2181 \
	--conf spark.driver.port=44104 \
	--conf spark.kyuubi.session.idle.timeout=PT5S \
	--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
	--conf spark.app.name=kyuubi_USER_SPARK_SQL_anonymous_default_c379bd55-a8fe-4838-8823-e4c2c7fb970b \
	--conf spark.decommission.enabled=true \
	--conf spark.kubernetes.authenticate.serviceAccountName=kyuubi-spark \
	--conf spark.kyuubi.ha.engine.ref.id=c379bd55-a8fe-4838-8823-e4c2c7fb970b \
	--conf spark.kubernetes.container.image=spark:spark3.2.1-hadoop3.2-delta1.2.1-scala2.12 \
	--conf spark.master=k8s://https://kubernetes.default.svc \
	--conf spark.yarn.tags=KYUUBI \
	--conf spark.kyuubi.ha.zookeeper.namespace=/kyuubi_USER_SPARK_SQL/anonymous/default \
	--conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
	--conf spark.kubernetes.driver.pod.name=kyuubi-server-6964b97f8-xf6bg \
	--proxy-user anonymous /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.4.1-incubating.jar

I'm reffering to this doc

Affects Version(s)

1.4.1-incubating

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

22/06/22 21:36:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2022-06-22 21:36:19.247 INFO util.SignalRegister: Registering signal handler for TERM
2022-06-22 21:36:19.248 INFO util.SignalRegister: Registering signal handler for HUP
2022-06-22 21:36:19.248 INFO util.SignalRegister: Registering signal handler for INT
2022-06-22 21:36:19.280 INFO conf.HiveConf: Found configuration file file:/opt/spark-3.2.1-bin-hadoop3.2/conf/hive-site.xml
2022-06-22 21:36:19.394 INFO spark.SparkContext: Running Spark version 3.2.1
2022-06-22 21:36:19.421 INFO resource.ResourceUtils: ==============================================================
2022-06-22 21:36:19.422 INFO resource.ResourceUtils: No custom resources configured for spark.driver.
2022-06-22 21:36:19.422 INFO resource.ResourceUtils: ==============================================================
2022-06-22 21:36:19.423 INFO spark.SparkContext: Submitted application: kyuubi_USER_SPARK_SQL_anonymous_default_c379bd55-a8fe-4838-8823-e4c2c7fb970b
2022-06-22 21:36:19.444 INFO resource.ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
2022-06-22 21:36:19.457 INFO resource.ResourceProfile: Limiting resource is cpus at 1 tasks per executor
2022-06-22 21:36:19.459 INFO resource.ResourceProfileManager: Added ResourceProfile id: 0
2022-06-22 21:36:19.515 INFO spark.SecurityManager: Changing view acls to: kyuubi,anonymous
2022-06-22 21:36:19.516 INFO spark.SecurityManager: Changing modify acls to: kyuubi,anonymous
2022-06-22 21:36:19.516 INFO spark.SecurityManager: Changing view acls groups to: 
2022-06-22 21:36:19.516 INFO spark.SecurityManager: Changing modify acls groups to: 
2022-06-22 21:36:19.517 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(kyuubi, anonymous); groups with view permissions: Set(); users  with modify permissions: Set(kyuubi, anonymous); groups with modify permissions: Set()
2022-06-22 21:36:19.730 INFO util.Utils: Successfully started service 'sparkDriver' on port 44104.
2022-06-22 21:36:19.759 INFO spark.SparkEnv: Registering MapOutputTracker
2022-06-22 21:36:19.791 INFO spark.SparkEnv: Registering BlockManagerMaster
2022-06-22 21:36:19.813 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2022-06-22 21:36:19.814 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
2022-06-22 21:36:19.817 INFO spark.SparkEnv: Registering BlockManagerMasterHeartbeat
2022-06-22 21:36:19.844 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-2a1d5526-04a0-4ed4-8668-b0296e8804d6
2022-06-22 21:36:19.866 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MiB
2022-06-22 21:36:19.882 INFO spark.SparkEnv: Registering OutputCommitCoordinator
2022-06-22 21:36:19.964 INFO util.log: Logging initialized @2125ms to org.sparkproject.jetty.util.log.Slf4jLog
2022-06-22 21:36:20.051 INFO server.Server: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 1.8.0_332-b09
2022-06-22 21:36:20.073 INFO server.Server: Started @2234ms
2022-06-22 21:36:20.107 INFO server.AbstractConnector: Started ServerConnector@27cbfddf{HTTP/1.1, (http/1.1)}{0.0.0.0:35437}
2022-06-22 21:36:20.107 INFO util.Utils: Successfully started service 'SparkUI' on port 35437.
2022-06-22 21:36:20.130 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@14faa38c{/jobs,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.133 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@759d81f3{/jobs/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.134 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5a4c638d{/jobs/job,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.136 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1f12e153{/jobs/job/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.137 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5a101b1c{/stages,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.137 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@29f0802c{/stages/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.138 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@60f2e0bd{/stages/stage,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.139 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@203dd56b{/stages/stage/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.139 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6d64b553{/stages/pool,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.140 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1d3e6d34{/stages/pool/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.140 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@26a94fa5{/storage,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.141 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2873d672{/storage/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.142 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@577f9109{/storage/rdd,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.142 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@757529a4{/storage/rdd/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.143 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5c41d037{/environment,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.143 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5ec77191{/environment/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.144 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1450078a{/executors,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.144 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@69c6161d{/executors/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.145 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2e1792e7{/executors/threadDump,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.145 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3eb631b8{/executors/threadDump/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.155 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6bff19ff{/static,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.156 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3b0ca5e1{/,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.157 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@54dcbb9f{/api,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.158 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5bb8f9e2{/jobs/job/kill,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.158 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5f78de22{/stages/stage/kill,null,AVAILABLE,@Spark}
2022-06-22 21:36:20.160 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://kyuubi-hs.kyuubi-test.svc:35437
2022-06-22 21:36:20.173 INFO spark.SparkContext: Added JAR file:/opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.4.1-incubating.jar at spark://kyuubi-hs.kyuubi-test.svc:44104/jars/kyuubi-spark-sql-engine_2.12-1.4.1-incubating.jar with timestamp 1655922979386
2022-06-22 21:36:20.231 INFO k8s.SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
2022-06-22 21:36:21.129 INFO k8s.ExecutorPodsAllocator: Going to request 2 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 0, sharedSlotFromPendingPods: 2147483647.
2022-06-22 21:36:21.150 INFO submit.KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : hive-site.xml
2022-06-22 21:36:21.173 INFO submit.KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : hive-site.xml
2022-06-22 21:36:21.188 INFO features.BasicExecutorFeatureStep: Adding decommission script to lifecycle
2022-06-22 21:36:21.244 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36477.
2022-06-22 21:36:21.244 INFO netty.NettyBlockTransferService: Server created on kyuubi-hs.kyuubi-test.svc:36477
2022-06-22 21:36:21.246 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2022-06-22 21:36:21.252 INFO submit.KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : hive-site.xml
2022-06-22 21:36:21.254 INFO features.BasicExecutorFeatureStep: Adding decommission script to lifecycle
2022-06-22 21:36:21.256 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, kyuubi-hs.kyuubi-test.svc, 36477, None)
2022-06-22 21:36:21.261 INFO storage.BlockManagerMasterEndpoint: Registering block manager kyuubi-hs.kyuubi-test.svc:36477 with 366.3 MiB RAM, BlockManagerId(driver, kyuubi-hs.kyuubi-test.svc, 36477, None)
2022-06-22 21:36:21.264 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, kyuubi-hs.kyuubi-test.svc, 36477, None)
2022-06-22 21:36:21.265 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, kyuubi-hs.kyuubi-test.svc, 36477, None)
2022-06-22 21:36:21.282 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3291b443{/metrics/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:24.277 INFO k8s.ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 1, sharedSlotFromPendingPods: 2147483647.
2022-06-22 21:36:24.279 INFO submit.KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : hive-site.xml
2022-06-22 21:36:24.280 INFO features.BasicExecutorFeatureStep: Adding decommission script to lifecycle
2022-06-22 21:36:25.117 INFO storage.BlockManagerMaster: Removal of executor 1 requested
2022-06-22 21:36:25.117 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 1
2022-06-22 21:36:25.122 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
2022-06-22 21:36:25.124 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 2 from BlockManagerMaster.
2022-06-22 21:36:25.124 INFO storage.BlockManagerMaster: Removal of executor 2 requested
2022-06-22 21:36:25.124 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 2
2022-06-22 21:36:25.298 INFO k8s.ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 1, sharedSlotFromPendingPods: 2147483646.
2022-06-22 21:36:25.300 INFO submit.KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : hive-site.xml
2022-06-22 21:36:25.301 INFO features.BasicExecutorFeatureStep: Adding decommission script to lifecycle
2022-06-22 21:36:27.285 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.17.4.68:34348) with ID 3,  ResourceProfileId 0
2022-06-22 21:36:27.386 INFO storage.BlockManagerMasterEndpoint: Registering block manager 172.17.4.68:33413 with 413.9 MiB RAM, BlockManagerId(3, 172.17.4.68, 33413, None)
2022-06-22 21:36:28.199 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 4 from BlockManagerMaster.
2022-06-22 21:36:28.199 INFO storage.BlockManagerMaster: Removal of executor 4 requested
2022-06-22 21:36:28.199 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 4
2022-06-22 21:36:28.326 INFO k8s.ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 1, sharedSlotFromPendingPods: 2147483647.
2022-06-22 21:36:28.327 INFO submit.KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : hive-site.xml
2022-06-22 21:36:28.329 INFO features.BasicExecutorFeatureStep: Adding decommission script to lifecycle
2022-06-22 21:36:32.225 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 5 from BlockManagerMaster.
2022-06-22 21:36:32.226 INFO storage.BlockManagerMaster: Removal of executor 5 requested
2022-06-22 21:36:32.226 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 5
2022-06-22 21:36:32.350 INFO k8s.ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 1, sharedSlotFromPendingPods: 2147483647.
2022-06-22 21:36:32.351 INFO submit.KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : hive-site.xml
2022-06-22 21:36:32.352 INFO features.BasicExecutorFeatureStep: Adding decommission script to lifecycle
2022-06-22 21:36:35.249 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 6 from BlockManagerMaster.
2022-06-22 21:36:35.249 INFO storage.BlockManagerMaster: Removal of executor 6 requested
2022-06-22 21:36:35.249 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 6
2022-06-22 21:36:35.370 INFO k8s.ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 1, sharedSlotFromPendingPods: 2147483647.
2022-06-22 21:36:35.371 INFO submit.KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : hive-site.xml
2022-06-22 21:36:35.372 INFO features.BasicExecutorFeatureStep: Adding decommission script to lifecycle
2022-06-22 21:36:39.276 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 7 from BlockManagerMaster.
2022-06-22 21:36:39.276 INFO storage.BlockManagerMaster: Removal of executor 7 requested
2022-06-22 21:36:39.277 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 7
2022-06-22 21:36:39.387 INFO k8s.ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 1, sharedSlotFromPendingPods: 2147483647.
2022-06-22 21:36:39.388 INFO submit.KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : hive-site.xml
2022-06-22 21:36:39.389 INFO features.BasicExecutorFeatureStep: Adding decommission script to lifecycle
2022-06-22 21:36:42.300 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster.
2022-06-22 21:36:42.300 INFO storage.BlockManagerMaster: Removal of executor 8 requested
2022-06-22 21:36:42.300 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 8
2022-06-22 21:36:42.406 INFO k8s.ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 1, sharedSlotFromPendingPods: 2147483647.
2022-06-22 21:36:42.408 INFO submit.KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : hive-site.xml
2022-06-22 21:36:42.409 INFO features.BasicExecutorFeatureStep: Adding decommission script to lifecycle
2022-06-22 21:36:46.325 INFO storage.BlockManagerMaster: Removal of executor 9 requested
2022-06-22 21:36:46.325 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 9 from BlockManagerMaster.
2022-06-22 21:36:46.325 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 9
2022-06-22 21:36:46.424 INFO k8s.ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 1, sharedSlotFromPendingPods: 2147483647.
2022-06-22 21:36:46.425 INFO submit.KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : hive-site.xml
2022-06-22 21:36:46.426 INFO features.BasicExecutorFeatureStep: Adding decommission script to lifecycle
2022-06-22 21:36:49.348 INFO storage.BlockManagerMaster: Removal of executor 10 requested
2022-06-22 21:36:49.348 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 10 from BlockManagerMaster.
2022-06-22 21:36:49.348 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 10
2022-06-22 21:36:49.440 INFO k8s.ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 1, sharedSlotFromPendingPods: 2147483647.
2022-06-22 21:36:49.442 INFO submit.KubernetesClientUtils: Spark configuration files loaded from Some(/opt/spark/conf) : hive-site.xml
2022-06-22 21:36:49.443 INFO features.BasicExecutorFeatureStep: Adding decommission script to lifecycle
2022-06-22 21:36:51.120 INFO k8s.KubernetesClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000000000(ns)
2022-06-22 21:36:51.283 INFO internal.SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir.
2022-06-22 21:36:51.429 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
2022-06-22 21:36:51.441 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2022-06-22 21:36:51.441 INFO impl.MetricsSystemImpl: s3a-file-system metrics system started
2022-06-22 21:36:51.893 INFO internal.SharedState: Warehouse path is 's3a://warehouse/'.
2022-06-22 21:36:51.915 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7f5614f9{/SQL,null,AVAILABLE,@Spark}
2022-06-22 21:36:51.916 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6c3f1658{/SQL/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:51.917 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2de9ca6{/SQL/execution,null,AVAILABLE,@Spark}
2022-06-22 21:36:51.917 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2ee39e73{/SQL/execution/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:51.926 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.17.0.212:51570) with ID 11,  ResourceProfileId 0
2022-06-22 21:36:51.926 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@c4d2c44{/static/sql,null,AVAILABLE,@Spark}
2022-06-22 21:36:52.009 INFO storage.BlockManagerMasterEndpoint: Registering block manager 172.17.0.212:33063 with 413.9 MiB RAM, BlockManagerId(11, 172.17.0.212, 33063, None)
2022-06-22 21:36:54.512 INFO hive.HiveUtils: Initializing HiveMetastoreConnection version 2.3.9 using Spark classes.
2022-06-22 21:36:54.567 INFO conf.HiveConf: Found configuration file file:/opt/spark-3.2.1-bin-hadoop3.2/conf/hive-site.xml
2022-06-22 21:36:54.762 INFO client.HiveClientImpl: Warehouse location for Hive client (version 2.3.9) is s3a://warehouse/
2022-06-22 21:36:54.829 INFO hive.metastore: Trying to connect to metastore with URI thrift://hive-metastore-cs.hive-metastore:9083
2022-06-22 21:36:54.847 INFO hive.metastore: Opened a connection to metastore, current connections: 1
2022-06-22 21:36:54.865 WARN security.ShellBasedUnixGroupsMapping: unable to return groups for user anonymous
PartialGroupNameException The user name 'anonymous' is not found. id: ‘anonymous’: no such user
id: ‘anonymous’: no such user

        at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:294)
        at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:207)
        at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97)
        at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51)
        at org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:387)
        at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:321)
        at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:270)
        at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
        at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
        at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
        at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
        at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache.get(LocalCache.java:3962)
        at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3985)
        at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4946)
        at org.apache.hadoop.security.Groups.getGroups(Groups.java:228)
        at org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1734)
        at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1722)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:496)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:245)
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1740)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3607)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3659)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3639)
        at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1563)
        at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1552)
        at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$databaseExists$1(HiveClientImpl.scala:396)
        at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
        at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:305)
        at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:236)
        at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:235)
        at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:285)
        at org.apache.spark.sql.hive.client.HiveClientImpl.databaseExists(HiveClientImpl.scala:396)
        at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$databaseExists$1(HiveExternalCatalog.scala:224)
        at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:102)
        at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:224)
        at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:150)
        at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:140)
        at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:45)
        at org.apache.spark.sql.hive.HiveSessionStateBuilder.$anonfun$catalog$1(HiveSessionStateBuilder.scala:60)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog$lzycompute(SessionCatalog.scala:118)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog(SessionCatalog.scala:118)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listDatabases(SessionCatalog.scala:298)
        at org.apache.spark.sql.execution.datasources.v2.V2SessionCatalog.listNamespaces(V2SessionCatalog.scala:205)
        at org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension.listNamespaces(DelegatingCatalogExtension.java:116)
        at org.apache.spark.sql.execution.datasources.v2.ShowNamespacesExec.run(ShowNamespacesExec.scala:42)
        at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
        at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
        at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:110)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:110)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:106)
        at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
        at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:106)
        at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:93)
        at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:91)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:219)
        at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
        at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
        at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$createSpark$4(SparkSQLEngine.scala:110)
        at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$createSpark$4$adapted(SparkSQLEngine.scala:107)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.kyuubi.engine.spark.SparkSQLEngine$.createSpark(SparkSQLEngine.scala:107)
        at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:157)
        at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2022-06-22 21:36:54.875 INFO hive.metastore: Connected to metastore.
2022-06-22 21:36:55.199 INFO codegen.CodeGenerator: Code generated in 177.63998 ms
2022-06-22 21:36:55.251 INFO codegen.CodeGenerator: Code generated in 6.67447 ms
2022-06-22 21:36:55.428 INFO codegen.CodeGenerator: Code generated in 10.057525 ms
2022-06-22 21:36:55.494 INFO spark.SparkContext: Starting job: isEmpty at SparkSQLEngine.scala:110
2022-06-22 21:36:55.510 INFO scheduler.DAGScheduler: Got job 0 (isEmpty at SparkSQLEngine.scala:110) with 1 output partitions
2022-06-22 21:36:55.510 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (isEmpty at SparkSQLEngine.scala:110)
2022-06-22 21:36:55.510 INFO scheduler.DAGScheduler: Parents of final stage: List()
2022-06-22 21:36:55.511 INFO scheduler.DAGScheduler: Missing parents: List()
2022-06-22 21:36:55.515 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at isEmpty at SparkSQLEngine.scala:110), which has no missing parents
2022-06-22 21:36:55.596 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 6.6 KiB, free 366.3 MiB)
2022-06-22 21:36:55.623 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.5 KiB, free 366.3 MiB)
2022-06-22 21:36:55.624 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on kyuubi-hs.kyuubi-test.svc:36477 (size: 3.5 KiB, free: 366.3 MiB)
2022-06-22 21:36:55.627 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1478
2022-06-22 21:36:55.640 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at isEmpty at SparkSQLEngine.scala:110) (first 15 tasks are for partitions Vector(0))
2022-06-22 21:36:55.640 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0
2022-06-22 21:36:55.673 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (172.17.4.68, executor 3, partition 0, PROCESS_LOCAL, 4645 bytes) taskResourceAssignments Map()
2022-06-22 21:36:55.875 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.17.4.68:33413 (size: 3.5 KiB, free: 413.9 MiB)
2022-06-22 21:36:56.557 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 894 ms on 172.17.4.68 (executor 3) (1/1)
2022-06-22 21:36:56.559 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
2022-06-22 21:36:56.564 INFO scheduler.DAGScheduler: ResultStage 0 (isEmpty at SparkSQLEngine.scala:110) finished in 1.038 s
2022-06-22 21:36:56.567 INFO scheduler.DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
2022-06-22 21:36:56.567 INFO scheduler.TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
2022-06-22 21:36:56.568 INFO scheduler.DAGScheduler: Job 0 finished: isEmpty at SparkSQLEngine.scala:110, took 1.074265 s
2022-06-22 21:36:56.596 INFO events.EventLoggingService: Service[EventLogging] is initialized.
2022-06-22 21:36:56.596 INFO events.EventLoggingService: Service[EventLogging] is started.
2022-06-22 21:36:56.600 INFO util.ThreadUtils: SparkSQLSessionManager-exec-pool: pool size: 100, wait queue size: 100, thread keepalive time: 60000 ms
2022-06-22 21:36:56.602 INFO operation.SparkSQLOperationManager: Service[SparkSQLOperationManager] is initialized.
2022-06-22 21:36:56.603 INFO session.SparkSQLSessionManager: Service[SparkSQLSessionManager] is initialized.
2022-06-22 21:36:56.603 INFO spark.SparkSQLBackendService: Service[SparkSQLBackendService] is initialized.
2022-06-22 21:36:56.630 INFO spark.SparkThriftBinaryFrontendService: Initializing SparkThriftBinaryFrontendService on host kyuubi-server-6964b97f8-xf6bg at port 40303 with [9, 999] worker threads
2022-06-22 21:36:56.711 INFO imps.CuratorFrameworkImpl: Starting
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:host.name=kyuubi-server-6964b97f8-xf6bg
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_332
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Azul Systems, Inc.
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/zulu8-ca-amd64/jre
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/opt/spark/conf/:/opt/spark/jars/derby-10.14.2.0.jar:/opt/spark/jars/kubernetes-model-scheduling-5.4.1.jar:/opt/spark/jars/paranamer-2.8.jar:/opt/spark/jars/delta-storage-1.2.1.jar:/opt/spark/jars/spark-hive-thriftserver_2.12-3.2.1.jar:/opt/spark/jars/parquet-hadoop-1.12.2.jar:/opt/spark/jars/kubernetes-model-admissionregistration-5.4.1.jar:/opt/spark/jars/commons-io-2.8.0.jar:/opt/spark/jars/commons-text-1.6.jar:/opt/spark/jars/oro-2.0.8.jar:/opt/spark/jars/commons-pool-1.5.4.jar:/opt/spark/jars/xbean-asm9-shaded-4.20.jar:/opt/spark/jars/curator-framework-2.13.0.jar:/opt/spark/jars/scala-xml_2.12-1.2.0.jar:/opt/spark/jars/json4s-ast_2.12-3.7.0-M11.jar:/opt/spark/jars/ivy-2.5.0.jar:/opt/spark/jars/mesos-1.4.0-shaded-protobuf.jar:/opt/spark/jars/ST4-4.0.4.jar:/opt/spark/jars/json4s-scalap_2.12-3.7.0-M11.jar:/opt/spark/jars/jersey-client-2.34.jar:/opt/spark/jars/hive-metastore-2.3.9.jar:/opt/spark/jars/aircompressor-0.21.jar:/opt/spark/jars/metrics-graphite-4.2.0.jar:/opt/spark/jars/kubernetes-model-batch-5.4.1.jar:/opt/spark/jars/leveldbjni-all-1.8.jar:/opt/spark/jars/spark-tags_2.12-3.2.1-tests.jar:/opt/spark/jars/jdo-api-3.0.1.jar:/opt/spark/jars/jackson-dataformat-yaml-2.12.3.jar:/opt/spark/jars/spark-sql_2.12-3.2.1.jar:/opt/spark/jars/hive-service-rpc-3.1.2.jar:/opt/spark/jars/zstd-jni-1.5.0-4.jar:/opt/spark/jars/wildfly-openssl-1.0.7.Final.jar:/opt/spark/jars/scala-collection-compat_2.12-2.1.1.jar:/opt/spark/jars/super-csv-2.2.0.jar:/opt/spark/jars/slf4j-api-1.7.30.jar:/opt/spark/jars/spark-repl_2.12-3.2.1.jar:/opt/spark/jars/kubernetes-model-common-5.4.1.jar:/opt/spark/jars/commons-collections-3.2.2.jar:/opt/spark/jars/snappy-java-1.1.8.4.jar:/opt/spark/jars/kubernetes-model-storageclass-5.4.1.jar:/opt/spark/jars/json-1.8.jar:/opt/spark/jars/jersey-hk2-2.34.jar:/opt/spark/jars/py4j-0.10.9.3.jar:/opt/spark/jars/jakarta.servlet-api-4.0.3.jar:/opt/spark/jars/jackson-mapper-asl-1.9.13.jar:/opt/spark/jars/jta-1.1.jar:/opt/spark/jars/parquet-format-structures-1.12.2.jar:/opt/spark/jars/hive-serde-2.3.9.jar:/opt/spark/jars/aws-java-sdk-bundle-1.11.901.jar:/opt/spark/jars/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/spark/jars/hadoop-yarn-server-web-proxy-3.3.1.jar:/opt/spark/jars/compress-lzf-1.0.3.jar:/opt/spark/jars/univocity-parsers-2.9.1.jar:/opt/spark/jars/okhttp-3.12.12.jar:/opt/spark/jars/spark-mllib_2.12-3.2.1.jar:/opt/spark/jars/spark-graphx_2.12-3.2.1.jar:/opt/spark/jars/jodd-core-3.5.2.jar:/opt/spark/jars/istack-commons-runtime-3.0.8.jar:/opt/spark/jars/core-1.1.2.jar:/opt/spark/jars/spark-launcher_2.12-3.2.1.jar:/opt/spark/jars/htrace-core4-4.1.0-incubating.jar:/opt/spark/jars/kubernetes-model-extensions-5.4.1.jar:/opt/spark/jars/spire-util_2.12-0.17.0.jar:/opt/spark/jars/scala-parser-combinators_2.12-1.1.2.jar:/opt/spark/jars/jakarta.ws.rs-api-2.1.6.jar:/opt/spark/jars/kubernetes-model-policy-5.4.1.jar:/opt/spark/jars/stream-2.9.6.jar:/opt/spark/jars/json4s-core_2.12-3.7.0-M11.jar:/opt/spark/jars/hive-beeline-2.3.9.jar:/opt/spark/jars/jcl-over-slf4j-1.7.30.jar:/opt/spark/jars/hive-storage-api-2.7.2.jar:/opt/spark/jars/parquet-jackson-1.12.2.jar:/opt/spark/jars/transaction-api-1.1.jar:/opt/spark/jars/JTransforms-3.1.jar:/opt/spark/jars/threeten-extra-1.5.0.jar:/opt/spark/jars/jackson-core-asl-1.9.13.jar:/opt/spark/jars/parquet-common-1.12.2.jar:/opt/spark/jars/antlr-runtime-3.5.2.jar:/opt/spark/jars/kubernetes-model-networking-5.4.1.jar:/opt/spark/jars/hadoop-aws-3.3.1.jar:/opt/spark/jars/hk2-utils-2.6.1.jar:/opt/spark/jars/hk2-locator-2.6.1.jar:/opt/spark/jars/jakarta.annotation-api-1.3.5.jar:/opt/spark/jars/jackson-annotations-2.12.3.jar:/opt/spark/jars/orc-shims-1.6.12.jar:/opt/spark/jars/kubernetes-model-core-5.4.1.jar:/opt/spark/jars/jackson-datatype-jsr310-2.11.2.jar:/opt/spark/jars/hive-common-2.3.9.jar:/opt/spark/jars/parquet-column-1.12.2.jar:/opt/spark/jars/commons-lang-2.6.jar:/opt/spark/jars/lz4-java-1.7.1.jar:/opt/spark/jars/kubernetes-model-discovery-5.4.1.jar:/opt/spark/jars/hive-jdbc-2.3.9.jar:/opt/spark/jars/gson-2.2.4.jar:/opt/spark/jars/xz-1.8.jar:/opt/spark/jars/json4s-jackson_2.12-3.7.0-M11.jar:/opt/spark/jars/netty-all-4.1.68.Final.jar:/opt/spark/jars/jline-2.14.6.jar:/opt/spark/jars/breeze-macros_2.12-1.2.jar:/opt/spark/jars/joda-time-2.10.10.jar:/opt/spark/jars/objenesis-2.6.jar:/opt/spark/jars/tink-1.6.0.jar:/opt/spark/jars/datanucleus-core-4.1.17.jar:/opt/spark/jars/kubernetes-model-certificates-5.4.1.jar:/opt/spark/jars/jaxb-runtime-2.3.2.jar:/opt/spark/jars/hive-cli-2.3.9.jar:/opt/spark/jars/osgi-resource-locator-1.0.3.jar:/opt/spark/jars/javolution-5.5.1.jar:/opt/spark/jars/spark-hive_2.12-3.2.1.jar:/opt/spark/jars/macro-compat_2.12-1.1.1.jar:/opt/spark/jars/jakarta.xml.bind-api-2.3.2.jar:/opt/spark/jars/jersey-container-servlet-core-2.34.jar:/opt/spark/jars/flatbuffers-java-1.9.0.jar:/opt/spark/jars/scala-reflect-2.12.15.jar:/opt/spark/jars/opencsv-2.3.jar:/opt/spark/jars/bonecp-0.8.0.RELEASE.jar:/opt/spark/jars/commons-lang3-3.12.0.jar:/opt/spark/jars/hive-exec-2.3.9-core.jar:/opt/spark/jars/kubernetes-model-rbac-5.4.1.jar:/opt/spark/jars/metrics-core-4.2.0.jar:/opt/spark/jars/metrics-json-4.2.0.jar:/opt/spark/jars/annotations-17.0.0.jar:/opt/spark/jars/spark-sketch_2.12-3.2.1.jar:/opt/spark/jars/hive-shims-scheduler-2.3.9.jar:/opt/spark/jars/metrics-jvm-4.2.0.jar:/opt/spark/jars/spark-network-shuffle_2.12-3.2.1.jar:/opt/spark/jars/HikariCP-2.5.1.jar:/opt/spark/jars/guava-14.0.1.jar:/opt/spark/jars/curator-recipes-2.13.0.jar:/opt/spark/jars/kubernetes-model-coordination-5.4.1.jar:/opt/spark/jars/hive-shims-2.3.9.jar:/opt/spark/jars/janino-3.0.16.jar:/opt/spark/jars/commons-logging-1.1.3.jar:/opt/spark/jars/commons-crypto-1.1.0.jar:/opt/spark/jars/jul-to-slf4j-1.7.30.jar:/opt/spark/jars/spark-catalyst_2.12-3.2.1.jar:/opt/spark/jars/cats-kernel_2.12-2.1.1.jar:/opt/spark/jars/javassist-3.25.0-GA.jar:/opt/spark/jars/spark-yarn_2.12-3.2.1.jar:/opt/spark/jars/metrics-jmx-4.2.0.jar:/opt/spark/jars/log4j-1.2.17.jar:/opt/spark/jars/automaton-1.11-8.jar:/opt/spark/jars/arrow-memory-core-2.0.0.jar:/opt/spark/jars/arpack_combined_all-0.1.jar:/opt/spark/jars/velocity-1.5.jar:/opt/spark/jars/kubernetes-model-apiextensions-5.4.1.jar:/opt/spark/jars/commons-math3-3.4.1.jar:/opt/spark/jars/orc-mapreduce-1.6.12.jar:/opt/spark/jars/kryo-shaded-4.0.2.jar:/opt/spark/jars/breeze_2.12-1.2.jar:/opt/spark/jars/delta-core_2.12-1.2.1.jar:/opt/spark/jars/audience-annotations-0.5.0.jar:/opt/spark/jars/jersey-server-2.34.jar:/opt/spark/jars/hk2-api-2.6.1.jar:/opt/spark/jars/spire-macros_2.12-0.17.0.jar:/opt/spark/jars/hive-llap-common-2.3.9.jar:/opt/spark/jars/jaxb-api-2.2.11.jar:/opt/spark/jars/avro-1.10.2.jar:/opt/spark/jars/spark-tags_2.12-3.2.1.jar:/opt/spark/jars/orc-core-1.6.12.jar:/opt/spark/jars/okio-1.14.0.jar:/opt/spark/jars/arrow-format-2.0.0.jar:/opt/spark/jars/jackson-core-2.12.3.jar:/opt/spark/jars/commons-cli-1.2.jar:/opt/spark/jars/kubernetes-model-events-5.4.1.jar:/opt/spark/jars/zookeeper-3.6.2.jar:/opt/spark/jars/slf4j-log4j12-1.7.30.jar:/opt/spark/jars/avro-mapred-1.10.2.jar:/opt/spark/jars/parquet-encoding-1.12.2.jar:/opt/spark/jars/avro-ipc-1.10.2.jar:/opt/spark/jars/scala-compiler-2.12.15.jar:/opt/spark/jars/javax.jdo-3.2.0-m3.jar:/opt/spark/jars/hive-vector-code-gen-2.3.9.jar:/opt/spark/jars/spark-network-common_2.12-3.2.1.jar:/opt/spark/jars/arrow-vector-2.0.0.jar:/opt/spark/jars/RoaringBitmap-0.9.0.jar:/opt/spark/jars/commons-compiler-3.0.16.jar:/opt/spark/jars/shims-0.9.0.jar:/opt/spark/jars/spark-kubernetes_2.12-3.2.1.jar:/opt/spark/jars/spark-core_2.12-3.2.1.jar:/opt/spark/jars/activation-1.1.1.jar:/opt/spark/jars/lapack-2.2.1.jar:/opt/spark/jars/spire_2.12-0.17.0.jar:/opt/spark/jars/kubernetes-model-metrics-5.4.1.jar:/opt/spark/jars/spark-kvstore_2.12-3.2.1.jar:/opt/spark/jars/logging-interceptor-3.12.12.jar:/opt/spark/jars/hive-shims-0.23-2.3.9.jar:/opt/spark/jars/kubernetes-model-autoscaling-5.4.1.jar:/opt/spark/jars/protobuf-java-2.5.0.jar:/opt/spark/jars/rocksdbjni-6.20.3.jar:/opt/spark/jars/jakarta.validation-api-2.0.2.jar:/opt/spark/jars/spark-mesos_2.12-3.2.1.jar:/opt/spark/jars/spire-platform_2.12-0.17.0.jar:/opt/spark/jars/blas-2.2.1.jar:/opt/spark/jars/scala-library-2.12.14.jar:/opt/spark/jars/curator-client-2.13.0.jar:/opt/spark/jars/httpclient-4.5.13.jar:/opt/spark/jars/zjsonpatch-0.3.0.jar:/opt/spark/jars/jtds-1.3.1.jar:/opt/spark/jars/scala-library-2.12.15.jar:/opt/spark/jars/hadoop-shaded-guava-1.1.1.jar:/opt/spark/jars/generex-1.0.2.jar:/opt/spark/jars/jersey-common-2.34.jar:/opt/spark/jars/kubernetes-model-node-5.4.1.jar:/opt/spark/jars/spark-mllib-local_2.12-3.2.1.jar:/opt/spark/jars/minlog-1.3.0.jar:/opt/spark/jars/datanucleus-rdbms-4.1.19.jar:/opt/spark/jars/antlr4-runtime-4.8.jar:/opt/spark/jars/libfb303-0.9.3.jar:/opt/spark/jars/commons-dbcp-1.4.jar:/opt/spark/jars/chill-java-0.10.0.jar:/opt/spark/jars/shapeless_2.12-2.3.3.jar:/opt/spark/jars/aopalliance-repackaged-2.6.1.jar:/opt/spark/jars/jackson-databind-2.12.3.jar:/opt/spark/jars/hive-shims-common-2.3.9.jar:/opt/spark/jars/commons-net-3.1.jar:/opt/spark/jars/libthrift-0.12.0.jar:/opt/spark/jars/snakeyaml-1.27.jar:/opt/spark/jars/jpam-1.1.jar:/opt/spark/jars/pyrolite-4.30.jar:/opt/spark/jars/commons-compress-1.21.jar:/opt/spark/jars/arpack-2.2.1.jar:/opt/spark/jars/jackson-module-scala_2.12-2.12.3.jar:/opt/spark/jars/jsr305-3.0.0.jar:/opt/spark/jars/jersey-container-servlet-2.34.jar:/opt/spark/jars/hadoop-client-api-3.3.1.jar:/opt/spark/jars/kubernetes-model-flowcontrol-5.4.1.jar:/opt/spark/jars/spark-unsafe_2.12-3.2.1.jar:/opt/spark/jars/jakarta.inject-2.6.1.jar:/opt/spark/jars/chill_2.12-0.10.0.jar:/opt/spark/jars/hadoop-client-runtime-3.3.1.jar:/opt/spark/jars/JLargeArrays-1.5.jar:/opt/spark/jars/kubernetes-client-5.4.1.jar:/opt/spark/jars/arrow-memory-netty-2.0.0.jar:/opt/spark/jars/spark-streaming_2.12-3.2.1.jar:/opt/spark/jars/zookeeper-jute-3.6.2.jar:/opt/spark/jars/commons-codec-1.15.jar:/opt/spark/jars/datanucleus-api-jdo-4.2.4.jar:/opt/spark/jars/algebra_2.12-2.0.1.jar:/opt/spark/jars/stax-api-1.0.1.jar:/opt/spark/jars/kubernetes-model-apps-5.4.1.jar:/opt/spark/jars/httpcore-4.4.14.jar
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:os.version=4.18.0-305.3.1.el8_4.x86_64
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:user.name=kyuubi
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/kyuubi
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/kyuubi/work/anonymous
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:os.memory.free=133MB
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:os.memory.max=910MB
2022-06-22 21:36:56.720 INFO zookeeper.ZooKeeper: Client environment:os.memory.total=366MB
2022-06-22 21:36:56.723 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=zookeeper-connect.kafka.svc:2181 sessionTimeout=60000 watcher=org.apache.kyuubi.shade.org.apache.curator.ConnectionState@37816ea6
2022-06-22 21:36:56.727 INFO common.X509Util: Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2022-06-22 21:36:56.731 INFO zookeeper.ClientCnxnSocket: jute.maxbuffer value is 1048575 Bytes
2022-06-22 21:36:56.737 INFO zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled=false
2022-06-22 21:36:56.744 INFO client.EngineServiceDiscovery: Service[EngineServiceDiscovery] is initialized.
2022-06-22 21:36:56.744 INFO spark.SparkThriftBinaryFrontendService: Service[SparkThriftBinaryFrontendService] is initialized.
2022-06-22 21:36:56.745 INFO spark.SparkSQLEngine: Service[SparkSQLEngine] is initialized.
2022-06-22 21:36:56.748 INFO zookeeper.ClientCnxn: Opening socket connection to server zookeeper-connect.kafka.svc/10.10.179.151:2181.
2022-06-22 21:36:56.748 INFO zookeeper.ClientCnxn: SASL config status: Will not attempt to authenticate using SASL (unknown error)
2022-06-22 21:36:56.749 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.17.1.205:48112, server: zookeeper-connect.kafka.svc/10.10.179.151:2181
2022-06-22 21:36:56.751 INFO operation.SparkSQLOperationManager: Service[SparkSQLOperationManager] is started.
2022-06-22 21:36:56.751 INFO session.SparkSQLSessionManager: Service[SparkSQLSessionManager] is started.
2022-06-22 21:36:56.751 INFO spark.SparkSQLBackendService: Service[SparkSQLBackendService] is started.
2022-06-22 21:36:56.755 INFO zookeeper.ClientCnxn: Session establishment complete on server zookeeper-connect.kafka.svc/10.10.179.151:2181, session id = 0x1000000d4d500b0, negotiated timeout = 40000
2022-06-22 21:36:56.765 INFO state.ConnectionStateManager: State change: CONNECTED
2022-06-22 21:36:56.767 INFO client.EngineServiceDiscovery: Zookeeper client connection state changed to: CONNECTED
2022-06-22 21:36:56.804 INFO client.ServiceDiscovery: Created a /kyuubi_USER_SPARK_SQL/anonymous/default/serviceUri=kyuubi-server-6964b97f8-xf6bg:40303;version=1.4.1-incubating;refId=c379bd55-a8fe-4838-8823-e4c2c7fb970b;sequence=0000000036 on ZooKeeper for KyuubiServer uri: kyuubi-server-6964b97f8-xf6bg:40303
2022-06-22 21:36:56.806 INFO client.EngineServiceDiscovery: Service[EngineServiceDiscovery] is started.
2022-06-22 21:36:56.807 INFO spark.SparkThriftBinaryFrontendService: Service[SparkThriftBinaryFrontendService] is started.
2022-06-22 21:36:56.807 INFO spark.SparkSQLEngine: Service[SparkSQLEngine] is started.
2022-06-22 21:36:56.807 INFO spark.SparkThriftBinaryFrontendService: Starting and exposing JDBC connection at: jdbc:hive2://kyuubi-server-6964b97f8-xf6bg:40303/
2022-06-22 21:36:56.814 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1aea759d{/kyuubi,null,AVAILABLE,@Spark}
2022-06-22 21:36:56.814 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5ead245{/kyuubi/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:56.815 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1cc42abe{/kyuubi/session,null,AVAILABLE,@Spark}
2022-06-22 21:36:56.815 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@138d978e{/kyuubi/session/json,null,AVAILABLE,@Spark}
2022-06-22 21:36:56.826 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1bfb60b7{/kyuubi/stop,null,AVAILABLE,@Spark}
2022-06-22 21:36:56.832 INFO spark.SparkSQLEngine: 
    Spark application name: kyuubi_USER_SPARK_SQL_anonymous_default_c379bd55-a8fe-4838-8823-e4c2c7fb970b
          application ID:  spark-application-1655922981028
          application web UI: http://kyuubi-hs.kyuubi-test.svc:35437
          master: k8s://https://kubernetes.default.svc
          version: 3.2.1
          driver: [cpu: 1, mem: 1g]
          executor: [cpu: 2, mem: 1g, maxNum: 2]
    Start time: Wed Jun 22 21:36:19 EEST 2022
    
    User: anonymous (shared mode: USER)
    State: LATENT
    
2022-06-22 21:36:57.808 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on kyuubi-hs.kyuubi-test.svc:36477 in memory (size: 3.5 KiB, free: 366.3 MiB)
2022-06-22 21:36:57.816 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on 172.17.4.68:33413 in memory (size: 3.5 KiB, free: 413.9 MiB)
2022-06-22 21:36:57.820 INFO spark.SparkThriftBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10
2022-06-22 21:36:57.825 INFO session.SparkSQLSessionManager: Opening session for anonymous@172.17.1.205
2022-06-22 21:36:57.977 INFO session.SparkSQLSessionManager: anonymous's session with SessionHandle [904b6fb6-bd6d-4aa2-8ee3-acaffa133820] is opened, current opening sessions 1

Kyuubi Server Configurations

kyuubi.authentication           NONE
spark.master                                        k8s://https://kubernetes.default.svc
spark.driver.host                                   kyuubi-hs.kyuubi-test.svc
spark.driver.port                                   44104
spark.kubernetes.driver.pod.name                    $HOSTNAME
spark.kubernetes.container.image                    spark:spark3.2.1-hadoop3.2-delta1.2.1-scala2.12
spark.kubernetes.namespace                          kyuubi-test
spark.kubernetes.authenticate.serviceAccountName    kyuubi-spark
spark.decommission.enabled                          true

#delta
spark.sql.extensions                                io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog                     org.apache.spark.sql.delta.catalog.DeltaCatalog

kyuubi.frontend.thrift.binary.bind.host     0.0.0.0
kyuubi.frontend.thrift.binary.bind.port     10009
kyuubi.session.idle.timeout                 PT5S
#
kyuubi.ha.zookeeper.quorum  zookeeper-connect.kafka.svc:2181

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Answered by pan3793

Jun 23, 2022

beeline -u jdbc:hive2://localhost:10009/default;#spark.sql.shuffle.partitions=2;spark.executor.memory=5g

You should quote the JDBC url by ', and we recommend placing spark confs after ? instead of #

Ref: https://kyuubi.apache.org/docs/latest/client/hive_jdbc.html

View full answer

@hanna-liashchuk · 2022-06-22T18:43:44Z

github-actions[bot]
bot Jun 22, 2022

Hello @hanna-liashchuk,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi (Incubating).

0 replies

pan3793 · 2022-06-23T01:40:39Z

pan3793
Jun 23, 2022
Collaborator

beeline -u jdbc:hive2://localhost:10009/default;#spark.sql.shuffle.partitions=2;spark.executor.memory=5g

You should quote the JDBC url by ', and we recommend placing spark confs after ? instead of #

Ref: https://kyuubi.apache.org/docs/latest/client/hive_jdbc.html

0 replies

hanna-liashchuk · 2022-06-23T12:37:50Z

hanna-liashchuk
Jun 23, 2022
Author

Indeed, seemed that I tried all the combinations of quotes and ?# and it still was not working.
Thanks and sorry for disturbing you :)

1 reply

cxzl25 Jun 23, 2022
Collaborator

You can try this.

beeline -u 'jdbc:hive2://localhost:10009/default;#spark.sql.shuffle.partitions=2;spark.executor.memory=5g'

luohaifang · 2022-11-03T07:45:32Z

luohaifang
Nov 3, 2022

hello，How to solve the problem，this is a bug?

3 replies

pan3793 Nov 3, 2022
Collaborator

The question has been got answered, I think the answer should be clear enough.

imperio-wxm Dec 20, 2022

I also reported the same error when I used zk as url.
Is there something wrong with my url? Oddly, does not image normal execution.
jdbc:kyuubi://127.0.0.1:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;user=tom;kyuubi.engine.pool.size=2;

14:42:31.115 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - org.apache.hadoop.security.ShellBasedUnixGroupsMapping$PartialGroupNameException: The user name 'tom' is not found. id: tom: no such user
14:42:31.115 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - id: tom: no such user
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:294) ~[hadoop-client-api-3.3.4.jar:?]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:207) ~[hadoop-client-api-3.3.4.jar:?]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97) ~[hadoop-client-api-3.3.4.jar:?]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51) ~[hadoop-client-api-3.3.4.jar:?]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:387) ~[hadoop-client-api-3.3.4.jar:?]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:321) ~[hadoop-client-api-3.3.4.jar:?]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:270) ~[hadoop-client-api-3.3.4.jar:?]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) ~[hadoop-client-runtime-3.3.4.jar:1.1.1]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) ~[hadoop-client-runtime-3.3.4.jar:1.1.1]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155) ~[hadoop-client-runtime-3.3.4.jar:1.1.1]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) ~[hadoop-client-runtime-3.3.4.jar:1.1.1]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache.get(LocalCache.java:3962) ~[hadoop-client-runtime-3.3.4.jar:1.1.1]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3985) ~[hadoop-client-runtime-3.3.4.jar:1.1.1]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.thirdparty.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4946) ~[hadoop-client-runtime-3.3.4.jar:1.1.1]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.security.Groups.getGroups(Groups.java:228) ~[hadoop-client-api-3.3.4.jar:?]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1734) ~[hadoop-client-api-3.3.4.jar:?]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1722) ~[hadoop-client-api-3.3.4.jar:?]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.kyuubi.session.HadoopGroupProvider.groups(HadoopGroupProvider.scala:36) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.kyuubi.session.HadoopGroupProvider.primaryGroup(HadoopGroupProvider.scala:33) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.kyuubi.session.KyuubiSessionImpl.engine$lzycompute(KyuubiSessionImpl.scala:83) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.kyuubi.session.KyuubiSessionImpl.engine(KyuubiSessionImpl.scala:80) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.kyuubi.session.KyuubiSessionImpl.renewEngineCredentials(KyuubiSessionImpl.scala:223) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.kyuubi.session.KyuubiSessionImpl.engineCredentials$lzycompute(KyuubiSessionImpl.scala:78) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.kyuubi.session.KyuubiSessionImpl.engineCredentials(KyuubiSessionImpl.scala:78) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:128) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:126) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) ~[kyuubi-ha_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:126) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_352]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_352]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_352]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_352]
14:42:31.116 [engine-launch-log] INFO org.apache.kyuubi.jdbc.hive.KyuubiConnection - 	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_352]

pan3793 Dec 21, 2022
Collaborator

The "org.apache.hadoop.security.ShellBasedUnixGroupsMapping$PartialGroupNameException: The user name 'tom' is not found. id: tom: no such user" is not a critical error, it does not break the query execution.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Question] Spark Configurations Via JDBC Connection URL #2937

{{title}}

Replies: 4 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[Question] Spark Configurations Via JDBC Connection URL #2937

hanna-liashchuk Jun 22, 2022

Code of Conduct

Search before asking

Describe the bug

Affects Version(s)

Kyuubi Server Log Output

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

Are you willing to submit PR?

Replies: 4 comments · 4 replies

github-actions[bot] bot Jun 22, 2022

pan3793 Jun 23, 2022 Collaborator

hanna-liashchuk Jun 23, 2022 Author

cxzl25 Jun 23, 2022 Collaborator

luohaifang Nov 3, 2022

pan3793 Nov 3, 2022 Collaborator

imperio-wxm Dec 20, 2022

pan3793 Dec 21, 2022 Collaborator

hanna-liashchuk
Jun 22, 2022

Replies: 4 comments 4 replies

github-actions[bot]
bot Jun 22, 2022

pan3793
Jun 23, 2022
Collaborator

hanna-liashchuk
Jun 23, 2022
Author

cxzl25 Jun 23, 2022
Collaborator

luohaifang
Nov 3, 2022

pan3793 Nov 3, 2022
Collaborator

pan3793 Dec 21, 2022
Collaborator