Kyuubi - Kerberos Authentication #1432
-
|
The data access flow is DBeaver -> Hive JDBC -> Kyuubi -> Spark-SQL engine -> Delta Lake tables. It works without Authentication enabled. I would like to enable Authentication/Authorization and I tried Kyuubi documentation (1.3.0) to enable Kerberos authentication on Mac OS but it is not working. In the MacOS, both Kerberos server and client are configured and running. Please, can someone provide the requirements for Kerberos integration with Kyuubi? Also, share the details or steps is highly appreciated. |
Beta Was this translation helpful? Give feedback.
Replies: 7 comments 24 replies
-
|
hi @mmuru could you provide the url you used to access kyuubi with kerberos authentication? Does it format like |
Beta Was this translation helpful? Give feedback.
-
|
I see that there is some doc for integrate DBeaver with kerberos enabled HiveServer |
Beta Was this translation helpful? Give feedback.
-
|
@turboFei: Thanks for the information. When I start bin/kyuubi run and throws the following exception Please, Let me know if you need more information. |
Beta Was this translation helpful? Give feedback.
-
|
@mmuru On MacOS the default client does not fall back to TCP. in your krb.conf prefix your kdc with tcp/ to force the client to use TCP if your network blocks UPD traffic (As some network admins might do). please modify your krb5.conf as below: |
Beta Was this translation helpful? Give feedback.
-
|
In fact, I met this issue before, #1291 |
Beta Was this translation helpful? Give feedback.
-
|
hi @mmuru , do your |
Beta Was this translation helpful? Give feedback.
-
|
@ulysses-you: I don't have core-site.xml file but I've hive-site.xml file in $SPARK_HOME/conf. Kyuubi uses the default kyuubi-env.sh file. Do I need to set it in kyuubi-env.sh? Please, can you share sample core-site.xml file? |
Beta Was this translation helpful? Give feedback.
-
|
@mmuru You can try with these steps:
|
Beta Was this translation helpful? Give feedback.
-
seems that you manually logged in with Maybe you can use a separate OS user to do client-side authentication and other operations |
Beta Was this translation helpful? Give feedback.
-
|
After with core-site.xml, Kyuubi server starts with Kerberos authentication enabled (see log file below) When I tried to connect from DBeaver, getting the below error Here is my DBeaver connection URL Please, let me know what am I missing here? |
Beta Was this translation helpful? Give feedback.
-
|
it seems that the client uses PLAIN type authentication instead of kerberos. |
Beta Was this translation helpful? Give feedback.
-
|
the server side now works well. |
Beta Was this translation helpful? Give feedback.
-
|
@turboFei: When I start Kyuubi server with Kerbeos authentication enabled, the following shows in the log Please, note jdbc:hive2://192.168.1.69:10009/ why Kyuubi didn't add principal part? In DBeaver side, I tried jdbc:hive2://192.168.1.69:10009/;principal=spark/[email protected];auth=KERBEROS but it didn't even hitting Kyuubi server. Please, can you provide the DBeaver steps? I tried google suggested steps and no of them working. Thanks for your help. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @mmuru I just solved an issue when trying to connect to kerberized kyuubi. Hope this will also be helpful for you. ProblemAt first, I used following connection settings to connect to kyuubi. And result in Kyuubi server log looks like this: Solution
Executing
EnvironmentOS: Ubuntu 20.04 ReferenceHere is an article that explains this JVM argument |
Beta Was this translation helpful? Give feedback.
-
|
Another solution is create a jar with a And add it to DBeaver's Restarting DBeaver is also required. I prefer this solution as it keeps JVM default GSS initiate behaviour. |
Beta Was this translation helpful? Give feedback.
-
|
@zhouyifan279: Awesome, thanks for the solution. I had to add this java property in dbeaver.ini and restart DBeaver. However, I can not use the two different users. i.e Kyuubi started with spark/[email protected] and in DBeaver I tried with another user muru/[email protected]. Kyuubi throws checksum exception. Kyuubi server is started with Kerberos user has token will expire after 1 day. Does Kyuubi server automatically renewal the token? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. If principal and keytab is in |
Beta Was this translation helpful? Give feedback.
-
If so, we can add environment viariable
Besides, ensure your default TGT cache file (
|
Beta Was this translation helpful? Give feedback.
-
|
If problem still exists, please enable GSS logging and provide the log. Enable GSS logging for Kyuubi
Enable GSS logging for DBeaver
|
Beta Was this translation helpful? Give feedback.
-
|
@mmuru Here is a conceptual graph about Kyuubi kerberos authentication
So when client and server are running with same OS user, please set environment variable |
Beta Was this translation helpful? Give feedback.
-
|
@zhouyifan279: Thanks for the clarification and the authentication flow diagram was very helpful. I had KRB5CCNAME environment variable set in kyuubi-env.sh. In Mac OS, my OS user cache is in KCM. Now, it works as expected. Thanks again for your help. |
Beta Was this translation helpful? Give feedback.
-
@zhouyifan279 As the graph shows, kyuubi does not need KDC serverce? thanks |
Beta Was this translation helpful? Give feedback.
-
@jia-zhengwei Generally, |
Beta Was this translation helpful? Give feedback.
-
Got it, thanks, @pan3793 . |
Beta Was this translation helpful? Give feedback.
-
|
Glad to hear that the problem has been solved, the Q&A here is wonderful and maybe we can summarise some and make a useful guide to others in our codebase. WDYT @mmuru @zhouyifan279 |
Beta Was this translation helpful? Give feedback.
-
|
@yaooqinn Glad to do it. |
Beta Was this translation helpful? Give feedback.
Another solution is create a jar with a
core-site.xmlfile with content:And add it to DBeaver's
Global librariesRestarting DBeaver is also required.
I prefer this solution as it keeps JVM default GSS initiate behaviour.