Kerberos authentication with Ranger KMS

[maron546]

Hi.

We're trying to integrate Ranger KMS and Ozone. Both have Kerberos enabled but Ozone seems not to require a delegation token while creating a new bucket.

• core-site.xml

<property>  
  <name>hadoop.security.key.provider.path</name>  
  <value>kms://https@xxx:443/kms</value>  
</property>  

• kms-site.xml

<property>  
  <name>hadoop.kms.authentication.type</name>  
  <value>kerberos</value>  
</property>  

Are we missing some properties/configurations? In attachment the creation command with HADOOP_ROOT_LOGGER set to 'DEBUG,console'.
error.log

[jojochuang]

simple commands do not require delegation tokens. They just need Kerberos credentials. This is consistent with Hadoop.

DT is required for MR, Spark jobs because their executors do not have Kerberos credentials. These frameworks issues FileSystem.addDelegationTokens()

BTW, Ozone silently drops acquiring delegation tokens if ozone.security.enabled = false. This is a known issue: https://issues.apache.org/jira/browse/HDDS-10350

[jojochuang]

The message "kinit: KDC can't fulfill requested option while renewing credentials" suggests it's a Kerberos problem.
You probably need to reconfigure KDC to allow kerberos renewal.

Another thing to check is KMS authorization rules. You log in as user 'om' and you need to make sure creating a bucket at xxx-s3g/encrypted is allowed for om.

[maron546]

Sorry, I used wrong terms, I meant that there is no Kerberos ticket for KMS issued by om.
Security is enabled (ozone.security.enabled) and the principal we are using for the command is the one configured in 'ozone.om.kerberos.principal' property. It can create volumes and buckets without any problems.
Furthermore, everything perfectly works if we execute key commands via curl with negotiation.

[jojochuang]

If the curl requires spnego (curl --negotiate -u :) to execute key commands then that means KMS requires Kerberos credentials, and Ozone client can access encrypted buckets that means communication between Ozone and KMS are authenticated without problems.

So what is the problem if everything's working?

[maron546]

We can't even create encrypted buckets due to the following error:

INTERNAL_ERROR org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: https://xxx:443/kms/v1/key/xxx/_metadata

[jojochuang]

If curl with negotiation works that means KMS server is Kerberized properly. So it must be the Ozone CLI configuration issue.
The AuthenticationException unfortunately doesn't show much useful info at client side. Can you check if there are additional messages on the KMS server side? Try start KMS in trace log level. There are a variety of possibilities that can result in this exxception.

[maron546]

Ranger KMS is installed inside an AWS EKS, while the Ozone cluster is on virtual machines.
The authentication problem was caused by either the AWS NLB, or the Nginx ingress. We simply opened a TCP connection and everything is working fine.
Thanks for your help!