AES Encryption Issue with YARN Service in Cloudera CDP Private Cloud 7.1.9.3 & freeIPA

[bacikerik]

I recently deployed a new test cluster using Cloudera CDP Private Cloud version 7.1.9.3 (later with hotfix 7.1.9.4) with Cloudera Manager version 7.11.3. The deployment includes a configuration using freeIPA for secure authentication:

$ ansible-galaxy collection list freeipa.ansible_freeipa

# /home/ebacik/.ansible/collections/ansible_collections
Collection              Version
----------------------- -------
freeipa.ansible_freeipa 1.12.1

However, I was encountering an issue with the YARN service related to AES encryption, specifically 'aes256-cts-hmac-sha384-192' encryption type, which caused problems with service principals:

WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: Authentication exception: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP-REQ - AES256 CTS mode with HMAC SHA384-192)

Based on the log message, I found the following article and tried setting permitted_enctypes = aes256-cts aes128-cts in /etc/krb5.conf, which didn't help at all.

After further investigation, I found out that the configuration is managed in the following file:

[root@ipa1 ~]# readlink -f /etc/krb5.conf.d/crypto-policies
/usr/share/crypto-policies/DEFAULT/krb5.txt
[root@ipa1 ~]# cat /usr/share/crypto-policies/DEFAULT/krb5.txt
[libdefaults]
# permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac
permitted_enctypes = aes256-cts aes128-cts

Solution/Executed steps:

1. Stop Cluster

2. Stop CMS

3. modify /usr/share/crypto-policies/DEFAULT/krb5.txt (contains the same enctypes as hdfs.keytab), comment out the original value and set 'aes256-cts aes128-cts' only:

[root@ipa ~]# readlink -f /etc/krb5.conf.d/crypto-policies
/usr/share/crypto-policies/DEFAULT/krb5.txt
[root@ipa ~]# cat /usr/share/crypto-policies/DEFAULT/krb5.txt
[libdefaults]
# permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac
permitted_enctypes = aes256-cts aes128-cts

4. ipactl restart

5. CM -> Administration -> Security -> Select all principals -> Regenerate Selected

6. Start CMS

7. Start Cluster

Conclusion

The configuration using freeIPA playbooks manage permitted_enctypes in /usr/share/crypto-policies/DEFAULT/krb5.txt instead of /etc/krb5.conf as assumed.

Environment Details:

• Cloudera CDP Private Cloud Version: 7.1.9.3 (later with hotfix 7.1.9.4)
• Cloudera Manager Version: 7.11.3
• Security Configuration: freeipa.ansible_freeipa 1.12.1
• Affected Service: YARN

[clevesque]

What version of RHEL?
I have seen this issue in RHEL 8.9 (did not see this in RHEL8.8...where the change in crypto policies & SHA1 hash seems caused kerberos issues
I moved on to RHEL 9.1 and did NOT see this side effect

[bacikerik]

RHEL 8.9, as you mentioned:

# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.9 (Ootpa)

[clevesque]

As of today, Cloudera has not yet done any product testing/certification on RHEL 8.9
Usually RHEL minor upgrades dont impact us so much, but this one def breaks YARN and I don't have any workaround.

See: https://supportmatrix.cloudera.com/
for the official matrix