Health check recommendations

[Lucas12138]

Hi community,

We recently found our application having issues for cache warming, i.e. saw errors like:
2024-08-01 23:39:53,684 | ERROR | S3GAudit | user=XXX | ip=xxx.xxx.xxx.xxx | op=CREATE_MULTIPART_KEY {bucket=[d72cf3e8-2a4e-4794-bcfd-4ad9fd6b7592], path=[CACHE.123/a.caf], uploadId=[ea13c5ff-9504-4ed0-8aee-d153ebec5263-112889363969409050], partNumber=[1]} | ret=FAILURE | INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: DestHost:destPort XXX-ozone-scm-qa-2.XXX-ozone-scm-qa-hs.XXX-qa.svc.cluster.local.:9863 , LocalHost:localPort XXX-ozone-om-qa-2/240.152.241.3:0. Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Hostname must match: XXX-ozone-scm-qa-2.XXX-ozone-scm-qa-hs:9863 vs XXX-ozone-scm-qa-2.XXX-ozone-scm-qa-hs.XXX-qa.svc.cluster.local./xxx.xxx.xxx.xxx:9863

since our health check is just things like nc -vz localhost 9860

The error is not caught by it.

Do you have any recommendations for better health check for each ozone applications, i.e. om, scm, etc.?
Thank you

[kerneltime]

One approach would be update a key as part of the health check. To be able to update a key, OM, SCM and some of the Datanodes need to be healthy.