HDDS-11450. Avoid managing versions for transitive-only dependencies

[adoroszlai]

What changes were proposed in this pull request?

Ozone should not override version of transitive dependencies which are not used directly. E.g. re2j via Hadoop, is only used by Hadoop. Bumping its version in Ozone may cause incompatibilities at runtime.

Currently:

• When building each Ozone component, its dependencies are listed in a file, to be used at runtime as classpath.
• Dependencies are copied to the share/ozone/lib directory when building the ozone-dist module. It depends on all other Ozone modules directly or transitively. Since ozone-dist is a single module, third-party transitive dependency versions are resolved by Maven to a single version. Only that version is copied to share/ozone/lib. However, various Ozone components may depend on different versions of the same third-party module. These different versions can end up in the classpath files, but will not be found at runtime.
• To fix the problem of missing jars due to different versions, Ozone started to manage versions of third-party dependencies (HDDS-7372, but even before that to some degree). Now dependency versions may be changed independently, and version of transitive dependency may be too old or too new for the direct dependency.
  • New version of some dependency bumps its own dependencies, but we keep using older versions of the transitive dependencies.
  • We bump transitive dependency version without bumping the direct dependency that requires it.
    Both of these could result in incompatibilities, which may go unnoticed in CI.

This PR proposes to:

• Allow multiple versions of the same dependency by copying them when building each module. dist-layout-stitching copies dependencies from per-module directories to the final location when building ozone-dist.
• Remove some transitive-only dependencies. Some exclusions are necessary to resolve dependency convergence errors.
• Keep dependencies upgraded due to security issues.

Dependency changes:

@@ -3,6 +3,7 @@
 annotations-4.1.1.4.jar
 aopalliance-1.0.jar
 aopalliance-repackaged-2.6.1.jar
+asm-9.1.jar
 asm-9.2.jar
 asm-analysis-9.2.jar
 asm-commons-9.2.jar
@@ -39,7 +40,7 @@
 curator-framework-4.2.0.jar
 derby-10.14.2.0.jar
 disruptor-3.4.4.jar
-dnsjava-2.1.9.jar
+dnsjava-2.1.7.jar
 error_prone_annotations-2.29.2.jar
 failureaccess-1.0.1.jar
 gethostname4j-0.0.2.jar
@@ -62,7 +63,7 @@
 hadoop-common-3.3.6.jar
 hadoop-hdfs-3.3.6.jar
 hadoop-hdfs-client-3.3.6.jar
-hadoop-shaded-guava-1.2.0.jar
+hadoop-shaded-guava-1.1.1.jar
 hadoop-shaded-protobuf_3_7-1.1.1.jar
 hdds-annotation-processing-1.5.0-SNAPSHOT.jar
 hdds-client-1.5.0-SNAPSHOT.jar
@@ -158,7 +159,7 @@
 jnr-ffi-2.2.16.jar
 jnr-posix-3.1.19.jar
 jnr-x86asm-1.0.2.jar
-joda-time-2.12.7.jar
+joda-time-2.10.6.jar
 jooq-3.11.10.jar
 jooq-codegen-3.11.10.jar
 jooq-meta-3.11.10.jar
@@ -254,7 +255,7 @@
 ratis-server-api-3.1.0.jar
 ratis-thirdparty-misc-1.0.6.jar
 ratis-tools-3.1.0.jar
-re2j-1.7.jar
+re2j-1.1.jar
 reflections-0.10.2.jar
 reload4j-1.2.25.jar
 rocksdb-checkpoint-differ-1.5.0-SNAPSHOT.jar
@@ -271,11 +272,12 @@
 spring-jdbc-5.3.39.jar
 spring-tx-5.3.39.jar
 sqlite-jdbc-3.46.1.0.jar
-stax2-api-4.2.2.jar
+stax2-api-4.2.1.jar
 txw2-2.3.9.jar
 vault-java-driver-5.1.0.jar
 weld-servlet-shaded-3.1.9.Final.jar
 woodstox-core-5.4.0.jar
+woodstox-core-6.5.1.jar
 zookeeper-3.8.4.jar
 zookeeper-jute-3.8.4.jar
 zstd-jni-1.5.6-4.jar

https://issues.apache.org/jira/browse/HDDS-11450

How was this patch tested?

CI:
https://github.com/adoroszlai/ozone/actions/runs/10825596616

[adoroszlai]

Additionally, we could also get rid of the dependency convergence check to avoid the need for exclusions (see adoroszlai@bba79f2).

[adoroszlai]

Extracted part of this to #7325.

[adoroszlai]

I plan to create separate PR for each dependency for easier review.