[WIP] HADOOP-19785. mvn site fails in JDK17

[aajisaka]

Description of PR

Fix the following error while running mvn site with JDK17

JIRA: HADOOP-19785

Error:  /home/runner/work/hadoop/hadoop/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java:26: error: cannot find symbol
Error:  import com.codahale.metrics.JmxReporter;
Error:                             ^
Error:    symbol:   class JmxReporter
Error:    location: package com.codahale.metrics
Error:  /home/runner/work/hadoop/hadoop/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSWebApp.java:69: error: cannot find symbol
Error:    private JmxReporter jmxReporter;
Error:            ^
Error:    symbol:   class JmxReporter
Error:    location: class KMSWebApp
Error:  /home/runner/work/hadoop/hadoop/hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java:36: error: cannot find symbol
Error:  import org.apache.hadoop.hdfs.MiniDFSCluster;
Error:                               ^
Error:    symbol:   class MiniDFSCluster
Error:    location: package org.apache.hadoop.hdfs
Error:  /home/runner/work/hadoop/hadoop/hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java:41: error: cannot find symbol
Error:  import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
Error:                                               ^
Error:    symbol:   class DataNodeTestUtils
Error:    location: package org.apache.hadoop.hdfs.server.datanode
Error:  /home/runner/work/hadoop/hadoop/hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/SimulatedDataNodes.java:42: error: cannot find symbol
Error:  import org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset;
Error:                                               ^
Error:    symbol:   class SimulatedFSDataset
Error:    location: package org.apache.hadoop.hdfs.server.datanode
Error: [ERROR] 5 errors

How was this patch tested?

Ran the following command and verified the above error has been fixed:

mvn clean install -DskipTests -DskipShade
mvn site

However, the Hadoop custom Doclet is failing due to the change in https://bugs.openjdk.org/browse/JDK-8253736

[ERROR] java.lang.ClassCastException: class jdk.proxy1.$Proxy0 cannot be cast to
 class jdk.javadoc.internal.tool.DocEnvImpl (jdk.proxy1.$Proxy0 is in module jdk.proxy1 of loader 'app'; jdk.javadoc.internal.tool.DocEnvImpl is in module jdk.javadoc of 
loader 'app')
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.WorkArounds.<init>(WorkArounds.java:104)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.BaseConfiguration.initConfiguration(BaseConfiguration.java:251)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.HtmlConfiguration.initConfiguration(HtmlConfiguration.java:220)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.run(AbstractDoclet.java:104)
[ERROR]     at jdk.javadoc/jdk.javadoc.doclet.StandardDoclet.run(StandardDoclet.java:103)
[ERROR]     at org.apache.hadoop.classification.tools.IncludePublicAnnotationsStandardDoclet.run(IncludePublicAnnotationsStandardDoclet.java:152)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Start.parseAndExecute(Start.java:556)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:393)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:342)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Main.execute(Main.java:63)
[ERROR]     at jdk.javadoc/jdk.javadoc.internal.tool.Main.main(Main.java:52)
[ERROR] 2 errors

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

• The PR includes the phrase "Contains content generated by "
  where is the name of the AI tool used.
• My use of AI contributions follows the ASF legal policy
  https://www.apache.org/legal/generative-tooling.html

[aajisaka]

Hi @zhtttylz, you've implemented new custom Doclet to support JDK17 in #8038, however, unfortunately mvn site fails in Doclet part. The detail is below:

1. RootDocProcessor.process(env) creates a Proxy object that implements DocletEnvironment interface
2. This proxy is passed to StandardDoclet.run(filtered)
3. Inside the StandardDoclet, the javadoc internals (specifically WorkArounds.()) try to cast this proxy to the concrete implementation class jdk.javadoc.internal.tool.DocEnvImpl: https://github.com/jonathan-gibbons/jdk/blob/bc1e60c1bf91678ef18652a00aa2ce55b0446caa/src/jdk.javadoc/share/classes/jdk/javadoc/internal/doclets/toolkit/WorkArounds.java#L112
4. The cast fails because a proxy object cannot be cast to a concrete implementation class

It seems our custom doclet implementation is prohibited after https://bugs.openjdk.org/browse/JDK-8253736:

One particularly annoying wart is the cast on DocletEnvironment to DocEnvImpl, which effectively prevents using subtypes to carry additional info. It is not clear (even now) what the best way is to replace that logic.

Now I feel it's becoming really hard to maintain Hadoop's custom Doclets, and therefore I would like to drop the custom implementation. The primary change is we are going to build Hadoop JavaDoc with @LimitedPrivate, @Private or @Unstable classes, which are now excluded by our custom Doclets.

@slfan1989 @cnauroth @zhtttylz What do you think?

[aajisaka]

Added because Hadoop Dynamometer production classes depend on HDFS test jar

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 14s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 2s		trunk passed
+1 💚	compile	19m 2s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	19m 26s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
-1 ❌	mvnsite	10m 10s	/branch-mvnsite-root.txt	root in trunk failed.
-1 ❌	javadoc	9m 7s	/branch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	root in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	javadoc	8m 32s	/branch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	root in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
+1 💚	shadedclient	141m 26s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	34m 27s		the patch passed
+1 💚	compile	18m 20s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	18m 20s		the patch passed
+1 💚	compile	19m 49s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	19m 49s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-1 ❌	mvnsite	8m 16s	/patch-mvnsite-root.txt	root in the patch failed.
-1 ❌	javadoc	7m 15s	/patch-javadoc-root-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	root in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	javadoc	7m 23s	/patch-javadoc-root-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	root in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
+1 💚	shadedclient	57m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	913m 20s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 51s		The patch does not generate ASF License warnings.
		1190m 50s

Reason	Tests
Failed junit tests	hadoop.yarn.service.TestYarnNativeServices
	hadoop.yarn.server.resourcemanager.TestRMHA
	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
	hadoop.yarn.server.router.webapp.TestRouterWebServicesREST
	hadoop.yarn.server.router.subcluster.fair.TestYarnFederationWithFairScheduler
	hadoop.yarn.sls.appmaster.TestAMSimulator
	hadoop.hdfs.tools.TestDFSAdmin
	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList
	hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
	hadoop.mapreduce.v2.app.TestRuntimeEstimators

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/artifact/out/Dockerfile
GITHUB PR	#8182
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux c4c5b5d51744 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 2996b57
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/testReport/
Max. process+thread count	3935 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8182/1/console
versions	git=2.25.1 maven=3.9.11
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.