HADOOP-19259. ugrade to jackson 2.18.0 #7022

pjfanning · 2024-09-01T21:28:04Z

Description of PR

Use 2.18.0-rc1 until full release is made. Not for merge until 2.18.0 is out.

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2024-09-02T10:53:10Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 10s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 45s		trunk passed
+1 💚	compile	9m 0s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	8m 30s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	mvnsite	15m 40s		trunk passed
+1 💚	javadoc	5m 2s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	4m 59s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	shadedclient	30m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 35s		Maven dependency ordering for patch
-1 ❌	mvninstall	9m 36s	/patch-mvninstall-root.txt	root in the patch failed.
+1 💚	compile	8m 46s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	8m 46s		the patch passed
+1 💚	compile	8m 20s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	8m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	12m 16s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	4m 56s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	5m 14s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
-1 ❌	shadedclient	22m 21s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	640m 15s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 6s		The patch does not generate ASF License warnings.
		804m 4s

Reason	Tests
Failed junit tests	hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesContainers
	hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesAuxServices
	hadoop.yarn.server.nodemanager.webapp.TestNMWebServices
	hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesApps
	hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestRuncContainerRuntime
	hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem
	hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7022/1/artifact/out/Dockerfile
GITHUB PR	#7022
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 6d24d7ce68ff 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `675c5ad`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7022/1/testReport/
Max. process+thread count	4607 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7022/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2024-09-02T13:30:46Z

some of those tests have jackson in the stack trace...

Caused by: com.fasterxml.jackson.core.JsonParseException: localhost:44185: Unrecognized token 'HdfsLocatedFileStatus': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 33]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:850)
	... 56 more

…/versions/21 classes

hadoop-yetus · 2024-09-03T07:29:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 3s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 40s		trunk passed
+1 💚	compile	9m 22s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	9m 14s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	mvnsite	16m 23s		trunk passed
+1 💚	javadoc	5m 7s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	5m 1s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	shadedclient	29m 46s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	18m 8s		the patch passed
+1 💚	compile	8m 42s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	8m 42s		the patch passed
+1 💚	compile	8m 28s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	8m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	12m 13s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	4m 56s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	4m 58s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	shadedclient	30m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	642m 26s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	0m 53s		The patch does not generate ASF License warnings.
		826m 15s

Reason	Tests
Failed junit tests	hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesContainers
	hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesAuxServices
	hadoop.yarn.server.nodemanager.webapp.TestNMWebServices
	hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesApps
	hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestRuncContainerRuntime
	hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem
	hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7022/2/artifact/out/Dockerfile
GITHUB PR	#7022
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 368e12666bf7 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `aaed920`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7022/2/testReport/
Max. process+thread count	4349 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7022/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-09-03T12:17:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 2s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 34s		trunk passed
+1 💚	compile	9m 51s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	8m 59s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	2m 31s		trunk passed
+1 💚	mvnsite	19m 40s		trunk passed
+1 💚	javadoc	5m 6s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	4m 56s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+0 🆗	spotbugs	0m 12s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
-1 ❌	spotbugs	18m 39s	/branch-spotbugs-root-warnings.html	root in trunk has 2 extant spotbugs warnings.
+1 💚	shadedclient	41m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 35s		Maven dependency ordering for patch
+1 💚	mvninstall	22m 42s		the patch passed
+1 💚	compile	10m 27s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	10m 27s		the patch passed
+1 💚	compile	9m 24s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	9m 24s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
+1 💚	checkstyle	3m 0s		the patch passed
+1 💚	mvnsite	11m 27s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	5m 13s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	4m 55s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+0 🆗	spotbugs	0m 11s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	42m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	634m 57s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 7s		The patch does not generate ASF License warnings.
		886m 24s

Reason	Tests
Failed junit tests	hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesContainers
	hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesAuxServices
	hadoop.yarn.server.nodemanager.webapp.TestNMWebServices
	hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesApps
	hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestRuncContainerRuntime

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7022/3/artifact/out/Dockerfile
GITHUB PR	#7022
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 576981244d06 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b34b9d1`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7022/3/testReport/
Max. process+thread count	4217 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-hdfs-project/hadoop-hdfs-client . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7022/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

pjfanning · 2024-09-03T12:57:51Z

this is not ready yet - the latest issue breaking some unit tests is happening with jettison - it seems like the upgraded jackson may be serializing some messages differently - and this needs investigation

steveloughran

might be good to isolate maven change to its own PR...we can merge in earlier, see what surprises surface etc

steveloughran · 2024-09-04T12:34:05Z

...s-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java

+            IOException newIoe = null;
+            if (cause != null) {
+              try {
+                ioe.getClass().getConstructor(String.class, Throwable.class)


our new DynMethods classes (copied from Parquet) may help here. If they don't that's an opportunity for improvement

pjfanning · 2024-09-04T12:56:16Z

might be good to isolate maven change to its own PR...we can merge in earlier, see what surprises surface etc

I'll create a separate PR for that. I'm relatively pessimistic about this PR generally. The exception handling changes fixed one set of issues but there are other issues that appear to suggest that Jersey 1 does not work well with this version of Jackson.

use jackson 2.18.0-rc1

675c5ad

pjfanning marked this pull request as draft September 1, 2024 21:28

github-actions bot added build trunk labels Sep 1, 2024

pjfanning added 2 commits September 2, 2024 18:41

update maven-shade-plugin because older version can't handle META-INF…

aaed920

…/versions/21 classes

try to make exception manipulation more robust

b34b9d1

github-actions bot added the HDFS label Sep 2, 2024

steveloughran requested changes Sep 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-19259. ugrade to jackson 2.18.0 #7022

HADOOP-19259. ugrade to jackson 2.18.0 #7022

pjfanning commented Sep 1, 2024

hadoop-yetus commented Sep 2, 2024

steveloughran commented Sep 2, 2024

hadoop-yetus commented Sep 3, 2024

hadoop-yetus commented Sep 3, 2024

pjfanning commented Sep 3, 2024 •

edited

Loading

steveloughran left a comment

steveloughran Sep 4, 2024

pjfanning commented Sep 4, 2024

HADOOP-19259. ugrade to jackson 2.18.0 #7022

Are you sure you want to change the base?

HADOOP-19259. ugrade to jackson 2.18.0 #7022

Conversation

pjfanning commented Sep 1, 2024

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Sep 2, 2024

steveloughran commented Sep 2, 2024

hadoop-yetus commented Sep 3, 2024

hadoop-yetus commented Sep 3, 2024

pjfanning commented Sep 3, 2024 • edited Loading

steveloughran left a comment

Choose a reason for hiding this comment

steveloughran Sep 4, 2024

Choose a reason for hiding this comment

pjfanning commented Sep 4, 2024

pjfanning commented Sep 3, 2024 •

edited

Loading