YARN-11916. FileSystemTimelineReaderImpl vulnerable to race conditions #8164

steveloughran · 2026-01-05T18:09:52Z

Move the class. It was never public API.
Added the test artifact at test scope to yarn-client for its tests.
Javadocs make really, really clear this isn't for production.
General resilience hardening and tests while there.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

Move the class. It was never public API. added the test artifact at test scope to yarn-client for its tests.

cnauroth

+1. Thank you @steveloughran . I flagged a minor nitpick on some comment text. Feel free to fix that when you commit. (No need for second review IMO.)

cnauroth · 2026-01-05T18:13:35Z

...java/org/apache/hadoop/yarn/server/timelineservice/storage/FileSystemTimelineWriterImpl.java

+ * To implement an atomic append it reads all the data in the original file,
+ * writes that to a temporary file, appends the new
+ * data there and renames that temporary file to the original path.
+ * This is makes the update operation slower and slower the longer an application runs.


Nitpick: remove the word "is."

hadoop-yetus · 2026-01-05T21:23:04Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	15m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	10m 39s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 45s		trunk passed
+1 💚	compile	7m 6s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	6m 45s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	1m 31s		trunk passed
+1 💚	mvnsite	1m 40s		trunk passed
+1 💚	javadoc	1m 34s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 30s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	spotbugs	2m 27s		trunk passed
+1 💚	shadedclient	27m 28s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	27m 55s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 52s		the patch passed
+1 💚	compile	6m 14s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	6m 14s		the patch passed
+1 💚	compile	6m 44s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	6m 44s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 34s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt	hadoop-yarn-project/hadoop-yarn: The patch generated 11 new + 1 unchanged - 0 fixed = 12 total (was 1)
+1 💚	mvnsite	1m 32s		the patch passed
+1 💚	javadoc	0m 41s		hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 0 new + 223 unchanged - 6 fixed = 223 total (was 229)
+1 💚	javadoc	0m 41s		hadoop-yarn-client in the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
+1 💚	javadoc	0m 39s		hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 0 new + 213 unchanged - 6 fixed = 213 total (was 219)
+1 💚	javadoc	0m 43s		hadoop-yarn-client in the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
+1 💚	spotbugs	2m 38s		the patch passed
+1 💚	shadedclient	27m 48s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 53s		hadoop-yarn-server-timelineservice in the patch passed.
+1 💚	unit	27m 53s		hadoop-yarn-client in the patch passed.
+1 💚	asflicense	1m 4s		The patch does not generate ASF License warnings.
		191m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8164/1/artifact/out/Dockerfile
GITHUB PR	#8164
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux 64f4515a5fd8 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `272e2f8`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8164/1/testReport/
Max. process+thread count	635 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8164/1/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2026-01-05T21:23:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 9s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	10m 22s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 29s		trunk passed
+1 💚	compile	7m 6s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	6m 44s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	1m 31s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 33s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 29s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	spotbugs	2m 29s		trunk passed
+1 💚	shadedclient	27m 22s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	27m 50s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 51s		the patch passed
+1 💚	compile	6m 19s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	6m 19s		the patch passed
+1 💚	compile	6m 41s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	6m 41s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 34s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt	hadoop-yarn-project/hadoop-yarn: The patch generated 11 new + 1 unchanged - 0 fixed = 12 total (was 1)
+1 💚	mvnsite	1m 32s		the patch passed
+1 💚	javadoc	0m 41s		hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 0 new + 223 unchanged - 6 fixed = 223 total (was 229)
+1 💚	javadoc	0m 43s		hadoop-yarn-client in the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
+1 💚	javadoc	0m 39s		hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 0 new + 213 unchanged - 6 fixed = 213 total (was 219)
+1 💚	javadoc	0m 43s		hadoop-yarn-client in the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
+1 💚	spotbugs	2m 40s		the patch passed
+1 💚	shadedclient	27m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 58s		hadoop-yarn-server-timelineservice in the patch passed.
+1 💚	unit	28m 16s		hadoop-yarn-client in the patch passed.
+1 💚	asflicense	1m 5s		The patch does not generate ASF License warnings.
		176m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8164/2/artifact/out/Dockerfile
GITHUB PR	#8164
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux 7114e268da0d 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `272e2f8`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8164/2/testReport/
Max. process+thread count	769 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8164/2/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao

LGTM. +1 from my side. I am curious if anyone enable it on prod environment?

#8164) Move the class. It was never public API. added the test artifact at test scope to yarn-client for its tests.

…onditions (#8164)" This reverts commit 11e58a4.

…onditions (#8164)" This reverts commit 0dc79ad.

apache#8164) Move the class. It was never public API. added the test artifact at test scope to yarn-client for its tests.

#8164) (#8170) Move the class. It was never public API. added the test artifact at test scope to yarn-client for its tests. branch-3.4 backport adds classic junit as a dependency to hadoop-yarn-server-timelineservice Contributed by Steve Loughran

steveloughran added 2 commits January 5, 2026 16:53

YARN-11916. FileSystemTimelineReaderImpl vulnerable to race conditions

5820e8d

YARN-11916. FileSystemTimelineReaderImpl vulnerable to race conditions

fb9e4dd

Move the class. It was never public API. added the test artifact at test scope to yarn-client for its tests.

github-actions bot added YARN build trunk labels Jan 5, 2026

cnauroth approved these changes Jan 5, 2026

View reviewed changes

YARN-11916. Typo

272e2f8

Hexiaoqiao approved these changes Jan 6, 2026

View reviewed changes

steveloughran merged commit aadeaa0 into apache:trunk Jan 6, 2026
4 checks passed

asf-gitbox-commits pushed a commit that referenced this pull request Jan 6, 2026

YARN-11916. FileSystemTimelineReaderImpl vulnerable to race conditions (

0dc79ad

#8164) Move the class. It was never public API. added the test artifact at test scope to yarn-client for its tests.

steveloughran added a commit that referenced this pull request Jan 6, 2026

YARN-11916. FileSystemTimelineReaderImpl vulnerable to race conditions (

11e58a4

#8164) Move the class. It was never public API. added the test artifact at test scope to yarn-client for its tests.

steveloughran added a commit that referenced this pull request Jan 7, 2026

Revert "YARN-11916. FileSystemTimelineReaderImpl vulnerable to race c…

49cedf9

…onditions (#8164)" This reverts commit 11e58a4.

asf-gitbox-commits pushed a commit that referenced this pull request Jan 7, 2026

Revert "YARN-11916. FileSystemTimelineReaderImpl vulnerable to race c…

665cbcb

…onditions (#8164)" This reverts commit 0dc79ad.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

YARN-11916. FileSystemTimelineReaderImpl vulnerable to race conditions #8164

YARN-11916. FileSystemTimelineReaderImpl vulnerable to race conditions #8164

Uh oh!

steveloughran commented Jan 5, 2026

Uh oh!

cnauroth left a comment

Uh oh!

cnauroth Jan 5, 2026

Uh oh!

hadoop-yetus commented Jan 5, 2026

Uh oh!

hadoop-yetus commented Jan 5, 2026

Uh oh!

Hexiaoqiao left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

YARN-11916. FileSystemTimelineReaderImpl vulnerable to race conditions #8164

YARN-11916. FileSystemTimelineReaderImpl vulnerable to race conditions #8164

Uh oh!

Conversation

steveloughran commented Jan 5, 2026

For code changes:

Uh oh!

cnauroth left a comment

Choose a reason for hiding this comment

Uh oh!

cnauroth Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jan 5, 2026

Uh oh!

hadoop-yetus commented Jan 5, 2026

Uh oh!

Hexiaoqiao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants