Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HBASE-27931 Update hadoop.version from 3.3.5 to 3.3.6 #6232

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

HBASE-27931 Update hadoop.version from 3.3.5 to 3.3.6 #6232

wants to merge 2 commits into from
+4 −4

Conversation

stoty
Copy link
Contributor

@stoty stoty commented Sep 9, 2024

No description provided.

@stoty
Copy link
Contributor Author

stoty commented Sep 9, 2024

targeting master and branch-3

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 41s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 4m 30s master passed
+1 💚 compile 10m 55s master passed
+1 💚 spotless 1m 11s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚 mvninstall 4m 34s the patch passed
+1 💚 compile 12m 24s the patch passed
+1 💚 javac 12m 24s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 xmllint 0m 0s No new issues.
+1 💚 hadoopcheck 15m 36s Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚 spotless 0m 54s patch has no errors when running spotless:check.
_ Other Tests _
+1 💚 asflicense 0m 15s The patch does not generate ASF License warnings.
60m 12s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #6232
Optional Tests dupname asflicense javac codespell detsecrets xmllint hadoopcheck spotless compile
uname Linux c8f983a40ea2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 42d0484
Default Java Eclipse Adoptium-17.0.11+9
Max. process+thread count 189 (vs. ulimit of 30000)
modules C: . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/1/console
versions git=2.34.1 maven=3.9.8 xmllint=20913
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 35s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 3m 5s master passed
+1 💚 compile 2m 10s master passed
+1 💚 javadoc 2m 6s master passed
+1 💚 shadedjars 5m 16s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 2m 59s the patch passed
+1 💚 compile 2m 10s the patch passed
+1 💚 javac 2m 10s the patch passed
+1 💚 javadoc 2m 4s the patch passed
+1 💚 shadedjars 5m 16s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚 unit 300m 10s root in the patch passed.
332m 46s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR #6232
Optional Tests javac javadoc unit shadedjars compile
uname Linux d7e2b20c2342 5.4.0-192-generic #212-Ubuntu SMP Fri Jul 5 09:47:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 42d0484
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/1/testReport/
Max. process+thread count 8612 (vs. ulimit of 30000)
modules C: . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/1/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache9
Copy link
Contributor

Apache9 commented Sep 10, 2024

Maybe I missed some discussions on the dev list. Why we want to bump the default hadoop version to 3.3.6? We do not want to support 3.3.5 since it has new CVEs?

@stoty
Copy link
Contributor Author

stoty commented Sep 11, 2024

Yes, I started a discuss thread on the topic a few days ago.

I will reply there to keep the discussion in one place.

@ndimiduk
Copy link
Member

You'll need to make changes in several more places. Search the repo on github for "3.3.5" and you'll get a sense for this:

  • the Jenkinsfiles download and cache specific hadoop build versions
  • the hbase_personality.sh has a hadoopcheck function that builds against myriad hadoop versions
  • the book includes a support matrix, which needs to be updated (on master only) according to these changes.

ndimiduk
ndimiduk requested changes Sep 11, 2024
View reviewed changes
Copy link
Member

@ndimiduk ndimiduk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my previous comment.

@stoty
Copy link
Contributor Author

stoty commented Sep 11, 2024

Thank you @ndimiduk .
Much of that work has already been done.
The docs refers to 3.3.5+, which should already cover 3.3.6, and 3.3.6 is already tested in HADOOPCHECK.

I have replaced 3.3.6 with 3.3.5 in QUICK_HADOOPCHECK, as 3.3.6 should already be tested as the default.
I have also updated the cached version you pointed out.

@Apache-HBase
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/console in case of problems.

1 similar comment
@Apache-HBase
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/console in case of problems.

@stoty
Copy link
Contributor Author

stoty commented Sep 11, 2024

Added a rather rambling summary of my thoughts to the discuss thread, @Apache9 :
https://lists.apache.org/thread/h53ltygosg2gqo0ftdvqngf6h9z6brb1

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 38s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 💚 mvninstall 3m 55s master passed
+1 💚 compile 9m 14s master passed
+1 💚 spotless 0m 45s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚 mvninstall 3m 8s the patch passed
+1 💚 compile 7m 55s the patch passed
+1 💚 javac 7m 55s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 shellcheck 0m 1s No new issues.
+1 💚 xmllint 0m 0s No new issues.
+1 💚 hadoopcheck 11m 34s Patch does not cause any errors with Hadoop 3.3.5 3.4.0.
+1 💚 spotless 0m 44s patch has no errors when running spotless:check.
_ Other Tests _
+1 💚 asflicense 0m 13s The patch does not generate ASF License warnings.
45m 4s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #6232
Optional Tests dupname asflicense codespell detsecrets shellcheck shelldocs spotless javac xmllint hadoopcheck compile
uname Linux d93118e90735 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / e8a70d9
Default Java Eclipse Adoptium-17.0.11+9
Max. process+thread count 192 (vs. ulimit of 30000)
modules C: . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/console
versions git=2.34.1 maven=3.9.8 shellcheck=0.8.0 xmllint=20913
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 28s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚 mvninstall 3m 1s master passed
+1 💚 compile 2m 2s master passed
+1 💚 javadoc 1m 53s master passed
+1 💚 shadedjars 5m 33s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 2m 51s the patch passed
+1 💚 compile 2m 4s the patch passed
+1 💚 javac 2m 4s the patch passed
+1 💚 javadoc 1m 52s the patch passed
+1 💚 shadedjars 5m 28s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚 unit 289m 11s root in the patch passed.
320m 58s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR #6232
Optional Tests javac javadoc unit shadedjars compile
uname Linux 8b037ce5be12 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / e8a70d9
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/testReport/
Max. process+thread count 8913 (vs. ulimit of 30000)
modules C: . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6232/2/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk
ndimiduk reviewed Sep 16, 2024
View reviewed changes
Copy link
Member

@ndimiduk ndimiduk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You've resolved my comments, but Duo's post to the mailing list thread on 11 Sept is a valid point about insufficient verification with existing hadoop checks. We need to either run a reasonably thorough test suite against the older versions or we need to explicitly drop support for them. Let's continue discussing on the mailing list.

@ndimiduk ndimiduk self-requested a review September 16, 2024 10:16
@Apache9
Copy link
Contributor

Apache9 commented Sep 16, 2024

You've resolved my comments, but Duo's post to the mailing list thread on 11 Sept is a valid point about insufficient verification with existing hadoop checks. We need to either run a reasonably thorough test suite against the older versions or we need to explicitly drop support for them. Let's continue discussing on the mailing list.

Thanks Nick. For me, I think we could at least extend the current client integration tests, to run multiple combinations for client and server versions. We could continue the discussion on the dev mailing list.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Apache9 Apache9 Awaiting requested review from Apache9

@ndimiduk ndimiduk Awaiting requested review from ndimiduk

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@stoty @Apache-HBase @Apache9 @ndimiduk