Skip to content

HADOOP-19605. Upgrade Protobuf 3.25.5 for docker images #7780

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 2 commits into
base: trunk
Choose a base branch
from
Draft

HADOOP-19605. Upgrade Protobuf 3.25.5 for docker images #7780

wants to merge 2 commits into from
+48 −39

Conversation

pan3793
Copy link
Member

@pan3793 pan3793 commented Jul 7, 2025

Description of PR

HADOOP-19289 upgraded protobuf-java 3.25.5, we should use same version for protobuf installed in docker images.

How was this patch tested?

Changes are covered by CI, also manually tested in local

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7780/1/console in case of problems.

pan3793
pan3793 commented Jul 7, 2025
View reviewed changes
dev-support/docker/pkg-resolver/install-protobuf.sh
make install &&
cmake -S . -B build -Dprotobuf_BUILD_TESTS=OFF &&
cmake --build build --parallel $(nproc) &&
cmake --install build --prefix /opt/protobuf &&
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

protocolbuffers/protobuf#10132 (since protobuf 3.22.0) removes autotools support, the supported toolchains are bazel and cmake.

pan3793
pan3793 commented Jul 7, 2025
View reviewed changes
dev-support/docker/vcpkg/vcpkg.json
@@ -10,7 +10,7 @@
"overrides": [
{
"name": "protobuf",
"version": "3.21.12"
"version": "3.25.5"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose this works, but I don't have Windows env to verify this.

cc @GauthamBanasandra

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for PR @pan3793. Please let me know once you're done with all the changes and I can verify it on Windows.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 0s Docker mode activated.
-1 ❌ docker 7m 58s Docker failed to build run-specific yetus/hadoop:tp-25385}.
Subsystem Report/Notes
GITHUB PR #7780
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7780/1/console
versions git=2.34.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

pan3793
pan3793 commented Jul 7, 2025
View reviewed changes
dev-support/docker/pkg-resolver/install-protobuf.sh
-o /opt/protobuf.tar.gz &&
tar xzf /opt/protobuf.tar.gz --strip-components 1 -C /opt/protobuf-src &&
mkdir -p /opt/abseil-cpp-src &&
curl -L -s -S \
https://github.com/abseil/abseil-cpp/archive/refs/tags/20230802.1.tar.gz \
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recent protobuf versions have hard dependency on abseil, the version is defined at
https://github.com/protocolbuffers/protobuf/blob/v3.25.5/protobuf_deps.bzl#L46

@hadoop-yetus
Copy link

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7780/2/console in case of problems.

pan3793
pan3793 commented Jul 7, 2025
View reviewed changes
BUILDING.txt
$ cd protobuf-3.25.5
$ cmake -S . -B build -Dprotobuf_BUILD_TESTS=OFF
$ cmake --build build --parallel $(nproc)
$ cmake --install build
$ protoc --version
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now it returns 25.5 (seems protobuf changed version policy recently)

chengpan@a1847ca27697:~/hadoop$ protoc --version
libprotoc 25.5

@pan3793
Copy link
Member Author

pan3793 commented Jul 7, 2025

cc @pjfanning @steveloughran @slfan1989

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 20m 28s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 hadolint 0m 0s hadolint was not available.
+0 🆗 shellcheck 0m 0s Shellcheck was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+0 🆗 jsonlint 0m 0s jsonlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 ❌ mvninstall 25m 27s /branch-mvninstall-root.txt root in trunk failed.
-1 ❌ mvnsite 2m 17s /branch-mvnsite-root.txt root in trunk failed.
+1 💚 shadedclient 48m 22s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
-1 ❌ mvninstall 18m 7s /patch-mvninstall-root.txt root in the patch failed.
+1 💚 blanks 0m 0s The patch has no blanks issues.
-1 ❌ mvnsite 2m 17s /patch-mvnsite-root.txt root in the patch failed.
+1 💚 shadedclient 22m 32s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 6m 5s /patch-unit-root.txt root in the patch failed.
+1 💚 asflicense 0m 45s The patch does not generate ASF License warnings.
117m 43s
Subsystem Report/Notes
Docker ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7780/2/artifact/out/Dockerfile
GITHUB PR #7780
Optional Tests dupname asflicense codespell detsecrets hadolint shellcheck shelldocs mvnsite unit jsonlint
uname Linux 910c4b75b506 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1be1417
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7780/2/testReport/
Max. process+thread count 699 (vs. ulimit of 5500)
modules C: . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7780/2/console
versions git=2.9.5 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@pan3793
Copy link
Member Author

pan3793 commented Jul 7, 2025

CentOS 7 CI failure caused by GLIBC version too low, which is already tracked by YARN-11794 and HADOOP-19489

/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7780/centos-7/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/target/protoc-plugins/protoc-gen-grpc-java-1.69.0-linux-x86_64.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7780/centos-7/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/target/protoc-plugins/protoc-gen-grpc-java-1.69.0-linux-x86_64.exe)
--grpc-java_out: protoc-gen-grpc-java: Plugin failed with status code 1.

@pan3793 pan3793 marked this pull request as draft July 7, 2025 06:26
@pan3793
Copy link
Member Author

pan3793 commented Jul 7, 2025

hadoop-hdfs-native-client building fails, further fix is required.

convert to draft now, will investigate later

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GauthamBanasandra GauthamBanasandra GauthamBanasandra left review comments

Assignees
No one assigned
Labels
Infra trunk
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pan3793 @hadoop-yetus @GauthamBanasandra