Skip to content

HADOOP-19605. Upgrade Protobuf 3.25.5 for docker images #7780

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: trunk
Choose a base branch
from
+48 −39
Draft

HADOOP-19605. Upgrade Protobuf 3.25.5 for docker images #7780

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
17d30f0
HADOOP-19605. Upgrade Protobuf 3.25.5 for docker images
pan3793 Jul 7, 2025
1be1417
fix
pan3793 Jul 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 29 additions & 24 deletions BUILDING.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Requirements:
* JDK 1.8
* Maven 3.3 or later
* Boost 1.86.0 (if compiling native code)
* Protocol Buffers 3.21.12 (if compiling native code)
* Protocol Buffers 3.25.5 (if compiling native code)
* CMake 3.19 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* Cyrus SASL devel (if compiling native code)
Expand Down Expand Up @@ -78,13 +78,15 @@ Refer to dev-support/docker/Dockerfile):
$ ./bootstrap
$ make -j$(nproc)
$ sudo make install
* Protocol Buffers 3.21.12 (required to build native code)
$ curl -L https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.21.12.tar.gz > protobuf-3.21.12.tar.gz
$ tar -zxvf protobuf-3.21.12.tar.gz && cd protobuf-3.21.12
$ ./autogen.sh
$ ./configure
$ make -j$(nproc)
$ sudo make install
* Protocol Buffers 3.25.5 (required to build native code)
$ curl -L https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.25.5.tar.gz > protobuf-3.25.5.tar.gz
$ curl -L https://github.com/abseil/abseil-cpp/archive/refs/tags/20230802.1.tar.gz > abseil-cpp-20230802.1.tar.gz
$ tar -zxvf protobuf-3.25.5.tar.gz
$ tar -zxvf abseil-cpp-20230802.1.tar.gz --strip-components 1 -C protobuf-3.25.5/third_party/abseil-cpp
$ cd protobuf-3.25.5
$ cmake -S . -B build -Dprotobuf_BUILD_TESTS=OFF
$ cmake --build build --parallel $(nproc)
$ sudo cmake --install build
* Boost
$ curl -L https://sourceforge.net/projects/boost/files/boost/1.86.0/boost_1_86_0.tar.bz2/download > boost_1_86_0.tar.bz2
$ tar --bzip2 -xf boost_1_86_0.tar.bz2 && cd boost_1_86_0
Expand Down Expand Up @@ -434,14 +436,15 @@ Installing required dependencies for clean install of macOS 10.14:
* Install native libraries, only openssl is required to compile native code,
you may optionally install zlib, lz4, etc.
$ brew install openssl
* Protocol Buffers 3.21.12 (required to compile native code)
$ curl -L https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.21.12.tar.gz > protobuf-3.21.12.tar.gz
$ tar -zxvf protobuf-3.21.12.tar.gz && cd protobuf-3.21.12
$ ./autogen.sh
$ ./configure
$ make
$ make check
$ make install
* Protocol Buffers 3.25.5 (required to build native code)
$ curl -L https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.25.5.tar.gz > protobuf-3.25.5.tar.gz
$ curl -L https://github.com/abseil/abseil-cpp/archive/refs/tags/20230802.1.tar.gz > abseil-cpp-20230802.1.tar.gz
$ tar -zxvf protobuf-3.25.5.tar.gz
$ tar -zxvf abseil-cpp-20230802.1.tar.gz --strip-components 1 -C protobuf-3.25.5/third_party/abseil-cpp
$ cd protobuf-3.25.5
$ cmake -S . -B build -Dprotobuf_BUILD_TESTS=OFF
$ cmake --build build --parallel $(nproc)
$ cmake --install build
$ protoc --version
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now it returns 25.5 (seems protobuf changed version policy recently)

chengpan@a1847ca27697:~/hadoop$ protoc --version
libprotoc 25.5


Note that building Hadoop 3.1.1/3.1.2/3.2.0 native code from source is broken
Expand Down Expand Up @@ -473,13 +476,15 @@ Building on CentOS 8
* Install python2 for building documentation.
$ sudo dnf install python2

* Install Protocol Buffers v3.21.12.
$ curl -L https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.21.12.tar.gz > protobuf-3.21.12.tar.gz
$ tar -zxvf protobuf-3.21.12.tar.gz && cd protobuf-3.21.12
$ ./autogen.sh
$ ./configure --prefix=/usr/local
$ make
$ sudo make install
* Protocol Buffers 3.25.5 (required to build native code)
$ curl -L https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.25.5.tar.gz > protobuf-3.25.5.tar.gz
$ curl -L https://github.com/abseil/abseil-cpp/archive/refs/tags/20230802.1.tar.gz > abseil-cpp-20230802.1.tar.gz
$ tar -zxvf protobuf-3.25.5.tar.gz
$ tar -zxvf abseil-cpp-20230802.1.tar.gz --strip-components 1 -C protobuf-3.25.5/third_party/abseil-cpp
$ cd protobuf-3.25.5
$ cmake -S . -B build -Dprotobuf_BUILD_TESTS=OFF
$ cmake --build build --parallel $(nproc)
$ sudo cmake --install build --prefix /usr/local
$ cd ..

* Install libraries provided by CentOS 8.
Expand Down Expand Up @@ -531,7 +536,7 @@ Requirements:
* JDK 1.8
* Maven 3.0 or later (maven.apache.org)
* Boost 1.86.0 (boost.org)
* Protocol Buffers 3.21.12 (https://github.com/protocolbuffers/protobuf/tags)
* Protocol Buffers 3.25.5 (https://github.com/protocolbuffers/protobuf/tags)
* CMake 3.19 or newer (cmake.org)
* Visual Studio 2019 (visualstudio.com)
* Windows SDK 8.1 (optional, if building CPU rate control for the container executor. Get this from
Expand Down
2 changes: 1 addition & 1 deletion dev-support/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
ENV SPOTBUGS_HOME /opt/spotbugs

#######
# Set env vars for Google Protobuf 3.21.12
# Set env vars for Google Protobuf 3.25.5
#######
ENV PROTOBUF_HOME /opt/protobuf
ENV PATH "${PATH}:/opt/protobuf/bin"
Expand Down
2 changes: 1 addition & 1 deletion dev-support/docker/Dockerfile_aarch64
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-arm64
ENV SPOTBUGS_HOME /opt/spotbugs

#######
# Set env vars for Google Protobuf 3.21.12
# Set env vars for Google Protobuf 3.25.5
#######
ENV PROTOBUF_HOME /opt/protobuf
ENV PATH "${PATH}:/opt/protobuf/bin"
Expand Down
2 changes: 1 addition & 1 deletion dev-support/docker/Dockerfile_centos_7
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ ENV JAVA_HOME /usr/lib/jvm/java-1.8.0
ENV SPOTBUGS_HOME /opt/spotbugs

#######
# Set env vars for Google Protobuf 3.21.12
# Set env vars for Google Protobuf 3.25.5
#######
ENV PROTOBUF_HOME /opt/protobuf
ENV PATH "${PATH}:/opt/protobuf/bin"
Expand Down
2 changes: 1 addition & 1 deletion dev-support/docker/Dockerfile_centos_8
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ ENV JAVA_HOME /usr/lib/jvm/java-1.8.0
ENV SPOTBUGS_HOME /opt/spotbugs

#######
# Set env vars for Google Protobuf 3.21.12
# Set env vars for Google Protobuf 3.25.5
#######
ENV PROTOBUF_HOME /opt/protobuf
ENV PATH "${PATH}:/opt/protobuf/bin"
Expand Down
2 changes: 1 addition & 1 deletion dev-support/docker/Dockerfile_debian_10
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64
ENV SPOTBUGS_HOME /opt/spotbugs

#######
# Set env vars for Google Protobuf 3.21.12
# Set env vars for Google Protobuf 3.25.5
#######
ENV PROTOBUF_HOME /opt/protobuf
ENV PATH "${PATH}:/opt/protobuf/bin"
Expand Down
2 changes: 1 addition & 1 deletion dev-support/docker/Dockerfile_ubuntu_24
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64
ENV SPOTBUGS_HOME /opt/spotbugs

#######
# Set env vars for Google Protobuf 3.21.12
# Set env vars for Google Protobuf 3.25.5
#######
ENV PROTOBUF_HOME /opt/protobuf
ENV PATH "${PATH}:/opt/protobuf/bin"
Expand Down
20 changes: 12 additions & 8 deletions dev-support/docker/pkg-resolver/install-protobuf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,29 +27,33 @@ if [ $? -eq 1 ]; then
exit 1
fi

default_version="3.21.12"
default_version="3.25.5"
version_to_install=$default_version
if [ -n "$2" ]; then
version_to_install="$2"
fi

if [ "$version_to_install" != "3.21.12" ]; then
if [ "$version_to_install" != "3.25.5" ]; then
echo "WARN: Don't know how to install version $version_to_install, installing the default version $default_version instead"
version_to_install=$default_version
fi

if [ "$version_to_install" == "3.21.12" ]; then
if [ "$version_to_install" == "3.25.5" ]; then
# hadolint ignore=DL3003
mkdir -p /opt/protobuf-src &&
curl -L -s -S \
https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.21.12.tar.gz \
https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.25.5.tar.gz \
-o /opt/protobuf.tar.gz &&
tar xzf /opt/protobuf.tar.gz --strip-components 1 -C /opt/protobuf-src &&
mkdir -p /opt/abseil-cpp-src &&
curl -L -s -S \
https://github.com/abseil/abseil-cpp/archive/refs/tags/20230802.1.tar.gz \
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recent protobuf versions have hard dependency on abseil, the version is defined at
https://github.com/protocolbuffers/protobuf/blob/v3.25.5/protobuf_deps.bzl#L46

-o /opt/abseil-cpp.tar.gz &&
tar xzf /opt/abseil-cpp.tar.gz --strip-components 1 -C /opt/protobuf-src/third_party/abseil-cpp &&
cd /opt/protobuf-src &&
./autogen.sh &&
./configure --prefix=/opt/protobuf &&
make "-j$(nproc)" &&
make install &&
cmake -S . -B build -Dprotobuf_BUILD_TESTS=OFF &&
cmake --build build --parallel $(nproc) &&
cmake --install build --prefix /opt/protobuf &&
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

protocolbuffers/protobuf#10132 (since protobuf 3.22.0) removes autotools support, the supported toolchains are bazel and cmake.

cd /root &&
rm -rf /opt/protobuf-src
else
Expand Down
2 changes: 1 addition & 1 deletion dev-support/docker/vcpkg/vcpkg.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"overrides": [
{
"name": "protobuf",
"version": "3.21.12"
"version": "3.25.5"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose this works, but I don't have Windows env to verify this.

cc @GauthamBanasandra

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for PR @pan3793. Please let me know once you're done with all the changes and I can verify it on Windows.

}
]
}