Skip to content

Commit

Permalink
Update the FBPCF Dependency Docker Images to be built with the Haswel…
Browse files Browse the repository at this point in the history
…l architecture (#475)

Summary:
Pull Request resolved: #475

## Context
After attempting to build the images with the "x86-64" architecture for a while, I determined that we wouldn't be able to make it work as the EMP library used an instruction that wasing included in the "x86-64" architecture (specifically ). This meant that I needed to explore other options. My next choice was to find the architecture of the current build machines that we use (c4.4xlarge) and go with that since we know that we were building it previously natively for those machines. The c4.4xlarge uses an Intel Xeon E5-2666 based on the cpu info from the machine:
```
ubuntu@ip-172-31-15-42:~$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
...
Model name:                      Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
...
```

According to the Intel website, this processor uses the Haswell architecture: https://www.intel.com/content/www/us/en/products/sku/81706/intel-xeon-processor-e52660-v3-25m-cache-2-60-ghz/specifications.html

With that information, I decided to rebuild everything for the haswell architecture since it should match our current build system and allow us all the symbols necessary to run the computation.

## This Diff
This diff updates all of the dependent FBPCF images (AWS, EMP, and Folly) to be built with the haswell architechture specified. After this has been done and confirmed to work. I will produce new haswell images and update the main FBPCF build in the next diff.

Reviewed By: robotal, ramesc

Differential Revision: D42588188

fbshipit-source-id: 18767d04911027139c40125c1833228616ce4939
  • Loading branch information
musebc authored and facebook-github-bot committed Jan 27, 2023
1 parent 78b356f commit bd87f69
Show file tree
Hide file tree
Showing 5 changed files with 29 additions and 26 deletions.
21 changes: 11 additions & 10 deletions .github/workflows/aws-s3-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@ name: Build and publish aws s3 dependency
on:
workflow_dispatch:
inputs:
name:
description: "Build and publish an fbpcf/aws-s3-core image for a particular version"
default: "Run"
image_tag:
description: "The tag to apply to the AWS SDK image. Previously this was the AWS SDK release version (e.g. 1.8.177)"
required: true
type: string
aws_release:
description: "The aws s3 version to build and publish (e.g. 1.8.177)"
required: true
Expand Down Expand Up @@ -44,16 +45,16 @@ jobs:
- name: Build image
run: |
docker build \
--build-arg os_release=${{ github.event.inputs.os_release }} \
--build-arg aws_release=${{ github.event.inputs.aws_release }} \
-t "fbpcf/${{ github.event.inputs.os }}-aws-s3-core:${{ github.event.inputs.aws_release }}" \
-f "docker/aws-s3-core/Dockerfile.${{ github.event.inputs.os }}" .
--build-arg os_release=${{ inputs.os_release }} \
--build-arg aws_release=${{ inputs.aws_release }} \
-t "fbpcf/${{ inputs.os }}-aws-s3-core:${{ inputs.image_tag }}" \
-f "docker/aws-s3-core/Dockerfile.${{ inputs.os }}" .
- name: Tag image
run: |
docker tag fbpcf/${{ github.event.inputs.os }}-aws-s3-core:${{ github.event.inputs.aws_release }} \
${{ env.REGISTRY }}/${{ github.repository }}/${{ github.event.inputs.os }}-aws-s3-core:${{ github.event.inputs.aws_release }}
docker tag fbpcf/${{ inputs.os }}-aws-s3-core:${{ inputs.image_tag }} \
${{ env.REGISTRY }}/${{ github.repository }}/${{ inputs.os }}-aws-s3-core:${{ inputs.image_tag }}
- name: Publish image
run: |
docker push --all-tags ${{ env.REGISTRY }}/${{ github.repository }}/${{ github.event.inputs.os }}-aws-s3-core
docker push --all-tags ${{ env.REGISTRY }}/${{ github.repository }}/${{ inputs.os }}-aws-s3-core
23 changes: 12 additions & 11 deletions .github/workflows/emp-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@ name: Build and publish emp dependency
on:
workflow_dispatch:
inputs:
name:
description: "Build and publish an fbpcf/emp image for a particular version"
default: "Run"
image_tag:
description: "The tag to apply to the emp image. Previously this was the emp tool release version (e.g. 0.2.3)"
required: true
type: string
emp_release:
description: "The emp version to build and publish (e.g. 0.2.2)"
required: true
Expand Down Expand Up @@ -48,17 +49,17 @@ jobs:
- name: Build image
run: |
docker build \
--build-arg os_release=${{ github.event.inputs.os_release }} \
--build-arg emp_tool_release=${{ github.event.inputs.emp_tool_release }} \
--build-arg emp_release=${{ github.event.inputs.emp_release }} \
-t "fbpcf/${{ github.event.inputs.os }}-emp:${{ github.event.inputs.emp_tool_release }}" \
-f "docker/emp/Dockerfile.${{ github.event.inputs.os }}" .
--build-arg os_release=${{ inputs.os_release }} \
--build-arg emp_tool_release=${{ inputs.emp_tool_release }} \
--build-arg emp_release=${{ inputs.emp_release }} \
-t "fbpcf/${{ inputs.os }}-emp:${{ inputs.image_tag }}" \
-f "docker/emp/Dockerfile.${{ inputs.os }}" .
- name: Tag image
run: |
docker tag fbpcf/${{ github.event.inputs.os }}-emp:${{ github.event.inputs.emp_tool_release }} \
${{ env.REGISTRY }}/${{ github.repository }}/${{ github.event.inputs.os }}-emp:${{ github.event.inputs.emp_tool_release }}
docker tag fbpcf/${{ inputs.os }}-emp:${{ inputs.image_tag }} \
${{ env.REGISTRY }}/${{ github.repository }}/${{ inputs.os }}-emp:${{ inputs.image_tag }}
- name: Publish image
run: |
docker push --all-tags ${{ env.REGISTRY }}/${{ github.repository }}/${{ github.event.inputs.os }}-emp
docker push --all-tags ${{ env.REGISTRY }}/${{ github.repository }}/${{ inputs.os }}-emp
2 changes: 1 addition & 1 deletion docker/aws-s3-core/Dockerfile.ubuntu
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ WORKDIR /root/build/aws-sdk-cpp
RUN git checkout tags/${aws_release} -b ${aws_release}
# -DCUSTOM_MEMORY_MANAGEMENT=0 is added to avoid Aws::String and std::string issue
# -DENABLE_TESTING=OFF for a weird failing test HttpClientTest.TestRandomURLWithProxyAndOtherDeclaredAsNonProxyHost
RUN cmake . -DBUILD_ONLY="s3;core" -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_SHARED_LIBS=OFF -DCUSTOM_MEMORY_MANAGEMENT=0 -DENABLE_TESTING=OFF
RUN cmake . -DBUILD_ONLY="s3;core" -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_SHARED_LIBS=OFF -DCUSTOM_MEMORY_MANAGEMENT=0 -DENABLE_TESTING=OFF -DCMAKE_CXX_FLAGS="-march=haswell"
RUN . /root/build/get_make_options.sh && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD && make install -j $MAKE_JOBS -l $MAKE_MAX_LOAD

FROM ubuntu:${os_release}
Expand Down
6 changes: 3 additions & 3 deletions docker/emp/Dockerfile.ubuntu
Original file line number Diff line number Diff line change
Expand Up @@ -33,21 +33,21 @@ RUN git clone -b${emp_tool_release} https://github.com/emp-toolkit/emp-tool.git
WORKDIR /root/build/emp-tool
RUN sed -i "s/SHARED/STATIC/" CMakeLists.txt
RUN cmake -DBUILD_SHARED_LIBS=OFF -DTHREADING=ON .
RUN . /root/build/get_make_options.sh && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD install
RUN . /root/build/get_make_options.sh && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD CFLAGS="-march=haswell" CXXFLAGS="-march=haswell" && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD install

# emp-ot build and install
WORKDIR /root/build
RUN git clone -b${emp_release} https://github.com/emp-toolkit/emp-ot.git
WORKDIR /root/build/emp-ot
RUN cmake -DBUILD_SHARED_LIBS=OFF -DTHREADING=ON .
RUN . /root/build/get_make_options.sh && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD install
RUN . /root/build/get_make_options.sh && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD CFLAGS="-march=haswell" CXXFLAGS="-march=haswell" && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD install

# emp-sh2pc build and install
WORKDIR /root/build
RUN git clone -b${emp_release} https://github.com/emp-toolkit/emp-sh2pc.git
WORKDIR /root/build/emp-sh2pc
RUN cmake -DBUILD_SHARED_LIBS=OFF -DTHREADING=ON .
RUN . /root/build/get_make_options.sh && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD install
RUN . /root/build/get_make_options.sh && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD CFLAGS="-march=haswell" CXXFLAGS="-march=haswell" && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD install

FROM ubuntu:${os_release}
RUN mkdir /usr/local/cmake
Expand Down
3 changes: 2 additions & 1 deletion docker/folly/Dockerfile.ubuntu
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ RUN git clone https://github.com/facebook/folly.git
WORKDIR /root/build/folly
RUN git checkout tags/v${folly_release} -b v${folly_release}

RUN cmake DBUILD_SHARED_LIBS=OFF -DFOLLY_USE_JEMALLOC=0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-march=x86-64" .
RUN cmake DBUILD_SHARED_LIBS=OFF -DFOLLY_USE_JEMALLOC=0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-march=haswell" .

RUN . /root/build/get_make_options.sh && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD && make -j $MAKE_JOBS -l $MAKE_MAX_LOAD install

FROM ubuntu:${os_release}
Expand Down

0 comments on commit bd87f69

Please sign in to comment.