Skip to content

Use attested-tls-proxy rather than cvm-reverse-proxy on Buildernet #65

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ameba23 wants to merge 6 commits into
base: trunk/buildernet
Choose a base branch
from
Draft

Use attested-tls-proxy rather than cvm-reverse-proxy on Buildernet #65

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8755ec3
Use attested-tls-proxy
ameba23 Dec 16, 2025
af87af1
Make attested-tls-proxy build script executable
ameba23 Dec 16, 2025
0236678
Use dcap-tdx not gcp-tdx as attestation type on GCP
ameba23 Dec 16, 2025
b86c6c4
Use project-specific build directory
ameba23 Dec 16, 2025
8ec1533
Omit version numbers from dependencies and put in alphabetical order
ameba23 Dec 16, 2025
0cb2c2f
Use separate copy of TLS cert and key for this service
ameba23 Dec 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions ...roxy-client.service.d/azure-override.conf → ...roxy-client.service.d/azure-override.conf
View file
Open in desktop
Copy link

@bakhtin bakhtin Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file should probably be renamed to attested-tls-proxy-client.... Also the the main service file in buildernet image

Copy link
Contributor Author

@ameba23 ameba23 Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean the name of this file should match the service file it is overriding - in this case attested-tls-proxy-client.service?

Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,10 @@ After=tpm2.target
ExecStartPre=+/usr/bin/chmod 440 /sys/kernel/security/tpm0/binary_bios_measurements
ExecStartPre=+/usr/bin/chown root:tss /sys/kernel/security/tpm0/binary_bios_measurements
ExecStart=
ExecStart=/usr/bin/cvm-reverse-proxy-client \
--listen-addr=localhost:7937 \
--target-addr=${BUILDERNET_BUILDERHUB_URL} \
ExecStart=/usr/bin/attested-tls-proxy client \
--listen-addr 127.0.0.1:7937 \
--client-attestation-type azure-tdx \
--server-attestation-type none
--allowed-remote-attestation-type none
--tls-private-key-path var/lib/persistent/operator-api/key.pem \
Copy link

@bakhtin bakhtin Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it really point to a Let's Encrypt certificate? If yes, we need to extend acme-le posthook to copy the certificate for this service separately, not re-using the path of operator-api

Copy link
Contributor Author

@ameba23 ameba23 Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is the big difference between this and cvm-reverse-proxy - we require CA-signed certs. I agree its better to copy the pem files to a separate location for this service.

Copy link
Contributor Author

@ameba23 ameba23 Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did this now, but i guess i need to set up a username and group for attested-tls-proxy. I can't see where that is being done for operator-api.

--tls-certificate-path var/lib/persistent/operator-api/cert.pem \
${BUILDERNET_BUILDERHUB_URL}
8 changes: 8 additions & 0 deletions ...re/mkosi.extra/etc/systemd/system/attested-tls-proxy-server.service.d/azure-override.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[Service]
ExecStart=
ExecStart=/usr/bin/attested-tls-proxy server \
--listen-addr 0.0.0.0:7936 \
--server-attestation-type azure-tdx \
--tls-private-key-path var/lib/persistent/operator-api/key.pem \
Copy link

@bakhtin bakhtin Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
--tls-private-key-path var/lib/persistent/operator-api/key.pem \
--tls-private-key-path %S/persistent/operator-api/key.pem \

--tls-certificate-path var/lib/persistent/operator-api/cert.pem \
127.0.0.1:14727
7 changes: 0 additions & 7 deletions ...ure/mkosi.extra/etc/systemd/system/cvm-reverse-proxy-server.service.d/azure-override.conf
View file
Open in desktop

This file was deleted.

14 changes: 14 additions & 0 deletions ...-gcp/mkosi.extra/etc/systemd/system/attested-tls-proxy-client.service.d/gcp-override.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[Service]
ExecStart=
ExecStart=/usr/bin/attested-tls-proxy client \
--listen-addr 127.0.0.1:7937 \
--client-attestation-type gcp-tdx
--allowed-remote-attestation-type none \
--tls-private-key-path var/lib/persistent/operator-api/key.pem \
--tls-certificate-path var/lib/persistent/operator-api/cert.pem \
${BUILDERNET_BUILDERHUB_URL} \
SupplementaryGroups=
ProtectSystem=strict
ProtectHome=yes
AmbientCapabilities=CAP_DAC_OVERRIDE
ReadWritePaths=/sys/kernel/config/tsm/report
14 changes: 14 additions & 0 deletions ...-gcp/mkosi.extra/etc/systemd/system/attested-tls-proxy-server.service.d/gcp-override.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[Service]
ExecStart=
ExecStart=/usr/bin/attested-tls-proxy-server \
--listen-addr 0.0.0.0:7936 \
--server-attestation-type gcp-tdx
--tls-private-key-path var/lib/persistent/operator-api/key.pem \
--tls-certificate-path var/lib/persistent/operator-api/cert.pem \
--allowed-remote-attestation-type none \
127.0.0.1:14727
SupplementaryGroups=
ProtectSystem=strict
ProtectHome=yes
AmbientCapabilities=CAP_DAC_OVERRIDE
ReadWritePaths=/sys/kernel/config/tsm/report
11 changes: 0 additions & 11 deletions ...t-gcp/mkosi.extra/etc/systemd/system/cvm-reverse-proxy-client.service.d/gcp-override.conf
View file
Open in desktop
Copy link
Member

@MoeMahhouk MoeMahhouk Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the first iteration, they could be running side be side (the previous reverse proxy and the new attested proxy server), right?
This way, we could have a fallback till we verify stability in further iterations, wdyt?

Copy link
Contributor Author

@ameba23 ameba23 Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking the other way around - for the first iteration we just want to know whether this works, and having a fallback might make us think it works when it doesn't. Once we know it works, add cvm-reverse-proxy as a fallback in case it stops working.

This file was deleted.

11 changes: 0 additions & 11 deletions ...t-gcp/mkosi.extra/etc/systemd/system/cvm-reverse-proxy-server.service.d/gcp-override.conf
View file
Open in desktop

This file was deleted.

33 changes: 33 additions & 0 deletions mkosi.images/buildernet/mkosi.build.d/20-attested-tls-proxy.sh
View file
Open in desktop
Copy link
Member

@MoeMahhouk MoeMahhouk Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could consider creating a reproducible debian packaging pipeline for attested-tls-proxy to avoid building it manually while building the image. This would accelerate the image building process and rollout times

Copy link
Contributor Author

@ameba23 ameba23 Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. Here i was roughly following how the other rust projects on the main branch of this repo are build - with the build_rust_package.sh script, which isn't available on this branch.

It looks like we have a reproducible debian packaging pipeline for rbuilder - and since that is a rust project it would be probably quite easy to adapt it for this. In my experience i have had annoying issues with trying to get 100% reproducibility with docker - but if its working for rbuilder it will probably work for this project too.

Either way, i think it will be easiest to built the first few iterations like this, as it inevitable wont work the first few times.

Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/usr/bin/env bash
set -euo pipefail

REF=06aafe43335a5d228a3ea2d3b871d15d2d06e855
CARGO_HOME="$BUILDDIR/.cargo"
PATH="$BUILDDIR/rust-toolchain/bin:$PATH"
BUILDDIR="$BUILDDIR/vector"
export CARGO_HOME="$SRCDIR/mkosi.images/buildernet/mkosi.cache/cargo"

echo "Installing attested-tls-proxy..."

mkdir -p $BUILDDIR

curl -sSfL https://api.github.com/repos/flashbots/attested-tls-proxy/tarball/${REF} | \
tar xzf - -C $BUILDDIR --strip-components=1

cd $BUILDDIR

RUSTFLAGS="-C target-cpu=x86-64-v4 \
-C link-arg=-Wl,--build-id=none \
-C symbol-mangling-version=v0 \
-L /usr/lib/x86_64-linux-gnu"
CARGO_PROFILE_RELEASE_LTO='thin'
CARGO_PROFILE_RELEASE_CODEGEN_UNITS='1'
CARGO_PROFILE_RELEASE_PANIC='abort'
CARGO_PROFILE_RELEASE_INCREMENTAL='false'
CARGO_PROFILE_RELEASE_OPT_LEVEL='3'
CARGO_TARGET_DIR="$BUILDDIR/target"

cargo build --release --locked

mkdir -p $DESTDIR/usr/bin
cp $CARGO_TARGET_DIR/release/attested-tls-proxy $DESTDIR/usr/bin/attested-tls-proxy
2 changes: 2 additions & 0 deletions mkosi.images/buildernet/mkosi.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ Packages=cryptsetup
systemd-repart
systemd-resolved
tpm2-tools
libtss2-esys-3.0.2-0t64
Copy link

@bakhtin bakhtin Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can omit the package version. The main mkosi config file already points to a point in time snapshot repository which guarantees stable version of the packages.

Copy link

@bakhtin bakhtin Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The package names should be alphabetically sorted

libtss2-tctildr0t64
BuildPackages=build-essential
dpkg-dev
git
Expand Down
22 changes: 22 additions & 0 deletions mkosi.images/buildernet/mkosi.extra/etc/systemd/system/attested-tls-proxy-client.service
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
[Unit]
DefaultDependencies=no
Description=Attested TLS Proxy client
Wants=network-online.target
After=network.target network-online.target

[Service]
Type=exec
DynamicUser=yes
SupplementaryGroups=tss
Environment=BUILDERNET_BUILDERHUB_URL=__BUILDERNET_BUILDERHUB_URL
ExecStart=/usr/bin/attested-tls-proxy client \
--listen-addr=127.0.0.1:7937 \
--allowed-remote-attestation-type none \
--tls-private-key-path var/lib/persistent/operator-api/key.pem \
--tls-certificate-path var/lib/persistent/operator-api/cert.pem \
--client-attestation-type auto \
${BUILDERNET_BUILDERHUB_URL}
Restart=on-failure

[Install]
WantedBy=multi-user.target
20 changes: 20 additions & 0 deletions mkosi.images/buildernet/mkosi.extra/etc/systemd/system/attested-tls-proxy-server.service
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
[Unit]
Description=Attested TLS Proxy server
Wants=network-online.target
After=network.target network-online.target

[Service]
Type=exec
DynamicUser=yes
SupplementaryGroups=tss
ExecStart=/usr/bin/attested-tls-proxy server \
--listen-addr 0.0.0.0:7936 \
--allowed-remote-attestation-type none \
--server-attestation-type auto \
--tls-private-key-path var/lib/persistent/operator-api/key.pem \
--tls-certificate-path var/lib/persistent/operator-api/cert.pem \
127.0.0.1:14727
Restart=on-failure

[Install]
WantedBy=multi-user.target
19 changes: 0 additions & 19 deletions mkosi.images/buildernet/mkosi.extra/etc/systemd/system/cvm-reverse-proxy-client.service
View file
Open in desktop

This file was deleted.

16 changes: 0 additions & 16 deletions mkosi.images/buildernet/mkosi.extra/etc/systemd/system/cvm-reverse-proxy-server.service
View file
Open in desktop

This file was deleted.

4 changes: 2 additions & 2 deletions mkosi.images/buildernet/mkosi.extra/etc/systemd/system/render-config.service
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
[Unit]
DefaultDependencies=no
Description=Pull and render configs from BuilderHub
Wants=network-online.target cvm-reverse-proxy-client.service
After=network.target network-online.target cvm-reverse-proxy-client.service
Wants=network-online.target attested-tls-proxy-client.service
After=network.target network-online.target attested-tls-proxy-client.service

[Service]
Type=oneshot
Expand Down
2 changes: 1 addition & 1 deletion mkosi.images/buildernet/mkosi.postinst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ for var in "${!BUILDERNET_@}"; do
replace_underscore_template "$BUILDROOT/etc/systemd/system/persistent-setup.service" "${!var}"
;;
BUILDERNET_BUILDERHUB_URL)
replace_underscore_template "$BUILDROOT/etc/systemd/system/cvm-reverse-proxy-client.service" "${!var}"
replace_underscore_template "$BUILDROOT/etc/systemd/system/attested-tls-proxy-client.service" "${!var}"
;;
BUILDERNET_SSH_PUBLIC_KEY)
replace_underscore_template "$BUILDROOT/home/bnet/.ssh/authorized_keys" "${!var}"
Expand Down