WIP: Builder: Improve build process for builder image

[jschintag]

Move all files for building the image under hack/build/builder. Remove unused output-bazel-arch.sh file.
Make build scripts more similiar to kubevirt scripts. Do not use buildah but instead support docker/podman. Build multiarch image by default.
Gatekeeping when a new image is built will happen in project-infra. This allow the build script to be more easily used for local development. Expand the provided builder image architectures by adding s390x.

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:
This needed to be reverted already once, as the build would randomly hang (#145).

/cc @Barakmor1

Release note:

NONE

[kubevirt-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign vladikr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jschintag]

I have created this as a draft, since in local testing i have experienced it hanging already. So currently i'm debugging.

[jschintag]

As best as i can determine, the processes simply sleep forever:

[jschinta@xocpvirt ~]$ ps -aux | grep /usr/bin/qemu-s390x-static
jschinta 1849439  0.0  0.0 222456 11092 ?        Ssl  Sep17   0:00 /usr/bin/qemu-s390x-static /bin/sh -c source /etc/profile.d/gimme.sh && ?eval $(go env) && ?go install mvdan.cc/sh/cmd/shfmt@latest && ?go install github.com/mattn/goveralls@latest && ?go install golang.org/x/lint/golint@latest &&     go install -v github.com/golang/protobuf/protoc-gen-go@1643683 && ?go install github.com/rmohr/go-swagger-utils/swagger-doc@latest && ?go install sigs.k8s.io/controller-tools/cmd/[email protected] && ?go install github.com/securego/gosec/v2/cmd/gosec@latest && ?rm -rf "${GOPATH}/pkg"
jschinta 1858229  0.2  0.4 2647744 70108 ?       Sl   Sep17   3:43 /usr/bin/qemu-s390x-static /go/pkg/mod/golang.org/[email protected]/bin/go install github.com/securego/gosec/v2/cmd/gosec@latest
jschinta 1863188  0.0  0.1 2040780 17724 ?       Sl   Sep17   0:00 /usr/bin/qemu-s390x-static /go/pkg/mod/golang.org/[email protected]/pkg/tool/linux_s390x/cgo -objdir /tmp/go-build3589083031/b199/ -importpath net -ldflags="-O2" "-g" "-lresolv" -- -I /tmp/go-build3589083031/b199/ -O2 -g -fno-stack-protector /go/pkg/mod/golang.org/[email protected]/src/net/cgo_linux.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_resnew.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_socknew.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_unix_cgo.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_unix_cgo_res.go
jschinta 1863210  0.0  0.0 1974728 13272 ?       Sl   Sep17   0:00 /usr/bin/qemu-s390x-static /go/pkg/mod/golang.org/[email protected]/pkg/tool/linux_s390x/cgo -objdir /tmp/go-build3589083031/b199/ -importpath net -ldflags="-O2" "-g" "-lresolv" -- -I /tmp/go-build3589083031/b199/ -O2 -g -fno-stack-protector /go/pkg/mod/golang.org/[email protected]/src/net/cgo_linux.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_resnew.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_socknew.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_unix_cgo.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_unix_cgo_res.go
jschinta 2736484  0.0  0.0   6492  2120 pts/5    S+   10:06   0:00 grep --color=auto /usr/bin/qemu-s390x-static

This is currently hanging since yesterday at

go: downloading go.opentelemetry.io/auto/sdk v1.1.0

Though it does not seem to matter at which point package it hangs.
I'm continuing to search for more information.

[Barakmor1]

As best as i can determine, the processes simply sleep forever:

[jschinta@xocpvirt ~]$ ps -aux | grep /usr/bin/qemu-s390x-static
jschinta 1849439  0.0  0.0 222456 11092 ?        Ssl  Sep17   0:00 /usr/bin/qemu-s390x-static /bin/sh -c source /etc/profile.d/gimme.sh && ?eval $(go env) && ?go install mvdan.cc/sh/cmd/shfmt@latest && ?go install github.com/mattn/goveralls@latest && ?go install golang.org/x/lint/golint@latest &&     go install -v github.com/golang/protobuf/protoc-gen-go@1643683 && ?go install github.com/rmohr/go-swagger-utils/swagger-doc@latest && ?go install sigs.k8s.io/controller-tools/cmd/[email protected] && ?go install github.com/securego/gosec/v2/cmd/gosec@latest && ?rm -rf "${GOPATH}/pkg"
jschinta 1858229  0.2  0.4 2647744 70108 ?       Sl   Sep17   3:43 /usr/bin/qemu-s390x-static /go/pkg/mod/golang.org/[email protected]/bin/go install github.com/securego/gosec/v2/cmd/gosec@latest
jschinta 1863188  0.0  0.1 2040780 17724 ?       Sl   Sep17   0:00 /usr/bin/qemu-s390x-static /go/pkg/mod/golang.org/[email protected]/pkg/tool/linux_s390x/cgo -objdir /tmp/go-build3589083031/b199/ -importpath net -ldflags="-O2" "-g" "-lresolv" -- -I /tmp/go-build3589083031/b199/ -O2 -g -fno-stack-protector /go/pkg/mod/golang.org/[email protected]/src/net/cgo_linux.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_resnew.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_socknew.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_unix_cgo.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_unix_cgo_res.go
jschinta 1863210  0.0  0.0 1974728 13272 ?       Sl   Sep17   0:00 /usr/bin/qemu-s390x-static /go/pkg/mod/golang.org/[email protected]/pkg/tool/linux_s390x/cgo -objdir /tmp/go-build3589083031/b199/ -importpath net -ldflags="-O2" "-g" "-lresolv" -- -I /tmp/go-build3589083031/b199/ -O2 -g -fno-stack-protector /go/pkg/mod/golang.org/[email protected]/src/net/cgo_linux.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_resnew.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_socknew.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_unix_cgo.go /go/pkg/mod/golang.org/[email protected]/src/net/cgo_unix_cgo_res.go
jschinta 2736484  0.0  0.0   6492  2120 pts/5    S+   10:06   0:00 grep --color=auto /usr/bin/qemu-s390x-static

This is currently hanging since yesterday at

go: downloading go.opentelemetry.io/auto/sdk v1.1.0

Though it does not seem to matter at which point package it hangs. I'm continuing to search for more information.

interesting observation

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale