Skip to content

Commit 2c349e0

Browse files
author
jasonvigil
authored
[ENH]: Add workflow to build and publish service container images (#6944)
## Description of changes Adds the plumbing for this repo to build and push its own service container images (compactor-service, query-service, rust-log-service, sysdb-service, garbage-collector-service, etc.). Nothing calls the workflow yet — this is the dormant foundation for later wiring. - docker-bake.hcl: 11 service targets covering the Rust and Go service stages in rust/Dockerfile, go/Dockerfile, and go/Dockerfile.migration. HCL variables (REGISTRY_AWS/GCP/DOCKERHUB, LOCAL_BUILD, COMMIT_SHORT_SHA, ADDRESS_SANITIZER, ENABLE_AVX512) drive registry fan-out and tag generation, with conditional blocks so unset registries are skipped. - .github/actions/build_service_images: composite action that auths to ECR (OIDC), GCP Artifact Registry (Workload Identity), and Docker Hub, then bakes and pushes with a 4x exponential-backoff retry on push. Includes a short-circuit that runs "docker buildx bake --print" and probes every emitted tag via "docker buildx imagetools inspect"; if all targets already exist in every configured registry, the build and push steps are skipped. Deriving the probe set from bake --print means new targets added to docker-bake.hcl are picked up without editing the action. - .github/workflows/_build_release_service_images.yml: reusable workflow (workflow_call + workflow_dispatch) that wraps the action. Concurrency group keyed on commit SHA so duplicate triggers for the same commit serialize and then short-circuit. Emits "skipped" and "commit_short_sha" outputs for callers. ## Test plan Manually run for a couple of different SHAs, double-check: - [ ] The short-circuit works correctly - [ ] It is able to authenticate to the upstream registries (todo: add those secrets) - [ ] It is able to push images to the upstream registries ## Migration plan N/A ## Observability plan N/A ## Documentation Changes N/A
1 parent a36300a commit 2c349e0

3 files changed

Lines changed: 559 additions & 0 deletions

File tree

‎.github/actions/build_service_images/action.yaml‎

Lines changed: 243 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,243 @@
1+
name: 'Build Service Images'
2+
description: 'Build and publish Chroma service images via docker-bake.hcl, with a short-circuit that skips the build when every target already exists in every configured registry.'
3+
inputs:
4+
COMMIT_SHA:
5+
description: 'Full commit SHA to build. Defaults to the currently-checked-out commit.'
6+
required: false
7+
default: ''
8+
PUSH:
9+
description: 'Flag for whether to push built images, must be a string set to "true" to push'
10+
required: false
11+
default: 'false'
12+
FORCE:
13+
description: 'If "true", skip the short-circuit check and always rebuild/push.'
14+
required: false
15+
default: 'false'
16+
AWS_REGION:
17+
description: 'AWS region of ECR'
18+
required: true
19+
AWS_ECR_OIDC_ARN:
20+
description: 'AWS ARN of the OIDC role to assume for logging into ECR'
21+
required: true
22+
GCP_WORKLOAD_IDENTITY_PROVIDER:
23+
description: 'GCP workload identity provider'
24+
required: true
25+
GCP_GITHUB_ACTIONS_SERVICE_ACCOUNT_EMAIL:
26+
description: 'GCP service account email'
27+
required: true
28+
GCP_ARTIFACT_REGISTRY_REGION:
29+
description: 'GCP artifact registry region'
30+
required: true
31+
GCP_ARTIFACT_REGISTRY_PROJECT_ID:
32+
description: 'GCP artifact registry project ID'
33+
required: true
34+
GCP_ARTIFACT_REGISTRY_NAME:
35+
description: 'GCP artifact registry name'
36+
required: true
37+
DOCKERHUB_USERNAME:
38+
description: 'DockerHub username for authentication'
39+
required: true
40+
DOCKERHUB_TOKEN:
41+
description: 'DockerHub token for authentication'
42+
required: true
43+
ADDRESS_SANITIZER:
44+
description: 'Enable Address Sanitizer for builds. Set to "1" to enable.'
45+
required: false
46+
default: ''
47+
ENABLE_AVX512:
48+
description: 'Enable AVX512 for builds. Set to "1" to enable.'
49+
required: false
50+
default: ''
51+
52+
outputs:
53+
skipped:
54+
description: 'Whether the build+push was skipped because all target images already exist.'
55+
value: ${{ steps.short-circuit.outputs.skipped }}
56+
commit_short_sha:
57+
description: 'Short commit SHA used for image tags.'
58+
value: ${{ steps.short-shas.outputs.COMMIT_SHORT_SHA }}
59+
60+
runs:
61+
using: "composite"
62+
steps:
63+
- name: Setup Blacksmith Docker cache
64+
uses: useblacksmith/setup-docker-builder@v1
65+
66+
- name: Resolve commit SHA
67+
shell: bash
68+
id: short-shas
69+
env:
70+
COMMIT_SHA_INPUT: ${{ inputs.COMMIT_SHA }}
71+
run: |
72+
set -euo pipefail
73+
if [[ -n "$COMMIT_SHA_INPUT" ]]; then
74+
COMMIT_SHA="$COMMIT_SHA_INPUT"
75+
else
76+
COMMIT_SHA=$(git rev-parse HEAD)
77+
fi
78+
echo "COMMIT_SHORT_SHA=$(echo "$COMMIT_SHA" | cut -c1-7)" >> "$GITHUB_OUTPUT"
79+
80+
- name: Configure AWS Credentials
81+
if: inputs.PUSH == 'true'
82+
uses: aws-actions/configure-aws-credentials@v3
83+
with:
84+
role-to-assume: ${{ inputs.AWS_ECR_OIDC_ARN }}
85+
aws-region: ${{ inputs.AWS_REGION }}
86+
87+
- name: Login to Amazon ECR
88+
if: inputs.PUSH == 'true'
89+
id: login-ecr
90+
uses: aws-actions/amazon-ecr-login@v2
91+
92+
- name: Authenticate to Google Cloud
93+
if: inputs.PUSH == 'true'
94+
id: auth-gcp
95+
uses: google-github-actions/auth@v2
96+
with:
97+
workload_identity_provider: ${{ inputs.GCP_WORKLOAD_IDENTITY_PROVIDER }}
98+
service_account: ${{ inputs.GCP_GITHUB_ACTIONS_SERVICE_ACCOUNT_EMAIL }}
99+
100+
- name: Configure Docker for GCP Artifact Registry
101+
if: inputs.PUSH == 'true'
102+
shell: bash
103+
env:
104+
GCP_AR_REGION: ${{ inputs.GCP_ARTIFACT_REGISTRY_REGION }}
105+
run: gcloud auth configure-docker "${GCP_AR_REGION}-docker.pkg.dev" --quiet
106+
107+
- name: Login to DockerHub
108+
if: inputs.PUSH == 'true'
109+
uses: docker/login-action@v3
110+
with:
111+
username: ${{ inputs.DOCKERHUB_USERNAME }}
112+
password: ${{ inputs.DOCKERHUB_TOKEN }}
113+
114+
- name: Resolve registries
115+
shell: bash
116+
id: get-registries
117+
env:
118+
PUSH: ${{ inputs.PUSH }}
119+
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
120+
GCP_AR_REGION: ${{ inputs.GCP_ARTIFACT_REGISTRY_REGION }}
121+
GCP_AR_PROJECT: ${{ inputs.GCP_ARTIFACT_REGISTRY_PROJECT_ID }}
122+
GCP_AR_NAME: ${{ inputs.GCP_ARTIFACT_REGISTRY_NAME }}
123+
run: |
124+
set -euo pipefail
125+
126+
if [[ "$PUSH" == "true" ]]; then
127+
printf '%s\n' "REGISTRY_AWS=${ECR_REGISTRY}" >> "$GITHUB_OUTPUT"
128+
printf '%s\n' "REGISTRY_GCP=${GCP_AR_REGION}-docker.pkg.dev/${GCP_AR_PROJECT}/${GCP_AR_NAME}" >> "$GITHUB_OUTPUT"
129+
printf '%s\n' "REGISTRY_DOCKERHUB=chromadb" >> "$GITHUB_OUTPUT"
130+
else
131+
printf '%s\n' "REGISTRY_AWS=local" >> "$GITHUB_OUTPUT"
132+
printf '%s\n' "REGISTRY_GCP=local" >> "$GITHUB_OUTPUT"
133+
printf '%s\n' "REGISTRY_DOCKERHUB=local" >> "$GITHUB_OUTPUT"
134+
fi
135+
136+
# Derives the set of image refs to probe from `docker buildx bake
137+
# --print`, so adding a new target to docker-bake.hcl doesn't require
138+
# editing this step. LOCAL_BUILD=false ensures the printed tags are
139+
# registry-qualified (the bake file only emits unqualified local tags
140+
# when LOCAL_BUILD=="true").
141+
- name: Short-circuit if images already exist
142+
id: short-circuit
143+
shell: bash
144+
env:
145+
LOCAL_BUILD: 'false'
146+
REGISTRY_AWS: ${{ steps.get-registries.outputs.REGISTRY_AWS }}
147+
REGISTRY_GCP: ${{ steps.get-registries.outputs.REGISTRY_GCP }}
148+
REGISTRY_DOCKERHUB: ${{ steps.get-registries.outputs.REGISTRY_DOCKERHUB }}
149+
COMMIT_SHORT_SHA: ${{ steps.short-shas.outputs.COMMIT_SHORT_SHA }}
150+
ADDRESS_SANITIZER: ${{ inputs.ADDRESS_SANITIZER }}
151+
ENABLE_AVX512: ${{ inputs.ENABLE_AVX512 }}
152+
FORCE: ${{ inputs.FORCE }}
153+
PUSH: ${{ inputs.PUSH }}
154+
run: |
155+
set -euo pipefail
156+
157+
if [[ "$PUSH" != "true" ]]; then
158+
echo "PUSH != true; short-circuit check disabled."
159+
echo "skipped=false" >> "$GITHUB_OUTPUT"
160+
exit 0
161+
fi
162+
163+
if [[ "$FORCE" == "true" ]]; then
164+
echo "FORCE=true; skipping short-circuit check."
165+
echo "skipped=false" >> "$GITHUB_OUTPUT"
166+
exit 0
167+
fi
168+
169+
mapfile -t refs < <(
170+
docker buildx bake --print -f docker-bake.hcl \
171+
| jq -r '.target | to_entries[].value.tags[]?'
172+
)
173+
174+
if [[ ${#refs[@]} -eq 0 ]]; then
175+
echo "::error::No image tags emitted by docker-bake.hcl; refusing to short-circuit."
176+
exit 1
177+
fi
178+
179+
all_present=true
180+
for ref in "${refs[@]}"; do
181+
if docker buildx imagetools inspect "$ref" >/dev/null 2>&1; then
182+
echo "found: $ref"
183+
else
184+
echo "missing: $ref"
185+
all_present=false
186+
fi
187+
done
188+
189+
if [[ "$all_present" == "true" ]]; then
190+
echo "All images already present at tag ${COMMIT_SHORT_SHA}; skipping build+push."
191+
echo "skipped=true" >> "$GITHUB_OUTPUT"
192+
else
193+
echo "skipped=false" >> "$GITHUB_OUTPUT"
194+
fi
195+
196+
# Build pass: fails fast on real build errors. The push is split out
197+
# below so registry flakes (DockerHub especially) can retry without
198+
# rebuilding.
199+
- name: Build service images
200+
if: steps.short-circuit.outputs.skipped != 'true'
201+
uses: docker/bake-action@v5
202+
with:
203+
push: false
204+
files: docker-bake.hcl
205+
env:
206+
LOCAL_BUILD: ${{ inputs.PUSH == 'true' && 'false' || 'true' }}
207+
REGISTRY_AWS: ${{ steps.get-registries.outputs.REGISTRY_AWS }}
208+
REGISTRY_GCP: ${{ steps.get-registries.outputs.REGISTRY_GCP }}
209+
REGISTRY_DOCKERHUB: ${{ steps.get-registries.outputs.REGISTRY_DOCKERHUB }}
210+
COMMIT_SHORT_SHA: ${{ steps.short-shas.outputs.COMMIT_SHORT_SHA }}
211+
ADDRESS_SANITIZER: ${{ inputs.ADDRESS_SANITIZER }}
212+
ENABLE_AVX512: ${{ inputs.ENABLE_AVX512 }}
213+
214+
# Retry to absorb transient registry failures. Buildx cache + registry
215+
# layer dedup make retries cheap — only whatever failed gets re-pushed.
216+
- name: Push service images
217+
if: inputs.PUSH == 'true' && steps.short-circuit.outputs.skipped != 'true'
218+
shell: bash
219+
env:
220+
LOCAL_BUILD: ${{ inputs.PUSH == 'true' && 'false' || 'true' }}
221+
REGISTRY_AWS: ${{ steps.get-registries.outputs.REGISTRY_AWS }}
222+
REGISTRY_GCP: ${{ steps.get-registries.outputs.REGISTRY_GCP }}
223+
REGISTRY_DOCKERHUB: ${{ steps.get-registries.outputs.REGISTRY_DOCKERHUB }}
224+
COMMIT_SHORT_SHA: ${{ steps.short-shas.outputs.COMMIT_SHORT_SHA }}
225+
ADDRESS_SANITIZER: ${{ inputs.ADDRESS_SANITIZER }}
226+
ENABLE_AVX512: ${{ inputs.ENABLE_AVX512 }}
227+
run: |
228+
set -euo pipefail
229+
230+
attempts=4
231+
delay=30
232+
for i in $(seq 1 "$attempts"); do
233+
if docker buildx bake --push -f docker-bake.hcl; then
234+
exit 0
235+
fi
236+
if [[ "$i" -lt "$attempts" ]]; then
237+
echo "::warning::docker buildx bake --push failed on attempt $i/$attempts; retrying in ${delay}s..."
238+
sleep "$delay"
239+
delay=$((delay * 2))
240+
fi
241+
done
242+
echo "::error::docker buildx bake --push failed after $attempts attempts"
243+
exit 1

‎.github/workflows/_build_release_service_images.yml‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
name: Build and publish service images
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
commit-sha:
7+
description: 'Full commit SHA to build. Defaults to the currently-checked-out commit.'
8+
required: false
9+
default: ''
10+
address_sanitizer:
11+
description: 'Enable Address Sanitizer for builds. Set to "1" to enable.'
12+
required: false
13+
default: ''
14+
enable_avx512:
15+
description: 'Enable AVX512 for builds. Set to "1" to enable.'
16+
required: false
17+
default: ''
18+
force:
19+
description: 'Skip the short-circuit check and always rebuild/push.'
20+
required: false
21+
default: false
22+
type: boolean
23+
workflow_call:
24+
inputs:
25+
commit-sha:
26+
description: 'Full commit SHA to build. Defaults to the currently-checked-out commit.'
27+
type: string
28+
required: false
29+
default: ''
30+
address_sanitizer:
31+
description: 'Enable Address Sanitizer for builds. Set to "1" to enable.'
32+
type: string
33+
required: false
34+
default: ''
35+
enable_avx512:
36+
description: 'Enable AVX512 for builds. Set to "1" to enable.'
37+
type: string
38+
required: false
39+
default: ''
40+
force:
41+
description: 'Skip the short-circuit check and always rebuild/push.'
42+
type: boolean
43+
required: false
44+
default: false
45+
secrets:
46+
DOCKERHUB_USERNAME:
47+
description: 'DockerHub username for authentication'
48+
required: false
49+
DOCKERHUB_TOKEN:
50+
description: 'DockerHub token for authentication'
51+
required: false
52+
SLACK_BOT_TOKEN:
53+
description: 'Slack bot token for failure notifications'
54+
required: true
55+
SLACK_CHANNEL_ID:
56+
description: 'Slack channel ID for failure notifications'
57+
required: true
58+
outputs:
59+
skipped:
60+
description: 'Whether the build+push was skipped because all images already existed.'
61+
value: ${{ jobs.build.outputs.skipped }}
62+
commit_short_sha:
63+
description: 'Short commit SHA used for the image tags.'
64+
value: ${{ jobs.build.outputs.commit_short_sha }}
65+
66+
env:
67+
AWS_REGION: us-east-1
68+
69+
# Avoid two concurrent runs building the same commit. A second trigger
70+
# waits for the first; when it runs, the short-circuit will hit and it
71+
# exits quickly.
72+
concurrency:
73+
group: build-service-images-${{ inputs.commit-sha || github.sha }}
74+
cancel-in-progress: false
75+
76+
jobs:
77+
build:
78+
runs-on: blacksmith-16vcpu-ubuntu-2404
79+
permissions:
80+
contents: read
81+
id-token: write
82+
outputs:
83+
skipped: ${{ steps.bake.outputs.skipped }}
84+
commit_short_sha: ${{ steps.bake.outputs.commit_short_sha }}
85+
steps:
86+
- name: Checkout
87+
uses: actions/checkout@v4
88+
with:
89+
ref: ${{ inputs.commit-sha || github.sha }}
90+
91+
- name: Build and publish
92+
id: bake
93+
uses: ./.github/actions/build_service_images
94+
with:
95+
COMMIT_SHA: ${{ inputs.commit-sha || github.sha }}
96+
PUSH: 'true'
97+
FORCE: ${{ inputs.force && 'true' || 'false' }}
98+
AWS_REGION: ${{ env.AWS_REGION }}
99+
AWS_ECR_OIDC_ARN: ${{ vars.AWS_ECR_OIDC_ARN }}
100+
GCP_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.GCP_WORKLOAD_IDENTITY_PROVIDER }}
101+
GCP_GITHUB_ACTIONS_SERVICE_ACCOUNT_EMAIL: ${{ vars.GCP_GITHUB_ACTIONS_SERVICE_ACCOUNT_EMAIL }}
102+
GCP_ARTIFACT_REGISTRY_REGION: ${{ vars.GCP_ARTIFACT_REGISTRY_REGION }}
103+
GCP_ARTIFACT_REGISTRY_PROJECT_ID: ${{ vars.GCP_ARTIFACT_REGISTRY_PROJECT_ID }}
104+
GCP_ARTIFACT_REGISTRY_NAME: ${{ vars.GCP_ARTIFACT_REGISTRY_NAME }}
105+
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
106+
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
107+
ADDRESS_SANITIZER: ${{ inputs.address_sanitizer }}
108+
ENABLE_AVX512: ${{ inputs.enable_avx512 }}
109+
110+
notify-slack-on-failure:
111+
name: Notify Slack on build failure
112+
if: ${{ always() && contains(needs.*.result, 'failure') }}
113+
needs:
114+
- build
115+
runs-on: blacksmith-2vcpu-ubuntu-2404
116+
steps:
117+
- name: Notify Slack
118+
uses: slackapi/slack-github-action@v2.0.0
119+
with:
120+
token: '${{ secrets.SLACK_BOT_TOKEN }}'
121+
method: chat.postMessage
122+
payload: |
123+
channel: ${{ secrets.SLACK_CHANNEL_ID }}
124+
text: |
125+
:x: *Service Image Build Failed!*
126+
*Workflow:* ${{ github.workflow }}
127+
*Run:* <https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run>
128+
*Ref:* <https://github.com/${{ github.repository }}/commit/${{ github.sha }}|${{ github.ref_name }}>
129+
*Author:* ${{ github.actor }}
130+
*Commit SHA:* ${{ inputs.commit-sha || github.sha }}
131+
*Address Sanitizer:* ${{ inputs.address_sanitizer || 'disabled' }}
132+
*AVX512:* ${{ inputs.enable_avx512 || 'disabled' }}

0 commit comments

Comments
 (0)