Skip to content

fix: Remove unnecessary airflow files #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jiayuasu merged 1 commit into from
Dec 18, 2025
Merged

fix: Remove unnecessary airflow files #106

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 2 additions & 25 deletions .github/workflows/upload-notebook-to-s3.yml → .../workflows/upload-notebook-to-git-tag.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,14 @@
name: Convert Notebooks and Upload to Managed storage
name: Convert Notebooks to Python Scripts
on:
push:
branches:
- main

jobs:
convert-and-upload:
name: Convert Notebooks and Upload to Managed Storage
name: Convert Notebooks to Python Scripts
runs-on: ubuntu-latest
permissions:
id-token: write
contents: write

steps:
Expand Down Expand Up @@ -42,28 +41,6 @@ jobs:
tar -czf $RUNNER_TEMP/converted_notebooks.tar.gz -C $RUNNER_TEMP converted_notebooks
echo "output_path=$RUNNER_TEMP/converted_notebooks.tar.gz" >> $GITHUB_OUTPUT

- name: Configure AWS credentials for staging account
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.WHEROBOTS_STAGING_OFFICIAL_IAM_ROLE }}
role-session-name: integration-testing-file-upload
aws-region: us-west-2

- name: Upload python scripts to staging Managed storage
run: |
aws s3 cp $RUNNER_TEMP/converted_notebooks s3://wbts-wbc-rcv7vl73oy/djrm9bs9uf/data/shared/integration-testing-airflow/scripts/ --recursive

- name: Configure AWS credentials for prod account
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.WHEROBOTS_PROD_OFFICIAL_IAM_ROLE }}
role-session-name: integration-testing-file-upload
aws-region: us-west-2

- name: Upload python scripts to prod Managed storage
run: |
aws s3 cp $RUNNER_TEMP/converted_notebooks s3://wbts-wbc-m97rcg45xi/dfqlwcrxlk/data/shared/integration-testing-airflow/scripts/ --recursive

# We use a mutable git tag / github release called `converted-notebooks` to always point to the latest converted notebooks

- name: Update Converted Notebooks git tag
Expand Down