Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add scripting and documentation for pushing releases to PyPI. #519

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

ScottTodd
Copy link
Member

@ScottTodd ScottTodd commented Nov 14, 2024

Progress on #400.

This scripting allows us to publish .whl files from https://github.com/nod-ai/SHARK-Platform/releases/tag/dev-wheels to PyPI. Here are the basic steps:

  1. Download wheels for a specific pre-release (e.g. 2.9.1rc20241114)
  2. Edit the versions in the downloaded wheels to remove the rcYYYYMMDD suffix
  3. Build the shark-ai meta package using the versions in those whls (NOTE: currently this uses the versions in the source tree, see below)
  4. Upload all wheels to PyPI

Logs of this running: https://gist.github.com/ScottTodd/9c7418d5bbbebc8aea72a39bc2ac37b0 (to push 2.9.1 to PyPI)

The new pypi_deploy.sh script is based on the similar script we maintain in IREE: https://github.com/iree-org/iree/blob/main/build_tools/python_deploy/pypi_deploy.sh . The README.md file is also forked from IREE.

Known sharp edges with the publishing process:

  • This currently mixes local information (versions from the source tree where this is running) with remote information (versions from the nightly release packages). Publishing nightly meta packages will let us simplify and make this safer to run from arbitrary working trees
  • The local build of the shark-ai meta package copies all files in shark-ai/build_tools/wheelhouse/ to the working directory that gets sent to twine upload *. If there are preexisting files in that directory they will be published.
  • The step that downloads releases from GitHub uses pip download so it can leverage PyPI's version resolution logic, but I couldn't figure out a way to download 3.13t wheels without running a 3.13t python interpreter. I happen to have 3.13t installed on my system, but we shouldn't require that in the scripting for all release engineers. We could try using the gh tool as in IREE, if we properly filter the download to just the version we want: https://github.com/iree-org/iree/blob/9eaa4ef7d6b439d8c444b533beddd82146578e25/build_tools/python_deploy/pypi_deploy.sh#L69-L72 (that has one release per nightly, while here we have a single release shared between all nightlies)

Currently downloads some wheels given a specified version. Has code to edit release versions and upload, taken from other similar scripts, but not tested e2e.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant