[Discussion] Release nightly cross-build-envs

[ryanking13]

Related: pyodide/pyodide#4918

---

Proposal

1. Create a new repository e.g. pyodide/cross-build-env-nightly (open to other names)
2. Store tip-of-tree cross-build environments in this repository.

Context

Currently, there is no official method for package maintainers to build their packages against unreleased versions of Pyodide. While we do publish tip-of-tree cross-build environments to a private S3 storage, it is not publicly accessible, and recommending its use to package maintainers isn't ideal.

To address this, I propose releasing nightly cross-build environments. These would allow package maintainers to build packages targeting the latest changes in Pyodide's development.

Release Frequency

We could approach releases in one of two ways:

1. Regularly (e.g., daily, weekly, or monthly).
2. As needed, particularly when significant or breaking changes occur (e.g., upgrading Emscripten, Python).

Option 2 seems more practical, as breaking changes occur infrequently.

[agriyakhetarpal]

Thanks for opening this! I would side with the first option, because publishing them as needed when breaking changes occur would mean that they would not really be "nightlies", so even if we need extra infrastructure to build and host them, it would be worth adding/keeping. My preference for the schedule would be that we publish them weekly, with an option to manually trigger a build and release using a workflow dispatch trigger.

Another aspect would be the reproducibility of successful builds and failures with nightlies for those who debug their builds with them, so it would be great that we don't overwrite them (like many Python packages do for their nightly wheels), and store them in GitHub releases where the release name would be appended with the week number (or better across years, the date) they were published on. For multiple releases on the same day, we will need to append a build number to the release, so that we don't face any errors, or overwrite the release files, like the https://github.com/softprops/action-gh-release action does.

If we do end up publishing them regularly, I am thinking forward and I have a question about what an interface for working with these nightlies would look like. Which of these options would be best, assuming we will follow CalVer?

1. Include the date of the release with a --nightly flag

pyodide xbuildenv install --nightly 2025-MM-DD

2. Same as point 1, but with a --pre flag instead, which is similar to interfaces like pip and uv, and hence likely to be more familiar with users

pyodide xbuildenv install --pre 2025-MM-DD

3. Include the xbuildenv version and append the string "nightly-2025-MM-DD" to it, so that users know which Pyodide version the xbuildenv largely corresponds to

pyodide xbuildenv install 0.27.0-nightly-2025-MM-DD

4. Use the same interface that we have, but allow passing --nightly separately to a particular xbuildenv version. The drawback here would be that a user won't know if we have broken the ABI (say, past a certain date, we don't have nightlies for 0.27.0, but for 0.28.0).

pyodide xbuildenv install 0.27.0 --nightly 2025-MM-DD

I think the third option would be the best one.

Also, should we add a pyodide xbuildenv search --nightly flag as well to go ahead with it, so that we can display compatibility data corresponding to a pyodide-build version? We would need to host two cross-build metadata files anyway (one for the regular xbuildenvs and one for these nightlies), and turning on --nightly in the search query would mean we will need to merge results from both JSON files before printing out the table (or JSON, using the --json option).

I was thinking we should probably fix pyodide/pyodide-build#72 before we enable the nightlies, (and maybe provide a pyodide xbuildenv purge option), because the xbuildenv folders would then clog up a lot of space on the system. But, then, the downside of storing them in the cache directory would be that it would be more difficult for users to see which xbuildenv would be used when they try to build their packages out-of-tree, which is where storing them in the same folder (usually next to a Pyodide virtual environment) would be nicer...

[hoodmane]

storing them in the same folder (usually next to a Pyodide virtual environment) would be nicer...

What about using a symlink for this?

[agriyakhetarpal]

Yes, a symlink would work, and there are no compatibility concerns either because we don't support Windows.

[hoodmane]

On linux, bazel puts the build in a cache and then symlinks directories bazel-out and bazel-bin in the local directory to the build cache. I'm not sure how it works on Windows, presumably it has some annoying workaround.

[ryanking13]

My preference for the schedule would be that we publish them weekly, with an option to manually trigger a build and release using a workflow dispatch trigger.

Sounds reasonable to me.

assuming we will follow CalVer?

Yes, let's use CalVer.

what an interface for working with these nightlies would look like.

Yes, I like the third option. We'll need to think of how we should manage the "compatible" version though.

Also, should we add a pyodide xbuildenv search --nightly flag as well to go ahead with it, so that we can display compatibility data corresponding to a pyodide-build version? We would need to host two cross-build metadata files anyway (one for the regular xbuildenvs and one for these nightlies), and turning on --nightly in the search query would mean we will need to merge results from both JSON files before printing out the table (or JSON, using the --json option).

We'll need to think about this. Maybe we can make a single metadata file and put everything there, but if we think there will be too many entries, separating the two would be better.

In the first step, let's start with not providing any interfaces. As users can install them through URL directory (pyodide xbuildenv install https://...).

Then, after releasing the nightly xbuildenvs gets stabilized, we can make a public interface for that.

[agriyakhetarpal]

In the first step, let's start with not providing any interfaces. As users can install them through URL directory (pyodide xbuildenv install https://...).

Thanks, that makes sense to me. I don't estimate a lot of users will be using these nightlies, so we can share additional information about the Emscripten and Pyodide versions on a case-by-case basis if they need it and ask about it for a particular nightly release.

[ryanking13]

I started to push nightly xbuildenvs to pyodide-build-environment-nightly repo. Making a release in that repository will build a fresh xbuildenv and upload it to Github Releases.

We'll need to figure out how to add more metadata in the release, but I think we can now tell package maintainers to use the xbuildenv in that repository when building packages against unreleased Pyodide versions.

I would like to say thanks to @agriyakhetarpal, as I referenced your code in pyodide-numpy-2.0-rebuilds a lot when writing the CI script.

[hoodmane]

@ryanking13 could you add a readme to:
https://github.com/pyodide/pyodide-build-environment-nightly

I'd like to know from it:

• Under what conditions is a release made (currently manually IIRC)
• How do I trigger a release?
• How do I use a release?

If they are going to be manual instead of scheduled we should have a designated place for downstream to ask for a new one.

[joemarshall]

Could it maybe have a scheduled job that checked for changes in emsdk version / python version?

[ryanking13]

@hoodmane. Yes, let me add it tomorrow.

Could it maybe have a scheduled job that checked for changes in emsdk version / python version?

That would be a little bit of overengineering, IMO. I think a scheduled job with regular releases per week (+ manual releases when needed) would be sufficient.