Could future Docker Stacks be based on pre-defined Software Packs shared across Jupyter?

[krassowski]

What docker image(s) is this feature applicable to?

new community stack

What change(s) are you proposing?

In my possibly naive understanding docker-stacks does two things:

• provides Docker images optimized for (certain) Jupyter use cases
• curates the set of software available in those images

Around JupyterLab (jupyterlab-contrib/jupyterlab-contrib.github.io#23) and JupyterLab Desktop (jupyterlab/jupyterlab-desktop#213, jupyterlab/jupyterlab-desktop#1072) we independently arrived at a need to specify the set of software that can be easily installed for the user. However, Docker is not a dependency those applications can expect from the user.

From that an idea of a Jupyter software pack was born. See:

• the discussion on Zulip: #jupyterhub > Would pixi-based software packs be a useful thing for Hub?
• proposal for software packs development

How does this affect the user?

If we used the same set of software packs:

• a user who relies on docker-stacks in an environment someone provisions to them (e.g. in a JupyterHub deployment provided by their uni) could then install JupyterLab Desktop on their laptop and easily start with a similar set of packages without a need to slowly replicate what they had available in the provisioned environment
• a user who starts their journey with JupyterLab Desktop could ask the admin to provision a new docker image from docker-stacks after trying a similar set of packages locally

Anything else?

A software pack would be:

• name
• description
• maturity label (e.g. beta)
• set of packages with pinned ranges
• a lock file specifying a "known good" set of versions that get tested on CI

As implementation detail, I'm looking at backing this with pixi but this is not something that is central to the idea.

For Desktop users this could show up as:

Or when selecting combinations that are less popular:

[bollwyvl]

backing this with pixi but this is not something that is central to the idea

Perhaps, but there is a lot to be said for pixi's ability to scale what can be reproducibly acquired, resolved, tested, and verified, inside and outside of containers and CI. However, forcing use of pixi on a user machine introduces a level of complexity that not every user will need.

To get there, I still see a place for a dedicated constructor-based repo which results in platform-specific installers as intermediate, testable artifacts. These would offer "click link, get coffee, double-click, get desktop shortcut" on a Win/Linux/Mac desktop, usable directly from offline media. By including a package manager (micromamba is still a good choice) these can be transactionally upgraded after deployment, with sensible rollback.

Recent releases of constructor support creating installers with extra environments (ideal for kernels) and the Windows .msi output (a big win for managed environments).

With a little bit of plumbing, a single pixi.toml and matrixed .gitlab/workflows/ci.yml could build and test any number of confections, including allowed-to-fail pre-releases. In the very worst case, this could include nightly/weekly builds containing many jupyter* packages, built from source, when preparing for "release train"-style.

Given the above, consuming installers in containers (or flatpak, etc.) boils down to:

ARG THE_INSTALLER
ARG THE_ENV
FROM something/with-curl:latest
RUN curl -O https://somewhere/$THE_INSTALLER && bash $THE_INSTALLER -fbp /opt/$THE_ENV && rm $THE_INSTALLER
ENTRYPOINT ["what", "ever"]