Skip to content

Update bootstrap file for KPNO package access.#78

Open
sybenzvi wants to merge 14 commits into
mainfrom
kpno-projects-update
Open

Update bootstrap file for KPNO package access.#78
sybenzvi wants to merge 14 commits into
mainfrom
kpno-projects-update

Conversation

@sybenzvi

Copy link
Copy Markdown

This bootstrap works with GitHub access restrictions at KPNO to download DESI packages. The desiutil kpno-projects-update branch is required for the script to work on the DESI cluster.

Appropriate guards in the bootstrap script should leave its behavior unchanged for users at NERSC and elsewhere.

@sbailey

sbailey commented Sep 3, 2025

Copy link
Copy Markdown
Contributor

Wow, what a pain to support installation at KPNO.

I like the idea of creating a config file with package names and versions, and then having the bootstrap script use that so that it can bootstrap more than just main, but also bootstrap the installation of sets of tags.

I don't like the idea of making that a verbose .ini format with custom parsing within a bash script, and having to maintain separate ini files for each site for each software release version. That feels fragile to maintain, e.g. ensuring that versions are consistent across sites, and when creating a new release making sure that each site has the correct subset of packages.

I suggest trying again, with

  • boostrap-desi.sh re-written into a python script for easier option and config file parsing
  • one config file per software version (main, 25.3, etc.) giving just the package name and version
    • for simplicity, I'd prefer trying a CSV or space-separated format read by the csv module rather than the long .ini format
  • bootstrap-desi.py --site SITENAME option to provide any customizations needed for KPNO, datatran, etc., e.g. limiting which packages are installed, or whether they are fetched via github or otherwise.
    • perhaps this should be a SITENAME.ini file? That would provide flexibility for updating the subset of packages list separate from the package versions, but I'm not sure what to suggest for a syntax to specify using git@desi-general:PRODUCT instead of https://github.com/desihub/PRODUCT. Maybe something like defining "URLPREFIX" with "https://github.com/desihub/" as the default, but could be overridden with "git@desi-general:"?
  • scope creep: this could also dump the "module load PACKAGE/VERSION" commands to add to a desimodules file, which is yet another place where a set of package names and versions are maintained.

There are probably other gotchas, but @sybenzvi does that core concept make sense? Thoughts / refinements?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants