Add lsdb to desc td env

[rknop]

This will be generally useful going forward.

(Short-term, what I need is nested_pandas, but that's a prereq for lsdb, and lsdb is probably something we want anyway.)

[heather999]

I started playing with this - but doing a mamba install -c conda-forge lsdb results in attempts to downgrade cfitsio and numpy. That's not a big deal but currently we're basing the td_env on the LSST Sci Pipelines which is rather strict about its requirements?

I'd like to review with TD whether we truly need the LSST Sci Pipelines in the td_env. Rick had also asked about adding RAIL - which will also introduce some requirements on package versions. I could imagine making a TD env without LSST Sci pipelines and one with and perhaps another without RAIL.

Who can I bug about this? In the meantime.. I'll see what happens if I just add nested_pandas for now.

 - cfitsio                             4.6.2  h44b4e7a_0            conda-forge     Cached
  + cfitsio                             4.5.0  h44b4e7a_0            conda-forge      742kB
  - numpy                               2.2.6  py312h72c5963_0       conda-forge     Cached
  + numpy                               2.0.2  py312h58c1407_1       conda-forge        8MB

[heather999]

nested-pandas similarly attempts to downgrade these packages

[hombit]

Thank you @heather999! Both lsdb and nested-pandas do not depend on cfitsio, and I'm surprised about they attempting to downgrade numpy. nested-pandas has a short list of dependencies, see it here.

When I try to create a new environment with lsdb and python 3.10 with conda create -n lsdb python=3.10 lsdb the resolved numpy version is the same as yours. The same with nested-pandas.

I believe it may be a deeper problem with some transitive dependencies.

What is a good way to debug it? conda create --clone lsst-scipipe-4.1.0-exact on Perlmutter?

[heather999]

Thanks for looking! Unfortunately Perlmutter is down today for scheduled maintenance - we can try again when they are back up. If we could get that resolved it would help. There are other packages that similarly are causing downgrades. We're currently basing the environment on the latest LSST Sci Pipelines using python 3.12.

I would like to reduce the complexity of the td_env - I don't know that anyone is truly using the LSST Sci Pipelines, rather td_env CPU-only seems to be primarily to support FastDB, SNANA, resspect, and firecrown/cosmosis. .. and then a separate environment for GPU-enabled code. Dropping the LSST Science Pipelines would greatly simplify things, but I'd like to make sure that is ok with the TD group.

[heather999]

I've repeated this at NERSC, just installing w_2025_31

The install of w_2025_31 was done, where $1==w_2025_31

module load PrgEnv-gnu
module load cpu
module load cray-mpich-abi/8.1.30

unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH PYTHONPATH

export PYTHONNOUSERSITE=1

# Build Steps
curl -LO https://ls.st/lsstinstall
bash ./lsstinstall -X $1 
source ./loadLSST.bash
eups distrib install -t $1 lsst_distrib --nolocks

and then attempting to mamba install -c conda-forge lsdb which results in:

mamba install -c conda-forge lsdb |& tee install-lsdb.log

Pinned packages:

  - python=3.12


Transaction

  Prefix: /pscratch/sd/h/heatherk/share/2025-08-01-38-48/conda/envs/lsst-scipipe-10.0.0-exact

  Updating specs:

   - lsdb


  Package                             Version  Build                 Channel           Size
─────────────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────────────

  + bokeh                               3.7.3  pyhd8ed1ab_0          conda-forge        5MB
  + cdshealpix                          0.7.1  py312h0ccc70a_2       conda-forge        1MB
  + cytoolz                             1.0.1  py312h66e93f0_0       conda-forge      394kB
  + dask                             2025.7.0  pyhe01879c_0          conda-forge       12kB
  + distributed                      2025.7.0  pyhe01879c_0          conda-forge      848kB
  + hats                                0.6.2  pyhd8ed1ab_0          conda-forge        1MB
  + jproperties                         2.1.2  pyhd8ed1ab_1          conda-forge       22kB
  + libopentelemetry-cpp               1.21.0  hd1b1c89_0            conda-forge      882kB
  + libopentelemetry-cpp-headers       1.21.0  ha770c72_0            conda-forge      360kB
  + lsdb                                0.6.2  pyhd8ed1ab_0          conda-forge        3MB
  + lsst-sphgeom                  28.2025.200  py312hf79963d_0       conda-forge      552kB
  + lz4                                 4.4.4  py312hf0f0c11_0       conda-forge       40kB
  + mocpy                              0.18.0  py312h0ccc70a_0       conda-forge        1MB
  + msgpack-python                      1.1.1  py312h68727a3_0       conda-forge      103kB
  + narwhals                            2.0.1  pyhe01879c_0          conda-forge      241kB
  + nested-pandas                       0.4.7  pyhd8ed1ab_0          conda-forge       88kB
  + nlohmann_json                      3.12.0  h3f2d84a_0            conda-forge      136kB
  + prometheus-cpp                      1.3.0  ha5d0236_0            conda-forge      200kB
  + universal-pathlib                   0.2.6  hd8ed1ab_1            conda-forge        7kB
  + universal_pathlib                   0.2.6  pyhd8ed1ab_1          conda-forge       45kB
  + xyzservices                      2025.4.0  pyhd8ed1ab_0          conda-forge       49kB
  + zict                                3.0.0  pyhd8ed1ab_1          conda-forge       36kB

  Change:
─────────────────────────────────────────────────────────────────────────────────────────────

  - fitsverify                           4.22  h0ab0aa2_4            conda-forge     Cached
  + fitsverify                           4.22  ha2d72c2_3            conda-forge        8kB
  - healpy                             1.18.1  py312h78ed882_1       conda-forge     Cached
  + healpy                             1.18.1  py312h703ab46_0       conda-forge        3MB
  - rubin-env                          10.0.0  py312h5fef794_16      conda-forge     Cached
  + rubin-env                          10.0.0  py312h5fef794_4       conda-forge       11kB
  - rubin-env-nosysroot                10.0.0  py312hed51c57_16      conda-forge     Cached
  + rubin-env-nosysroot                10.0.0  py312hc70b54f_4       conda-forge       15kB
  - wcslib                              8.2.2  hb28848a_6            conda-forge     Cached
  + wcslib                              8.2.2  h60ba7f6_3            conda-forge      637kB

  Upgrade:
─────────────────────────────────────────────────────────────────────────────────────────────

  - libarrow                           17.0.0  hd64ae6c_62_cpu       conda-forge     Cached
  + libarrow                           20.0.0  h019e7cd_8_cuda       conda-forge        9MB
  - libarrow-acero                     17.0.0  hcb10f89_62_cpu       conda-forge     Cached
  + libarrow-acero                     20.0.0  hb826db4_8_cuda       conda-forge      627kB
  - libarrow-dataset                   17.0.0  hcb10f89_62_cpu       conda-forge     Cached
  + libarrow-dataset                   20.0.0  hb826db4_8_cuda       conda-forge      603kB
  - libarrow-substrait                 17.0.0  h1bed206_62_cpu       conda-forge     Cached
  + libarrow-substrait                 20.0.0  h69308b4_8_cuda       conda-forge      503kB
  - libparquet                         17.0.0  h081d1f1_62_cpu       conda-forge     Cached
  + libparquet                         20.0.0  h3f30f2e_8_cuda       conda-forge        1MB
  - pyarrow                            17.0.0  py312h9cebb41_2       conda-forge     Cached
  + pyarrow                            20.0.0  py312h7900ff3_0       conda-forge       26kB
  - pyarrow-core                       17.0.0  py312h01725c0_2_cpu   conda-forge     Cached
  + pyarrow-core                       20.0.0  py312h09cf70e_0_cuda  conda-forge        5MB

  Downgrade:
─────────────────────────────────────────────────────────────────────────────────────────────

  - cfitsio                             4.6.2  ha0b56bc_1            conda-forge     Cached
  + cfitsio                             4.5.0  h44b4e7a_0            conda-forge      742kB
  - numpy                               2.2.6  py312h72c5963_0       conda-forge     Cached
  + numpy                               2.0.2  py312h58c1407_1       conda-forge        8MB

  Summary:

  Install: 22 packages
  Change: 5 packages
  Upgrade: 7 packages
  Downgrade: 2 packages

  Total download: 45MB

[nevencaplar]

Is this acceptable, or we still need to solve cfitsio and numpy downgrade from lsdb side?

[heather999]

I think we really need to honor the version of packages like cfitsio and numpy that rubin_env installs, so I think we do need to sort out why lsdb or its dependencies might be looking to downgrade.

[heather999]

I'm now playing on my Ubuntu laptop rather than on NERSC Perlmutter where I could reproduce the downgrading if I install w_2025_31 LSST Science pipelines and then try to do mamba install -c conda-forge lsdb. I also found that if I do conda install -cconda-forge lsdb, it installs lsdb 0.2.6. If I force conda to install 0.6.3, conda install returns an error saying it cannot satisfy the request due to conflicts, which seem to point back to pyarrow:

Could not solve for environment specs
The following packages are incompatible
├─ lsdb ==0.6.3 * is installable and it requires
│  └─ nested-pandas >=0.4.6,<0.5.0 *, which requires
│     └─ pyarrow >=18 * with the potential options
│        ├─ pyarrow [18.0.0|18.1.0|...|21.0.0] would require
│        │  └─ python >=3.10,<3.11.0a0 *, which can be installed;
│        ├─ pyarrow [18.0.0|18.1.0|...|21.0.0] would require
│        │  └─ python >=3.11,<3.12.0a0 *, which can be installed;
│        ├─ pyarrow 18.0.0 would require
│        │  └─ pyarrow-core [==18.0.0 *_0_*|==18.0.0 *_1_*|==18.0.0 *_2_*] with the potential options
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0] would require
│        │     │  └─ __cuda >=11.8 *, which is missing on the system;
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0] would require
│        │     │  └─ python >=3.10,<3.11.0a0 *, which can be installed;
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0] would require
│        │     │  └─ python >=3.11,<3.12.0a0 *, which can be installed;
│        │     ├─ pyarrow-core [18.0.0|18.1.0|...|21.0.0] would require
│        │     │  └─ python >=3.13,<3.14.0a0 *, which can be installed;
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0] would require
│        │     │  └─ python >=3.9,<3.10.0a0 *, which can be installed;
│        │     └─ pyarrow-core 18.0.0 would require
│        │        └─ libarrow =18.0.0 *cpu with the potential options
│        │           ├─ libarrow 18.0.0 would require
│        │           │  └─ aws-crt-cpp >=0.29.4,<0.29.5.0a0 * with the potential options
│        │           │     ├─ aws-crt-cpp [0.29.3|0.29.4] would require
│        │           │     │  └─ aws-c-io >=0.15.1,<0.15.2.0a0 * with the potential options
│        │           │     │     ├─ aws-c-io [0.15.0|0.15.1] would require
│        │           │     │     │  └─ s2n >=1.5.6,<1.5.7.0a0 *, which can be installed;
│        │           │     │     └─ aws-c-io [0.15.1|0.15.2] would require
│        │           │     │        └─ s2n >=1.5.7,<1.5.8.0a0 *, which can be installed;
│        │           │     └─ aws-crt-cpp [0.29.4|0.29.5] would require
│        │           │        └─ aws-c-io >=0.15.2,<0.15.3.0a0 * with the potential options
│        │           │           ├─ aws-c-io [0.15.2|0.15.3] would require
│        │           │           │  └─ s2n >=1.5.9,<1.5.10.0a0 *, which can be installed;
│        │           │           └─ aws-c-io [0.15.1|0.15.2], which can be installed (as previously explained);
│        │           ├─ libarrow 18.0.0 would require
│        │           │  └─ aws-crt-cpp >=0.29.3,<0.29.4.0a0 * with the potential options
│        │           │     ├─ aws-crt-cpp [0.29.0|0.29.1|0.29.3] would require
│        │           │     │  └─ aws-c-io >=0.15.0,<0.15.1.0a0 * with the potential options
│        │           │     │     ├─ aws-c-io [0.14.20|0.15.0] would require
│        │           │     │     │  └─ s2n >=1.5.5,<1.5.6.0a0 *, which can be installed;
│        │           │     │     └─ aws-c-io [0.15.0|0.15.1], which can be installed (as previously explained);
│        │           │     └─ aws-crt-cpp [0.29.3|0.29.4], which can be installed (as previously explained);
│        │           ├─ libarrow 18.0.0 would require
│        │           │  └─ aws-crt-cpp >=0.29.5,<0.29.6.0a0 *, which can be installed (as previously explained);
│        │           └─ libarrow 18.0.0 would require
│        │              └─ aws-crt-cpp [>=0.29.0,<0.29.1.0a0 *|>=0.29.1,<0.29.2.0a0 *] with the potential options
│        │                 ├─ aws-crt-cpp 0.29.0 would require
│        │                 │  └─ aws-c-io >=0.14.20,<0.14.21.0a0 *, which can be installed (as previously explained);
│        │                 └─ aws-crt-cpp [0.29.0|0.29.1|0.29.3], which can be installed (as previously explained);
│        ├─ pyarrow [18.0.0|18.1.0|...|21.0.0] would require
│        │  └─ python >=3.13,<3.14.0a0 *, which can be installed;
│        ├─ pyarrow [18.0.0|18.1.0|...|21.0.0] would require
│        │  └─ python >=3.9,<3.10.0a0 *, which can be installed;
│        ├─ pyarrow 18.1.0 would require
│        │  └─ pyarrow-core ==18.1.0 *_0_* with the potential options
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which cannot be installed (as previously explained);
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│        │     ├─ pyarrow-core 18.1.0, which can be installed;
│        │     ├─ pyarrow-core [18.0.0|18.1.0|...|21.0.0], which can be installed (as previously explained);
│        │     └─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│        ├─ pyarrow 19.0.0 would require
│        │  └─ pyarrow-core ==19.0.0 *_0_* with the potential options
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which cannot be installed (as previously explained);
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│        │     ├─ pyarrow-core [18.0.0|18.1.0|...|21.0.0], which can be installed (as previously explained);
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│        │     └─ pyarrow-core 19.0.0 would require
│        │        └─ libarrow =19.0.0 *cpu, which requires
│        │           └─ aws-crt-cpp [>=0.29.8,<0.29.9.0a0 *|>=0.29.9,<0.29.10.0a0 *], which requires
│        │              └─ aws-c-io >=0.15.3,<0.15.4.0a0 * with the potential options
│        │                 ├─ aws-c-io 0.15.3 would require
│        │                 │  └─ s2n >=1.5.11,<1.5.12.0a0 *, which can be installed;
│        │                 ├─ aws-c-io 0.15.3 would require
│        │                 │  └─ s2n >=1.5.10,<1.5.11.0a0 *, which can be installed;
│        │                 └─ aws-c-io [0.15.2|0.15.3], which can be installed (as previously explained);
│        ├─ pyarrow 19.0.1 would require
│        │  └─ pyarrow-core ==19.0.1 *_0_* with the potential options
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which cannot be installed (as previously explained);
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│        │     ├─ pyarrow-core [18.0.0|18.1.0|...|21.0.0], which can be installed (as previously explained);
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│        │     └─ pyarrow-core 19.0.1, which can be installed;
│        ├─ pyarrow 20.0.0 would require
│        │  └─ pyarrow-core ==20.0.0 *_0_* with the potential options
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which cannot be installed (as previously explained);
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│        │     ├─ pyarrow-core [18.0.0|18.1.0|...|21.0.0], which can be installed (as previously explained);
│        │     ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│        │     └─ pyarrow-core 20.0.0, which can be installed;
│        └─ pyarrow 21.0.0 would require
│           └─ pyarrow-core ==21.0.0 *_0_* with the potential options
│              ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which cannot be installed (as previously explained);
│              ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│              ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│              ├─ pyarrow-core [18.0.0|18.1.0|...|21.0.0], which can be installed (as previously explained);
│              ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│              └─ pyarrow-core 21.0.0, which can be installed;
├─ pin on python =3.12 * is not installable because it requires
│  └─ python =3.12 *, which conflicts with any installable versions previously reported;
├─ pyarrow-core ==17.0.0 * is installable with the potential options
│  ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which cannot be installed (as previously explained);
│  ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│  ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│  ├─ pyarrow-core [17.0.0|18.0.0|...|21.0.0], which can be installed (as previously explained);
│  ├─ pyarrow-core 17.0.0 conflicts with any installable versions previously reported;
│  └─ pyarrow-core 17.0.0 would require
│     └─ python >=3.8,<3.9.0a0 *, which can be installed;
└─ s2n ==1.5.22 * is not installable because it conflicts with any installable versions previously reported.

Pins seem to be involved in the conflict. Currently pinned specs:
 - python=3.12

To simplify things, instead of using the LSST Science Pipelines, I just grabbed the latest Miniforge installer, and then set up an environment installing; python==3.12.* numpy==2.2.6 cfitsio==4.6.2 pyarrow==17.0.0 (these are the versions LSST Sci Pipelines uses in w_2025_31) followed by attempting to install lsdb using mamba and conda, and I see the same behavior.. the mamba install without any qualifiers on the versions would upgrade pyarrow and install lsdb 0.5.1. Pinning to lsdb 0.6.3 and pyarrow 17.0.0 results in a conflict. Here is what I did:

bash ./Miniforge3-Linux-x86_64.sh -b -p $PWD/py
source py/bin/activate
conda create -n py312 "python>=3.12,<3.13" numpy==2.2.6 cfitsio==4.6.2 pyarrow==17.0.0
conda activate py312
mamba install -c conda-forge "python>=3.12,<3.13" lsdb==0.6.3 pyarrow==17.0.0 

The error: (there's a similar conflict error when using conda install)

error    libmamba Could not solve for environment specs
    The following packages are incompatible
    ├─ lsdb ==0.6.3 * is installable and it requires
    │  └─ nested-pandas >=0.4.6,<0.5.0 *, which requires
    │     └─ pyarrow >=18 *, which can be installed;
    └─ pyarrow ==17.0.0 * is not installable because it conflicts with any installable versions previously reported.
critical libmamba Could not solve for environment specs

Looking at the recipes for lsdb (https://github.com/conda-forge/lsdb-feedstock/blob/main/recipe/meta.yaml) and nested-pandas (https://github.com/conda-forge/nested-pandas-feedstock/blob/main/recipe/meta.yaml), it does require pyarrow>=18

[heather999]

An immediate fix would be to pin nested-pandas to a version that allows pyarrow v17... which seems to be nested-pandas v0.3.8 and that requires pinning lsdb to some version before 0.6.2

I guess I either need LSST Sci Pipelines to bump up to pyarrow 18 or lsdb/nested-pandas need to support pyarrow 17. Are there still plans to add lsdb to LSST Sci Pipelines?

[hombit]

We can try to support pyarrow 17 with nested-pandas, could you please open an issue?

https://github.com/lincc-frameworks/nested-pandas/issues

[hombit]

From my understanding yes, LSST pipelines still plan to add LSDB