Nesting columns prior to crossmatching creates delayed error

[gitosaurus]

Bug report

# LSDB version 0.4.4
import lsdb
ztf_db = lsdb.read_hats(
    '/data3/epyc/data3/hats/catalogs/ztf_dr22/ztf_lc',
    margin_cache='/data3/epyc/data3/hats/catalogs/ztf_dr22/ztf_lc_10arcs')
nest_lc = ['hmjd', 'mag', 'magerr']
ztf_db = ztf_db.nest_lists(
    base_columns=[c for c in ztf_db.columns if c not in nest_lc],
    list_columns=nest_lc,
    name='lc')

gaia3 = lsdb.read_hats(
    'https://data.lsdb.io/hats/gaia_dr3/gaia', 
    margin_cache='https://data.lsdb.io/hats/gaia_dr3/gaia_10arcs')

cm = gaia3.crossmatch(ztf_db)
cmc = cm.compute()

The above yields the following error:

File ~/.conda/envs/dtj1s-py3.12/lib/python3.12/site-packages/pyarrow/array.pxi:4079, in pyarrow.lib.StructArray.from_arrays()

TypeError: Expected Array, got <class 'pyarrow.lib.ChunkedArray'>

If ztf_db is not nested prior to the crossmatch, the crossmatch succeeds.

Before submitting
Please check the following:

• I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
• I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
• If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.

[hombit]

I think it is fixed in nested-pandas v0.3.4, see lincc-frameworks/nested-pandas#190.

But now I have a different issue

import lsdb
ztf_db = lsdb.read_hats(
    '/data3/epyc/data3/hats/catalogs/ztf_dr22/ztf_lc',
    columns=['objectid', 'objra', 'objdec', 'hmjd', 'mag', 'magerr'],
    margin_cache='/data3/epyc/data3/hats/catalogs/ztf_dr22/ztf_lc_10arcs')
nest_lc = ['hmjd', 'mag', 'magerr']
ztf_db = ztf_db.nest_lists(
    base_columns=[c for c in ztf_db.columns if c not in nest_lc],
    list_columns=nest_lc,
    name='lc')

gaia3 = lsdb.read_hats(
    'https://data.lsdb.io/hats/gaia_dr3/gaia',
    margin_cache='https://data.lsdb.io/hats/gaia_dr3/gaia_10arcs')

cm = gaia3.partitions[0].crossmatch(ztf_db)
cmc = cm.compute()

File ~/.virtualenvs/default/lib/python3.10/site-packages/dask/dataframe/utils.py:363, in check_meta(x, meta, funcname, numeric_equal)
    357         return x
    358     errmsg = "Partition type: `{}`\n{}".format(
    359         typename(type(meta)),
    360         asciitable(["", "dtype"], [("Found", x.dtype), ("Expected", meta.dtype)]),
    361     )
--> 363 raise ValueError(
    364     "Metadata mismatch found%s.\n\n"
    365     "%s" % ((" in `%s`" % funcname if funcname else ""), errmsg)
    366 )

ValueError: Metadata mismatch found in `from_delayed`.

Partition type: `nested_pandas.nestedframe.core.NestedFrame`
+-------------+-------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| Column      | Found                                                 | Expected                                                                                           |
+-------------+-------------------------------------------------------+----------------------------------------------------------------------------------------------------+
| 'lc_ztf_lc' | nested<hmjd: [double], mag: [float], magerr: [float]> | nested<hmjd: [list<element: double>], mag: [list<element: float>], magerr: [list<element: float>]> |
+-------------+-------------------------------------------------------+----------------------------------------------------------------------------------------------------+

This expected dtype is wrong

[nevencaplar]

I confirm that the bug is still there as of March 23, 2025

[hombit]

@nevencaplar for me both codes run with no errors. I modified both codes to use data.lsdb.io and put search_filter=lsdb.ConeSearch(1, 1, 600) into all read_hats calls to reduce the data volume; I did check that the result is not empty. I used a fresh virtual environment with lsdb 0.5.0, nested_pandas 0.3.8, and Python 3.12.

What versions do you run with?

[nevencaplar]

Sees good with the setup that @hombit mentioned above.