Unwrapped list of nested frame columns

[camposandro]

It would be useful to have a frame property that returns the list of columns with unwrapped nested column names in the following format: {base_column}.{col_name}. In LSDB we are already doing this unwrapping of column names to specify columns to load, default columns, etc:

all_columns = self._ddf._meta.all_columns
base_columns = all_columns.pop("base", [])
nonnested_basecols = [c for c in base_columns if c not in self._ddf._meta.nested_columns]
loading_columns = [
    [f"{base_name}.{col}" for col in all_columns[base_name]] for base_name in all_columns
]
columns = nonnested_basecols + [c for cs in loading_columns for c in cs]

We will need to use it more broadly in other contexts (e.g. in astronomy-commons/lsdb#1029) so it would be better to have it done upstream.

[dougbrn]

@camposandro @hombit @nevencaplar
As it so happens, I recently added a new function to the v0.6 branch #363 called _get_subcolumns which is very similar in scope. Some questions on what exactly we/you would want:

• Is it important for this to be a public function? Currently _get_subcolumns is private, because it's more targeted for internal use but I could see value in a public function as well
• Is it preferred for this to be a property? _get_subcolumns takes in a list of columns to retrieve subcolumns for, so to get all subcolumns from it currently you would do nf._get_subcolumns(nf.nested_columns), would it be nice to have either a property that directly grabs from all cols in nf.nested_columns, or even just have nf._get_subcolumns() default to nf.nested_columns when not specified.

[dougbrn]

We discussed it briefly, but will make _get_subcolumns public with default behavior of returning the subcolumn "dot" names from all nested columns. In v0.6 (#363 ) the path of least resistance for the above codeblock will be:

columns = nf.base_columns + nf.get_subcolumns()

[camposandro]

Thanks @dougbrn! I was just about to update this thread. Your suggestion looks very reasonable for the time being!