Use polars instead of pandas for faster reading?

[arunkannawadi]

Axel Guinot has some code he put together to read the catalogs faster primarily by switching from pandas to polars. I think it'd be beneficial to upstream some of the changes, and have a unit test that makes sure that both these libraries give identical results.

[JoanneBogart]

I don't use pandas for reading. I use pyarrow.parquet.
What code is it you were thinking of changing?

[sidneymau]

Polars and pyarrow should give comparable performance. There might be some ways to optimize how the IO is done here using pyarrow.dataset, but that may require some architectural changes.