DataIO refactoring

[israelmcmc]

This issue superseeds #249 and #338, closed by #342.

This is meant to start the discussion on how DataIO will look like longer term. None of this is critical, it's only refactoring, and therefore it's low priotirity

I'll start with my thoughts. I propose:

• UnbinnedData and BinnedData should both derive from DataIO. DataIO should contain all common functions and properties (currently UnbinnedData derived from DataIO, and BinnedData from UnbinnedData)
• Both UnbinnedData and BinnedData should have the methods open(), write() and plot(). For BinnedData, these will mostly call histpy methods, instead of duplicating the IO code (tagging @jdbuhler since he had thoughts about this)
• Methods like get_raw_spectrum and get_raw_lightcurve should return another BinnedData object (with a reduced number of axes). They'll now how to open(), write() and plot() themselves already. Similarly for other axes --e.g. psichi, phi.
• BinnedData should have a classmethod from_unbinned() which takes an UnbinnedData object and bins it.
• UnbinnedData should have a method called to_binned() which internally calls BinnedData.from_unbinned(). This is a convenience method, but doesn't duplicate code.
• config file should become optional, only if the user finds them convenient. There should be a way to use DataIO programatically without config files.
• (Done in Interfaces refactoring #369 )~We should create an e.g. EventSelection class. This will take an event (or event list), and output True of False (or an array that works as a mask) depending on whether an event passed a cut or not. Derived classes from EventSelection can implement different types of cuts --e.g. energy, time, if it's in the SAA, if it's a GTI, number of hits, hit location, etc. Multiple event EventSelection objects can be concatenated. The UnbinnedData object should be able to take these as an input to filter out events. This mechanism should replace specific methods like select_data_energy, select_data_time, filter_good_data, cut_SAA_events, etc.
• The binning information should come from a histpy Axes, which can in turn from from a response.

[ckarwin]

@israelmcmc Thanks for starting this conversation. Generally I think your suggestions are good. I would like to try to work on this over the next year, in order to have it ready by DC4. I would also like to add better and more complete methods for chunking.

I see #249 and #338 are still open. Will they close automatically when v0.3.x are merged to main?

[israelmcmc]

I forgot to click the button. They are closed now, thanks for pointing it out.

[israelmcmc]

When we fix this, fix also the light curve tutorial. See
israelmcmc#22

[jdbuhler]

Two things I'd like to see in BinnedData:

(1) I'd like the ability to build up a binned representation by successively loading a series of unbinned files and calling fill() for each to add its contents to the BinnedData object. This would be a much more efficient idiom for combining and then binning unbinned data sets than actually concatenating the unbinned files, writing the result back out, and then rereading it for binning.

(2) For analyses where event timestamps are not needed (basically, all the binned analysis we do now), we should be able to specify that binning should discard event timestamps and produce a dense Histogram by binning on the remaining dimensions. Getting rid of the time axis in a large sparse BinnedData object is extremely expensive, and we do it all the time in the tutorials when we project onto the CDS dimensions. In the spectral fitting tutorials, this projection can take considerably more time than the actual fitting.