Updated dependencies, added a few functions and other edits

[palc001]

README has been edited to clearly state the edits.

[zhangzzk]

I find it still extremely slow to generate uv file with ska512 even if I use the fast file. It is possible to accelarate it using multi-process?

[palc001]

I find it still extremely slow to generate uv file with ska512 even if I use the fast file. It is possible to accelarate it using multi-process?

Hi @zhangzzk, it has been quite some time since I worked on this code so I've lost track of what exactly those methods imported from aipy are doing. I'd tried adding multiprocessing during earlier development but had run into some issues. However, now with better knowledge of python, I can definitely take a re-look at adding multiprocessing. But it will take time due to other commitments I have... It will be faster for your research if you do it yourself instead.

There could be 3 ways to go about it to make the code faster

1. There's is a for loop in the code. Try to implement it using numpy methods. You could look at numpy.apply_along_axis or numpy.apply_over_axes.
2. Wherever there's a function using numpy arrays, you could modify them to use JIT (Just-in-time) compiler by adding a simple decorator using the numba package. In fact, if possible, you could create a function for the for loop and add JIT compiler to it.
3. Finally, if the above still don't suffice your case, we could go with multiprocessing, but that comes with it's own set of complications.

Let me know your thoughts.

[zhangzzk]

Thank you very much for the detailed reply. I'll try your suggestions if neccessary. Besides, for your SKA sensitivity calculation, is there any work/papers I can refer to?

[palc001]

Thank you very much for the detailed reply. I'll try your suggestions if neccessary. Besides, for your SKA sensitivity calculation, is there any work/papers I can refer to?

@zhangzzk, firstly, my code is simply a fork of @jpober's repo. So, it would be wrong to call it "my" calculation in every sense, all credit goes to Prof. Pober.

The code is based on a paper that he published which is linked to in his repo. He has given a very rich literature review which cites various related works including ones that led to the calculations used in the code development.

Over in my fork of the repo, you'll find a PDF project report which lists my findings while I studied these sensitivity calculations and worked on a short project over it. That too has some citations which may help you.

Yesterday, when you commented, I had already presumed there would be some extended works existing on this and I searched them to refer you over. However, much to my surprise, I didn't find papers superceding Pober's. You could do a search for all latest papers that cite Pober's work on sensitivity calc too and let me know if you find anything.

You could also take a look at some latest works by Prof. A. Datta (DAASE, IIT Indore, India) on 21cm. Back when I did my project with him, there were ongoing discussions to use the same sensitivity calculations for SKA to train some neural networks and it's possible that they may have some helpful references. However, this is just my speculation, I'm not sure if they have extended this or if they found anything that supercedes this.

[zhangzzk]

Again, thank you very much for your help!

[jpober]

The original 21cmSense code has been deprecated for quite some time now. It's my fault for not formally updating this Github page and marking the code as such -- I do hope to that soon. A (much much much) more modern version of the code can be found here: https://github.com/steven-murray/21cmSense. There are still a few things to do before that becomes the official release, but that's where things are headed.

[palc001]

Thanks @jpober for leading us in the right direction. :)