Updated dependencies, added a few functions and other edits

README

@@ -4,28 +4,80 @@ Developer: Jonathan Pober (jpober <at> uw <dot> edu).
 
 ===============================================================================
 
+This code has been modified by Pallav Chanda (pallav16 <at> iiserb <dot> ac <dot> in) during a summer internship at IIT Indore under the guidance of Abhirup Datta (abhirup <dot> datta <at> iiti <dot> ac <dot> in) in Summer 2019.
+
+The work involves understanding the code, upgrading the dependencies, increasing its efficiency, adding new funtions and more. Please read project_report.pdf to understand how to run the code. It also contains more details about the new fuctions and the changes.
+
+DO THE FOLLOWING TO RUN:
+
+python fast_mk_array_file.py -C hera127
+This generates an array file for calc_sense. Change calibration file as needed from calibration_files folder (DO NOT include folder_name and .py, eg. mwa_compact replaces hera127 for MWA Compact).
+
+python calc_sense.py [*.npz]
+Calculates sensitivity
+([*.npz] is name of generated array file from fast_mk_array_file.py)
+(Result stored in another array file)
+
+python plot_sense.py [*.npz]
+Image of the calculated sensitivity and saves a .csv file for the same.
+([*.npz] is name of earlier generated array file from calc_sense.py)
+
+Please direct any Questions, comments, or feedback about the edits to pallav16 <at> iiserb <dot> ac <dot> in.
+
+===============================================================================
+
 Dependencies:
 
-numpy
-scipy
-aipy (https://github.com/AaronParsons/aipy)
+Python 2
+
+numpy (Tested on v1.16.6)
+scipy (Tested on v1.2.3)
+aipy (https://github.com/AaronParsons/aipy) (Tested on v3.0.1)
+healpy (Tested on v1.13.0)
 
 ===============================================================================
 
 There are two main code pieces: mk_array_file.py and calc_sense.py.  The array files produced by mk_array_file are the inputs to calc_sense.py.  The inputs to mk_array_file.py are calibration files, which are also python scripts, but not executable.  Each of these components will be described in turn.
 
+
 calibration files:
 
-These files will contain all the information about the array you are trying to calculate sensitivities for.  Most importantly, they contain the positions of all the antennas, but also information about the antenna size and system temperature.  An example calibration file for use with this package has been included: hera127.py.  To modify for other arrays, you should only need to change the values in the "ARRAY SPECIFIC PARAMETERS" section.
+These files will contain all the information about the array you are trying to calculate sensitivities for.  Most importantly, they contain the positions of all the antennas, but also information about the antenna size and system temperature.  Few calibration files have been included in the folder calibration_files for use with this package.  To modify for other arrays, you should only need to change the values in the "ARRAY SPECIFIC PARAMETERS" section.
+[Edits by Pallav Chanda: Added calibration files for MWA Compact Baseline, MWA Long Baseline, SKA and VLA arrays.]
+
 
 mk_array_file.py:
 
 This code takes calibration files as input (note that when calling a calibration file from the command line, one needs to omit the '.py' on the end), and returns a .npz file that can be read by calc_sense.py.  As it stands, this code should require no user modification for a drift scan with any antenna array.  For a tracked scan, the opts.track keyword should be set to the length of the track in hours.  Note that no correction is made for the dipole beam of phased array tiles while tracking.  Therefore, this calculation will break down for very long tracks.  The opts.bl_max keyword can also be specified to set a maximum size of the uv plane simulated (by default the longest baseline in the array is always included).  This can be useful for speeding up calculations where outrigger antennas greatly increase the size of the uv plane but provide little EoR sensitivity.
+[Edits by Pallav Chanda: Upgraded dependencies and minor changes.]
+
+
+fast_mk_array_file.py:
+[Added by Pallav Chanda]
+
+Implements 3-D arrays for storing data and also a few function definitions in mk_array_file.py so as to speed up the code by as much as 10-12% while consuming 4-6 times the memory. The user can choose to use it depending on machine configuration.
+
+
+calc_sense.py:
+
+This is the main portion of the package.  To run with defaults, it should only need an array file as input.  Through the command line, the user can change the foreground avoidance/subtraction model, the observing frequency, the epoch of reionization model (which is required to calculate sample variance), and the total observing time.   A redshift 9.5 ~50% ionization model produced by 21cmFAST has been included.  The code can natively handle any power spectrum output from 21cmFAST, but should be easily modifiable to include other models.  Since the calculated sensitivities are for a single frequency, one should change the observing frequency if one uses a power spectrum from a redshift other than the default (z = 9.5).  Changing more detailed parameters, like the sky temperature model, will require editing of the code itself.  The code also now contains the opts.no_ns option, which excludes the u = 0 column of the uv plane; this effectively removes north/south oriented baselines from the sensitivity ccalculation, which can be corrupted by systematics due to their low fringe rate.
+[Edits by Pallav Chanda: Changed system, antenna temperature model and minor changes.]
+
+
+calc_sense_mcquinn.py
+[Written by Pallav Chanda, available in main-pallav branch]
+
+This code modifies calc_sense.py to implement formulae from McQuinn (2008). Please check main-pallav branch for further details.
+
+
+plot_sense.py:
+[Added by Pallav Chanda]
+
+Plots the calculated sensitivity and saves a CSV file for the same.  Code doesn't automatically name files according to input. Do rename both output files manually or it will get replaced in the next run for plot_sense.py
+
 
-calc_sense.py: 
+[Further edits by Pallav Chanda: Solved various dependencies. Updated README with dependencies. Few files sorted for better understanding and codes modified accordingly. Added instructions to use the code in README.]
 
-This is the main portion of the package.  To run with defaults, it should only need an array file as input.  Through the command line, the user can change the foreground avoidance/subtraction model, the observing frequency, the epoch of reionization model (which is required to calculate sample variance), and the total observing time.   A redshift 9.5 ~50% ionization model produced by 21cmFAST has been included.  The code can natively handle any power spectrum output from 21cmFAST, but should be easily modifiable to include other models.  Since the calculated sensitivities are for a single frequency, one should change the observing frequency if one uses a power spectrum from a redshift other than the default (z = 9.5).  Changing more detailed parameters, like the sky temperature model, will require editing of the code itself.  The code also now contains the opts.no_ns option,
-which excludes the u = 0 column of the uv plane; this effectively removes north/south oriented baselines from the sensitivity ccalculation, which can be corrupted by systematics due to their low fringe rate.
 
 ===========================================================================
 
@@ -59,4 +111,4 @@ noise_interp2d:
 
 
 
-Questions, comments, or feedback should be directed to jpober <at> uw <dot> edu.If you use this code in any of your work, please acknowledge Pober et al. 2013AJ....145...65P and Pober et al. 2014ApJ...782...66P and provide a link to this repository.
+Questions, comments, or feedback should be directed to jpober <at> uw <dot> edu. If you use this code in any of your work, please acknowledge Pober et al. 2013AJ....145...65P and Pober et al. 2014ApJ...782...66P and provide a link to this repository.

calc_sense.py

@@ -88,7 +88,8 @@ def find_nearest(array,value):
 nchan = opts.nchan
 kpls = dk_deta(z) * n.fft.fftfreq(nchan,B/nchan)
 
-Tsky = 60e3 * (3e8/(array['freq']*1e9))**2.55  # sky temperature in mK
+Tsky = 180e3 * (array['freq']*1e3/180)**-2.5  # sky temperature in mK
+Tsys = Tsky #Trx not being added here as Tsky being calculated according to given formula
 n_lstbins = opts.n_per_day*60./obs_duration
 
 #===============================EOR MODEL===================================
@@ -143,7 +144,6 @@ def find_nearest(array,value):
        if k > n.max(mk): continue
        tot_integration = uv_coverage[iv,iu] * opts.ndays
        delta21 = p21(k)
-       Tsys = Tsky + Trx
        bm2 = bm/2. #beam^2 term calculated for Gaussian; see Parsons et al. 2014
        bm_eff = bm**2 / bm2 # this can obviously be reduced; it isn't for clarity
        scalar = X2Y(z) * bm_eff * B * k**3 / (2*n.pi**2)
@@ -175,8 +175,11 @@ def find_nearest(array,value):
     sense1d[ind] = kbin**-.5
     Tsense1d[ind] = Tsense1d[ind]**-.5
 
+#save the power spectrum in an array
+power = p21(kmag)
+
 #save results to output npz
-n.savez('%s_%s_%.3f.npz' % (name,opts.model,array['freq']),ks=kmag,errs=sense1d,T_errs=Tsense1d)
+n.savez('%s_%s_%.3f.npz' % (name,opts.model,array['freq']), ks=kmag, errs=sense1d, T_errs=Tsense1d, ps21=power)
 
 #calculate significance with least-squares fit of amplitude
 A = p21(kmag)

calibration_files/antpos_for_other_arrays/mwa_compact_antpos.csv

@@ -0,0 +1,129 @@
+Tile, Lat, Long, Height, m East, m North
+11,-26.70094317,116.6692851,377.018,-149.785,265.814
+12,-26.70090368,116.6698319,377.291,-95.365,270.188
+13,-26.70094128,116.6699007,377.315,-88.512,266.021
+14,-26.70100978,116.6699993,377.297,-78.698,258.431
+15,-26.70109983,116.6699165,377.344,-86.94,248.452
+16,-26.70113922,116.6697619,377.264,-102.325,244.088
+17,-26.70115554,116.6698521,377.275,-93.346,242.279
+18,-26.70135417,116.6692997,377.22,-148.33,220.271
+21,-26.70079159,116.6694543,377.01,-132.939,282.611
+22,-26.70085578,116.6693847,377.026,-139.867,275.498
+23,-26.70077134,116.6695475,377.058,-123.667,284.854
+24,-26.70080288,116.6698567,377.261,-92.889,281.358
+25,-26.70078951,116.670008,377.26,-77.835,282.839
+26,-26.70083713,116.6699513,377.307,-83.476,277.563
+27,-26.70088158,116.6700165,377.304,-76.988,272.637
+28,-26.7007258,116.6690216,376.632,-176.006,289.903
+31,-26.70152311,116.6705527,376.808,-23.62,201.546
+32,-26.70133534,116.6705509,376.876,-23.802,222.353
+33,-26.70121967,116.6703832,376.965,-40.489,235.17
+34,-26.70119849,116.6702422,377.128,-54.527,237.518
+35,-26.70107852,116.6701385,377.244,-64.851,250.813
+36,-26.70100101,116.670086,377.272,-70.075,259.402
+37,-26.70096243,116.6701866,377.186,-60.057,263.676
+38,-26.70095459,116.6702809,377.106,-50.676,264.546
+41,-26.70089542,116.6701069,377.242,-67.99,271.103
+42,-26.70088818,116.6701866,377.21,-60.058,271.905
+43,-26.70082579,116.6700778,377.259,-70.885,278.818
+44,-26.70087608,116.6703325,377.072,-45.54,273.245
+45,-26.7008488,116.670256,377.096,-53.156,276.268
+46,-26.70096226,116.6705624,376.954,-22.659,263.694
+47,-26.70077645,116.6702365,377.188,-55.097,284.285
+48,-26.70087043,116.6706403,376.916,-14.9,273.87
+61,-26.70138204,116.6697322,377.26,-105.281,217.18
+62,-26.70126489,116.6698001,377.265,-98.53,230.162
+63,-26.70127233,116.6699677,377.275,-81.845,229.337
+64,-26.701193,116.6699909,377.275,-79.541,238.128
+65,-26.70111292,116.6700353,377.26,-75.12,247.002
+66,-26.70121284,116.6700765,377.244,-71.017,235.929
+67,-26.70127548,116.6701614,377.214,-62.567,228.987
+68,-26.70133957,116.6702807,377.082,-50.691,221.885
+81,-26.69816917,116.6691778,374.801,-160.465,573.207
+82,-26.70017899,116.6695035,376.581,-128.046,350.494
+83,-26.70068851,116.670089,377.199,-69.771,294.031
+84,-26.70065938,116.669999,377.203,-78.731,297.259
+85,-26.70062641,116.6697457,377.058,-103.941,300.914
+86,-26.70073486,116.6697836,377.164,-100.172,288.896
+87,-26.69999265,116.6668168,374.538,-395.452,371.152
+88,-26.69982907,116.6681421,375.339,-263.547,389.274
+91,-26.70150438,116.6740669,373.957,326.141,203.608
+92,-26.70159479,116.6725325,375.037,173.425,193.595
+93,-26.70069923,116.6707648,376.899,-2.515,292.841
+94,-26.70074772,116.6703223,377.122,-46.554,287.469
+95,-26.70067144,116.670226,377.163,-56.141,295.922
+96,-26.70017396,116.6704514,376.986,-33.705,351.048
+97,-26.69850344,116.6712937,376.204,50.127,536.159
+98,-26.69865699,116.6716429,376.336,84.88,519.141
+HexE01,-26.700755,116.671683,376.24,86.373,284.121
+HexE02,-26.700755,116.671824,376.172,100.347,284.097
+HexE03,-26.700755,116.671964,375.922,114.346,284.092
+HexE04,-26.700755,116.672105,375.764,128.34,284.104
+HexE05,-26.700865,116.671613,376.232,79.369,271.986
+HexE06,-26.700865,116.671753,376.061,93.339,271.976
+HexE07,-26.700865,116.671894,375.941,107.365,271.957
+HexE08,-26.700865,116.672035,375.816,121.355,271.991
+HexE09,-26.700865,116.672175,375.651,135.367,271.962
+HexE10,-26.700974,116.671542,376.226,72.38,259.848
+HexE11,-26.700974,116.671683,376.091,86.349,259.844
+HexE12,-26.700974,116.671824,375.948,100.373,259.847
+HexE13,-26.700974,116.671964,375.814,114.357,259.839
+HexE14,-26.700974,116.672105,375.685,128.346,259.848
+HexE15,-26.700974,116.672246,375.498,142.36,259.849
+HexE16,-26.701084,116.671472,376.221,65.362,247.723
+HexE17,-26.701084,116.671613,376.088,79.366,247.742
+HexE18,-26.701084,116.671753,375.943,93.355,247.743
+HexE19,-26.701084,116.672035,375.71,121.369,247.744
+HexE20,-26.701084,116.672175,375.576,135.353,247.721
+HexE21,-26.701084,116.672316,375.392,149.358,247.731
+HexE22,-26.701193,116.671543,376.085,72.363,235.61
+HexE23,-26.701193,116.671683,375.969,86.344,235.606
+HexE24,-26.701193,116.671824,375.853,100.367,235.616
+HexE25,-26.701193,116.671964,375.74,114.335,235.598
+HexE26,-26.701193,116.672105,375.559,128.349,235.61
+HexE27,-26.701193,116.672245,375.442,142.34,235.586
+HexE28,-26.701302,116.671613,375.995,79.374,223.474
+HexE29,-26.701302,116.671753,375.867,93.343,223.488
+HexE30,-26.701303,116.671894,375.747,107.357,223.462
+HexE31,-26.701303,116.672035,375.56,121.372,223.488
+HexE32,-26.701303,116.672175,375.466,135.352,223.486
+HexE33,-26.701412,116.671683,375.877,86.362,211.341
+HexE34,-26.701412,116.671824,375.742,100.364,211.338
+HexE35,-26.701412,116.671964,375.593,114.359,211.353
+HexE36,-26.701412,116.672105,375.475,128.345,211.358
+HexS01,-26.701889,116.670715,376.542,-9.985,158.452
+HexS02,-26.70189,116.670855,376.411,4.006,158.456
+HexS03,-26.70189,116.670996,376.27,18.022,158.439
+HexS04,-26.70189,116.671137,376.158,32.013,158.442
+HexS05,-26.701999,116.670644,376.596,-17.013,146.309
+HexS06,-26.701999,116.670785,376.428,-2.985,146.316
+HexS07,-26.701999,116.670926,376.323,11.022,146.311
+HexS08,-26.701999,116.671067,376.16,25.002,146.32
+HexS09,-26.701999,116.671207,376.042,39.014,146.304
+HexS10,-26.702108,116.670574,376.578,-23.966,134.201
+HexS11,-26.702108,116.670715,376.416,-9.983,134.185
+HexS12,-26.702108,116.670855,376.314,4.005,134.206
+HexS13,-26.702108,116.670996,376.164,17.99,134.182
+HexS14,-26.702108,116.671137,376.056,32.009,134.184
+HexS15,-26.702108,116.671278,375.935,46.016,134.221
+HexS16,-26.702218,116.670504,376.602,-30.991,122.075
+HexS17,-26.702218,116.670644,376.457,-16.989,122.07
+HexS18,-26.702218,116.670785,376.332,-2.972,122.058
+HexS19,-26.702218,116.671067,376.089,25.015,122.055
+HexS20,-26.702218,116.671207,375.961,39.014,122.09
+HexS21,-26.702218,116.671348,375.822,53.019,122.052
+HexS22,-26.702327,116.670574,376.51,-23.994,109.937
+HexS23,-26.702327,116.670715,376.345,-9.98,109.944
+HexS24,-26.702327,116.670855,376.229,3.996,109.951
+HexS25,-26.702327,116.670996,376.088,18.025,109.959
+HexS26,-26.702327,116.671137,375.978,32.024,109.944
+HexS27,-26.702327,116.671277,375.834,46.022,109.949
+HexS28,-26.702437,116.670644,376.369,-16.987,97.81
+HexS29,-26.702437,116.670785,376.228,-2.998,97.819
+HexS30,-26.702437,116.670926,376.149,11.019,97.833
+HexS31,-26.702437,116.671067,375.997,25.004,97.836
+HexS32,-26.702437,116.671207,375.917,39.021,97.82
+HexS33,-26.702546,116.670715,376.251,-9.987,85.702
+HexS34,-26.702546,116.670855,376.169,4.018,85.698
+HexS35,-26.702546,116.670996,376.049,18.018,85.716
+HexS36,-26.702546,116.671137,375.92,32.013,85.713