uv.select uses huge amounts of ram (order 5x file size)

[AaronParsons]

recreate by running a pull_antpols with ~50 or so antennas

import aipy as a
uvi = a.miriad.UV('zen.2457471.18781.xx.uvc')
a.scripting.uv_selector(uvi, ants='(1,3,4,13,15,16,23,26,37,38,41,42,46,47,49,50,56,57,58,59,61,63,66,67,70,71,73,74,82,83,87,90,98,99,103,106,114,115,116,117,118,119,120,121,122,123,124,125,126,127)_(1,3,4,13,15,16,23,26,37,38,41,42,46,47,49,50,56,57,58,59,61,63,66,67,70,71,73,74,82,83,87,90,98,99,103,106,114,115,116,117,118,119,120,121,122,123,124,125,126,127)')
zen.2457471.18781.xx.uvc is 3.7GB
running this uv_selector uses about 20GB of ram.

The origin seems to be that each miriad.select does a malloc for each baseline that is order MAXANT^2. MAXANT was recently increased to 2048, making this issue more significant.

We are bypassing this issue for now in pull_antpols.py by selecting baselines in the uv.pipe with an mfunc

[AaronParsons]

From pkgw:

Inside MIRIAD, the UV selection code creates SELECT structures as UV selection terms are created. Each of these structures has an ants field that is of size order MAXANT**2 (see here) and MAXANT is currently 2048.

If you apply multiple "antennae" selections right in a row, MIRIAD will merge the selections in a single SELECT structure (relevant code around here), so that multiple antenna selections do not result in extra memory allocation.

However, the uv_selector function in scripting.py intersperses each selection term with "and" or "or" clauses, which breaks MIRIAD's merging, so that you end up consuming MAXANT**2 memory for every single term in the selection.

I do not understand the inclusion/exclusion logic of the low-level UV selection or of the uv_selector function, so it may not be possible to reproduce the logic in the full generality of cases supported by uv_selector right now. However, it is definitely possible to massively reduce the memory consumption in certain targeted cases by changing the way the uv_selector calls the uv.select function.