Remove channels when writing to varobs (FCI setup)

[Owen Lewis (orlewis)]

For FCI, some of the channels are required to be processed but not all are required to be written out into the varobs file.

This change enables the var_channels option to be used to reduce the number of channels which are written out into the varobs file.
For example if we had channels 5,6,7,9,10,11 and we only wanted channels 5 and 10 in the varobs file then var_channels would be set to 5, 10.
Previously the code would have put channels 5 and 10 into the 5 and 6 spaces. This change will now put the channels into the correct location of 5 and 10.

This however does hijack how var_channels is intended to be used (it was always assumed that the length of channels selected would be the same as the var_channels). It does mean if the length of var_channels is less then all_channels then this cannot be used for mapping the channels to new identities at the same time.

This has been run through here https://cylchub/services/cylc-review/view/owen.lewis?&suite=sith_fci_opsinputtest%2Frun1&no_fuzzy_time=0&path=log/job/20210701T1200Z/glu_var_jopa_anal_low/22/job.stats#957

ctests have also been created.

Running through all the ctests there are failures but that looks to be due to a missing file.
https://cylchub/services/cylc-review/taskjobs?per_page=15&user=owen.lewis&no_fuzzy_time=0&suite=mobbm-opsinputs%2Frun13&task_status=expired&task_status=failed&task_status=preparing&task_status=running&task_status=submitted&task_status=submit-failed&task_status=succeeded&task_status=waiting&page=1

Github copilot was used.

[PJLevensMO]

This looks to work for both SEVIRI and FCI in my testing so far - thanks for adding it.

• SEVIRI test, only keeping channels 5 and 10: https://cylchub/services/cylc-review/view/peter.levens?&suite=seviri_reducechan%2Frun3&no_fuzzy_time=0&path=log/job/20250812T1200Z/glu_var_jopa_anal_low/01/job.stats#889
• FCI test, only keeping channels 10 and 15: https://cylchub/services/cylc-review/view/peter.levens?&suite=fci_reducechan%2Frun9&no_fuzzy_time=0&path=log/job/20250812T1200Z/glu_var_jopa_anal_low/01/job.stats#901

To note, though, I had failures for FCI when I had the increase_chan_array varobs writer option set to true. Not sure why that was the case, but the output/error can be seen in this run: https://cylchub/services/cylc-review/view/peter.levens?&suite=fci_reducechan%2Frun9&no_fuzzy_time=0&path=log/job/20250812T1200Z/glu_jopa_process_background_fciclr/01/job.out - I can't see any obvious reason for the failure in the output. But removing that option appears to work as expected.

[PJLevensMO]

This however does hijack how var_channels is intended to be used (it was always assumed that the length of channels selected would be the same as the var_channels). It does mean if the length of var_channels is less then all_channels then this cannot be used for mapping the channels to new identities at the same time.

I think it may be worth at least opening an issue on this point to be looked at soon-ish - just in case we ever need to pull a channel from ABI or AHI which do use the varChannels option for mapping.

[Michael Cooke (mikecooke77)]

I've just been discussing this with Owen and I can't figure out why this should be needed. The mapping issue looks exactly the same issue as with SEVIRI but with different numbers. If I'm understanding the varobs writer correctly you should need something like this for FCI:

filter: VarObs Writer
    reject_obs_with_all_variables_failing_qc: true
    variables_for_quality_control:
    - name: brightnessTemperature
      channels: 10, 11, 12, 14, 15, 16
    varChannels: 10, 11, 12, 14, 15, 16
    compress_var_channels: false
    size_of_varobs_array: 16

PJLevensMO could you explain why this does not work?