IndexError: list index out of range

[MrKri03]

Hello, I am trying to use ArrNorm for batch processing, using the command
arrnorm -i 15 -p 3 -ref _20220612_SR_B4.tif _19990801_SR_B4.tif
in anaconda prompt, with conda environment with all the requirements activated. I have removed the whole path in the following output:

==============================================================

ArrNorm - Automatic Relative Radiometric Normalization

Some code base on: Dr. Mort Canty
                   https://github.com/mortcanty/CRCDocker

Copyright (c) SMByC-IDEAM
Author: Xavier C. Llano <[email protected]>
Sistema de Monitoreo de Bosques y Carbono - SMByC
IDEAM, Colombia

==============================================================

Reference image:  _20220612_SR_B4.tif
Creating 3 multiprocesses

PROCESSING IMAGE: _19990801_SR_B4.tif (1)

======================================
Clipping the ref image with target: Image (1) _20220612_SR_B4.tif
Extent: 298335.0 6552495.0 309375.0 6514815.0
Clipped ref image successfully: _20220612_SR_B4__19990801_SR_B4_clip.tif

======================================
iMad process for: Image (1) _19990801_SR_B4.tif
------------IRMAD -------------
Fri Feb 23 16:20:59 2024
time1: \arr_norm\_20220612_SR_B4__19990801_SR_B4_clip.tif
time2: _19990801_SR_B4.tif

Stop condition: max iteration 15 with auto selection
the best delta for the final result:
 Image (1) -> iteration: 0, delta: 1.0 (Fri Feb 23 16:20:59 2024)

 WARNING: Occurred a exception value error for the last iteration No. 0,
 then the ArrNorm will be use the best result at the moment calculated, you
 should check the result and all bands in input file if everything is correct.

 Error: singular matrix
[]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):

File "\arr_env\Lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "\arr_env\Scripts\arrnorm", line 185, in meta_process
    process(*args)
  File "\arr_env\Scripts\arrnorm", line 181, in process
    norm_instance.run()
  File "\arr_env\Scripts\arrnorm", line 78, in run
    self.imad()
  File "\arr_env\Scripts\arrnorm", line 130, in imad
    self.img_imad = iMad.main(self.img_ref_clip, img_target,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\arr_env\Lib\site-packages\arrnorm\iMad.py", line 230, in main
    best_results = sorted(results, key=itemgetter(0))[0]
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "\arr_env\Scripts\\arrnorm", line 249, in <module>
    [proc for proc in imap_it]
  File "\arr_env\Scripts\\arrnorm", line 249, in <listcomp>
    [proc for proc in imap_it]
  File "\arr_env\Lib\multiprocessing\pool.py", line 873, in next
    raise value
IndexError: list index out of range

As I can see in the script, the "results" list is empty, thus, it is not getting any parameters for selection (only reaches delta:1.0)
The images are in the same coordinate system (EPSG:32635) and overlap. I have tried the command with absolute paths (pointing at different folders ) and pointing to the images in the same folder as the script.
However, the QGIS plugin is doing the process correctly. Maybe I am missing some parameters.

[XavierCLL]

Hi @MrKri03

Mmm, most of the cases for the "singular matrix" error are because there is no "match" at all between reference and target, and I think your images are correct, then maybe the issue is one of the preprocess of the Arrnorm, check if this temporal clip image is ok or empty: \arr_norm\_20220612_SR_B4__19990801_SR_B4_clip.tif. Or maybe a issue related with a special character in the paths, try to run it in a path that only has English characters without spaces.

Let me know how it goes

[MrKri03]

Thank you for your support @XavierCLL
As my intention is to include ArrNorm in a pipeline, I was using my rescaled images converted to reflectance values (0.0 - 1.0 , float). I changed the inputs with my original 16-bit integer DN images and it worked.
When I said "it worked in QGIS" , it was because I was doing a "testing" with original Landsat images and of course, it worked.
Maybe it can be included in the README that the input requires to be in integer type and not in float (rescaled reflectance values).
Regards,

[XavierCLL]

Hi @MrKri03

Thanks for testing that, yes that is the issue, I always use the integer rescaled image for that but in theory the normalization should work with float values. I gonna let this open so I remember to check that.

Regards

[MrKri03]

Thank you,
I am using Landsat Level 2 - Collection 2, which are distributed in 16-bit integer pixel depth. Then, I will start using ArrNorm on the DN images and then convert the results to reflectance values. Similarly, for thermal images too.

[XavierCLL]

yeah, that is the way, normalize and then convert it to the reflectance values. Enable running with float values allows it to run with other types of images beyond Landsat.