Errors in extraction of thalweg polygons in the Alaska domain

[BahramKhazaei-NOAA]

@cuill After fixing the problem with impi, @SorooshMani-NOAA were trying to run the pyDEM package on a DEM in the Alaska domain which is extracted from GMRT. However, there was a problem with processing the DEM and his pyDEM script failed. We then tried another DEM that has coverage over the Cook Inlet and got another error that seems like is due to existence of missing values in the DEM (it has 'no data' value in some pixels). Here are the error messages in processing of each DEM:

Cook Inlet DEM:

(meshdev) [Soroosh.Mani@compute-0001 Bahram]$ python run_serial.py
./cook_1e7

A Priority-Flood+Epsilon
C Barnes, R., Lehman, C., Mulla, D., 2014. Priority-flood: An optimal depression-filling and watershed-labeling algorithm for digital elevation models. Computers & Geosciences 62, 117–127. doi:10.1016/j.cageo.2013.04.024

c topology = D8
p Setting up boolean flood array matrix...
p Adding cells to the priority queue...
p Performing Priority-Flood+Epsilon...
t succeeded in 2.66116 s
m Cells processed = 36056748
m Cells in pits = 30266840
W W In assigning negligible gradients to depressions, some depressions rose above the surrounding cells. This implies that a larger storage type should be used. The problem occured for 70 of 36056748.
total time=19.3s: ./cook_1e7
finish processing ./cook_1e7
---------compute watershed------------------------------------
---------sort watershed number--------------------------------
computing river network
Traceback (most recent call last):
  File "/lustre/meshing/river/extract/pyDEM_Samples/Bahram/run_serial.py", line 38, in <module>
    gdf = S.write_shapefile(npt_smooth=3)
  File "/home/Soroosh.Mani/sandbox/RiverMeshTools/pyDEM/pyDEM/dem.py", line 2387, in write_shapefile
    yi,xi=self.get_coordinates(SF.data[i])
  File "/home/Soroosh.Mani/sandbox/RiverMeshTools/pyDEM/pyDEM/dem.py", line 2304, in get_coordinates
    indy,indx=unravel_index(sind,ds)
  File "<__array_function__ internals>", line 200, in unravel_index
TypeError: only int indices permitted

GMRT DEM:

(meshdev) [Soroosh.Mani@compute-0001 Bahram]$ python run_serial.py
./gmrt_1e7

A Priority-Flood+Epsilon
C Barnes, R., Lehman, C., Mulla, D., 2014. Priority-flood: An optimal depression-filling and watershed-labeling algorithm for digital elevation models. Computers & Geosciences 62, 117–127. doi:10.1016/j.cageo.2013.04.024

c topology = D8
p Setting up boolean flood array matrix...
p Adding cells to the priority queue...
p Performing Priority-Flood+Epsilon...
t succeeded in 7.31966 s
m Cells processed = 60838806
m Cells in pits = 44421818
W W In assigning negligible gradients to depressions, some depressions rose above the surrounding cells. This implies that a larger storage type should be used. The problem occured for 37223 of 60838806.
total time=36.3s: ./gmrt_1e7
finish processing ./gmrt_1e7
---------compute watershed------------------------------------
---------sort watershed number--------------------------------
computing river network
Traceback (most recent call last):
  File "/lustre/meshing/river/extract/pyDEM_Samples/Bahram/run_serial.py", line 38, in <module>
    gdf = S.write_shapefile(npt_smooth=3)
  File "/home/Soroosh.Mani/sandbox/RiverMeshTools/pyDEM/pyDEM/dem.py", line 2402, in write_shapefile
    idxs=np.squeeze(np.argwhere(np.isnan(river[:,0])))
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

These errors occur when using the sample code from pyDEM readme (run_serial.py just with updated paths for our DEMs). He tried to perform some pre-processing on both DEMs but even with fixed DEMs he got the exact same errors.

I will send you the GMRT file (too big to attach here). It would be great if you can take a look at this and see if you have a solution/fix for this.

[cuill]

@BahramKhazaei-NOAA The dem file has "nan" values, which may cause this error because pyDEM takes "-9999" as nodata. The gdal command to change nodata value cannot process this file:
gdal_calc.py -A dem_with_ocean.tif --calc="numpy.where(A<-100000, -9999, A)" --outfile dem_with_ocen_nodata.tif --NoDataValue=-9999

Do you have any method to change 'nan' to -9999? @BahramKhazaei-NOAA @SorooshMani-NOAA

[cuill]

@BahramKhazaei-NOAA @SorooshMani-NOAA
Find a method to do it:
https://gis.stackexchange.com/questions/163685/reclassify-a-raster-value-to-9999-and-set-it-to-the-nodata-value-using-python-a/163705#163705

[BahramKhazaei-NOAA]

@BahramKhazaei-NOAA @SorooshMani-NOAA Find a method to do it: https://gis.stackexchange.com/questions/163685/reclassify-a-raster-value-to-9999-and-set-it-to-the-nodata-value-using-python-a/163705#163705

Thank you. I was trying with QGIS, but it crashed. Did that work for you?

[cuill]

@BahramKhazaei-NOAA
Yes, just use the last rule:

# Set large negative values to -9999
#Z[Z<-9999]= -9999
# Choose your preference: (comment either rule)
#Z[Z==-9999]= np.nan
# Or 
Z[np.isnan(Z)]= -9999

[SorooshMani-NOAA]

I'll try this out and let you know if we still see issues. Thanks @cuill

[BahramKhazaei-NOAA]

I'll try this out and let you know if we still see issues. Thanks @cuill

Thanks, Soroosh.

[SorooshMani-NOAA]

@cuill using GMRT DEM, I read the raster values and set all nan's to -9999, but I still get the same error as before:

Traceback (most recent call last):
  File "/lustre/meshing/river/extract/pyDEM_Samples/Bahram/run_serial.py", line 38, in <module>
    gdf = S.write_shapefile(npt_smooth=3)
  File "/home/Soroosh.Mani/sandbox/RiverMeshTools/pyDEM/pyDEM/dem.py", line 2402, in write_shapefile
    idxs=np.squeeze(np.argwhere(np.isnan(river[:,0])))
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed```

This is my raster now:

In [5]: with rio.open('./fixed_gmrt.tif', 'r') as ds:
   ...:     v = ds.read(1)
   ...:

In [6]: v
Out[6]:
array([[  -31.025713,   -31.001385,   -30.999046, ...,  1909.2639  ,
         1822.1892  ,  1735.6036  ],
       [  -30.996735,   -31.061066,   -31.002304, ...,  1906.1929  ,
         1839.0338  ,  1778.5045  ],
       [  -30.894072,   -30.888163,   -31.011707, ...,  1883.3633  ,
         1867.0518  ,  1846.1737  ],
       ...,
       [-5467.1333  , -5472.786   , -5457.6455  , ..., -3490.741   ,
        -3495.3052  , -3498.335   ],
       [-9999.      , -9999.      , -9999.      , ..., -9999.      ,
        -9999.      , -9999.      ],
       [-9999.      , -9999.      , -9999.      , ..., -9999.      ,
        -9999.      , -9999.      ]], dtype=float32)

[BahramKhazaei-NOAA]

Thank you Soroosh for testing again. Looks like river[:,0] array is defined as a 2D array in dem.py while a 1D array is passed to the function.

[BahramKhazaei-NOAA]

@cuill, any thoughts on this error message?

[cuill]

@BahramKhazaei-NOAA I just came back from vacation. I'll take a look at this error when I get a chance.

[BahramKhazaei-NOAA]

@BahramKhazaei-NOAA I just came back from vacation. I'll take a look at this error when I get a chance.

Appreciate your help and quick responses as always. Looks like for this error something is not compatible wit the definition of the river array.

[cuill]

@BahramKhazaei-NOAA @SorooshMani-NOAA
I cannot reproduce this error but I got another error related to the package GEOS (see issue here shapely/shapely#1345) if using shapely2.0. So, please upgrade GEOS to 3.12.0.

I've pushed the version that worked on my side, please pull it. Here are the thalwegs extracted with a threshold of 1e4.

[SorooshMani-NOAA]

@cuill thanks for your feedback, so you just used the original DEMs and set all the nans to -9999 and ran pyDEM?

[cuill]

@SorooshMani-NOAA correct.

[BahramKhazaei-NOAA]

Hi @cuill I was able to install pyDEM and run the both serial and parellel test cases provided here. However, when I try to run it based on my DEM it gives me the same error as Soroosh. Here is the error:

Traceback (most recent call last):
  File "/work2/noaa/nos-surge/bkhazaei/work/run_dir/Alaska_Modeling/pre_processing/grid/bearing_sea_ocsmesh/pyDEM/run_serial.py", line 47, in <module>
    gdf = S.write_shapefile(npt_smooth=3)
  File "/work2/noaa/nos-surge/bkhazaei/envs/miniconda3/envs/RiverMesh3_10/lib/python3.10/site-packages/pyDEM/dem.py", line 2400, in write_shapefile
    idxs=np.squeeze(np.argwhere(np.isnan(river[:,0])))
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

I made sure that the nan values in my DEM are replaced with -9999 as we discussed above. I'm using run_serial.py that is provided for the savannah river test case and didn't change anything in that script except for the path to the DEM file. I'm pasting the script here:

#!/usr/bin/env python3
import sys
import glob
import time

from pylib import *
from pyDEM.dem import *

if __name__ == '__main__':
    #input tiff files
    PathToInputDEMs = '/work2/noaa/nos-surge/bkhazaei/work/run_dir/alaska_domain/mesh/ocsmesh/cook_inlet/DEMs_inputs'
    PathToDomainBoundary = PathToInputDEMs+"/GMRTv4_1_1_20230815topo_NaN_removed.tif"
    print('Path to DEM input:', str(PathToInputDEMs))
    
    files = glob.glob(PathToDomainBoundary)
    files.sort()

    t0 = time.time()

    for fname in files:
        names = [fname]

        #output filename
        sname = f"./{fname.split('.')[0].split('/')[-1]}_1e7"
        print(sname)

        #declaer a dem object
        S=dem()

        #read data
        if not os.path.exists('{}.npz'.format(sname)):
            S.proc_demfile(names,sname=sname,depth_limit=[-100,1000],subdomain_size=2e10)
        S.read_data('{}.npz'.format(sname))

        #compuate watershed information
        S.compute_watershed()

        #extract river network (area_limit: catchment area)
        S.compute_river(acc_limit=1e7)

        #write shapefile for river network
        gdf = S.write_shapefile(npt_smooth=3)
        gdf.to_file(f'{sname}.shp')
        print(f'It took {(time.time() - t0)/60} mins!')

Did you use the same script to get the result you shared above? If not, could you please share that with me?