Spatial data preparation tools for Geo-X users. The Geo-X library requires spatial hexagon files for the area of interest with several spatial parameters attached as an input. These scripts are designed to assist in creating these input data. They allow users to move from raw data inputs to a Geo-X-ready input file containing hexagons, by interfacing with the Slope-Exclusion, Global Land Availability of Energy Systems (GLAES), and Spatially Integrated Development of Energy and Resources (SPIDER) repositories.
First, clone the repository and initialise the submodules in one step:
.../your/path % git clone --recurse-submodules https://github.com/ClimateCompatibleGrowth/Geo-X-data-prep.git
After cloning, navigate to the top-level folder of the repository.
The Python package requirements to use these tools are in the environment.yaml file.
You can install these requirements in a new environment using mamba package and environment manager (installation instructions here):
.../Geo-X-data-prep % mamba env create -f environment.yaml
This new environment can be activated using:
.../Geo-X-data-prep % mamba activate prep
Before proceeding to the next step, make sure the environment is deactivated using:
.../Geo-X-data-prep % mamba deactivate
You will need to create a separate environment for the SPIDER submodule to work.
Firstly, navigate to the ccg-spider/prep folder:
.../Geo-X-data-prep % cd ccg-spider/prep
Next, create a new environment using your package and environment manager. Below shows how to, using mamba:
.../prep % mamba create -n spider
Next, activate the environment using:
.../prep % mamba activate spider
Next, install some necessary packages:
.../prep % mamba install pip gdal
Next, install the SPIDER requirements:
.../prep % pip install -e .
You should now have a fully functioning environment named spider. You can deactivate this for now and return to the top-level of the repository.
Note
Where COUNTRY NAME is used, make sure to replace it with the country name spelling that matches those used in the Natural Earth country boundaries shapefile (downloaded in step 2.1).
Before running the preparation scripts, some data must be downloaded and placed in the data folder.
- The Global Oceans and Seas GeoPackage file can be downloaded from: https://www.marineregions.org/downloads.php
- The country boundaries shapefile can be downloaded from: https://www.naturalearthdata.com/downloads/50m-cultural-vectors/50m-admin-0-countries-2/
- OpenStreetMap Shapefile layers can be downloaded from (.shp.zip): https://download.geofabrik.de/
- The Corine Land Cover dataset (PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif) can be downloaded from: https://zenodo.org/records/3939050
Extra information:
- For the Global Oceans and Seas GeoPackage file, place the downloaded
GOaS_v1_20211214_gpkgfolder into thedatafolder. - For the OpenStreetMap files, extract the contents from the downloaded folder into a subfolder as follows
data/OSM/[COUNTRY NAME]for each country. - For the files from Natural Earth Data, place the downloaded
ne_50m_admin_0_countriesfolder into thedatafolder.
Important
Ensure that the config file you are using, either Country_config.yml or Country_config_hydro.yml, located in the inputs_spider folder, contains all the details you want SPIDER to use. Any removal or addition of features will require modification of the Geo-X codebase.
If you want hydropower to be used as a generator, you will need another input file. In the data folder, there is a template that can be filled in and name updated. It should be named [COUNTRY NAME]_hydropower_plants.csv and kept in that folder.
You can also use files from open-source datasets, like the Hydropower Database. You must place that file into the data folder, rename the file to [COUNTRY NAME]_hydropower_plants.csv and ensure that the required column titles match those provided in the template file. Extra columns do not need to be deleted, but they will not be taken into consideration when creating the GeoPackage file.
Input Data Requirements:
- The script is designed for datasets containing:
- Latitude & Longitude (plant location)
- Installed capacity (MW)
- Hydraulic head (m)
- Name (for user to keep track)
Slope-exclusion requires two input data files that must be downloaded and renamed:
- The country boundary GeoJSON file for each country can be downloaded from: opendatasoft. Place each GeoJSON file into the
Slope-Exclusion/datafolder and rename to[COUNTRY NAME]_boundary.geojson. - A 3-arc-second resolution conditioned Digital Elevation Model (DEM) file can be downloaded from: HydroSHEDS. Place the downloaded conditioned DEM file, for each continent you require, in the
Slope-Exclusion/datafolder and rename to[CONTINENT NAME]_full_dem.tif, whereCONTINENT NAMEis the name of the continent that the TIF file contains.
Note
The above naming conventions will allow you to place several files to run for several countries. At this point, each country will have to be run separately for some steps, this is expanded on in step 3.
There are two main scripts that are used, as well as the SPIDER submodule. As an optional step, the Slope-Exclusion submodule can be used before running any of the main steps to gather some input data for GLAES, which is part of the first main script. This data will be used to exlude land for solar and wind generators, based on slope.
Important
The size of the country can affect the runs as follows:
- The two main scripts may take more than 10 minutes to complete.
Note
This step must be repeated fully for each country that you wish to have slope exclusion files for.
Move to the Slope-Exclusion directory and activate the prep environment.
Copy the following command, replace [COUNTRY NAME] and [CONTINENT] as necessary, and paste it into your terminal:
.../Slope-Exclusion % python clip_raster_to_boundary.py --raster data/[CONTINENT]_full_dem.tif --boundary data/[COUNTRY NAME]_boundary.geojson --output data/dem.tif
For the second and third commands, there are some arguments that can be passed via the terminal. They are:
--type: (Only one required,stringtype) Should be eithersolar,wind, orboth.--solar-nea: (Default is6.28,floattype) The slope threshold for solar PV installations on north, east, and west-facing slopes.--solar-s: (Default is33,floattype) The slope threshold for solar PV installations on south-facing slopes.--wind-thresh: (Default is8.53,floattype) The slope threshold for wind turbine exclusion.--sigma: (Default is1,floattype) The standard deviation of the Gaussian filter used to smooth the DEM.--output: (Default isexclusion.tif,stringtype) Name of output file.
Below are the recommended commands to get the files required for GLAES. You can change any float arguments to match assumptions that you wish to use.
Copy the following command, replace [COUNTRY NAME] as necessary, and paste it into your terminal:
.../Slope-Exclusion % python exclude_slope.py --type solar --output [COUNTRY NAME]_slope_excluded_pv.tif
Copy the following command, replace [COUNTRY NAME] as necessary, and paste it into your terminal:
.../Slope-Exclusion % python exclude_slope.py --type wind --output [COUNTRY NAME]_slope_excluded_wind.tif
The output files can be found in Slope-Exclusion/output and will by GLAES as input data.
Make sure you are in the top-level folder, with the prep environment activated.
There are some arguments that you need to pass via the terminal. They are:
countries: (At least one required,stringtype) This should be the names of the countries you are preparing with a space between them. Make sure that the spellings used for country names match those used in the Natural Earth country boundaries shapefile.--hydro: (Default isFalse,booleantype) Only use this flag when you want hydropower to be considered, otherwise it will not be considered.-se: (Default isFalse,booleantype) Only use this flag when you have used the Slope-Exclusion submodule, otherwise it will not consider that the Slope-Exclusion submodule has been used.
Take the following command, replace [COUNTRY NAME] and keep/remove --hydro and -se as needed, and paste it into your terminal:
.../Geo-X-data-prep % python prep_before_spider.py [COUNTRY NAME] [COUNTRY NAME] --hydro -se
The above will first prepare a hydropower GeoPackage file, then pre-process the raw data, create a config file for SPIDER to use, and finally run GLAES. This will be done for each country provided.
Remember to deactive the prep environment before beginning the next step.
Important
Before running this step, it's important to know the following:
- You must check that the config file is correct for each country you intend to run results for. These can be found in the
ccg-spider/prepfolder. - The version of ccg-spider that is used is
Commit 2eed525. We recommend using this version for Geo-X-data-prep, but you can change it to best fit your needs. - Do not use multiple
&&symbols to run more than one country at once. Only one set ofblank.tifandblank_proj.tiffiles will be generated based on the first country, which will lead to inaccurate hexagon files for subsequent countries.
Now, you will need to move to the ccg-spider/prep directory, activate the spider environment, and use the SPIDER CLI.
Take the following command, replace [COUNTRY NAME] with the name of the country you are studying without spaces or periods, and paste it into your terminal:
.../prep % gdal_rasterize data/[COUNTRY NAME].gpkg -burn 1 -tr 0.1 0.1 data/blank.tif && gdalwarp -t_srs EPSG:4088 data/blank.tif data/blank_proj.tif && spi --config=[COUNTRY NAME]_config.yml [COUNTRY NAME]_hex.geojson
This command must be used for each country separately.
This will produce a set of hexagon tiles for each country using the parameters in the Country_config.yml file.
Note
Remember to move back to the top-level of the repository and deactivate the spider environment.
Activate the prep environment for this step.
There are some arguments that you need to pass via the terminal. They are:
countries: (At least one required,stringtype) This should be the name of the countries you are preparing with a space between them. Make sure that the spellings used for country names match those used in the Natural Earth country boundaries shapefile.-ic: (At least one required,stringtype) This is the two-letter ISO code for your countries. They must be in the same order as your countries.
Take the following command, replace [COUNTRY NAME] and [ISO CODE] as necessary, and paste it into your terminal:
.../Geo-X-data-prep % python prep_after_spider.py [COUNTRY NAME] [COUNTRY NAME] -ic [ISO CODE] [ISO CODE]
The above will combine the SPIDER and GLAES files. It will then assign the specified country name to the hexagons, as well as updating the CRS to match the world dataset. Lastly, this script removes the duplicated hexagons that belong to a country which is not the specified country.
The final file will be saved as hex_final_[COUNTRY ISO CODE].geojson for each country in the inputs_geox/final_data folder. These hex_final_[COUNTRY ISO CODE].geojson files can be placed into a copy of the Geo-X repository in the data folder, as the baseline input data for modelling.
If you set hydro to True, a [COUNTRY NAME]_hydropower_dams.gpkg file for each country will be generated into the inputs_geox/final_data folder. These files must be placed into the data/hydro folder of your Geo-X repository.
As the runs progress, you may not see all the files being generated, but rest assured they are there and taking up space. Once the runs have been completed, it's recommended to save the necessary files and review the listed folders below to delete any unnecessary files in order to free up space:
ccg-spider/prepccg-spider/prep/dataglaes/glaes/datainputs_geox/datainputs_geox/final_datainputs_glaes/processedSlope-Exclusion/data
If you decide to use this library and/or GeoH2, please kindly cite us using the following:
Halloran, C., Leonard, A., Salmon, N., Müller, L., & Hirmer, S. (2024). GeoH2 model: Geospatial cost optimization of green hydrogen production including storage and transportation. Pre-print submitted to MethodsX: https://doi.org/10.5281/zenodo.10568855. Model available on Github: https://github.com/ClimateCompatibleGrowth/GeoH2.
@techreport{halloran2024geoh2,
author = {Halloran, C and Leonard, A and Salmon, N and Müller, L and Hirmer, S},
title = {GeoH2 model: Geospatial cost optimization of green hydrogen production including storage and
transportation},
type = {Pre-print submitted to MethodsX},
year = {2024},
doi = {10.5281/zenodo.10568855},
note = {Model available on Github at https://github.com/ClimateCompatibleGrowth/GeoH2.}
}