AClimate V3 Historical Spatial ETL 🌍📊

🏷️ Version & Tags

Tags: climate-data, etl, geoprocessing, python, geoserver, chirps, copernicus, AGera5

---

📌 Introduction

Python package for processing spatial historical climate data with a complete ETL pipeline that includes:

• Data download from CHIRPS and Copernicus sources
• Spatial clipping by country boundaries
• Monthly aggregation and climatology calculations
• Climate indicators calculation (TR20, TXx, etc.)
• GeoServer integration for data publishing

Key Features:

• Automated processing of temperature, precipitation, and solar radiation data
• Parallel processing for downloads and indicator calculations
• Flexible indicator calculation with custom year ranges
• Flexible configuration for multiple countries and variables
• End-to-end pipeline from raw data to published layers
• Database-backed configuration management

---

✅ Prerequisites

• Python > 3.10
• Copernicus Climate Data Store (CDS) API key - Register here
• GeoServer
• PostgreSQL database for configuration storage

⚙️ Installation

pip install git+https://github.com/CIAT-DAPA/aclimate_v3_cut_spatial_data.git
pip install git+https://github.com/CIAT-DAPA/aclimate_v3_spatial_importer.git
pip install git+https://github.com/CIAT-DAPA/aclimate_v3_orm
pip install git+https://github.com/CIAT-DAPA/aclimate_v3_historical_spatial_etl

To install a specific version:

pip install git+https://github.com/CIAT-DAPA/[email protected]

🚀 Basic Usage

1. Command Line Interface

Standard Pipeline

python -m aclimate_v3_historical_spatial_etl.aclimate_run_etl \
  --country HONDURAS \
  --start_date 2020-01 \
  --end_date 2020-12 \
  --data_path /path/to/data \
  --climatology

Climate Indicators Calculation

python -m aclimate_v3_historical_spatial_etl.aclimate_run_etl \
  --country HONDURAS \
  --start_date 2025-01 \
  --end_date 2025-01 \
  --data_path /path/to/data \
  --indicators \
  --indicator_years 2015-2020

Skip Processing (Indicators Only)

python -m aclimate_v3_historical_spatial_etl.aclimate_run_etl \
  --country COLOMBIA \
  --start_date 2025-01 \
  --end_date 2025-01 \
  --data_path /path/to/data \
  --skip_download \
  --skip_processing \
  --indicators \
  --indicator_years 2010-2020

Note

New Options:

• --skip_download: Skip the data download step
• --skip_processing: Skip data processing (clipping, monthly aggregation)
• --climatology: Calculate monthly averages-climatology
• --indicators: Calculate climate indicators
• --indicator_years YYYY-YYYY: Specify year range for indicator calculation
• --no_cleanup: Keep intermediate files after processing

2. Programmatic Usage

from aclimate_v3_historical_spatial_etl.aclimate_run_etl import run_etl_pipeline

run_etl_pipeline(
    country="HONDURAS",
    start_date="2020-01",
    end_date="2020-12",
    data_path="/path/to/data",
    climatology=True,
    indicators=True,
    indicator_years="2015-2020"
)

🗂️ Directory Structure (Auto-generated)

data/
├── config/               # Must contain required JSON config files
├── raw_data/             # Downloaded raw datasets
├── process_data/         # Intermediate raster data
├── calc_data/
│   ├── climatology_data/ # Climatology outputs
│   ├── monthly_data/     # Monthly processed rasters
│   └── indicators_data/  # Climate indicators (TR20, TXx, etc.)
└── upload_geoserver/     # Output prepared for GeoServer publishing

🌡️ Climate Indicators

Supported Indicators

Indicator	Name	Description	Unit
TR20	Tropical Days	Annual count of days with Tmax > 20°C	days
TXx	Maximum Temperature Maximum	Annual maximum of daily maximum temperature	°C

Indicator Features

• Parallel Processing: Multiple years calculated simultaneously
• Flexible Year Ranges: Calculate indicators for any historical period
• Raster Output: Results saved as GeoTIFF files with proper georeferencing
• Multi-year Statistics: Automatic calculation of multi-year averages
• Memory Efficient: Processes data in chunks to handle large datasets

📋 Detailed Usage Guide: See INDICATORS_USAGE.md for comprehensive examples and configuration options.

🔧 Configuration

Database Configuration

All configurations are stored in the database. Ensure your database contains the required configuration entries for:

1. chirps_config - CHIRPS download settings
2. copernicus_config - Copernicus/ERA5 settings
3. clipping_config - Country boundaries and ISO codes
4. geoserver_config - GeoServer workspace and store names
5. naming_config - Output file naming conventions

Environment Variables

• Windows:

# GeoServer credentials
set GEOSERVER_URL=http://localhost:8086/geoserver/rest/
set GEOSERVER_USER=admin
set GEOSERVER_PASSWORD=password
set OTLP_ENDPOINT=localhost:4317
set ENABLE_SIGNOZ=false
set LOG_FILE_PATH=path/application.log
set DATABASE_URL=postgresql://postgres:admin@localhost:5432/acimate_v3

• Linux/Ubuntu:

# GeoServer credentials
export GEOSERVER_URL=http://localhost:8086/geoserver/rest/
export GEOSERVER_USER=admin
export GEOSERVER_PASSWORD=password
export OTLP_ENDPOINT=localhost:4317
export ENABLE_SIGNOZ=false
export LOG_FILE_PATH=path/application.log
export DATABASE_URL=postgresql://postgres:admin@localhost:5432/acimate_v3

Note

Options:

• GEOSERVER_URL: Geoserver URL
• GEOSERVER_USER: Geoserver user
• GEOSERVER_PASSWORD: Geoserver password
• OTLP_ENDPOINT: Signoz endpoint to send logs
• ENABLE_SIGNOZ: Flag to send logs to signoz
• LOG_FILE_PATH: Path to save logs
• DATABASE_URL: Connection string to database

🧪 Running Tests

# Install test requirements
pip install pytest pytest-mock

# Run tests
pytest tests/

🔄 CI/CD Pipeline Overview

Workflow Architecture

Our GitHub Actions pipeline implements a three-stage deployment process:

Code Push → Test Stage → Merge Stage → Release Stage

📊 Project Structure

aclimate_v3_historical_spatial_etl/
│
├── .github/
│ └── workflows/ # CI/CD pipeline configurations
├── src/
│   └── aclimate_v3_historical_spatial_etl/
│       ├── connectors/           # Downloaders: CHIRPS, Copernicus
│       ├── tools/                # Clipping and GeoServer tools
│       ├── climate_processing/   # Monthly and climatology processors
│       ├── config/               # Example config files
│       └── aclimate_run_etl.py   # Main ETL entry script
├── tests/                       # Unit and integration tests
├── requirements.txt             # Dependencies
└── pyproject.toml               # Packaging