Refactor Airflow container architecture and environment configuration

.gitignore

@@ -124,9 +124,7 @@ celerybeat.pid
 # Environments
 .env
 .venv
-#env/
 venv/
-ENV/
 env.bak/
 venv.bak/
 
@@ -160,3 +158,15 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# Data Folder
+data/*
+# TODO: Remove when you need the cut background file anymore
+# Raw data folder excluding only the background cut file
+# data/raw/
+
+# Modules Pool
+*.cfmodule
+
+# OS
+.DS_Store

README.md

@@ -1,101 +1,239 @@
-# cosiflow
+# Cosiflow
 
-The COSI SDOC pipeline based on Apache Airflow
+Cosiflow provides an Airflow-based orchestration environment for managing and monitoring scientific pipelines for COSI.
 
-## Build the cosiflow docker
+---
 
-We assume that the cosiflow repository is in your $HOME directory.
+## Quick Start Overview
 
-```bash
-cd $HOME/cosiflow/env
-```
+There are two main steps to using Cosiflow:
 
-Mac:
+1. **Set up and run the core Cosiflow environment** (this README):
+   - Configure `env/docker-compose.yaml` (UID/GID, Airflow admin password, ports, etc.).
+   - Build and start the Docker Compose stack.
+   - Access the Airflow Web UI.
 
-```bash
-docker build --platform linux/arm64 -t airflow:1.0.0 -f Dockerfile .
-```
+2. **(Optional) Install one or more pipeline modules** (e.g. analysis pipelines):
+   - Each module lives alongside `cosiflow/` (for example `fast-transient-analysis-pipeline/`).
+   - Modules provide DAGs and pipeline scripts that are plugged into the running Cosiflow instance.
+   - To install a module into Cosiflow, follow the guide in `env/README.md`.
 
-Linux:
+If you only want to bring up a plain Cosiflow environment to explore Airflow and the COSIDAG framework, step 1 is sufficient.  
+If you want to run a specific scientific pipeline (e.g. the Fast Transient Analysis Pipeline), you must also complete step 2.
 
-```bash
-docker build -t airflow:1.1.0 -f Dockerfile .
-```
+---
 
-## Execute the docker compose to start containers
+### 1. REQUIREMENTS
 
-```bash
-docker compose up -d
-```
+#### CONFIGURE ENVIRONMENT VARIABLES (NO `.env` FILE)
 
-If you want to enter into the postgre docker container: `docker compose exec postgres bash`
+All configuration is now done directly in `env/docker-compose.yaml` (there is **no** `.env` file anymore).
 
-If you want to enter into the postgre docker container: `docker compose exec airflow bash`
+1. Move into the `env` folder:
+   ```bash
+   cd env
+   ```
 
-## Connect to the web server using a browser
+2. Find your local user and group IDs:
 
-localhost:8080
+   ```bash
+   id -u   # YOUR_USER_ID
+   id -g   # YOUR_GROUP_ID
+   ```
 
-Note: if you use a remote server you can change the `docker-compose.yaml` file to use another port.
+3. Open `docker-compose.yaml` and locate the `x-common-env` block at the top.  
+   Replace the default values with your IDs:
 
-For example:
-
-  ```yaml
-  ports:
-    - "28080:8080"
-  ```
+   ```yaml
+   UID: ${UID:-<YOUR_USER_ID>}  # TOEDIT
+   GID: ${GID:-<YOUR_GROUP_ID>} # TOEDIT
+   ```
+
+4. In the same `x-common-env` block, set a secure password for the Airflow Web UI:
+
+   ```yaml
+   AIRFLOW_ADMIN_PASSWORD: ${AIRFLOW_ADMIN_PASSWORD:-<YOUR_AIRFLOW_PASSWORD>}  # TOEDIT
+   ```
+
+5. (Optional, but recommended to review)  
+   Still in `x-common-env`, check the variables marked with `# TOEDIT` comments  
+   (e.g. `HOST_IP`, `MAILHOG_WEBUI_PORT`, `AIRFLOW_WEBUI_PORT`) and adjust them
+   if the defaults are not suitable for your setup.
 
-then from your local pc you can forward the port in this way:
+#### PREPARE THE FOLDER FOR STORING POSTGRESS DATA
+```bash
+cd ..
+mkdir -p data/postgres_data
+```
+
+---
+
+### 2. BUILD THE COMPOSE
+
+Build all containers defined in `docker-compose.yml`:
 
 ```bash
-ssh -N -L 28080:localhost:28080 [user]@[remote machine]
+cd env
+docker compose build
 ```
 
-and open the airflow webpace from your local pc at `localhost:28080`
+⏱ Estimated build time: **~490 seconds**
+
+---
 
-Login with username: `admin`  password: `<password>`
+### 3. RUN THE CONTAINER
 
-To obtain the password `<password>` execute this command after the initialization of the containers
+To run with logs visible:
 
 ```bash
-docker compose logs | grep pass
+docker compose up
 ```
 
-### Shutdown the dockers
+To run in detached mode (no logs):
 
 ```bash
-docker compose down -v
+docker compose up -d
 ```
 
-## Test the cosipy DAG
+---
+
+### 4. ENTER THE CONTAINER
 
-Enter in the docker airflow
+To open a terminal inside the running Airflow container:
 
 ```bash
 docker compose exec airflow bash
 ```
 
-First download the data file from wasabi.
+---
 
-```bash
-cd /shared_dir/pipeline
-source activate cosipy
-python initialize_pipeline.py
-```
+### 5. CONNECT TO THE AIRFLOW WEB UI
+
+1. Open your web browser and go to:
+
+   [http://localhost:8080/home](http://localhost:8080/home)
 
-This script downloads the input file from wasabi and move it in `/home/gamma/workspace/data`
+2. Insert the user credentials:
+   ```text
+   user:     admin
+   password: <YOUR_AIRFLOW_PASSWORD>
+   ```
 
-Now we must activate the DAG named `"cosipt_test_v0"` from the airflow website
+---
 
-Then we have to copy the file in the input directory to trigger the DAG
+### 6. STOP THE CONTAINER
+
+To stop and remove all running containers, networks, and volumes:
 
 ```bash
-cd /home/gamma/workspace/data
-cp GalacticScan.inc1.id1.crab2hr.extracted.tra.gz input
+docker compose down -v
 ```
 
-We should see that the DAG started to process the data.
+---
+
+### 7. CONFIGURATIONS
+
+Below is the list of the main environment variables configured in `env/docker-compose.yaml`
+inside the `x-common-env` block (and related sections), with their purpose:
+
+| Variable | Description |
+|-----------|--------------|
+| **UID** | User ID used inside containers (must match your local user) |
+| **GID** | Group ID used inside containers (must match your local group) |
+| **DISPLAY** | Display variable for X11 forwarding (optional) |
+| **AIRFLOW_ADMIN_USERNAME** | Default Airflow Web UI username |
+| **AIRFLOW_ADMIN_EMAIL** | Email associated with Airflow admin user |
+| **AIRFLOW_ADMIN_PASSWORD** | Secure password for Airflow Web UI (must be set by you) |
+| **HOST_IP** | Host IP used to construct service URLs (e.g. Web UIs) |
+| **MAILHOG_WEBUI_PORT** | Port for the MailHog Web UI |
+| **AIRFLOW_WEBUI_PORT** | Port for the Airflow Web UI |
+| **POSTGRES_USER** | Username for the Airflow PostgreSQL database |
+| **POSTGRES_DB** | Database name for the Airflow PostgreSQL database |
+| **POSTGRES_PASSWORD** | Password for the Airflow PostgreSQL database |
+| **ALERT_USERS_LIST_PATH** | Path to YAML file containing user alert configurations |
+| **ALERT_SMTP_SERVER** | SMTP server used for alert notifications |
+| **ALERT_EMAIL_SENDER** | Email address used as sender for system alerts |
+| **ALERT_LOG_PATH** | Path to Airflow log file monitored by alert system |
+| **AIRFLOW__SMTP__SMTP_STARTTLS** | Enables/disables STARTTLS (default: False) |
+| **AIRFLOW__SMTP__SMTP_SSL** | Enables/disables SMTP over SSL (default: False) |
+| **COSI_DATA_DIR** | Root directory for COSI data |
+| **COSI_INPUT_DIR** | Directory for COSI input data |
+| **COSI_LOG_DIR** | Directory for COSI log files |
+| **COSI_OBS_DIR** | Directory for observation data |
+| **COSI_TRANSIENT_DIR** | Directory for transient event data |
+| **COSI_TRIGGER_DIR** | Directory for trigger event data |
+| **COSI_MAPS_DIR** | Directory for map data products |
+| **COSI_SOURCE_DIR** | Directory for source-level data products |
+
+---
+
+### NOTES
+
+- Configuration is done directly in `env/docker-compose.yaml`; there is no `.env` file.
+- Variables that are important to customize are explicitly marked with `# TOEDIT` comments in `docker-compose.yaml`.
+- To inspect container logs, use:
+  ```bash
+  docker compose logs -f airflow
+  ```
+
+- For details on how to install and manage modules (pipelines) within Cosiflow, see `env/README.md`.
+
+---
+
+**Cosiflow environment ready for use.**
+
+---
+
+## What is COSIDAG
+
+A **COSIDAG** (COSI DAG) is a structured abstraction built on top of Apache Airflow DAGs.
+
+It provides a **standardized workflow layout** for scientific pipelines, reducing boilerplate and enforcing consistent patterns across different analyses.
+
+In particular, a COSIDAG:
+
+* defines a common execution skeleton (input resolution, optional monitoring, result handling)
+* encapsulates best practices for:
+
+  * file discovery
+  * parameter propagation
+  * XCom-based communication
+* allows developers to focus only on **scientific tasks**, while orchestration logic is handled automatically
+
+COSIDAGs are used for all production scientific pipelines (e.g. Light Curve, TS Map), while standard DAGs are reserved for orchestration, testing, or utilities.
+
+**How to write and customize a COSIDAG** is explained in detail in the [tutorial section](modules/README.md).
+
+---
+
+## Tutorials and developer guide
+
+A complete, step-by-step guide on how to:
+
+* understand the COSIDAG execution model
+* write new COSIDAGs
+* add custom tasks
+* use XCom correctly
+* integrate external Python environments
+
+is available in:
+
+[tutorial section](tutorials/README.md).
+
+This is the **recommended starting point for developers**.
+
+---
+
+## Available DAGs and COSIDAGs
+
+A complete and up-to-date list of all DAGs and COSIDAGs implemented in this repository — including:
+
+* workflow purpose
+* inputs and outputs
+* task structure
+* operators used
+* XCom usage
 
-This directory `/home/gamma/workspace/heasarc/dl0` contains several folders with this format `2025-01-24_14-31-56`.
+is documented in: [DAG and COSIDAG LIST README](dags/README.md)
 
-Inside the folder we have the results of the analysis.
+This document serves as the **catalog and reference** for all workflows available in Cosiflow.