Merge pull request #298 from mapswipe/dev

Stuctural changes to the configuration and data path handling.

Author: Matthias-Schaub

README.md

@@ -15,17 +15,48 @@ Please refer to the documentation for more information: https://mapswipe-workers
 
 ## Ressources
 
-- GitHub repository of the MapSwipe Back-End: https://github.com/mapswipe/python-mapswipe-workers
-- GitHub repository of the MapSwipe Website: https://github.com/mapswipe/mapswipe.github.io
-- GitHub repository of the MapSwipe App https://github.com/mapswipe/mapswipe
-- Website of MapSwipe: https://mapswipe.org
-- OSM-Wiki page of MapSwipe: https://wiki.openstreetmap.org/wiki/MapSwipe
+- MapSwipe Back-End: https://github.com/mapswipe/python-mapswipe-workers
+- MapSwipe App https://github.com/mapswipe/mapswipe
+- MapSwipe Website: https://mapswipe.org
+- MapSwipe OSM-Wiki: https://wiki.openstreetmap.org/wiki/MapSwipe
+
+
+## Contributing Guidelines
+
+### Feature Branch
+
+To contribute to the MapSwipe back-end please create dedicated feature branches based on the `dev` branch. After the changes create a Pull Request of the `feature` branch into the `dev` branch on GitHub:
+
+```bash
+git checkout dev
+git checkout -b featureA
+# Hack away ...
+git commit -am 'Describe changes.'
+git push -u origin featureA
+# Create a Pull Request from feature branch into the dev branch on GitHub.
+```
+
+> Note: If a bug in production (master branch) needs fixing before a new versions of MapSwipe Workers gets released (merging dev into master branch), a hotfix branch should be created. In the hotfix branch the bug should be fixed and then merged with the master branch (and also dev).
+
+
+### Style Guide
+
+This project uses [black](https://github.com/psf/black) and [flake8](https://gitlab.com/pycqa/flake8) to achieve a unified style.
+
+Use [pre-commit](https://pre-commit.com/) to run `black` and `flake8` prior to any git commit. `pre-commit`, `black` and `flake8` should already be installed in your virtual environment since they are listed in `requirements.txt`. To setup pre-commit simply run:
+
+```
+pre-commit install
+```
+
+From now on `black` and `flake8` should run automatically whenever `git commit` is executed.
 
 
 ## License
 
 This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details
 
+
 ## Authors
 
 * **Benjamin Herfort** - HeiGIT - [Hagellach37](https://github.com/Hagellach37)
@@ -34,6 +65,7 @@ This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENS
 
 See also the list of [contributors](contributors.md) who participated in this project.
 
+
 ## Acknowledgements
 
 * Humanitarian organizations can't help people if they can't find them.

docker-compose.yaml

@@ -30,7 +30,7 @@ services:
         build:
             context: api/
         volumes:
-            - ./api-data:/usr/share/nginx/html/api/:ro
+            - ./mapswipe-data/api:/usr/share/nginx/html/api/:ro
         restart: unless-stopped
         expose:
             - "80"
@@ -55,15 +55,7 @@ services:
             - postgres
         command: mapswipe_workers --verbose run --schedule
         volumes:
-            - ./api-data:/var/lib/mapswipe_workers/api-data/:rw
-            - ./api-data/agg_results:/var/lib/mapswipe_workers/api-data/agg_results:rw
-            - ./api-data/groups:/var/lib/mapswipe_workers/api-data/groups:rw
-            - ./api-data/history:/var/lib/mapswipe_workers/api-data/history:rw
-            - ./api-data/projects:/var/lib/mapswipe_workers/api-data/projects:rw
-            - ./api-data/results:/var/lib/mapswipe_workers/api-data/results:rw
-            - ./api-data/tasks:/var/lib/mapswipe_workers/api-data/tasks:rw
-            - ./api-data/yes_maybe:/var/lib/mapswipe_workers/api-data/yes_maybe:rw
-            - ./api-data/hot_tm:/var/lib/mapswipe_workers/api-data/hot_tm:rw
+            - ./mapswipe-data:/root/.local/share/mapswipe_workers
         restart: "no"
         networks:
             - mapswipe_workers

docs/source/contributing.md

@@ -1,39 +1,22 @@
 # Contributing
 
-
-## Clone from GitHub
-
-```bash
-git clone https://github.com/mapswipe/python-mapswipe-workers.git
-cd python-mapswipe-workers
-git checkout dev
-```
-
-
-## Install MapSwipe Workers
-
-Create a Python virtual environment and activate it. Install MapSwipe Workers using pip:
-
-```bash
-python -m venv venv
-source venv/bin/activate
-pip install --editable .
-```
+This document describes how to setup Python MapSwipe Workers locally without using the provided Dockerfile for development purposes.
 
 
 ## Feature Branch
 
-To contribute to the MapSwipe back-end please create dedicated feature branches from dev:
+To contribute to the MapSwipe back-end please create dedicated feature branches based on the `dev` branch:
 
 ```bash
 git checkout dev
 git checkout -b featureA
-git commit -am 'add new project type'
+# Hack away ...
+git commit -am 'Describe changes.'
 git push -u origin featureA
-git request-pull origin/dev featureA
+# Create a Pull Request from feature branch into the dev branch on GitHub.
 ```
 
-> Note: If a bug in production (master branch) needs fixing before a new versions of MapSwipe Workers gets released (merging dev into master branch), a hotfix branch should be created. In the hotfix branch the bug should be fixed and then merged back with master and also dev.
+> Note: If a bug in production (master branch) needs fixing before a new versions of MapSwipe Workers gets released (merging dev into master branch), a hotfix branch should be created. In the hotfix branch the bug should be fixed and then merged with the master branch (and also dev).
 
 
 ## Style Guide

docs/source/dev_setup.md

@@ -1,47 +1,69 @@
 # Development Setup
 
-In this document some tips and workflows for development are loosely collected. Those are independend of the production setup and Docker. A working Firebase Project (including Firebase Functions and Database Rules) is presupposed.
+In this document some tips and workflows for development are loosely collected. Those are independend of the production setup using Docker. A working Firebase Project (including Firebase Functions and Database Rules) is presupposed.
 
 
-1. Install GDAL (python-gdal)
-2. Setup virtual environment with system-site-packages enabled (To get access to GDAL)
-    - `python3 -m venv --system-site-packages venv`
-3. Activate virtual environment
-    - `source venv/bin/activate`
-4. Install mapswipe_workers
-    - `pip install -e .`
-5. Configure MapSwipe Workers as described in the section 'MapSwipe Workers Setup' in the [Setup](setup.md) chapter
-    - Make sure configuration and data paths are created and permissions are set as described in the next section [Configuration and data path](#Configurationa_and_data_path).
-    - Move the configuration (`config/configuration.json`) and the Service Account Key (`config/serviceAccountKey.json`) to `/usr/share/config/mapswipe_workers/`
-6. Setup a postgres instance
-    - Use the Docker image of MapSwipe
-        - `cd postgres/`
-        - `docker build -t mapswipe_postgres .`
-        - `docker run -d -p 5432:5432 --name mapswipe_postgres -e POSTGRES_DB=mapswipe -e POSTGRES_USER=mapswipe_workers -e POSTGRES_PASSWORD=your_password mapswipe_postgres`
-    - Or set up your own using the `initdb.sql` file in the `postgres/` folder
-7. Run Mapswipe Workers using the command: `mapswipe_wokers`
-    - eg. `mapswipe_wokers --help`
+## Installation
 
+### Requirements
 
-## Configuration and data path
+MapSwipe Workers requires GDAL/OGR (`gdal-bin`) and GDAL for Python (`libgdal-dev`, `python-gdal`) to be installed.
 
-Mapsipe Workers needs access to three directories:
 
-- CONFIG_PATH (`/usr/share/config/mapswipe_workers/`)
-- DATA_PATH (`/var/lib/mapswipe_workers/`)
-- LOG_PATH (`/var/log/mapswipe_workers/`)
+### Clone from GitHub
+
+```bash
+git clone https://github.com/mapswipe/python-mapswipe-workers.git
+cd python-mapswipe-workers
+git checkout dev
+```
+
+
+### Configuration
+
+MapSwipe Workers looks for configuration in `~/.config/mapswipe_workers`. (XDG Base Directory Specification is respected). It expects three files:
+
+- `configuration.json`
+- `serviceAccountKey.json`
+- `logging.cfg`
+
+Please refer to the [configuration](configuration.md) and [setup](setup.md) documentation for further details.
+
+In addition the data directory for MapSwipe Workers needs to be created:
+`mkdir --parents .local/share/mapswipe_workers`
 
-Make sure those are existing and accessible:
 
-- Use `mkdir DATA_PATH` to create the directory.
-- Use `chown -R $USER:$USER PATH` (eg. `chown -R $USER:$USER /user/share/config/mapswipe_workers/`) to give write permission to current user.
+### Install MapSwipe Workers Python Package
+
+1. Create a Python virtual environment with `system-site-packages` option enabled to get access to GDAL/OGR Python packages
+2. Activate the vitrual environment.
+3. Install MapSwipe Workers using pip.
+4. Run it.
+
+```bash
+python -m venv --system-site-packages venv
+source venv/bin/activate
+pip install --editable .
+mapswipe_workers --help
+```
+
+
+## Postgres
+
+Setup a local Postgres instance for MapSwipe Workers using the provided Dockerfile.
+
+```bash
+cd postgres/`
+docker build -t mapswipe_postgres .
+docker run -d -p 5432:5432 --name mapswipe_postgres -e POSTGRES_DB=mapswipe -e POSTGRES_USER=mapswipe_workers -e POSTGRES_PASSWORD=your_password mapswipe_postgres
+```
 
-Alternatively you can change the PATH variables in `definitions.py` to your desired path. Except of the path for logs. This is defined in `logging.cfg`.
+Or set up your own using the `initdb.sql` file in the `postgres/` folder
 
 
 ## Logging
 
-Mapswipe workers logs are generated using the Python logging module of the standard library (See [Official docs](https://docs.python.org/3/library/logging.html) or this [Tutorial](https://realpython.com/python-logging/#the-logging-module). The configuration file of the logging module is located at `mapswipe_workers/logging.cfg`. With this configuration a logger object is generated in the `definitions` module (`mapswipe_workers.definitions.py`) and is imported in other modules to write logs.
+Mapswipe workers logs are generated using the Python logging module of the standard library (See [Official docs](https://docs.python.org/3/library/logging.html) or this [Tutorial](https://realpython.com/python-logging/#the-logging-module). The configuration file of the logging module is located at `~/.config/mapswipe_workers/logging.cfg`. With this configuration a logger object is generated in the `definitions` module (`mapswipe_workers/definitions.py`) and is imported by other modules for writing logs.
 
 ```python
 from mapswipe_workers.definitions import logger
@@ -50,12 +72,12 @@ logger.waring('warning')
 
 # Include stack trace in the log
 try:
-    print(something)
+    do_something()
 except Exception:
-    logger.exception('something')
+    logger.exception('Additional information.')
 ```
 
-Default logging level is Warning. To change the logging level edit the configuration (`mapswipe_workers/logging.cfg`).
+Default logging level is Info. To change the logging level edit the configuration. Logs are written to STDOUT and `~/.local/share/mapswipe_workers/mapswipe_workers.log`.
 
 Per default logging of third-party packages is disabled. To change this edit the definition module (`mapswipe_workers/defintions.md`). Set the `disable_existing_loggers` parameter of the `logging.config.fileConfig()` function to False.
 

docs/source/installation.md

@@ -251,28 +251,16 @@ certbot certonly \
     --non-interactive
 ```
 
+> Note: Certbot systemd timer for renewal of certificate will not work for standalone certificates because the service (docker nginx) which occupies port 80 has to be stopped before renewal.
 
-Enable and start Certbot systemd timer for renewal of certificates:
+For certificate renewal a cronjob is used:
 
 ```bash
-systemctl enable certbot.timer
-systemctl start certbot.timer
-# To check if certbot.timer is enabled run:
-systemctl list-units --type timer | grep certbot
-```
-
-Add renewal post hook to reload nginx after certificate renwal:
-
-- `mkdir -p /etc/letsencrypt/renewal-hooks/deploy`
-- `vim /etc/letsencrypt/renewal-hooks/deploy/nginx`
+SHELL=/bin/sh
+PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
 
+0 */12 * * * certbot -q renew --pre-hook "docker stop nginx" --post-hook "docker start nginx"
 ```
-#!/usr/bin/env bash
-
-docker container restart nginx
-```
-
-- `chmod +x /etc/letsencrypt/renewal-hooks/deploy/nginx`
 
 
 ### Nginx

mapswipe_workers/Dockerfile

@@ -2,44 +2,45 @@
 # Image includes python3.6, gdal-python, gdal-bin
 FROM osgeo/gdal:ubuntu-small-latest
 
+# Set C.UTF-8 locale as default (Needed by the Click library)
 ENV LC_ALL=C.UTF-8
 ENV LANG=C.UTF-8
 
 # Install pip
 RUN apt-get update
 RUN apt-get --yes install python3-pip
 
-# Create directories for config, logs and data
-ARG config_dir=/usr/share/config/mapswipe_workers/
-ARG repo_dir=/usr/local/mapswipe_workers/
-ARG data_dir=/var/lib/mapswipe_workers/
-ARG logs_dir=/var/log/mapswipe_workers/
-
-RUN mkdir -p $config_dir
-RUN mkdir -p $repo_dir
-RUN mkdir -p $logs_dir
-RUN mkdir -p $data_dir
-RUN mkdir -p $data_dir"api-data"
-RUN mkdir -p $data_dir"api-data/agg_results/"
-RUN mkdir -p $data_dir"api-data/groups/"
-RUN mkdir -p $data_dir"api-data/history/"
-RUN mkdir -p $data_dir"api-data/projects/"
-RUN mkdir -p $data_dir"api-data/results/"
-RUN mkdir -p $data_dir"api-data/tasks/"
-RUN mkdir -p $data_dir"api-data/yes_maybe/"
-RUN mkdir -p $data_dir"api-data/hot_tm/"
+####
+# Due to unresolved permission error of shared volume
+# fall back to root user.
+#
+## Create user mapswipe
+# RUN groupadd --system mapswipe
+# RUN useradd --create-home --no-log-init --system --gid mapswipe mapswipe
+## Change user to mapswipe
+# USER mapswipe
+# WORKDIR /home/mapswipe
+####
+
+WORKDIR /root
+
+# Create directories for config, data and logs
+RUN mkdir --parents .config/mapswipe_workers
+RUN mkdir --parents .local/share/mapswipe_workers
 
 # Copy mapswipe workers repo from local repo
-WORKDIR $repo_dir
 COPY mapswipe_workers/ mapswipe_workers/
 COPY sample_data/ sample_data/
 COPY tests/ tests/
 COPY requirements.txt .
 COPY setup.py .
-COPY config $config_dir
+COPY config/ .config/mapswipe_workers/
 
 # Update setuptools and install mapswipe-workers with dependencies (requirements.txt)
 RUN pip3 install --upgrade setuptools
 RUN pip3 install .
 
+# Add platform-specific user binary directory for Python applications to $PATH
+# ENV PATH="$PATH:/home/mapswipe/.local/bin"
+
 # Don't use a CMD here, this will be defined in docker-compose.yaml

mapswipe_workers/config/logging.cfg

@@ -27,7 +27,7 @@ args=(sys.stdout,)
 class=handlers.TimedRotatingFileHandler
 level=INFO
 formatter=mapswipeFormatter
-args=('/var/log/mapswipe_workers/mapswipe_workers.log','D', 1, 14,)
+args=('%(logfilename)s','D', 1, 14,)
 
 [formatter_mapswipeFormatter]
 format=%(asctime)s - %(levelname)s - %(filename)s - %(funcName)s - %(message)s