Environment file and Readme updated and edited the table populating s…

…cripts.

README.md

@@ -1,11 +1,9 @@
 # TOST - Trip Outlier Scoring Tool
 
-# Backend
-## Requirements
+## Backend
+### Requirements
 
 - [python](https://www.python.org/) >= 3.6
-- [postgres](https://www.postgresql.org/)
-- [postgis](https://postgis.net/) - Is an postgres extention to work with geometries and geography
 - [psycopg2](https://github.com/psycopg/psycopg2) - PostgreSQL database adapter for Python
 
 We are using p3 venv
@@ -18,13 +16,35 @@ This will create a conda environment named `tost`. You can then verify the new e
 
 To activate the environment run `conda activate tost`
 
-# Server
+## Dataset and Database
+### Requirements
+- [Postgres](https://www.postgresql.org/) - PostgreSQL, A relational DB solution
+- [Postgis](https://postgis.net/) - A Postgres extention to work with geometries and geographical datatype
 
-For the sake of consistency we are using the same language for the server as for the scripts.
+The dataset used in this work is composed of trips between the ports of Houston and New Orleans between 2009 and 2014. This dataset is composed of two csv files.
 
-## Requirements
+The sample data is available at this google drive shared [folder](https://drive.google.com/drive/folders/1B7WlfLfyh9IBGbic61i9bDycJSFasKiA?usp=sharing)
 
-*[flask](https://palletsprojects.com/p/flask/)
+A PostgreSQL (aka Postgres) database is used to store the raw trip and vessel data for ease of pre-processing and querying. To populate the database from the raw AIS data, the below steps need to be followed - 
+
+  1. Create a database in Postgre named `tost-db`
+  2. Run the sql create statements in the `/database-scripts` directory in the given sequence in the filename prefix (v01-v06). Running all these create statements will create all the required table in the `tost-db` database
+  3. When the tables are there, to fill in the tables from the raw AIS data in the available csv files, first download the csv files from the shared Google drive folder. Then, in the `/scripts` directory there are python scripts to read in the csv files and write in the database table. <br>
+  Change the username and password to your Postgres username and password in the _db_connection.py_ file
+
+  1. Running all the python scripts in the sequence _script1_ to _script9_, will 
+
+
+
+
+
+## Server
+
+For the sake of consistency we are using the same language for the server as for the scripts. The server-side is a python _flask_ application.
+
+### Requirements
+
+- [flask](https://palletsprojects.com/p/flask/)
 
 ### Flask
 

backend/environment.yml

@@ -20,6 +20,8 @@ dependencies:
     - click==7.1.2
     - flask==1.1.2
     - flask-cors==3.0.10
+    - geographiclib==1.50
+    - geopy==2.1.0
     - itsdangerous==1.1.0
     - jinja2==2.11.3
     - markupsafe==1.1.1

scripts/db_connection.py

@@ -7,7 +7,7 @@ class DBConnection(object):
     def __init__(self):
         """Initialize your database connection here."""
         try:
-            connection = psycopg2.connect(database = "thesis")
+            connection = psycopg2.connect(database = "tost-db", user="postgres", password="12345678")
             connection.autocommit = True
             self.connection = connection
         except (Exception, psycopg2.Error) as error :

scripts/script1.py

@@ -1,16 +1,11 @@
+# Used to populate the trip_raw table from the raw csv data
 import csv
 import psycopg2
+from db_connection import DBConnection
 
-csv_file_path = '/Users/fernando/Projects/thesis_project/dataset/tais_voyages_sample_hou_nol_points_by_voyage_w_ref.csv'
-
-connection = None
-
-try:
-    connection = psycopg2.connect(database = "thesis")
-    connection.autocommit = True
-except (Exception, psycopg2.Error) as error :
-    print ("Error while connecting to PostgreSQL", error)
+csv_file_path = 'D:/Projects/TOST/SampleDataset/tais_voyages_sample_hou_nol_points_by_voyage_w_ref.csv'
 
+db_connection = DBConnection.get_instance().get_connection()
 cursor = connection.cursor()
 
 postgres_insert_query = """ INSERT INTO trip_raw (trip_id, timestamp, mmsi, lng_lat, heading, sog, rot, cog, ship_type) VALUES (%s,%s,%s, %s,%s,%s, %s,%s,%s) ON CONFLICT DO NOTHING;"""

scripts/script2.py

@@ -1,4 +1,4 @@
-# this scrip add data to vessel table
+# this script extract  data to vessel table
 from db_connection import DBConnection
 
 db_connection = DBConnection.get_instance().get_connection()