GPII-3138: Update Prefs Safes and their GPII Keys in the data base

Dockerfile

@@ -2,17 +2,8 @@ FROM node:8-alpine
 
 WORKDIR /home/node
 
-RUN apk add --no-cache curl git && \
-    git clone https://github.com/GPII/universal.git && \
-    cd universal && \
-    rm -f package-lock.json && \
-    npm install json5 && \
-    npm install fs && \
-    npm install rimraf && \
-    npm install mkdirp && \
-    node scripts/convertPrefs.js testData/preferences/ build/dbData/ && \
-    apk del git
+RUN apk add --no-cache curl git
 
-COPY loadData.sh /usr/local/bin
+COPY deleteAndLoadSnapsets.sh /usr/local/bin/
 
-CMD ["/usr/local/bin/loadData.sh"]
+CMD ["/usr/local/bin/deleteAndLoadSnapsets.sh"]

README.md

@@ -1,6 +1,12 @@
 # CouchDB Data Loader
 
-Builds a [sidecar container](http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html) with the CouchDB data from the GPII/universal repository baked in and a mechanism for loading them into a CouchDB database.
+Builds a [sidecar container](http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html) that contains the `git` command and a shell script for setting up a CouchDB database.  When the docker image is run, this sequence is executed:
+1. Clones the latest version of [GPII universal](https://github.com/gpii/universal/),
+1. Converts the preferences in universal into snapset Prefs Safes and GPII Keys,
+1. Creates a CouchDB database if none exits,
+1. Optionally clears an existing database of all its records,
+1. Deletes any snapsets currently in the database,
+1. Loads the latest snapsets created at the second step into the database.
 
 ## Building
 
@@ -28,8 +34,15 @@ $ docker run -d -p 8081:8081 --name preferences --link couchdb -e NODE_ENV=gpii.
 
 ```
 
-Loading couchdb data from a different location (e.g. /home/vagrant/sync/universal/testData/dbData for static data directory and /home/vagrant/sync/universal/build/dbData for build data directory):
+Below are two versions of loading couchdb data from a different location (e.g. /home/vagrant/sync/universal/testData/dbData for static data directory and /home/vagrant/sync/universal/build/dbData for build data directory).  The first version has the optional `CLEAR_INDEX` set to erase and reset the database prior to other database changes:
 
 ```
 $ docker run --name dataloader --link couchdb -v /home/vagrant/sync/universal/testData/dbData:/static_data -e STATIC_DATA_DIR=/static_data -v /home/vagrant/sync/universal/build/dbData:/build_data -e BUILD_DATA_DIR=/build_data -e COUCHDB_URL=http://couchdb:5984/gpii -e CLEAR_INDEX=1 gpii/gpii-dataloader
 ```
+
+The second version has `CLEAR_INDEX` set to nothing such that any existing database is left intact prior to subsequent changes to it (e.g., deleting the snapsets):
+
+```
+$ docker run --name dataloader --link couchdb -v /home/vagrant/sync/universal/testData/dbData:/static_data -e STATIC_DATA_DIR=/static_data -v /home/vagrant/sync/universal/build/dbData:/build_data -e BUILD_DATA_DIR=/build_data -e COUCHDB_URL=http://couchdb:5984/gpii -e CLEAR_INDEX= gpii/gpii-dataloader
+```
+```

deleteAndLoadSnapsets.sh

@@ -0,0 +1,59 @@
+#!/bin/sh
+
+STATIC_DATA_DIR=${STATIC_DATA_DIR:-/home/node/universal/testData/dbData}
+BUILD_DATA_DIR=${BUILD_DATA_DIR:-/home/node/universal/build/dbData/snapset}
+
+log() {
+  echo "$(date +'%Y-%m-%d %H:%M:%S') - $1"
+}
+
+# Verify variables
+if [ -z "$COUCHDB_URL" ]; then
+  echo "COUCHDB_URL environment variable must be defined"
+  exit 1
+fi
+
+if [ ! -d "$STATIC_DATA_DIR" -o ! "$(ls -A $STATIC_DATA_DIR/*.json)" ]; then
+  echo "STATIC_DATA_DIR ($STATIC_DATA_DIR) must exist and contain data"
+  exit 1
+fi
+
+if [ ! -d "$BUILD_DATA_DIR" -o ! "$(ls -A $BUILD_DATA_DIR/*.json)" ]; then
+  echo "BUILD_DATA_DIR ($BUILD_DATA_DIR) must exist and contain data"
+  exit 1
+fi
+
+log "Starting"
+log "CouchDB: $COUCHDB_URL"
+log "Clear index: $CLEAR_INDEX"
+log "Static: $STATIC_DATA_DIR"
+log "Build: $BUILD_DATA_DIR"
+log "Working directory: `pwd`"
+
+# Set up universal
+git clone --depth 1 https://github.com/GPII/universal.git
+cd universal
+
+npm install json5
+npm install fs
+npm install rimraf
+npm install mkdirp
+npm install infusion
+rm -f package-lock.json
+node scripts/convertPrefs.js testData/preferences/ build/dbData/snapset snapset
+
+# Initialize (possibly clear) data base
+if [ ! -z "$CLEAR_INDEX" ]; then
+  log "Deleting database at $COUCHDB_URL"
+  if ! curl -fsS -X DELETE "$COUCHDB_URL"; then
+    log "Error deleting database"
+  fi
+fi
+
+log "Creating database at $COUCHDB_URL"
+if ! curl -fsS -X PUT "$COUCHDB_URL"; then
+  log "Database already exists at $COUCHDB_URL"
+fi
+
+# Submit data
+node scripts/deleteAndLoadSnapsets.js $COUCHDB_URL $STATIC_DATA_DIR $BUILD_DATA_DIR