Dataseed (#209)

* tidied up code for build_hospdeath_geoid_fixedIFR_par(); made output file faster and smaller * Add python argument to make_makefile.R * Remove file autogenerated by importation * Try to print a fake secret to see if env is working * Update ci.yml * Update ci.yml * to all caps * Update README.md * Updated to use optimize branch of covidImportation * Getting the version of covidImportation that the script works with * Run if not there instead of the opposite * Print error if config not specified * Need parallel back in here for detectCores * Fix missing comma * adding option to switch hosp script version * Bad merge * Modify report generation targets in Makefile * Remove unused include_in_report column from test1 geodata * Add integration test for report generation * making hosp_run backwards compatible, adding toggle in config to run age adjusted or non age adjusted hosp function * clarifying which time to death distribution should be used for each hosp script. Minor fixes to ensure backwards compatibility * removing hard coded p_death_rate * Fixed a few issues intrudocued in the updated state report template. * Update README.md * Create seeding.csv * Fixed testing config * minor text changes * Removing comments to help refresh repo * Trying to specify a specific commit for integration testing * Remove packages customer in favor of local_install for containerization * Change if clause for dockerhub deployment * Fix typo in dockerfile * Add awscli in dockerfile * Add report::continue_on_error config. Use notebooks directory instead of report. And runs importation for report generation test. * add check models R script * Changed state usps as a paramter and added better error handling. * added time from ventilation to discharge and number of people currently using ventilators to hospitalization * Add state_usps to report * state_usps and changing assert * export vent data * add vent figures * comment out vents figs * Improve hospitalization performance * Removes unnecessary casting and copying of data frames and simplifies data frame merge in a critical section * New age-adjusted hospitalization is 50% faster with 35% less memory pressure * Old legacy hospitalization is 68% faster with ~50% less memory pressure * create filterupdate for other data * fixing NAs in curr hosp, ICU, vent * Add CLI args for check_model_reports.R to config it like our other scripts * Let us control the number of simulations from the CLI for importation * bug fix * so bad at coding * incidVent exported in reportloaddata * export vents, breaks in table * Remove this invalid config checking pattern from our scripts * changing default vent_dur if not specified in old configs * fixing leading 0s on read in issue * new age specific parameter estimates with DC, PR * Remove extra comma on check_report CLI * update figs to compare data & model * fig updates for CA report * Modified base load functions to take advantage of a parallel back end using foreach. * Add ability for filtering helpers to take arguments. * Fixed bug of missing ... with scenarios filter. * Update make_makefile.R * changing hosp load function to include ventilation outcomes * adding table function with ventilator outcomes, fixking peak ICU occupancy from table * updating namespace for new vent function * fixing threshold axis ordering * adding function for scenario specific table with ventilations * function to format current ICU, ventilator data for heatmap figures * Updated covidcommon to have lognormal distribution. Added a new function to hospitalization package that (slowly) includes both updated config and uses random distributions for delay/duration. Needs to be optimized before use, but at least this way people can look at it. * import only one death rate in data/model comparison * update methods and static chunk text * change limits for model-obs comparison * uncomment current vent usage * include only necc geoids in check script * Documentation first draft for R script runners * reporting rate of 20% for modeled "cases" * Fixed some italics * plot mean and .2*infections * Create create_pipeline_docs.R * Add warning for special R comments * Update .gitignore * Some formatting cleanup * Update hosp_run.R * separate random distr tests & test mn/var * pre or post process aren't fxns test * Fixed issues around new function arguments (e.g. num_files) breaking existing code * export vent_curr where appropriate * mean incident deaths figure * change back silly caps name * Updated all scripts and packages (except importation to support feather * Fixing seed problems in all hospitalization scripts * fix bug when incl_geoids=NULL * Change documentation from knitr to Doxygen * Add integration test_hosp_age_adjust and rename test1 to test_simple * Updated loading code * Altered feather to parquet and updated the skeleton * Update requirements.txt to include the dvc[s3] dependencies * Updated makefile stuff, and also everything should now support feather * Updated docker requirements * Write parameters and npi from SEIR model * Add model_parameters to integration test * Clean up some configs * Version with pyarrow included * Dependencies for arrow in R as well * Fixed check_model script * Updated for feather integration * Updated test cases since `n` is reserved in yml * adding make_excess_heatmap function for hosp outcomes * Fixing parallelization mistake * Minor fixes - Use the "optimize" covidImportation version - Always upgrade local packages if upgrade available (vs silently ignore) - check_model_reports should ensure axis are dates * new figure relative to threshold heatmap * Update importation.R to match covidImportation package updates * Updated model code to use the new covidImportation package, and also seed to E instead of I (and keep population fixed * Fixed typo * Final fix to avoid numba * Fixed path to install_local script * Added package * Fixed seeding creation * rm NAs and fix create_seeding.R * add new cum hosp/deaths check to check_models scr * update indexes in check model script * long form mobility * Update reference to geoid-params.csv inside of hosp_run.R * 10x seeding file * Write the npi when writing parquet output * template * report after simulation * Removed geodata read from hosp_run.R since it's not being used * Updated things that feed into mobility * Updated build_US_setup.R to account for the move * These files got removed in a previous commit * Removing unused (as far as I can tell anyway) data * Fix bug when the places are also a number * Changing back test cases to use size/prob instead of n/p * Updated name to pass checks on case sensitive OS * Updated to use file_extension argument` * Fix broken tests, though I recommend we eliminate the mean and var checks since they'll be flaky * Updated build_US_setup.R to work with the current setup * Renamed parameters to avoid confusion; print out simid as 9 digits SEIR and hospitalization phases have more standardized file format * read parquet file times correctly * Revert "read parquet file times correctly" This reverts commit 521dd25. * parquet date fixes (#207) Co-authored-by: hrmeredith12 <[email protected]> * Report devel (#208) * fix unit test code * fix unit test for real * fix unit tests * adding ability to filter geoids in relative heatmap function * adding template for county-specific report for a given state * lower tolerance for distribution tests * planning_models chunk * planning scenario chunk * add names to dev team Co-authored-by: eclee25 <[email protected]> Co-authored-by: Kyra Grantz <[email protected]> Co-authored-by: hrmeredith12 <[email protected]> * Adding Javier (#210) Co-authored-by: hrmeredith12 <[email protected]> Co-authored-by: Elizabeth Lee <[email protected]> * Delete build-model-input.R (#217) * Dataseed merge (#215) * Adding Javier * Adding commute data back in * rm fixed param and comment out bad plot * commit namesapce report gen * fix NVentCurr name * formatting changes to county report template, removing defaults that should be modified for each report * adding references for county report template * change importation seeding * table formatting * limitations chunk considering age specific hosp calculations * removing build_hospdeath_geoid_par - old version not used in hosprun.R * removing legacy hospitalization scripts. everything runs through hosp_run.R now * using current default durations to minimize confusion Co-authored-by: hrmeredith12 <[email protected]> Co-authored-by: Elizabeth Lee <[email protected]> Co-authored-by: Kyra Grantz <[email protected]> * Removing config.yml and changing the variable name in create_seeding to be truthful. (#219) * Fixed the low in followup issue (#224) * Fixed the low in followup issue * Adding initial ^ * adding county report template yaml (#221) Co-authored-by: jkamins7 <[email protected]> * Fix load-bearing typo (#225) * Fix load-bearing typo * pretty sure it's supposed to be this Co-authored-by: Josh Wills <[email protected]> Co-authored-by: kkintaro <[email protected]> Co-authored-by: salauer <[email protected]> Co-authored-by: kkintaro <[email protected]> Co-authored-by: Josh Wills <[email protected]> Co-authored-by: Kyra Grantz <[email protected]> Co-authored-by: jlessler <[email protected]> Co-authored-by: eclee25 <[email protected]> Co-authored-by: Pierre-Yves Aquilanti <[email protected]> Co-authored-by: Sam Shah <[email protected]> Co-authored-by: Stephen A Lauer <[email protected]> Co-authored-by: shauntruelove <[email protected]> Co-authored-by: chadi <[email protected]> Co-authored-by: hrmeredith12 <[email protected]> Co-authored-by: Josh Wills <[email protected]>
HopkinsIDD · Apr 21, 2020 · 769b0e1 · 769b0e1
1 parent 308508c
commit 769b0e1
Show file tree

Hide file tree

Showing 128 changed files with 11,537 additions and 206,867 deletions.
diff --git a/.github/workflows/aws.yaml b/.github/workflows/aws.yaml
@@ -14,7 +14,7 @@ on:
     paths:
       - 'requirements.txt'
       - 'packages.R'
-      - 'packages-custom.R'
+      - 'local_install.R'
       - 'Dockerfile'
       - 'R/pkgs/**'
 

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -40,3 +40,12 @@ jobs:
           setwd("R/pkgs/report_generation")
           devtools::test(stop_on_failure=TRUE)
         shell: Rscript {0}
+      - name: Run integration tests
+        env: 
+          CENSUS_API_KEY: ${{ secrets.CENSUS_API_KEY }}
+        run: |
+          Rscript local_install.R
+          cd test
+          /home/app/python_venv/bin/pytest run_tests.py
+        shell: bash
+
diff --git a/.github/workflows/dockerhub.yaml b/.github/workflows/dockerhub.yaml
@@ -12,7 +12,7 @@ on:
     paths:
       - 'requirements.txt'
       - 'packages.R'
-      - 'packages-custom.R'
+      - 'local_install.R'
       - 'Dockerfile'
       - 'R/pkgs/**'
 
@@ -38,7 +38,7 @@ jobs:
 
     - name: Build Push Master
       uses: docker/build-push-action@v1
-      if: contains( ${{ github.ref }}, 'master')
+      if: contains( github.ref, 'master') || contains( github.base_ref, 'master' )
       with:
         username: ${{ secrets.DOCKER_USERNAME }}
         password: ${{ secrets.DOCKER_PASSWORD }}
@@ -47,19 +47,19 @@ jobs:
 
     - name: Build Push Dev
       uses: docker/build-push-action@v1
-      if: contains( ${{ github.ref }}, 'dev')
       with:
         username: ${{ secrets.DOCKER_USERNAME }}
         password: ${{ secrets.DOCKER_PASSWORD }}
         repository: hopkinsidd/covidscenariopipeline
         tags: latest-dev,${{steps.tags.outputs.shashort}},${{steps.tags.outputs.reftag}},${{ github.sha }}
+      if: contains( github.ref , 'dev') || contains( github.base_ref, 'dev' )
 
 
     - name: Build Push Dataseed
       uses: docker/build-push-action@v1
-      if: contains( ${{ github.ref }}, 'dataseed')
       with:
         username: ${{ secrets.DOCKER_USERNAME }}
         password: ${{ secrets.DOCKER_PASSWORD }}
         repository: hopkinsidd/covidscenariopipeline
         tags: latest-dataseed,${{steps.tags.outputs.shashort}},${{steps.tags.outputs.reftag}},${{ github.sha }}
+      if: contains( github.ref , 'dataseed') || contains( github.base_ref, 'dataseed' )
diff --git a/.gitignore b/.gitignore
@@ -9,6 +9,9 @@ data/RAW/
 data/dataside/
 dev-versions/
 reports/
+doc/html
+doc/latex
+
 # ^Joseph's stuff
 
 
@@ -51,4 +54,3 @@ vignettes/*.pdf
 
 # R Environment Variables
 .Renviron
->>>>>>> bee4416083d1cd42367acf7b0f1bcf872718f101
diff --git a/Dockerfile b/Dockerfile
@@ -66,6 +66,7 @@ RUN apt-get update && \
     libncurses-dev \
     libreadline-dev \
     supervisor \
+    awscli \
     r-base-dev=$R_VERSION \
     # make sure we have up-to-date CA certs or curling some https endpoints (like python.org) may fail
     ca-certificates \
@@ -90,9 +91,9 @@ COPY packages.R $HOME
 RUN Rscript packages.R
 
 # install custom packages from R/pkgs/**
-COPY packages-custom.R $HOME
-COPY R/pkgs $HOME/pkgs
-RUN Rscript packages-custom.R
+COPY local_install.R $HOME
+COPY R/pkgs $HOME/R/pkgs
+RUN Rscript local_install.R
 
 
 #####

diff --git a/R/pkgs/covidcommon/NAMESPACE b/R/pkgs/covidcommon/NAMESPACE
@@ -1,4 +1,5 @@
 # Generated by roxygen2: do not edit by hand
 
 export(as_evaled_expression)
+export(as_random_distribution)
 export(load_config)
diff --git a/R/pkgs/covidcommon/R/config.R b/R/pkgs/covidcommon/R/config.R
@@ -67,6 +67,7 @@ as_evaled_expression <- function(l) {
 ##'@param obj the string to evaluate
 ##'@return a float evaluation of the expression
 ##'
+##'@export
 as_random_distribution <- function(obj) {
   require(purrr)
 
@@ -75,12 +76,11 @@ as_random_distribution <- function(obj) {
   } else if (obj$distribution == "poisson") {
     return(purrr::partial(rpois, lambda=as_evaled_expression(obj$lam)))
   } else if (obj$distribution == "binomial") {
-    return(purrr::partial(rbinom, size=as_evaled_expression(obj$n), prob=as_evaled_expression(obj$p)))
+    return(purrr::partial(rbinom, size=as_evaled_expression(obj$size), prob=as_evaled_expression(obj$prob)))
+  } else if (obj$distribution == "lognormal") {
+    return(purrr::partial(rlnorm, meanlog=as_evaled_expression(obj$meanlog), sdlog=as_evaled_expression(obj$sdlog)))
   } else {
       stop("unknown distribution")
   }
 }
 
-.onLoad <- function(libname, pkgname) {
-  config <<- load_config()
-}
diff --git a/R/pkgs/covidcommon/tests/testthat/test-as_random_distribution.R b/R/pkgs/covidcommon/tests/testthat/test-as_random_distribution.R
@@ -0,0 +1,85 @@
+test_that("as_random_distribution works", {
+  expect_true({
+    rn <- as_random_distribution(list(distribution = "uniform", high = "0.9", low = .3))
+    all(is.numeric(rn(100)) & (rn(100) >= .3) & (rn(100) <= .9))
+  })
+
+  expect_error({
+    rn <- as_random_distribution(list(distribution = "poisson", high = "0.9", low = .3))
+    rn(100)
+  })
+
+  expect_error({
+    rn <- as_random_distribution(list(distribution = "binomial", high = "0.9", low = .3))
+    rn(100)
+  })
+
+  expect_true({
+    rn <- as_random_distribution(list(distribution = "binomial", size = 5, prob = "1/10"))
+    all(is.numeric(rn(100)) & (rn(100) >= 0) & (rn(100) <= 5))
+  })
+
+  expect_error({
+    rn <- as_random_distribution(list(distribution = "uniform", size = 5, prob = "1/10"))
+    rn(100)
+  })
+
+  expect_error({
+    rn <- as_random_distribution(list(distribution = "poisson", size = 5, prob = "1/10"))
+    rn(100)
+  })
+
+  expect_true({
+    rn <- as_random_distribution(list(distribution = "poisson",lam="5"))
+    all(is.numeric(rn(100)))
+  })
+
+  expect_error({
+    rn <- as_random_distribution(list(distribution = "binomial",lam="5"))
+    rn(100)
+  })
+
+  expect_error({
+    rn <- as_random_distribution(list(distribution = "uniform",lam="5"))
+    rn(100)
+  })
+
+  expect_error({
+    rn <- as_random_distribution(list(something = 1))
+  })
+
+  expect_error({
+    rn <- as_random_distribution(list(distribution = "fish"))
+  })
+
+  expect_true({
+    rn <- as_random_distribution(list(distribution = "uniform",low = 0, high = 1))
+    mean(rn(100000)) > 0.5*.97 & mean(rn(100000)) < 0.5*1.03
+  })
+
+  expect_true({
+    rn <- as_random_distribution(list(distribution = "uniform",low = 0, high = 1))
+    var(rn(100000)) > (1/12*(1-0)^2)*.97 & var(rn(100000)) < (1/12*(1-0)^2)*1.03
+  })
+
+  expect_true({
+    rn <- as_random_distribution(list(distribution = "poisson",lam=4))
+    mean(rn(100000)) > 4*.97 & mean(rn(100000)) < 4*1.03
+  })
+
+  expect_true({
+    rn <- as_random_distribution(list(distribution = "poisson",lam=4))
+    var(rn(100000)) > 4*.97 & var(rn(100000)) < 4*1.03
+  })
+
+  expect_true({
+    rn <- as_random_distribution(list(distribution = "binomial", size = 5, prob = "1/10"))
+    mean(rn(100000)) > 5*1/10*.97 & mean(rn(100000)) < 5*1/10*1.03
+  })
+
+  expect_true({
+    rn <- as_random_distribution(list(distribution = "binomial", size = 5, prob = "1/10"))
+    var(rn(100000)) > 5*1/10*9/10*.97 & var(rn(100000)) < 5*1/10*9/10*1.03
+  })
+
+})
diff --git a/R/pkgs/covidcommon/tests/testthat/test-load_config.R b/R/pkgs/covidcommon/tests/testthat/test-load_config.R
@@ -51,57 +51,3 @@ test_that("unsafe evaluation prevented", {
 
 })
 
-test_that("as_random_distribution works", {
-  expect_true({
-    rn <- as_random_distribution(list(distribution = "uniform", high = "0.9", low = .3))
-    all(is.numeric(rn(100)) & (rn(100) >= .3) & (rn(100) <= .9))
-  })
-
-  expect_error({
-    rn <- as_random_distribution(list(distribution = "poisson", high = "0.9", low = .3))
-    rn(100)
-  })
-
-  expect_error({
-    rn <- as_random_distribution(list(distribution = "binomial", high = "0.9", low = .3))
-    rn(100)
-  })
-
-  expect_true({
-    rn <- as_random_distribution(list(distribution = "binomial",n=5,p="1/10"))
-    all(is.numeric(rn(100)) & (rn(100) >= 0) & (rn(100) <= 5))
-  })
-
-  expect_error({
-    rn <- as_random_distribution(list(distribution = "uniform",n=5,p="1/10"))
-    rn(100)
-  })
-
-  expect_error({
-    rn <- as_random_distribution(list(distribution = "poisson",n=5,p="1/10"))
-    rn(100)
-  })
-
-  expect_true({
-    rn <- as_random_distribution(list(distribution = "poisson",lam="5"))
-    all(is.numeric(rn(100)))
-  })
-
-  expect_error({
-    rn <- as_random_distribution(list(distribution = "binomial",lam="5"))
-    rn(100)
-  })
-
-  expect_error({
-    rn <- as_random_distribution(list(distribution = "uniform",lam="5"))
-    rn(100)
-  })
-
-  expect_error({
-    rn <- as_random_distribution(list(something = 1))
-  })
-
-  expect_error({
-    rn <- as_random_distribution(list(distribution = "fish"))
-  })
-})