search_index.json

[
["index.html", "The Predictive Ecosystem Analyzer Welcome", " The Predictive Ecosystem Analyzer 2017-04-06 Welcome Ecosystem science, policy, and management informed by the best available data and models Our Mission: Develop and promote accessible tools for reproducible ecosystem modeling and forecasting PEcAn Website Public Chat Room Github Repository "],
["project-overview.html", "Project Overview", " Project Overview The Predictive Ecosystem Analyzer (PEcAn) is an integrated informatics toolbox for ecosystem modeling (Dietze et al. 2013, LeBauer et al. 2013). PEcAn consists of: An application program interface (API) that encapsulates an ecosystem model, providing a common interface, inputs, and output. Core utilities for handling and tracking model runs and the flows of information and uncertainties into and out of models and analyses An accessible web-based user interface and visualization tools An extensible collection of modules to handle specific types of analyses (sensitivity, uncertainty, ensemble), model-data syntheses (benchmarking, parameter data assimilation, state data assimilation), and data processing (model inputs and data constraints) This project is motivated by the fact that many of the most pressing questions about global change are limited by our ability to synthesize existing data and strategically prioritize the collection of new data. This project seeks to improve this ability by developing a framework for integrating multiple data sources in a sensible manner. The workflow system allows ecosystem modeling to be more reproducible, automated, and transparent in terms of operations applied to data, and thus ultimately more comprehensible to both peers and the public. It reduces the redundancy of effort among modeling groups, facilitate collaboration, and make models more accessible the rest of the research community. PEcAn is not itself an ecosystem model, and it can be used to with a variety of different ecosystem models; integrating a model involves writing a wrapper to convert inputs and outputs to and from the standards used by PEcAn. Currently, PEcAn supports multiple models listed PEcAn Models. Acknowledgements This material is based upon work supported by the National Science Foundation under Grant No. 1062547, 1062204, 1241894, 1261582, 1318164, 1346748, 1458021, 1638577, the National Aeronautics and Space Administration (NASA) Grant No. 13-TE13-0060, and the Energy Biosciences Institute. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or NASA. PEcAn is a collaboration among research groups at the Department of Earth And Environment at Boston University, the Energy Biosciences Institute at the University of Illinois, the Image Spatial Data Analysis group at NCSA, the Department of Atmospheric & Oceanic Sciences at the University Wisconsin-Madison, and the Terrestrial Ecosystem Science & Technology (TEST) Group at Brookhaven National Lab. BETY-db is a product of the Energy Biosciences Institute at the University of Illinois at Urbana-Champaign. We gratefully acknowledge the great effort of other researchers who generously made their own data available for further study. PEcAn Publications Shiklomanov. A, MC Dietze, T Viskari, PA Townsend, SP Serbin. 2016 “Quantifying the influences of spectral resolution on uncertainty in leaf trait estimates through a Bayesian approach to RTM inversion” Remote Sensing of the Environment 183: 226-238 Viskari et al. 2015 Model-data assimilation of multiple phenological observations to constrain and forecast leaf area index. Ecological Applications 25(2): 546-558 Dietze, M. C., S. P. Serbin, C. Davidson, A. R. Desai, X. Feng, R. Kelly, R. Kooper, D. LeBauer, J. Mantooth, K. McHenry, and D. Wang (2014) A quantitative assessment of a terrestrial biosphere model’s data needs across North American biomes. Journal of Geophysical Research-Biogeosciences doi:10.1002/2013jg002392 LeBauer, D.S., D. Wang, K. Richter, C. Davidson, & M.C. Dietze. (2013). Facilitating feedbacks between field measurements and ecosystem models. Ecological Monographs. doi:10.1890/12-0137.1 Wang, D, D.S. LeBauer, and M.C. Dietze(2013) Predicting yields of short-rotation hybrid poplar (Populus spp.) for the contiguous US through model-data synthesis. Ecological Applications doi:10.1890/12-0854.1 Dietze, M.C., D.S LeBauer, R. Kooper (2013) On improving the communication between models and data. Plant, Cell, & Environment doi:10.1111/pce.12043 Longer / auto-updated list of publications that mention PEcAn’s full name in Google Scholar "],
["contributor-covenant-code-of-conduct.html", "Contributor Covenant Code of Conduct", " Contributor Covenant Code of Conduct Our Pledge In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. Our Standards Examples of behavior that contributes to creating a positive environment include: Using welcoming and inclusive language Being respectful of differing viewpoints and experiences Gracefully accepting constructive criticism Focusing on what is best for the community Showing empathy towards other community members Examples of unacceptable behavior by participants include: The use of sexualized language or imagery and unwelcome sexual attention or advances Trolling, insulting/derogatory comments, and personal or political attacks Public or private harassment Publishing others’ private information, such as a physical or electronic address, without explicit permission Other conduct which could reasonably be considered inappropriate in a professional setting Our Responsibilities Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. Scope This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at pecanproj@gmail.com. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project’s leadership. Attribution This Code of Conduct is adapted from the Contributor Covenant version 1.4, available at http://contributor-covenant.org/version/1/4. "],
["about-the-pecan-book.html", "1 About the PEcAn Book 1.1 How to Contribute 1.2 How this Book is organized", " 1 About the PEcAn Book This book serves as documentation for the PEcAn Project. It contains descriptions of topics necessary to inform a beginner and advanced user as well as requisite materials for developers. It does not contain low-level descriptions of functions within PEcAn. Our aim for this documentation is to educate you about the PEcAn software, the possibilities of its usage, and the standards,expectations, and core workflows for developers. 1.1 How to Contribute If you wish to contribute to this book, we greatly welcome it! The same general steps to contributing PEcAn code to the main line can be used to contribute to this book. 1.1.1 Editing the book yourself Navigate to the main PEcAn github repository: https://github.com/PecanProject/pecan and fork the repository to your github account. Use git clone on the machine your are using to create a local copy of the repository. General instructions can be found here Navigate to the book_source directory and edit the page you would like to change using any text editor. Save your files in the R markdown format (.Rmd). If you are adding a new page, you must edit the _bookdown.yml file also and list your page in order for it to be properly indexed by gitbook. Within the directory book_source diectory, execute make Use git add changedfile.Rmd to stage your changes and then commit git commit -m "your message" your changes and push (git push) them to your fork. Create a pull request, using github, with a description of the changes you have made. 1.1.2 Feedback/Comments/Suggestions We want your ideas, thoughts, comments, and suggestions! This is an open-sourced project so please feel free to reach out to us in the following ways: Github: https://github.com/PecanProject/pecan Serving as the main hub of communication surrounding PEcAn development, check out the issues section to see what is coming down the pipeline. If there is anything you feel you can contribute to, leave a comment. Gitter: http://pecanproject.github.io Gitter serves as our day to day mode of communication. Go to our homepage, linked above, and open the “OPEN CHAT” tab at the bottom to see what we are up to. All you need is a github account and you can chat with us about anything project related. Email: pecanproj[at]gmail.com If you do not wish your communication with the team to be public, send us an email at the address above and we will get back to you as soon as possible. 1.2 How this Book is organized This book is organized into three main sections: A user section that is split three ways into a basic user’s guide, advanced web interface guide, and an advanced user guide for using pecan from the command line. A section dedicated to expaining the core modules contained within the PEcAn workflow. A developer section meant to serve as a guide to anyone interested in contributing to the project. "],
["getting-started.html", "2 Getting Started 2.1 How PEcAn Works in a nutshell 2.2 Quick Start", " 2 Getting Started 2.1 How PEcAn Works in a nutshell PEcAn provides an interface to a variety of ecosystem models and attempts to standardize and automate the processes of model parameterization, execution, and analysis. First, you choose an ecosystem model, then the time and location of interest (a site), the plant community (or crop) that you are interested in simulating, and a source of atmospheric data from the BETY database (LeBauer et al, 2010). These are set in a “settings” file, commonly named pecan.xml which can be edited manually if desired. From here, PEcAn will take over and set up and execute the selected model using your settings. The key is that PEcAn uses models as-is, and all of the translation steps are done within PEcAn so no modifications are required of the model itself. Once the model is finished it will allow you to create graphs with the results of the simulation as well as download the results. It is also possible to see all past experiments and simulations. 2.2 Quick Start There are two ways of using PEcAn, via the web interface and directly within R. Even for users familiar with R, using the web interface is a good place to start because it provides a high level overview of the PEcAn workflow.The quickest way to get started is to download the virtual machine or use an AWS instance. 2.2.1 Working with the PEcAn VM PEcAn consists of a set of scripts and code that is compiled within a Linux operating system and saved in a “virtual machine (VM)”. Virtual machines allow for running consistent set-ups without worrying about differences between operating systems, library dependencies, compiling the code, etc. To run the PEcAn VM you will need to install VirtualBox, the program that runs the virtual machine http://www.virtualbox.org. On Windows you may see a warning about Logo testing, it is okay to ignore the warning. After you have Virtual Box installed you’ll need to download the PEcAn virtual machine: http://opensource.ncsa.illinois.edu/projects/artifacts.php?key=PECAN. The virtual machine is available under the “Files” header. Click the 32 or 64 bit “.ova” file and note that the download is ~5 GB so will take from several minutes to hours depending on the connection speed. To open up the virtual machine you’ll first want to open up VirtualBox. The first time you use the VM you’ll want to use File → Import Appliance in VirtualBox in order to import the VM. This will create a virtual machine from the disk image. When asked about the Appliance Import Settings make sure you select “Reinitialize the MAC address of all network cards”. This is not selected by default and can result in networking issues since multiple machines might claim to have the same network MAC Address. That said, users who have experienced network connection difficulties within the VM have sometimes had better luck after reinstalling without reinitializing. Next, click “Import”. You only have to do this Import step once, in the future you can just skip to the next step. Start Pecan by double clicking on the icon for the VM. A terminal window will pop up showing the machine booting up which may take a minute. It is done booting when you get to the “pecan32 login:” prompt. You do not need to login as the VM behaves like a server that we will be accessing through you web browser. Feel free to minimize the VM window. If you do want to login to the VM, the credentials are as follows: username: carya, password: illinois (after the pecan tree, Carya illinoinensis). To launch the PEcAn web interface, make sure that the VM is running somewhere in the background on your machine. Then, open any web browser on the same machine and navigate to localhost:6480/pecan to start the PEcAn workflow. 2.2.2 Using AmazonWeb Services (AWS) Login to Amazon Web Services (AWS) and select the EC2 Dashboard. If this is your first time using AWS you will need to set up an account before you are able to access the EC2 Dashboard. Important: You will need a credit card number and access to a phone to be able to verify AWS account registration. AWS is free for one year. Choose AMI On the top right next to your name, make sure the location setting is on U.S. East (N. Virginia), not U.S. West (Oregon) On the left click, click on EC2 (Virtual servers), then click on “AMIs”, also on the left In the search window toggle to change “Owned by me” to “Public images” Type “pecan” into the search window Click on the toggle button on the left next to PEcAn1.4.6 Click on the “Launch” button at the top Choose an Instance Type Select what type of machine you want to run. For this demo the default, t2.micro, will be adequate. Be aware that different machine types incur very different costs, from 1.3 cents/hour to over $5/hr https://aws.amazon.com/ec2/pricing/ Select t2.micro, then click “Next: Configure Instance Details” Configure Instance Details The defaults are OK. Click “Next: Add Storage” Add Storage The defaults are OK. Click “Next: Tag Instance” Tag Instance You can name your instance if you want. Click “Next: Configure Security Group” Configure Security Group You will need to add two new rules: Click “Add Rule” then select “HTTP” from the pull down menu. This rule allows you to access the webserver on PEcAn. Click “Add Rule”, leave the pull down on “Custom TCP Rule”, and then change the Port Range from 0 to 8787. Set “Source” to Anywhere. This rule allows you to access RStudio Server on PEcAn. Click “Review and Launch” . You will then see this pop-up: Select the default drive volume type and click Next Review and Launch Review the settings and then click “Launch”, which will pop up a select/create Key Pair window. Key Pair Select “Create a new key pair” and give it a name. You won’t actually need this key unless you need to SSH into your PEcAn server, but AWS requires you to create one. Click on “Download Key Pair” then on “Launch Instances”. Next click on “View Instances” at the bottom of the following page. Instances You will see the status of your PEcAn VM, which will take a minute to boot up. Wait until the Instance State reads “running”. The most important piece of information here is the Public IP, which is the URL you will need in order to access your PEcAn instance from within your web browser (see Demo 1 below). Be aware that it often takes ~1 hr for AWS instances to become fully operational, so if you get an error when you put the Public IP in you web browser, most of the time you just need to wait a bit longer. Congratulations! You just started a PEcAn server in the “cloud”! When you are done using PEcAn, you will want to return to the “Instances” menu to turn off your VM. To STOP the instance (which will turn the machine off but keep your work), select your PEcAn instance and click Actions > Instance state > Stop. Be aware that a stopped instance will still accrue a small storage cost on AWS. To restart this instance at any point in the future you do not want to repeat all the steps above, but instead you just need to select your instance and then click Actions > Instance state > Start To TERMINATE the instance (which will DELETE your PEcAn machine), select your instance and click Actions > Instance state > Terminate. Terminated instances will not incur costs. In most cases you will also want to go to the Volumes menu and delete the storage associated with your PEcAn VM.Remember, AWS is free for one year, but will automatically charge a fee in second year if account is not cancelled. "],
["the-model-site-page.html", "3 The model-site page 3.1 Selecting a model 3.2 Selecting a site", " 3 The model-site page This page is used to select the model to run and the site at which you would like to run that model. NOTE: If this page does not load for you, it may be related to a known Google Maps API key issue. See issue #1269 for a possible solution. 3.1 Selecting a model On the Select Host webpage use the Host pull-down menu to select the server you want to run on. Note that at the moment only the LOCAL server will run through the web-based workflow. Next, select the model you want to run under the Model pull-down menu If the model you want to run is not listed login to BETY and navigate to Runs > Models. If there are already entries for the model you want to run then most likely the PEcAn modules for that model have been implemented but PEcAn is unaware that the model has been installed on your server. Information on adding a model executable to the PEcAn database can be found here. After this is done, your model should appear on the PEcAn Select Host page after your refresh the page. If there a no entries for the model you want to run then most likely the PEcAn modules need to be implemented. See the instructions at Adding an Ecosystem Model If selecting your model causes your site to disappear from the Google Map then the site exists but there are no drivers for that site registered in the database. See the instruction here for more info on diagnosing what drivers are missing. 3.2 Selecting a site 3.2.1 Using existing sites Find the site on the map The simplest way of determining if a site exists in PEcAn is through the Google Map interface of the web-based workflow. You’ll want to make sure that the Host is set to All Sites and the Model is set to All Models. Find the site in BETY If the site is not on the map there’s a chance that it still in PEcAn but just lacking geographic information. To begin you’ll want to login to your local version of the BETY database. If you’re on the PEcAn VM this will be at localhost:3280/bety or localhost:6480/bety depending on whether you downloaded the 32 or 64 bit version. Within BETY navigate to Data > Sites and use the Search window on the page to try and locate your site. If you DO find your site you will want to click Edit and add geographic information so that the site will show up on the map. It is also worth noting that the site ID number shows up in the URL for the Show or Edit pages. This ID is frequently useful to know, for example if you have to set up a PEcAn settings file by hand. If you did not find you site you will want to follow the instructions below for adding a site 3.2.2 Adding a new site To add a new site to PEcAn you currently have to begin by logging in to BETY Navigate to Data > Citations. Before you add a Site you will need to establish a Citation to associate with the site. If you went to Data > Sites and just clicked on Add Site it will take you to an error page reminding you of this. Before adding a new citation, you should Search the system to make sure the Citation you are about to add doesn’t already exist. If it does you should be able to just click on the check-box icon to select that citation and then skip the next step To create a new citation click on the New Citation button, fill in the fields, and then click Create. The field URL should contain the web address that takes you to this publication on the publisher’s website. The Pdf field should similarly be the web address for the pdf for this citation. If there are no publications associated with the site you are interested in you can leave most of the fields blank, but you could add a descriptive title, such as “EBI Farm Field Data”, and a relevant contact person as the Author so that future users can associate any site-specific data with this citation. Once the Citation is created or selected this should automatically take you to the Sites page and list any Sites already associated with this citation. To create a new site click the New Site button. When creating a new cite the most critical information is the Site name and the Lat/Lon. The Lat/Lon can be entered by hand or by clicking on the site location on the Google Map interface. It is also helpful to fill in the other location information (Cite, State, Country) and the Notes to make it easier for other users to search for the site. The topographic, climate, and soils information is optional. When you are done click Create. At this point if you refresh the PEcAn site-level run webpage the site should automatically show up. 3.2.3 Troubleshooting 3.2.3.1 My site shows up when I don’t have any model selected, but disappears once I select the model I want to run Selecting a model will cause PEcAn to filter the available sites based on whether they possess the required Inputs for a given model (e.g. meteorology). To check what Inputs are missing for a site point your browser to the pecan/checksite.php webpage (e.g. localhost:3280/pecan/checksite.php). This page looks virtually identical to the site selection page, except that it has a Check button instead of Prev and Next. If you select a Machine, Model, and Site and then click Check the page should return a list of what Inputs are missing (listing both the name and the Format ID number). Don’t forget that its possible for PEcAn to have required Inputs in its database, but just not have them for the Machine where you want to run. To see more about what Inputs a given model can accept, and which of those are required, take a look at the MODEL_TYPE table entry in the database (e.g. go to localhost:3280/bety; Select Runs > Model Type; and then click on the model you want to run). For information about loading missing Inputs into the database visit How to insert new Input data, and also read the rest of the pages under this section, which will provide important information about the specific classes of Inputs (e.g. meteorology, vegetation, etc). Finally, we are in the process of developing workflows and standards for processing Input data in a model-agnostic way, starting with meteorology. So hopefully much of the work of building model driver files and uploading them into PEcAn will become automatic soon. "],
["basic-model-configuration.html", "4 Basic model configuration 4.1 Choosing meteorology 4.2 Choosing initial vegetation 4.3 Selecting a soils product", " 4 Basic model configuration This page is used for basic model configuration, including when your model will run and what input data it will use. 4.1 Choosing meteorology Once a Machine, Model, and Site have been selected, PEcAn will take you to the Input selection page. From this page you will select what Plant Functional Type (PFT) you want to run at a site, the start and end dates of the run, and various Input selections. The most common of these across all models is the need to specify meteorological forcing data. The exact name of the menu item for meteorology will vary by model because all of the Input requirements are generated individually for each model based on the MODEL_TYPE table. In general there are 3 possible cases for meteorology PEcAn already has driver files in its database PEcAn does not have drivers, but can generate them from publicly available data You need (or want) to upload your own drivers The first two cases will appear automatically in the the pull down menu. For meteorological files that already exist you will see the date range that’s available. By contrast, met that can be generated will appear as “Use ”, where is the origin of the data (e.g. “Use Ameriflux” will use the micromet from an Ameriflux eddy covariance tower, if one is present at the site). If you want to upload your own met data this can be done in three ways. The standard way is to upload data in PEcAn’s standard meteorological format (netCDF files, CF metadata). See here for details about variables and units. From this standard, PEcAn can then convert the file to the model-specific format required by the model you have chosen. This approach is preferred because PEcAn will also be able to convert the file into the format required by any other model as well. The second option for adding met data is to add it in a model-specific format, which is often easiest if you’ve already been running your model at a site and are just switching to using PEcAn. The final way to add met data is to incorporate it into the overall meteorological processing workflow. This is preferred if you are working with a common meteorological data product that is not yet in PEcAn’s workflow. Details are here, though at this stage you would also be strongly encouraged to contact the PEcAn development team. 4.1.1 Met workflow In a nutshell, the PEcAn met workflow is designed to reduce the problem of converting n possible met inputs into m possible model formats, which requires n x m conversion functions as well as numerous custom functions for downscaling, gap filling, etc. Instead, PEcAn works with a single met standard, and thus requires n conversion functions, one for converting each data source into the PEcAn standard, and then m conversion functions for converting from that standard to what an individual model requires. For a new model joining the PEcAn system the burden in particularly low – writing one conversion function provides access to n inputs. Similarly, PEcAn performs all other operations/manipulations (extracting a site, downscaling, gap filling, etc) within the PEcAn standard, which means these operations only need be implemented once. Consider a generic met data product named MET for simplicity. PEcAn will use a function, download.MET, to pull data for the selected year from a public data source (e.g. Ameriflux, North American Regional Reanalysis, etc). Next, PEcAn will use a function, met2CF.MET, to convert the data into the PEcAn standard. If the data is already at the site scale it will then gapfill the data. If the data is a regional or global data product, PEcAn will then permute the data to allow easier site-level extraction, then it will extract data for the requested site and data range. Modules to address the temporal and spatial downscaling of meteorological data products, as well as their uncertainties, are in development but not yet part of the operational workflow. All of these functions are located within the data.atmosphere module. Once data is in the standard format and processed, it will be converted to the model-specific format using a met2model.MODEL function (located in that MODEL’s module). 4.1.2 Troubleshooting meteorological conversions At the current moment (PEcAn 1.4.0), most of the issues below address possible errors that the Ameriflux meteorology workflow might report 4.1.2.1 Could not do gapfill … The following variables have NA’s This error message means that there were gaps in the downloaded data, for whatever variables that were listed, which were larger than the current algorithm could fill. Particularly common is missing radiation or PAR data, as Ameriflux frequently converts nighttime data to NULL, and work is in progress to detect this based on solar geometry. Also common are incomplete years (first or last year of tower operations). 4.1.2.2 Could not get information about . Is this an Ameriflux site? This message occurs when PEcAn believes that a site is part of Ameriflux (because it was listed on the Ameriflux or FLUXNET webpage and has a US-* site code), but no data is present on the Ameriflux server. The most common reasons for this is that you have selected a site that has not submitted data to Ameriflux yet (or that data hasn’t been processed yet), or you have selected a year that’s outside the tower’s operational period. Visit Ameriflux and FLUXNET for lists of available site years. 4.1.2.3 Could not download data for for the year This is similar to the previous error, but in this case PEcAn did find data for the site listed, but just not for the year requested. This can usually be fixed by just altering the years of the run to match those with available data. 4.1.2.4 I could not find the requested var (or dimvar) in the file! PEcAn could not find a required variable within the downloaded file. Most likely this is due to that variable not being measured at this site. The most common cause of failure is the absence of atmospheric pressure data (PRESS), but since most models have a low sensitivity to this variable we are working on methods to estimate this from other sources. 4.2 Choosing initial vegetation On the Input Selection webpage, in addition to selecting PFTs, start & end dates, and meteorology, many models also require some way of specifying the initial conditions for the vegetation, which may range from setting the aboveground biomass and LAI up to detailed inventory-like data on species composition and stand structure. At the moment, PEcAn has two cases for initial conditions and they only exist as required Inputs for the ED2 model: If files already exist in the database, they can simply be selected from the menu. For ED2, there are 3 different veg files (site, pss, css) and it is important that you select a complete set, not mix and match. If files don’t exist they can be uploaded following the instructions on [[How to insert new Input data]]. Information on the ED2-specific format is located here Two additional options are in development: Model spin-up Automated workflows 4.2.1 Spin up A number of ecosystem models are typically initialized by spinning up to steady state. At the moment PEcAn doesn’t handle spin up automatically (e.g. looping met, checking for stability), but there are various ways to achieve a spin-up within the system. Option 1: If there are model-specific settings in a model’s settings/config file, then that file can be accessed by clicking on the Edit model config check box. If this box is selected then PEcAn will pause the site run workflow after it has generated your model config file, but before it runs the model, and give you an opportunity to edit the file by hand, allowing you to change any model-specific spin up settings (e.g met recycling, spin up length) Option 2: Set start_year very early and set the met drivers to be a long time series (e.g. PalEON, something custom uploaded to Inputs) Option 3: In the MODEL_TYPE table, add your model’s restart format as an optional input, modify the model specific write.config function to use that restart, and then load a previous spin-up to the Inputs table Beyond these options, we hope to eventually develop more general, model-agnostic tools for spin up. In particular, we have started to explore the accelerated spin-up and semi-analytical techniques being developed by Yiqi Luo’s lab 4.2.2 Veg workflow As with meteorology, PEcAn is working to develop a model-agnostic workflow for converting various sources of vegetation data to common standards, developing common processing tools, and then writing out to model-specific formats. This process is in a much early stage than the meteorology workflow, as we are still researching what the options are for standard formats, but ultimately aims to be much more broad in scope, considering not just plot inventory data but also historical documentation, paleoecological proxies, satellite remote sensing (e.g. LANDSAT), airborne hyperspectral imagery, and active remote sensing (Lidar, Radar). At the moment, what is functional is a model-specific workflow for the ED2 model that can query the USFS Forest Inventory and Analysis and then construct initial condition files for ED2. This tool works with an internal copy of the FIA that is uploaded to a postGRES database along side BETY, however for space reasons this database does not ship with the PEcAn VM. To turn this feature on: Download and Install the FIA database For web-base runs, specify the database settings in the config.php For R-based runs, specify the database settings in the pecan.xml 4.3 Selecting a soils product Many models have requirements for soils information, which may include: site-specific soil texture and depth information; soil biogeochemical initial conditions (e.g. soil carbon and nitrogen pools); soil moisture initial conditions; and soil thermal initial conditions. As with [Choosing initial vegetation], we eventually hope to develop data standards, soils workflows, and spin-up tools, but at the moment model requirements need to be met by [[inserting Input data|How to insert new Input data]] into the database or using files that have already been uploaded. Finally, any other model-specific inputs (e.g. N deposition, land use history, etc), should be met by [[inserting Input data|How to insert new Input data]] into the database or using files that have already been uploaded. "],
["vm-desktop-conversion.html", "5 VM Desktop Conversion", " 5 VM Desktop Conversion sudo apt-get update sudo apt-get install xfce4 xorg For a more refined desktop environment, try sudo apt-get install --no-install-recommends xubuntu-desktop replace xubuntu- with ubuntu-, lubuntu-, or other preferred desktop enviornment the --no-install-recommends eliminates additional applications, removing it will add a word processor, a browser, and lots of other applications included in the default operating system. Reinstall Virtual Box additions for better integration adding X/mouse support sudo mount /dev/cdrom /mnt sudo /mnt/VBoxLinuxAdditions.run sudo umount /mnt 5.0.1 Install RStudio Desktop wget http://download1.rstudio.org/rstudio-0.97.551-amd64.deb apt-get install libjpeg621 dpkg -i rstudio-* rm rstudio-* "],
["advanced-web-interface-guide.html", "6 Advanced Web Interface Guide", " 6 Advanced Web Interface Guide This section will provide information to those wanting to take advantage of PEcAn’s customizations from the web interface. 6.0.1 What do all those other buttons on the web interface do? Using Browndog PEcAn XML Advanced Setup 6.0.2 How do I perform specific analyses? Under construction Ensemble prediction Sensitivity & Uncertainty analysis Parameter data assimilation State data assimilation Benchmarking "],
["using-browndog.html", "7 Using Browndog", " 7 Using Browndog "],
["pecan-xml-configuration.html", "8 PEcAn XML Configuration 8.1 PEcAn folders 8.2 Database Access 8.3 BETY Database Configuration 8.4 Brown Dog Configuration 8.5 PFT Selection 8.6 Meta Analysis 8.7 Ensemble Runs 8.8 Sensitivity Runs 8.9 Model Setup 8.10 Run Setup 8.11 State Data Assimilation Tags 8.12 Parameter Data Assimilation 8.13 Benchmarking", " 8 PEcAn XML Configuration The PEcAn system is configured using a xml file, often called settings.xml. The configuration file can be split in X seperate pieces: PEcAn Folders Database Access BETY Configuration Brown Dog PFT Selection Meta Analysis Ensemble Runs Sensitivity Runs Model Setup Run Setup State Data Assimilation 8.1 PEcAn folders The following are the tags that can be used to configure the folders used by PEcAn. All of these are optional. <outdir>/home/carya/testrun.pecan</outdir> outdir : [optional] specifies where PEcAn will write all outputs and create folders. If this is not specified the folder pecan in the current folder will be used. 8.2 Database Access The connection to the BETY database is configured using this section. In this section you will specify what driver to use to connect to the database (PostgreSQL by default) and the connection parameters to connect to the database. This section is very picky and will only accept parameters that are passed into the connection function for each database driver, any other entries will result in an error by the database driver. 8.2.0.1 functions: db.open(), and thus by db.query() and db.check(). New format (starting with PEcAn 1.3.6) <database> <bety> <dbname>bety</dbname> <username>bety</username> <password>bety</password> <host>localhost</host> <write>true</write> </bety> <fia> <dbname>fia5data</dbname> <username>bety</username> <password>bety</password> <host>localhost</host> </fia> </database> The connection information under bety will be used by most of the PEcAn system, only the extraction of pss/css/site information from the FIA database will use the connection information in the fia section. driver : [optional] the driver to use to connect to the database. Default value is PostgreSQL dbname : [optional] the name of the database (was name), default value is the username of the current user logged in. user : [optional] the username to connect to the database (was userid), default value is the username of the current user logged in (PostgreSQL uses user for this field). password : [optional] the password to connect to the database (was passwd), if not specified no password is used. host : [optional] the name of the host to connect to, default value is localhost. write : [optional] should any results be written back to the database, setting this to TRUE (the default) will write any results back into the database. (since PEcAn 1.3.6) For other database drivers these parameters will change. See the driver documentation in R for the right parameters. 8.3 BETY Database Configuration This section describes how to connect to the BETY Database. This section is used for versions of PEcAn prior to version 1.3.6, starting at 1.3.6 this parameter is part of <database><bety> and defaults to TRUE. 8.3.0.1 functions: db.check(), write.configs(), run.models() <bety> <write>TRUE</write> </bety> write : [optional] this can be TRUE/FALSE (the default is TRUE). If set to TRUE, runs, ensembles and workflows are written to the database. 8.4 Brown Dog Configuration This section describes how to connect to Brown Dog. This will allow for conversions of data (currently only met data). 8.4.0.1 functions: met.process() <browndog> <url>...</url> <username>...</username> <password>...</password> </browndog> url : [required] endpoint for Brown Dog to be used. username : [optional] username to be used with the endpoint for Brown Dog. password : [optional] password to be used with the endpoint for Brown Dog. 8.5 PFT Selection The PEcAn system requires at least 1 PFT (Plant Functional Type) to be specified inside the <pfts> section. 8.5.0.1 functions: get.traits() 8.5.0.2 tags <pfts> <pft> <name>sipnet.temperate.coniferous</name> <outdir>/home/carya/testrun.pecan/pft/1/</outdir> <constants> <num>1</num> </constants> </pft> </pfts> name : [required] the pft as is found inside the BETY database, this needs to be an exact match. outdir: [optional] path in which pft-specific output will be placed during meta-analysis and sensitivity analysis. If not specified it will be written into <outdir>/<pftname>. contants: [optional] this section contains information that will be written directly into the model specific configuration files. PEcAn does not look at the information in this section. 8.6 Meta Analysis 8.6.0.1 functions: run.meta.analysis() 8.6.0.2 tags <meta.analysis> <iter>1000</iter> <random.effects>FALSE</random.effects> <update>FALSE</update> <threshold>1.2</threshold> </meta.analysis> The section meta.analysis needs to exists for a meta.analysis to be executed, even though all tags inside are optional. iter : [optional] [MCMC](http:/en.wikipedia.org/wiki/Markov_chain_Monte_Carlo) (Markov Chain Monte Carlo) chain length, i.e. the total number of posterior samples in the meta-analysis, default is 3000. Smaller numbers will run faster but produce larger errors. random.effects : [optional] Whether to include random effects (site, treatment) in meta-analysis model. Can be set to FALSE to work around convergence problems caused by an over parameterized model (e.g. too many sites, not enough data). The default value is TRUE. update : [optional] Should previous results of meta.analysis and get.traits be re-used. If set to TRUE the meta-analysis and get.trait.data will always be executed. Setting this to FALSE will try and reuse existing results. Future versions will allow for AUTO as well which will try and reuse if the PFT/traits have not changed. The default value is FALSE. threshold threshold for Gelman-Rubin convergence diagnostic (MGPRF); default is 1.2 8.7 Ensemble Runs Only if this section is defined an ensemble analysis is done. 8.7.0.1 functions: write.configs(), run.ensemble.analysis() 8.7.0.2 tags <ensemble> <size>5</size> <variable>GPP</variable> <variable>NPP</variable> </ensemble> size : [required] the number of runs in the ensemble. variable: [optional] name of one (or more) variables the analysis should be run for. If not specified, sensitivity.analysis variable is used, otherwise default is GPP. Note: if the ensemble size is set to 1, PEcAn will select the posterior median parameter values rather than taking a single random draw from the posterior 8.8 Sensitivity Runs Only if this section is defined a sensitivity analysis is done. This section will have <quantile> or <sigma> nodes. If neither are given, the default is to use the median +/- [1 2 3] x sigma (e.g. the 0.00135 0.0228 0.159 0.5 0.841 0.977 0.999 quantiles); If the 0.5 (median) quantile is omitted, it will be added in the code. 8.8.0.1 functions: write.configs(), run.sensitivity.analysis() 8.8.0.2 tags <sensitivity.analysis> <quantiles> <quantile></quantile> <sigma>-3</sigma> <sigma>-2</sigma> <sigma>-1</sigma> <sigma>1</sigma> <sigma>2</sigma> <sigma>3</sigma> </quantiles> <variable>GPP</variable> <start.year>2004</start.year> <end.year>2006</end.year> </sensitivity.analysis> quantiles : [optional] Quantiles of parameter distributions at which the model should be evaluated when running sensitivity analysis. Values greater than 0 and less than 1 can be used. sigma : [optional] Any real number can be used to indicate the quantiles to be used in units of normal probability. quantile : [optional] Which quantile should be used. start.date : [required?] start date of the sensitivity analysis (in YYYY/MM/DD format) end.date : [required?] end date of the sensitivity analysis (in YYYY/MM/DD format) NOTE: start.date and end.date are distinct from values set in the run tag because this analysis can be done over a subset of the run. ** variable** : [optional] name of one (or more) variables the analysis should be run for. If not specified, sensitivity.analysis variable is used, otherwise default is GPP. 8.9 Model Setup This section is required and tells PEcAn what model to run. This section should either specify <id> or both <name> and <binary> of the model. If both id and name and/or binary are specified the id is used to check the specified name and/or binary. As of version 1.3.7 the <name> tag has been renamed <type>. The <type> tag refers to the “type” of model and is used for a) identifying appropriate pfts and b) identifying whether appropriate inputs (i.e. met data) are available for the given site and model (or if they can be created). To ensure compatability, the code will automatically convert from <name> to <type>. 8.9.0.1 functions: write.configs(), run.models() 8.9.0.2 tags <model> <id>7</id> <type>ED2</type> <binary>/usr/local/bin/ed2.r82</binary> <job.sh>module load hdf5</job.sh> <config.header> <!--...xml code passed directly to config file...--> </config.header> </model> id : [optional/required] id in the models database table, see above. name : OBSOLETE name of the model, use type from version 1.3.7 type : [optional/required] type of model, see above. binary : [optional/required] path to the model executable, see above. job.sh : [optional] additional options to add to the job.sh at the top config.headers : [optional] XML that will appear at the start of generated config files. 8.9.0.3 ED2 specific tags Following variables are ED specific and are used in the ED2 Configuration. Starting at 1.3.7 the tags for inputs have moved to <run><inputs>. This includes, veg, soil, psscss, inputs. <edin>/home/carya/runs/PEcAn_4/ED2IN.template</edin> <config.header> <radiation> <lai_min>0.01</lai_min> </radiation> <ed_misc> <output_month>12</output_month> </ed_misc> </config.header> <phenol.scheme>0</phenol.scheme> edin : [required] template used to write ED2IN file veg : OBSOLETE [required] location of VEG database, now part of ` soil : OBSOLETE [required] location of soild database, now part of ` psscss : OBSOLETE [required] location of site inforation, now part of <run><inputs>. Should be specified as <pss>, <css> and <site>. inputs : OBSOLETE [required] location of additional input files (e.g. data assimilation data), now part of <run><inputs>. Should be specified as <lu>' and`. 8.10 Run Setup 8.10.0.1 tags <run> <jobtemplate>/home/carya/path/to/template</jobtemplate> <start.date>2002-01-01 00:00:00</start.date> <end.date>2005-12-31 00:00:00</end.date> </dbfiles>/home/carya/.pecan/dbfiles</dbfiles> <site> <id>772</id> <name>Niwot Ridge Forest/LTER NWT1 (US-NR1)</name> <lat>40.032900</lat> <lon>-105.546000</lon> <met.start>2002/01/01</met.start> <met.end>2005/12/31</met.end> </site> <inputs> <met> <id>10000000001</id> <path>/fs/data1/pecan.data/input/</path> <source>Ameriflux</source> <username>carya</username> </met> </inputs> </run> <host> <name>localhost</name> <rundir>/home/carya/testrun.pecan/run/</rundir> <outdir>/home/carya/testrun.pecan/out/</outdir> <scratchdir>/tmp/carya</scratchdir> <clearscratch>TRUE</clearscratch> <qsub>qsub -N @NAME@ -o @STDOUT@ -e @STDERR@ -S /bin/bash</qsub> <qsub.jobid>Your job ([0-9]+) .*</qsub.jobid> <qstat>qstat -j @JOBID@ &> /dev/null || echo DONE</qstat> <job.sh>module load udunits R/R-3.0.0_gnu-4.4.6</job.sh> </host> jobtemplate : [optional] the template used when creating a job.sh file which is used to launch the actual model. Each model has it’s own template in the inst folder of the module. The following variables can e used: @SITE_LAT@, @SITE_LON@, @SITE_MET@, @START_DATE@, @END_DATE@, @OUTDIR@, @RUNDIR@ which all come variables in the pecan.xml file. The following two command can be used to copy and clean the results from a scratch folder (specified as scratch in the run section below, for example local disk vs network disk) : @SCRATCH_COPY@, @SCRATCH_CLEAR@ . start.date : [required] the first day of the simulation end.date : [required] the last day of the simulation dbfiles : [optional] location where pecan should write files that will be stored in the database. The default is store them in ${HOME}/.pecan/dbfiles Site specific information is specified in the <site> subsection. Either <id> or <name>, <lat> and <lon> should be specified. If id and any of the others are specified the values will be compared with those from BETYdb. id : [optional/required] id of the site in the BETY database, see above. name : [optional/required] site name, see above. lat : [optional/required] site latitude, see above. lon : [optional/required] site longitude, see above. Inputs specific information for each model type is specified in the <inputs> subsection. Each input will have three tags: * id : [optional/required] An id that points to the input if already in the database. * path : [optional/required] The path to the dataset if already downloaded. * source : [optional/required] The input data type. This tag name needs to match the names in the conversion functions. In general, the path should be filled in by PEcAn and not by the user. If PEcAn is processing all the input, id is not required. Alternatively, if you’re using data processed by hand, source is not required. One common input will be the weather data, often specified in <met>. Other model types will have different model specific inputs. PEcAn will find the location of the file on the host you run PEcAn on. met : [model type specific] most models will have a met tag to specify location of the weather data. pss : [ED2, required] location of patch file css : [ED2, required] location of cohort file site : [ED2, optional] location of site file lu : [ED2, required] location of land use file thsums : [ED2, required] location of thermal sums file veg : [ED2, required] location of vegetation data soil : [ED2, required] location of soil data Host on which the simulation will run is specified in the <host> subsection. If this section is not specified it is assumed the simulation will run on localhost. If qsub is specified (can be empty in which case the defaults will be used) the models will be executed using qsub. name : [optional] name of host server where model is located and executed, if not specified localhost is assumed. rundir : [optional/required] location where all the configuration files are written. For localhost this is optional (<outdir>/run is the default), for any other host this is required. outdir : [optional/required] location where all the outputs of the model are written. For localhost this is optional (<outdir>/out is the default), for any other host this is required. scratchdir : [optional] location where output is written. If specified the output from the model is written to this folder and copied to the outdir when the model is finished, this could significantly speed up the model execution (by using local or ram disk). clearscratch : [optional] if set to TRUE the scratchfolder is cleaned up after copying the results to the outdir, otherwise the folder will be left. The default is to clean up after copying. qsub : [optional] the command to submit a job to the queuing system. There are 3 parameters you can use when specifying the qsub command, you can add additional values for your specific setup (for example -l walltime to specify the walltime, etc). You can specify @NAME@ the pretty name, @STDOUT@ where to write stdout and @STDERR@, where to write stderr. You can specify an empty element () in which case it will use the default value is “qsub -V -N @NAME@ -o @STDOUT@ -e @STDERR@ -s /bin/bash”. qsub.jobid : [optional] the regular expression used to find the jobid returned from qsub. If not specified (and qsub is) it will use the default value is “Your job ([0-9]+) .*" qstat : [optional] the command to execute to check if a job is finished, this should return DONE if the job is finished. There is one parameter this command should take @JOBID@ which is the id of the job as returned by qsub.jobid. If not specified (and qsub is) it will use the default value is “qstat -j @JOBID@ || echo DONE” job.sh : [optional] additional options to add to the job.sh at the top 8.11 State Data Assimilation Tags The following tags can be used for state data assimilation. More detailed information can be found here: State Data Assimilation Documentation <state.data.assimilation> <process.variance>FALSE</process.variance> <sample.parameters>FALSE</sample.parameters> <state.variables> <variable>AGB.pft</variable> <variable>TotSoilCarb</variable> </state.variables> <spin.up> <start.date>2004/01/01</start.date> <end.date>2006/12/31</end.date> </spin.up> <forecast.time.step>1</forecast.time.step> <start.date>2004/01/01</start.date> <end.date>2006/12/31</end.date> </state.data.assimilation> process.variance : [optional] TRUE/FLASE flag for if process variance should be estimated (TRUE) or not (FALSE). If TRUE, a generalized ensemble filter will be used. If FALSE, an ensemble Kalman filter will be used. Default is FALSE. sample.parameters : [optional] TRUE/FLASE flag for if parameters should be sampled for each ensemble member or not. This allows for more spread in the intial conditions of the forecast. NOTE: If TRUE, you must also assign a vector of trait names to pick.trait.params within the sda.enkf function. state.variable : [required] State variable that is to be assimilated (in PEcAn standard format). Default is “AGB” - Above Ground Biomass. spin.up : [required] start.date and end.date for model spin up. NOTE: start.date and end.date are distinct from values set in the run tag because spin up can be done over a subset of the run. forecast.time.step : [optional] start.date and end.date for model spin up. start.date : [required?] start date of the state data assimilation (in YYYY/MM/DD format) end.date : [required?] end date of the state data assimilation (in YYYY/MM/DD format) NOTE: start.date and end.date are distinct from values set in the run tag because this analysis can be done over a subset of the run. 8.12 Parameter Data Assimilation The following tags can be used for state data assimilation. More detailed information can be found here: Parameter Data Assimilation Documentation 8.12.0.1 tags Coming soon… 8.13 Benchmarking Coming soon… "],
["parameter-data-assimilation-1.html", "9 Parameter Data Assimilation 9.1 All functions pertaining to Parameter Data Assimilation are housed within: pecan/modules/assim.batch", " 9 Parameter Data Assimilation 9.1 All functions pertaining to Parameter Data Assimilation are housed within: pecan/modules/assim.batch 9.1.1 pda.mcmc.R This is the main PDA code. It performs Bayesian MCMC on model parameters by proposing parameter values, running the model, calculating a likelihood (between model output and supplied observations), and accepting or rejecting the proposed parameters (Metropolis algorithm). Additional notes: The first argument is settings, followed by others that all default to NULL.settings is a list used throughout Pecan, which contains all the user options for whatever analyses are being done. The easiest thing to do is just pass that whole object all around the Pecan code and let different functions access whichever settings they need. That’s what a lot of the rest of the Pecan code does. But the flexibility to override most of the relevant settings in settings is there by providing them directly as arguments to the function. The if(FALSE)… : If you’re trying to step through the function you probably will have the settings object around, but those other variables will be undefined. If you set them all to NULL then they’ll be ignored without causing errors. It is there for debugging purposes. The next step calls pda.settings(), which is in the file pda.utils.R (see below). It checks whether any settings are being overridden by arguments, and in most cases supplies default values if it can’t find either. In the MCMC setup section The code is set up to allow you to start a new MCMC chain, or to continue a previous chain as specified in settings. The code writes a simple text file of parameter samples at every iteration, which lets you get some results and even re-start an MCMC that fails for some reason. The code has adaptive jump distributions. So you can see some initialization of the jump distributions and associated variables here. Finally, note that after all this setup a new XML settings file is saved. The idea is that the original pecan.xml you create is preserved for provenance, and then periodically throughout the workflow the settings (likely containing new information) are re-saved with descriptive filenames. MCMC loop Periodically adjust jump distribution to make acceptance rate closer to target Propose new parameters one at a time. For each: First, note that Pecan may be handling many more parameters than are actually being targeted by PDA. Pecan puts priors on any variables it has information for (in the BETY database), and then these get passed around throughout the analysis and every step (meta-, sensitivity, ensemble analyses, etc.). But for PDA, you specify a separate list of probably far fewer parameters to constrain with data. These are the ones that get looped over and varied here. The distinction between all parameters and only those dealt with in PDA is dealt with in the setup code above. First a new value is proposed for the parameter of interest. Then, a new model run is set up, identical to the previous except with the new proposed value for the one parameter being updated on this run. The model run is started, and outputs collected after waiting for it to finish. A new likelihood is calculated based on the model outputs and the observed dataset provided. Standard Metropolis acceptance criteria is used to decide whether to keep the proposed parameter. Periodically (at interval specified in settings), a diagnostic figure is saved to disk so you can check on progress. This works only for NEE currently 9.1.2 pda.mcmc.bs.R This file is basically identical to pda.mcm.R, but rather than propose parameters one at a time, it proposes new values for all parameters at once (“bs” stands for “block sampling”). You choose which option to use by specifying settings\\(assim.batch\\)method: * “bruteforce” means sample parameters one at a time * “bruteforce.bs” means use this version, sampling all parameters at once * “emulator” means use the emulated-likelihood version 9.1.3 pda.emulator This version of the PDA code again looks quite similar to the basic “bruteforce” one, but its mechanics are very different. The basic idea is, rather than running thousands of model iterations to explore parameter space via MCMC, run a relatively smaller number of runs that have been carefully chosen to give good coverage of parameter space. Then, basically interpolate the likelihood calculated for each of those runs (actually, fit a Gaussian process to it), to get a surface that “emulates” the true likelihood. Now, perform regular MCMC (just like the “bruteforce” approach), except instead of actually running the model on every iteration to get a likelihood, just get an approximation from the likelihood emulator. Since the latter step takes virtually no time, you can run as long of an MCMC as you need at little computational cost, once you have done the initial model runs to create the likelihood emulator. 9.1.4 pda.mcmc.recover.R This function is for recovering a failed PDA MCMC run. 9.1.5 pda.utils.R This file contains most of the individual functions used by the main PDA functions (pda.mcmc.*.R). assim.batch is the main function Pecan calls to do PDA. It checks which method is requested (bruteforce, bruteforce.bs, or emulator) and call the appropriate function described above. pda.setting handles settings. If a setting isn’t found, the code can usually supply a reasonable default. pda.load.priors is fairly self explanatory, except that it handles a lot of cases and gives different options priority over others. Basically, the priors to use for PDA parameters can come from either a Pecan prior.distns or post.distns object (the latter would be, e.g., the posteriors of a meta-analysis or previous PDA), or specified either by file path or BETY ID. If not told otherwise, the code tries to just find the most recent posterior in BETY, and use that as prior for PDA. pda.create.ensemble gets an ensemble ID for the PDA. All model runs associated with an individual PDA (any of the three methods) are considered part of a single ensemble. This function does is register a new ensemble in BETY, and return the ID that BETY gives it. pda.define.prior.fn creates R functions for all of the priors the PDA will use. pda.init.params sets up the parameter matrix for the run, which has one row per iteration, and one column per parameter. Columns include all Pecan parameters, not just the (probably small) subset that are being updated by PDA. This is for compatibility with other Pecan components. If starting a fresh run, the returned matrix is just a big empty matrix to fill in as the PDA runs. If continuing an existing MCMC, then it will be the previous params matrix, with a bunch of blank rows added on for filling in during this round of PDA. pda.init.run This is basically a big wrapper for Pecan’s write.config function (actually functions [plural], since every model in Pecan has its own version). For the bruteforce and bruteforce.bs methods this will be run once per iteration, whereas the emulator method knows about all its runs ahead of time and this will be a big batch of all runs at once. pda.adjust.jumps tweaks the jump distributions for the standard MCMC method, and pda.adjust.jumps.bs does the same for the block-sampled version. pda.calc.llik calculates the log-likelihood of the model given all datasets provided to compare it to. pda.generate.knots is for the emulator version of PDA. It uses a Latin hypercube design to sample a specified number of locations in parameter space. These locations are where the model will actually be run, and then the GP interpolates the likelihood surface in between. pda.plot.params provides basic MCMC diagnostics (trace and density) for parameters being sampled. pda.postprocess prepares the posteriors of the PDA, stores them to files and the database, and performs some other cleanup functions. pda.load.data.r This is the function that loads in data that will be used to constrain the PDA. It’s supposed to be eventually more integrated with Pecan, which will know how to load all kinds of data from all kinds of sources. For now, it can do NEE from Ameriflux. pda.define.llik.r A simple helper function that defines likelihood functions for different datasets. Probably in the future this should be queried from the database or something. For now, it is extremely limited. The original test case of NEE assimilation uses a heteroskedastic Laplacian distribution. pda.get.model.output.R Another function that will eventually grow to handle many more cases, or perhaps be replaced by a better system altogether. For now though, it again just handles Ameriflux NEE. 9.1.6 **get.da.data.*.R, plot.da.R** Old codes written by Carl Davidson. Defunct now, but may contain good ideas so currently left in. "],
["state-data-assimilation.html", "10 State Data Assimilation 10.1 sda.enkf.R is housed within: /pecan/modules/assim.sequential/R 10.2 The tree ring tutorial is housed within: /pecan/documentation/tutorials/StateAssimilation", " 10 State Data Assimilation 10.1 sda.enkf.R is housed within: /pecan/modules/assim.sequential/R 10.2 The tree ring tutorial is housed within: /pecan/documentation/tutorials/StateAssimilation 10.2.1 sda.enkf.R Description This is the main ensemble Kalman filter and generalized filter code. Originally, this was just ensemble Kalman filter code. Mike Dietze and Ann Raiho added a generalized ensemble filter to avoid filter divergence. The output of this function will be all the of run outputs, a PDF of diagnostics, and an Rdata object that includes three lists: FORECAST will be the ensemble forecasts for each year ANALYSIS will be the updated ensemble sample given the NPP observations enkf.params contains the prior and posterior mean vector and covariance matrix for each time step. 10.2.2 sda.enkf.R Arguments settings - (required) pecan.SDA.xml settings object obs.mean - (required) a list of observation means named with dates in YYYY/MM/DD format obs.cov - (required) a list of observation covariances names with dates in YYYY/MM/DD format IC - (optional) initial condition matrix (dimensions: ensemble memeber # by state variables). Default is NULL. Q - (optional) process covariance matrix (dimensions: state variable by state variables). Defualt is NULL. 10.2.3 State Data Assimilation Workflow Before running sda.enkf, these tasks must be completed (in no particular order), Read in a pecan.SDA.xml settings file with tags listed below. i.e. read.settings(‘pecan.SDA.xml’) Load data means (obs.mean) and covariances (obs.cov) as lists with PEcAn naming and unit conventions. Each observation must have a date in YYYY/MM/DD format (optional time) associated with it. If there are missing data, the date must still be represented in the list with an NA as the list object. Create initial conditions matrix (IC) that is state variables columns by ensemble members rows in dimension. sample.IC.MODEL can be used to create the IC matrix, but it is not required. This IC matrix is fed into write.configs for the initial model runs. The main parts of the SDA function are: Setting up for initial runs: Set parameters Load initial run inputs via split.inputs.MODEL Open database connection Get new workflow ids Create ensemble ids Performing the initial set of runs Set up for data assimilation Loop over time read.restart.MODEL - read model restart files corresponding to start.time and stop.time that you want to assimilate data into Analysis - There are four choices based on if process variance is TRUE or FALSE and if there is data or not. See explaination below. write.restart.MODEL - This function has two jobs. First, to insert adjusted state back into model restart file. Second, to update start.time, stop.time, and job.sh. run model Save outputs Create diagnostics 10.2.4 State Data Assimilation Tags Example <state.data.assimilation> <n.ensemble>25</n.ensemble> <process.variance>FALSE</process.variance> <sample.parameters>FALSE</sample.parameters> <state.variables> <variable> <variable.name>AGB.pft</variable.name> <unit>MgC/ha/yr</unit> <min_value>0</min_value> <max_value>100000000</max_value> </variable> <variable> <variable.name>TotSoilCarb</variable.name> <unit>KgC/m^2</unit> <min_value>0</min_value> <max_value>100000000</max_value> </variable> </state.variables> <spin.up> <start.date>1950/01/01</start.date> <end.date>1960/12/31</end.date> </spin.up> <forecast.time.step>1</forecast.time.step> <start.date>1961/01/01</start.date> <end.date>2010/12/31</end.date> </state.data.assimilation> 10.2.5 State Data Assimilation Tags Descriptions n.ensemble : [required] Number of ensemble members. Should be much larger than number of state variables you are assimilating. process.variance : [optional] TRUE/FLASE flag for if process variance should be estimated (TRUE) or not (FALSE). If TRUE, a generalized ensemble filter will be used. If FALSE, an ensemble Kalman filter will be used. Default is FALSE. sample.parameters : [optional] TRUE/FLASE flag for if parameters should be sampled for each ensemble member or not. This allows for more spread in the initial conditions of the forecast. state.variable : [required] State variable that is to be assimilated (in PEcAn standard format). spin.up : [required] start.date and end.date for initial model runs. NOTE: start.date and end.date are distinct from values set in the run tag because initial runs can be done over a subset of the full run. forecast.time.step : [optional] In the future, this will be used to allow the forecast time step to vary from the data time step. start.date : [optional] start date of the state data assimilation (in YYYY/MM/DD format) end.date : [optional] end date of the state data assimilation (in YYYY/MM/DD format) NOTE: start.date and end.date are distinct from values set in the run tag because this analysis can be done over a subset of the run. 10.2.6 Model Specific Functions for SDA Workflow 10.2.6.1 read.restart.MODEL.R The purpose of read.restart is to read model restart files and return a matrix that is site rows by state variable columns. The state variables must be in PEcAn names and units. The arguments are: outdir - output directory runid - ensemble member run ID stop.time - used to determine which restart file to read (in POSIX format) settings - pecan.SDA.xml settings object var.names - vector with state variable names with PEcAn standard naming. Example: c(‘AGB.pft’, ‘TotSoilCarb’) params - parameters used by ensemble member (same format as write.configs) 10.2.6.2 write.restart.MODEL.R This model specific function takes in new state and new parameter matrices from sda.enkf.R after the analysis step and translates new variables back to the model variables. Then, updates start.time, stop.time, and job.sh so that start.model.runs() does the correct runs with the new states. In write.restart.LINKAGES and write.restart.SIPNET, job.sh is updated by using write.configs.MODEL. outdir - output directory runid - run ID for ensemble member start.time - beginning of model run (in POSIX format) stop.time - end of model run (in POSIX format) settings - pecan.SDA.xml settings object new.state - matrix from analysis of updated state variables with PEcAn names (dimensions: site rows by state variables columns) new.params - In the future, this will allow us to update parameters based on states (same format as write.configs) inputs - model specific inputs from split.inputs.MODEL used to run the model from start.time to stop.time RENAME - [optional] Flag used in write.restart.LINKAGES.R for development. 10.2.6.3 split.inputs.MODEL.R This model specific function gives the correct met and/or other model inputs to settings\\(run\\)inputs. This function returns settings\\(run\\)inputs to an inputs argument in sda.enkf.R. But, the inputs will not need to change for all models and should return settings\\(run\\)inputs unchanged if that is the case. settings - pecan.SDA.xml settings object start.time - start time for model run (in POSIX format) stop.time - stop time for model run (in POSIX format) 10.2.6.4 sample.IC.MODEL.R This model specific function is optional. But, it can be used to create initial condition matrix (IC) with # state variables columns by # ensemble rows. This IC matrix is used for the initial runs in sda.enkf.R in the write.configs.MODEL function. ne - number of ensemble members state - matrix of state variables to get initial conditions from year - used to determine which year to sample initial conditions from 10.2.7 Analysis Options There are four options depending on whether process variance is TRUE/FALSE and whether or not there is data or not. If there is no data and process variance = FALSE, there is no analysis step. If there is no data and process variance = TRUE, process variance is added to the forecast. If there is data and process variance = TRUE, the generalized ensemble filter is implemented with MCMC. If there is data and process variance = FALSE, the Kalman filter is used and solved analytically. 10.2.8 The Generalized Ensemble Filter An ensemble filter is a sequential data assimilation algorithm with two procedures at every time step: a forecast followed by an analysis. The forecast ensembles arise from a model while the analysis makes an adjustment of the forecasts ensembles from the model towards the data. An ensemble Kalman filter is typically suggested for this type of analysis because of its computationally efficient analytical solution and its ability to update states based on an estimate of covariance structure. But, in some cases, the ensemble Kalman filter fails because of filter divergence. Filter divergence occurs when forecast variability is too small, which causes the analysis to favor the forecast and diverge from the data. Models often produce low forecast variability because there is little internal stochasticity. Our ensemble filter overcomes this problem in a Bayesian framework by including an estimation of model process variance. This methodology also maintains the benefits of the ensemble Kalman filter by updating the state vector based on the estimated covariance structure. This process begins after the model is spun up to equilibrium. The likelihood function uses the data vector \\(\\left(\\boldsymbol{y_{t}}\\right)\\) conditional on the estimated state vector \\(\\left(\\boldsymbol{x_{t}}\\right)\\) such that \\(\\boldsymbol{y}_{t}\\sim\\mathrm{multivariate\\:normal}(\\boldsymbol{x}_{t},\\boldsymbol{R}_{t})\\) where \\(\\boldsymbol{R}_{t}=\\boldsymbol{\\sigma}_{t}^{2}\\boldsymbol{I}\\) and \\(\\boldsymbol{\\sigma}_{t}^{2}\\) is a vector of data variances. To obtain an estimate of the state vector \\(\\left(\\boldsymbol{x}_{t}\\right)\\), we use a process model that incorporates a process covariance matrix \\(\\left(\\boldsymbol{Q}_{t}\\right)\\). This process covariance matrix differentiates our methods from past ensemble filters. Our process model contains the following equations \\(\\boldsymbol{x}_{t} \\sim \\mathrm{multivariate\\: normal}(\\boldsymbol{x}_{model_{t}},\\boldsymbol{Q}_{t})\\) \\(\\boldsymbol{x}_{model_{t}} \\sim \\mathrm{multivariate\\: normal}(\\boldsymbol{\\mu}_{forecast_{t}},\\boldsymbol{P}_{forecast_{t}})\\) where \\(\\boldsymbol{\\mu}_{forecast_{t}}\\) is a vector of means from the ensemble forecasts and \\(\\boldsymbol{P}_{forecast_{t}}\\) is a covariance matrix calculated from the ensemble forecasts. The prior for our process covariance matrix is \\(\\boldsymbol{Q}_{t}\\sim\\mathrm{Wishart}(\\boldsymbol{V}_{t},n_{t})\\) where \\(\\boldsymbol{V}_{t}\\) is a scale matrix and \\(n_{t}\\) is the degrees of freedom. The prior shape parameters are updated at each time step through moment matching such that \\(\\boldsymbol{V}_{t+1} = n_{t}\\bar{\\boldsymbol{Q}}_{t}\\) \\(n_{t+1} = \\frac{\\sum_{i=1}^{I}\\sum_{j=1}^{J}\\frac{v_{ijt}^{2}+v_{iit}v_{jjt}}{Var(\\boldsymbol{\\bar{Q}}_{t})}}{I\\times J}\\) where we calculate the mean of the process covariance matrix \\(\\left(\\bar{\\boldsymbol{Q}_{t}}\\right)\\) from the posterior samples at time t. Degrees of freedom for the Wishart are typically calculated element by element where \\(v_{ij}\\) are the elements of \\(\\boldsymbol{V}_{t}\\). \\(I\\) and \\(J\\) index rows and columns of \\(\\boldsymbol{V}\\). Here, we calculate a mean number of degrees of freedom for \\(t+1\\) by summing over all the elements of the scale matrix \\(\\left(\\boldsymbol{V}\\right)\\) and dividing by the count of those elements \\(\\left(I\\times J\\right)\\). We fit this model sequentially through time in the R computing environment using R package ‘rjags.’ "],
["specific-model-configurations.html", "11 Specific Model Configurations", " 11 Specific Model Configurations "],
["advanced-setup.html", "12 Advanced Setup", " 12 Advanced Setup ** Under construction ** "],
["advanced-command-line-guide.html", "13 Advanced Command Line Guide", " 13 Advanced Command Line Guide For those wanting to take full advantage of PEcAn and its tools, this section will provide you with the necessary information to do so. In this section we hope to answer the question: How can I use PEcAn to its fullest? Replicating a basic run without touching the web interface Working with the VM Replicating ensemble and sensitivity analysis Extra Tools within PEcAn "],
["run-pecan-from-command-line.html", "14 Run PEcAn from COmmand Line", " 14 Run PEcAn from COmmand Line "],
["working-with-the-vm.html", "15 Working with the VM 15.1 Connecting to the VM via SSH", " 15 Working with the VM 15.1 Connecting to the VM via SSH Once the VM is running anywhere on your machine, you can connect to it from a separate terminal via SSH as follows: ssh -p 6422 carya@localhost You will be prompted for a password. Like everywhere else in PEcAn, the username is carya and the password is illinois. The same password is used for any system maintenance you wish to do on the VM via sudo. As a shortcut, you can add the following to your ~/.ssh/config file (or create one if it does not exist). Host pecan-vm Hostname localhost Port 6422 user carya ForwardX11Trusted yes This will allow you to SSH into the VM with the simplified command, ssh pecan-vm. "],
["extra-tools-in-pecan.html", "16 Extra Tools In PEcAn", " 16 Extra Tools In PEcAn "],
["user-appendix.html", "17 User Appendix", " 17 User Appendix "],
["updating-pecan-code-and-bety-database.html", "18 Updating PEcAn Code and Bety Database 18.1 Updating BETY", " 18 Updating PEcAn Code and Bety Database Release notes for all releases can be found here. This page will only list any steps you have to do to upgrade an existing system. When updating PEcAn it is highly encouraged to update BETY. You can find instructions on how to do this, as well on how to update the database in the Updating BETYdb gitbook page. 18.0.1 Updating PEcAn The latest version of PEcAn code can be obtained from the PEcAn repository on GitHub: cd pecan # If you are not already in the PEcAn directory git pull The PEcAn build system is based on GNU Make. The simplest way to install is to run make from inside the PEcAn directory. This will update the documentation for all packages and install them, as well as all required dependencies. For more control, the following make commands are available: make document – Use devtools::document to update the documentation for all package. Under the hood, this uses the roxygen2 documentation system. make install – Install all packages and their dependnencies using devtools::install. By default, this only installs packages that have had their code changed and any dependent packages. make check – Perform a rigorous check of packages using devtools::check make test – Run all unit tests (based on testthat package) for all packages, using devtools::test make clean – Remove the make build cache, which is used to track which packages have changed. Cache files are stored in the .doc, .install, .check, and .test subdirectories in the PEcAn main directory. Running make clean will force the next invocation of make commands to operate on all PEcAn packages, regardless of changes. The following are some additional make tricks that may be useful: Install, check, document, or test a specific package – make .<cmd>/<pkg-dir>; e.g. make .install/utils or make .check/modules/rtm Force make to run, even if package has not changed – make -B <command> Run make commands in parallel – make -j<ncores>; e.g. make -j4 install to install packages using four parallel processes. All instructions for the make build system are contained in the Makefile in the PEcAn root directory. For full documentation on make, see the man pages by running man make from a terminal. Point of contact: Alexey Shiklomanov GitHub/Gitter: @ashiklom email: ashiklom@bu.edu 18.1 Updating BETY Moved to: https://pecan.gitbooks.io/betydb-documentation/content/updating_betydb_when_new_versions_are_released.html "],
["pecan-models.html", "19 PEcAn Models 19.1 BioCro 19.2 BioCro Configuration 19.3 CLM 19.4 DALEC 19.5 ED 19.6 ED2IN configuration variables 19.7 GDAY 19.8 LINKAGES 19.9 LPJ-GUESS 19.10 MAESPA 19.11 PRELES 19.12 Sipnet", " 19 PEcAn Models This section will contain information about all models and output variables that are supported by PEcAn. Model Name Available in the VM Prescribed Inputs Input Functions/Values Restart Function BioCro Yes Yes Yes No CLM No No No No DALEC Yes Yes Yes No ED Yes Yes Yes No FATES No Yes No GDAY No No No No LINKAGES Yes Yes Yes Yes LPJ-GUESS Yes Yes No No MAESPA Yes Yes No No PRELES Yes Yes Partially No Sipnet Yes Yes Yes No Output Variables PEcAn converts all model outputs to a single standard. This standard evolved out of MsTMIP project, which is itself based on NACP, LBA, and other model-intercomparison projects. This standard was expanded for the PalEON MIP and the needs of the PEcAn modeling community to support variables not in these standards. Model developers: do not add variables to your PEcAn output without first adding them to the PEcAn standard table! Also, do not create new variables equivalent to existing variables but just with different names or units. 19.1 BioCro Model Information Home Page Source Code License Authors PEcAn Integration Introduction Introduction about model PEcAn configuration file additions Should list the model specific additions to the PEcAn file here Model specific input files List of inputs required by model, such as met, etc. Model configuration files BioCro uses a config.xml file similar to ED2. At this time, no other template files are required. Installation notes This section contains notes on how to compile the model. The notes for the VM might work on other machines or configurations as well. VM 19.2 BioCro Configuration BioCro uses a config.xml file similar to ED2. At this time, no other template files are required. 19.3 CLM Model Information Home Page Source Code License Authors PEcAn Integration Introduction Introduction about model PEcAn configuration file additions Should list the model specific additions to the PEcAn file here Model specific input files List of inputs required by model, such as met, etc. Model configuration files MODEL is configured using 3 files which are placed in the run folder, as well as a symbolic link to the met file. file1 : template for this file is located at models/MODEL/inst/file1 and is not modified. file2 : template for this file is located at models/MODEL/inst/file2 and is not modified. file3 : template for this file is in models/MODEL/inst/file3 or it is specified in the <model> section as <template>. The values in this template are replaced by those computed in the earlier stages of PEcAN. Installation notes This section contains notes on how to compile the model. The notes for the VM might work on other machines or configurations as well. VM 19.4 DALEC Model Information Home Page Source Code License Authors PEcAn Integration Introduction Introduction about model PEcAn configuration file additions Should list the model specific additions to the PEcAn file here Model specific input files List of inputs required by model, such as met, etc. Model configuration files MODEL is configured using 3 files which are placed in the run folder, as well as a symbolic link to the met file. file1 : template for this file is located at models/MODEL/inst/file1 and is not modified. file2 : template for this file is located at models/MODEL/inst/file2 and is not modified. file3 : template for this file is in models/MODEL/inst/file3 or it is specified in the <model> section as <template>. The values in this template are replaced by those computed in the earlier stages of PEcAN. Installation notes This section contains notes on how to compile the model. The notes for the VM might work on other machines or configurations as well. VM 19.5 ED Model Information Home Page http://moorcroftlab.oeb.harvard.edu/ Source Code https://github.com/EDmodel/ED2 License Authors Paul Moorcroft, … PEcAn Integration Michael Dietze, Rob Kooper Introduction Introduction about ED model PEcAn configuration file additions Should list the model specific additions to the PEcAn file here Model specific input files List of inputs required by model, such as met, etc. Model configuration files ED2 is configured using 2 files which are placed in the run folder. ED2IN : template for this file is located at models/ed/inst/ED2IN.<revision>. The values in this template that need to be modified are described below and are surrounded with @ symbols. config.xml : this file is generated by PEcAn. Some values are stored in the pecan.xml in <pfts><pft><constants> section as well as in <model> section. An example of the template can be found in ED2IN.r82 The ED2IN template can contain the following variables. These will be replaced with actual values when the model configuration is written. **@ENSNAME@** : run id of the simulation, used in template for NL%EXPNME **@START_MONTH@** : start of simulation UTC time, from <run><start.date>, used in template for NL%IMONTHA **@START_DAY@** : start of simulation UTC time, from <run><start.date>, used in template for NL%IDATEA **@START_YEAR@** : start of simulation UTC time, from <run><start.date>, used in template for NL%IYEARA **@END_MONTH@** : end of simulation UTC time, from <run><end.date>, used in template for NL%IMONTHZ **@END_DAY@** : end of simulation UTC time, from <run><end.date>, used in template for NL%IDATEZ **@END_YEAR@** : end of simulation UTC time, from <run><end.date>, used in template for NL%IYEARZ **@SITE_LAT@** : site latitude location, from <run><site><lat>, used in template for NL%POI_LAT **@SITE_LON@** : site longitude location, from <run><site><lon>, used in template for NL%POI_LON **@SITE_MET@** : met header location, from <run><site><met>, used in template for NL%ED_MET_DRIVER_DB **@MET_START@** : first year of met data, from <run><site><met.start>, used in template for NL%METCYC1 **@MET_END@** : last year of met data, from <run><site><met.end>, used in template for NL%METCYCF **@PHENOL_SCHEME@** : phenology scheme, if this variabe is 1 the following 3 fields will be used, otherwise they will be set to empty strings, from <model><phenol.scheme>, used in template for NL%IPHEN_SCHEME **@PHENOL_START@** : first year for phenology, from <model><phenol.start>, used in template for NL%IPHENYS1 and NL%IPHENYF1 **@PHENOL_END@** : last year for phenology, from <model><phenol.end>, used in template for NL%IPHENYSF and NL%IPHENYFF **@PHENOL@** : path and prefix of the prescribed phenology data, from * <phenol>, used in template for NL%PHENPATH **@SITE_PSSCSS@** : path and prefix of the previous ecosystem state, from <model><psscss>, used in template for NL%SFILIN **@ED_VEG@** : path and prefix of the vegetation database, used only to determine the land/water mask, from <model><veg>, used in template for NL%VEG_DATABASE **@ED_SOIL@** : path and prefix of the soil database, used to determine the soil type, from <model><soil>, used in template for NL%SOIL_DATABASE **@ED_INPUTS@** : input directory with dataset to initialise chilling degrees and growing degree days, which is used to drive the cold-deciduous phenology, from <model><inputs>, used in template for NL%THSUMS_DATABASE **@FFILOUT@** : path and prefix for analysis files, generated from <run><host><outdir>/run.id/analysis, used in template for NL%FFILOUT **@SFILOUT@** : path and prefix for history files, generated from <run><host><outdir>/run.id/history, used in template for NL%SFILOUT **@CONFIGFILE@** : XML file containing additional parameter settings, this is always “config.xml”, used in template for NL%IEDCNFGF **@OUTDIR@** : location where output files are written (without the runid), from <run><host><outdir>, should not be used. **@SCRATCH@** : local scratch space for outputs, generated /scratch/<username>/run$scratch, should not be used right now since it only works on ebi-cluster Installation notes This section contains notes on how to compile the model. The notes for the VM might work on other machines or configurations as well. VM ** BU geo** TACC lonestar module load hdf5 curl -o ED.r82.tgz http://isda.ncsa.illinois.edu/~kooper/EBI/ED.r82.tgz tar zxf ED.r82.tgz rm ED.r82.tgz cd ED.r82/ED/build/bin curl -o include.mk.lonestar http://isda.ncsa.illinois.edu/~kooper/EBI/include.mk.lonestar make OPT=lonestar TACC stampede module load hdf5 curl -o ED.r82.tgz http://isda.ncsa.illinois.edu/~kooper/EBI/ED.r82.tgz tar zxf ED.r82.tgz rm ED.r82.tgz cd ED.r82/ED/build/bin curl -o include.mk.stampede http://isda.ncsa.illinois.edu/~kooper/EBI/include.mk.stampede make OPT=stampede 19.5.1 ED Configuration ED2 is configured using 2 files which are placed in the run folder. ED2IN : template for this file is located at models/ed/inst/ED2IN.<revision>. The values in this template that need to be modified are described below and are surrounded with @ symbols. config.xml : this file is generated by PEcAn. Some values are stored in the pecan.xml in <pfts><pft><constants> section as well as in <model> section. An example of the template can be found in ED2IN.r82 19.6 ED2IN configuration variables **@ENSNAME@** : run id of the simulation, used in template for NL%EXPNME **@START_MONTH@** : start of simulation UTC time, from <run><start.date>, used in template for NL%IMONTHA **@START_DAY@** : start of simulation UTC time, from <run><start.date>, used in template for NL%IDATEA **@START_YEAR@** : start of simulation UTC time, from <run><start.date>, used in template for NL%IYEARA **@END_MONTH@** : end of simulation UTC time, from <run><end.date>, used in template for NL%IMONTHZ **@END_DAY@** : end of simulation UTC time, from <run><end.date>, used in template for NL%IDATEZ **@END_YEAR@** : end of simulation UTC time, from <run><end.date>, used in template for NL%IYEARZ **@SITE_LAT@** : site latitude location, from <run><site><lat>, used in template for NL%POI_LAT **@SITE_LON@** : site longitude location, from <run><site><lon>, used in template for NL%POI_LON **@SITE_MET@** : met header location, from <run><site><met>, used in template for NL%ED_MET_DRIVER_DB **@MET_START@** : first year of met data, from <run><site><met.start>, used in template for NL%METCYC1 **@MET_END@** : last year of met data, from <run><site><met.end>, used in template for NL%METCYCF **@PHENOL_SCHEME@** : phenology scheme, if this variabe is 1 the following 3 fields will be used, otherwise they will be set to empty strings, from <model><phenol.scheme>, used in template for NL%IPHEN_SCHEME **@PHENOL_START@** : first year for phenology, from <model><phenol.start>, used in template for NL%IPHENYS1 and NL%IPHENYF1 **@PHENOL_END@** : last year for phenology, from <model><phenol.end>, used in template for NL%IPHENYSF and NL%IPHENYFF **@PHENOL@** : path and prefix of the prescribed phenology data, from <model><phenol>, used in template for NL%PHENPATH **@SITE_PSSCSS@** : path and prefix of the previous ecosystem state, from <model><psscss>, used in template for NL%SFILIN **@ED_VEG@** : path and prefix of the vegetation database, used only to determine the land/water mask, from <model><veg>, used in template for NL%VEG_DATABASE **@ED_SOIL@** : path and prefix of the soil database, used to determine the soil type, from <model><soil>, used in template for NL%SOIL_DATABASE **@ED_INPUTS@** : input directory with dataset to initialise chilling degrees and growing degree days, which is used to drive the cold-deciduous phenology, from <model><inputs>, used in template for NL%THSUMS_DATABASE **@FFILOUT@** : path and prefix for analysis files, generated from <run><host><outdir>/run.id/analysis, used in template for NL%FFILOUT **@SFILOUT@** : path and prefix for history files, generated from <run><host><outdir>/run.id/history, used in template for NL%SFILOUT **@CONFIGFILE@** : XML file containing additional parameter settings, this is always “config.xml”, used in template for NL%IEDCNFGF **@OUTDIR@** : location where output files are written (without the runid), from <run><host><outdir>, should not be used. **@SCRATCH@** : local scratch space for outputs, generated /scratch/<username>/run$scratch, should not be used right now since it only works on ebi-cluster ED Computation HPC TACC lonestar module load hdf5 curl -o ED.r82.tgz http://isda.ncsa.illinois.edu/~kooper/EBI/ED.r82.tgz tar zxf ED.r82.tgz rm ED.r82.tgz cd ED.r82/ED/build/bin curl -o include.mk.lonestar http://isda.ncsa.illinois.edu/~kooper/EBI/include.mk.lonestar make OPT=lonestar TACC stampede module load hdf5 curl -o ED.r82.tgz http://isda.ncsa.illinois.edu/~kooper/EBI/ED.r82.tgz tar zxf ED.r82.tgz rm ED.r82.tgz cd ED.r82/ED/build/bin curl -o include.mk.stampede http://isda.ncsa.illinois.edu/~kooper/EBI/include.mk.stampede make OPT=stampede 19.7 GDAY Model Information Home Page Source Code License Authors PEcAn Integration Introduction Introduction about model PEcAn configuration file additions Should list the model specific additions to the PEcAn file here Model specific input files List of inputs required by model, such as met, etc. Model configuration files MODEL is configured using 3 files which are placed in the run folder, as well as a symbolic link to the met file. file1 : template for this file is located at models/MODEL/inst/file1 and is not modified. file2 : template for this file is located at models/MODEL/inst/file2 and is not modified. file3 : template for this file is in models/MODEL/inst/file3 or it is specified in the <model> section as <template>. The values in this template are replaced by those computed in the earlier stages of PEcAN. Installation notes This section contains notes on how to compile the model. The notes for the VM might work on other machines or configurations as well. VM 19.8 LINKAGES Model Information Home Page Source Code License Authors PEcAn Integration Introduction Introduction about model PEcAn configuration file additions Should list the model specific additions to the PEcAn file here Model specific input files List of inputs required by model, such as met, etc. Model configuration files MODEL is configured using 3 files which are placed in the run folder, as well as a symbolic link to the met file. file1 : template for this file is located at models/MODEL/inst/file1 and is not modified. file2 : template for this file is located at models/MODEL/inst/file2 and is not modified. file3 : template for this file is in models/MODEL/inst/file3 or it is specified in the <model> section as <template>. The values in this template are replaced by those computed in the earlier stages of PEcAN. Installation notes This section contains notes on how to compile the model. The notes for the VM might work on other machines or configurations as well. VM 19.9 LPJ-GUESS Model Information Home Page Source Code License Authors PEcAn Integration Introduction Introduction about model PEcAn configuration file additions Should list the model specific additions to the PEcAn file here Model specific input files List of inputs required by model, such as met, etc. Model configuration files MODEL is configured using 3 files which are placed in the run folder, as well as a symbolic link to the met file. file1 : template for this file is located at models/MODEL/inst/file1 and is not modified. file2 : template for this file is located at models/MODEL/inst/file2 and is not modified. file3 : template for this file is in models/MODEL/inst/file3 or it is specified in the <model> section as <template>. The values in this template are replaced by those computed in the earlier stages of PEcAN. Installation notes This section contains notes on how to compile the model. The notes for the VM might work on other machines or configurations as well. VM 19.10 MAESPA Model Information Home Page http://maespa.github.io/ Source Code http://maespa.github.io/download.html License Authors Belinda Medlyn and Remko Duursma PEcAn Integration Tony Gardella, Martim DeKauwe, Remki Duursma Introduction PEcAn configuration file additions Model specific input files Model configuration files MODEL is configured using 3 files which are placed in the run folder, as well as a symbolic link to the met file. file1 : template for this file is located at models/MODEL/inst/file1 and is not modified. file2 : template for this file is located at models/MODEL/inst/file2 and is not modified. file3 : template for this file is in models/MODEL/inst/file3 or it is specified in the <model> section as <template>. The values in this template are replaced by those computed in the earlier stages of PEcAN. Installation notes Installing the MAESPA model requires cloning the MAESPA Bitbucket Repository, executing the makefile, and ensuring that the Maeswarp R package is correctly installed. To clone and compile the model, execute this code at the command line git clone https://bitbucket.org/remkoduursma/maespa.git cd maespa make clean make maespa.out is your executable. Example input files can be found in the inputfiles directory. Executing measpa.out from within one of the example directories will produce output. MAESPA developers have also developed a wrapper package called Maeswrap. The usual R package installation method install.packages may present issues with downloading an unpacking a dependency package called rgl. Here are a couple of solutions: Solution 1 From the Command Line sudo apt-get install r-cran-rgl then from within R install.packages("Maeswrap") Solution 2 From the Command line sudo apt-get install libglu1-mesa-dev then from within R install.packages("Maeswrap") This section contains notes on how to compile the model. The notes for the VM might work on other machines or configurations as well. VM 19.11 PRELES Model Information Home Page Source Code License Authors PEcAn Integration Introduction Introduction about model PEcAn configuration file additions Should list the model specific additions to the PEcAn file here Model specific input files List of inputs required by model, such as met, etc. Model configuration files MODEL is configured using 3 files which are placed in the run folder, as well as a symbolic link to the met file. file1 : template for this file is located at models/MODEL/inst/file1 and is not modified. file2 : template for this file is located at models/MODEL/inst/file2 and is not modified. file3 : template for this file is in models/MODEL/inst/file3 or it is specified in the <model> section as <template>. The values in this template are replaced by those computed in the earlier stages of PEcAN. Installation notes This section contains notes on how to compile the model. The notes for the VM might work on other machines or configurations as well. VM 19.12 Sipnet Model Information Home Page Source Code License Authors PEcAn Integration Michael Dietze, Rob Kooper Introduction Introduction about model PEcAn configuration file additions Should list the model specific additions to the PEcAn file here Model specific input files List of inputs required by model, such as met, etc. Model configuration files SIPNET is configured using 3 files which are placed in the run folder, as well as a symbolic link to the met file. sipnet.in : template for this file is located at models/sipnet/inst/sipnet.in and is not modified. sipnet.param-spatial : template for this file is located at models/sipnet/inst/template.param-spatial and is not modified. sipnet.param : template for this file is in models/sipnet/inst/template.param or it is specified in the <model> section as <default.param>. The values in this template are replaced by those computed in the earlier stages of PEcAN. Installation notes This section contains notes on how to compile the model. The notes for the VM might work on other machines or configurations as well. SIPNET version unk: if [ ! -e ${HOME}/sipnet_unk ]; then cd curl -o sipnet_unk.tar.gz http://isda.ncsa.illinois.edu/~kooper/PEcAn/models/sipnet_unk.tar.gz tar zxf sipnet_unk.tar.gz rm sipnet_unk.tar.gz fi cd ${HOME}/sipnet_unk/ make clean make sudo cp sipnet /usr/local/bin/sipnet.runk make clean SIPNET version 136: if [ ! -e ${HOME}/sipnet_r136 ]; then cd curl -o sipnet_r136.tar.gz http://isda.ncsa.illinois.edu/~kooper/EBI/sipnet_r136.tar.gz tar zxf sipnet_r136.tar.gz rm sipnet_r136.tar.gz sed -i 's#$(LD) $(LIBLINKS) \\(.*\\)#$(LD) \\1 $(LIBLINKS)#' ${HOME}/sipnet_r136/Makefile fi cd ${HOME}/sipnet_r136/ make clean make sudo cp sipnet /usr/local/bin/sipnet.r136 make clean VM SIPNET is configured using 3 files which are placed in the run folder, as well as a symbolic link to the met file. sipnet.in : template for this file is located at models/sipnet/inst/sipnet.in and is not modified. sipnet.param-spatial : template for this file is located at models/sipnet/inst/template.param-spatial and is not modified. sipnet.param : template for this file is in models/sipnet/inst/template.param or it is specified in the <model> section as <default.param>. The values in this template are replaced by those computed in the earlier stages of PEcAN. "],
["model-output-variables.html", "20 Model Output Variables", " 20 Model Output Variables Note: initial table derived from MsTMIP Num Group order Saveit Variable Name Units Long name Priority Category 3-hourly Monthly var_type ndim dim1 dim2 dim3 dim4 Description 1 1 1 Yes lon degrees_east Longitude 0 Grid Yes Yes real 1 lon na na na longitude at center of each grid cell 2 1 2 Yes lat degrees_north Latitude 0 Grid Yes Yes real 1 lat na na na latitude at center of each grid cell 3 1 3 Yes lon_bnds degrees_east Longitude west-east bounds 0 Grid Yes Yes real 2 nbnds lon na na (west boundary of grid cell, east boundary of grid cell) 4 1 4 Yes lat_bnds degrees_north Latitude south-north bounds 0 Grid Yes Yes real 2 nbnds lat na na (south boundary of grid cell, north boundary of grid cell) 5 2 1 Yes time days since 1700-01-01 00:00:00 UTC Time middle averaging period 0 Time Yes Yes double 1 time na na na julian days days since 1700-01-01 00:00:00 UTC for middle of time averaging period Proleptic_Gregorianc calendar 6 2 2 Yes time_bnds days since 1700-01-01 00:00:00 UTC Time beginning-end bounds 0 Time Yes Yes double 2 nbnds time na na (julian days days since 1700-01-01 beginning time ave period, julian days days since 1700-01-01 end time ave period) 7 2 3 Yes dec_date yr Decimal date middle averaging period 0 Time Yes Yes double 1 time na na na decimal date in fractional years for middle of time averaging period 8 2 4 Yes dec_date_bnds yr Decimal date beginning-end bounds 0 Time Yes Yes double 2 nbnds time na na (decimal date beginningĘĘtime ave period, decimal date end time ave period) 9 2 5 Yes cal_date_mid yr, mon, day, hr, min, sec Calender date middle averaging period 0 Time Yes Yes integer 2 ncal time na na calender date middle of time ave period: year, month, day, hour, minute, second for UTC time zone 10 2 6 Yes cal_date_beg yr, mon, day, hr, min, sec Calender date beginning averaging period 0 Time Yes Yes integer 2 ncal time na na calender date beginning of time ave period: year, month, day, hour, minute, second for UTC time zone 11 2 7 Yes cal_date_end yr, mon, day, hr, min, sec Calender date end averaging period 0 Time Yes Yes integer 2 ncal time na na calender date end of time ave period: year, month, day, hour, minute, second for UTC time zone 12 3 1 Yes GPP kg C m-2 s-1 Gross Primary Productivity 1 Carbon Fluxes Yes Yes real 3 lon lat time na Rate of photosynthesis (always positive) 13 3 2 Yes NPP kg C m-2 s-1 Net Primary Productivity 1 Carbon Fluxes Yes Yes real 3 lon lat time na Net Primary Productivity (NPP=GPP-AutoResp, positive into plants) 14 3 3 Yes TotalResp kg C m-2 s-1 Total Respiration 1 Carbon Fluxes Yes Yes real 3 lon lat time na Total respiration (TotalResp=AutoResp+heteroResp, always positive) 15 3 4 Yes AutoResp kg C m-2 s-1 Autotrophic Respiration 1 Carbon Fluxes Yes Yes real 3 lon lat time na Autotrophic respiration rate (always positive) 16 3 5 Yes HeteroResp kg C m-2 s-1 Heterotrophic Respiration 1 Carbon Fluxes Yes Yes real 3 lon lat time na Heterotrophic respiration rate (always positive) 17 3 6 Yes DOC_flux kg C m-2 s-1 Dissolved Organic Carbon flux 1 Carbon Fluxes Yes Yes real 3 lon lat time na Loss of organic carbon dissolved in ground water or rivers (positive out of grid cell) 18 3 7 Yes Fire_flux kg C m-2 s-1 Fire emissions 1 Carbon Fluxes Yes Yes real 3 lon lat time na Flux of carbon due to fires (always positive) 19 3 8 Yes NEE kg C m-2 s-1 Net Ecosystem Exchange 1 Carbon Fluxes Yes Yes real 3 lon lat time na Net Ecosystem Exchange (NEE=HeteroResp+AutoResp-GPP, positive into atmosphere) 21 4 2 Yes poolname (-) Name of each Carbon Pool 1 Carbon Pools No Yes character 2 nchar npool na na Name of each carbon pool (i.e., “wood | or”“Coarse Woody Debris”“)” 22 4 3 Yes CarbPools kg C m-2 Size of each carbon pool 1 Carbon Pools No Yes real 4 lon lat npool time Total size of each carbon pool vertically integrated over the entire soil column 23 4 4 Yes AbvGrndWood kg C m-2 Above ground woody biomass 1 Carbon Pools No Yes real 3 lon lat time na Total above ground wood biomass 24 4 5 Yes TotLivBiom kg C m-2 Total living biomass 1 Carbon Pools No Yes real 3 lon lat time na Total carbon content of the living biomass (leaves+roots+wood) 25 4 6 Yes TotSoilCarb kg C m-2 Total Soil Carbon 1 Carbon Pools No Yes real 3 lon lat time na Total soil and litter carbon content vertically integrated over the enire soil column 26 4 7 Yes LAI m2 m-2 Leaf Area Index 1 Carbon Pools No Yes real 3 lon lat time na Area of leaves per area ground 27 5 1 Yes Qh W m-2 Sensible heat 1 Energy Fluxes Yes Yes real 3 lon lat time na Sensible heat flux into the boundary layer (positive into atmosphere) 28 5 2 Yes Qle W m-2 Latent heat 1 Energy Fluxes Yes Yes real 3 lon lat time na Latent heat flux into the boundary layer (positive into atmosphere) 29 5 3 Yes Evap kg m-2 s-1 Total Evaporation 1 Energy Fluxes No Yes real 3 lon lat time na Sum of all evaporation sources (positive into atmosphere) 30 5 4 Yes TVeg kg m-2 s-1 Transpiration 1 Energy Fluxes No Yes real 3 lon lat time na Total Plant transpiration (always positive) 31 5 5 Yes LW_albedo (-) Longwave Albedo 1 Energy Fluxes No Yes real 3 lon lat time na Longwave Albedo 32 5 6 Yes SW_albedo (-) Shortwave Albedo 1 Energy Fluxes No Yes real 3 lon lat time na Shortwave albedo 33 5 7 Yes Lwnet W m-2 Net Longwave Radiation 1 Energy Fluxes No Yes real 3 lon lat time na Incident longwave radiation minus simulated outgoing longwave radiation (positive into grnd) 34 5 8 Yes SWnet W m-2 Net shortwave radiation 1 Energy Fluxes No Yes real 3 lon lat time na Incident shortwave radiation minus simulated outgoing shortwave radiation (positive into grnd) 35 5 9 Yes fPAR (-) Absorbed fraction incoming PAR 1 Energy Fluxes No Yes real 3 lon lat time na absorbed fraction incoming photosyntetically active radiation 37 6 2 Yes z_top m Soil Layer Top Depth 1 Physical Variables No Yes real 1 nsoil na na na Depth from soil surface to top of soil layer 38 6 3 Yes z_node m Soil Layer Node Depth 1 Physical Variables No Yes real 1 nsoil na na na Depth from soil surface to layer prognostic variables; typically center of soil layer 39 6 4 Yes z_bottom m Soil Layer Bottom Depth 1 Physical Variables No Yes real 1 nsoil na na na Depth from soil surface to bottom of soil layer 40 6 5 Yes SoilMoist kg m-2 Average Layer Soil Moisture 1 Physical Variables No Yes real 4 lon lat nsoil time Soil water content in each soil layer, including liquid, vapor and ice 41 6 5 Yes SoilMoistFrac (-) Average Layer Fraction of Saturation 1 Physical Variables No Yes real 4 lon lat nsoil time Fraction of saturation of soil water in each soil layer, including liquid and ice 42 6 6 Yes SoilWet (-) Total Soil Wetness 1 Physical Variables No Yes real 3 lon lat time na Vertically integrated soil moisture divided by maximum allowable soil moisture above wilting point 43 6 7 Yes Qs kg m-2 s-1 Surface runoff 1 Physical Variables No Yes real 3 lon lat time na Runoff from the landsurface and/or subsurface stormflow 44 6 8 Yes Qsb kg m-2 s-1 Subsurface runoff 1 Physical Variables No Yes real 3 lon lat time na Gravity soil water drainage and/or soil water lateral flow 45 6 9 Yes SoilTemp K Average Layer Soil Temperature 1 Physical Variables No Yes real 4 lon lat nsoil time Average soil temperature in each soil layer 46 6 10 Yes Tdepth m Active Layer Thickness 1 Physical Variables No Yes real 3 lon lat time na Thaw depth; depth to zero centigrade isotherm in permafrost 47 6 11 Yes Fdepth m Frozen Layer Thickness 1 Physical Variables No Yes real 3 lon lat time na Freeze depth; depth to zero centigrade isotherm in non-permafrost 48 6 12 Yes Tcan K Canopy Temperature 1 Physical Variables No Yes real 3 lon lat time na Canopy or vegetation temperature (or temperature used in photosynthesis calculations) 49 6 13 Yes SWE kg m-2 Snow Water Equivalent 1 Physical Variables No Yes real 3 lon lat time na Total water mass of snow pack, including ice and liquid water 50 6 14 Yes SnowDen kg m-3 Bulk Snow Density 1 Physical Variables No Yes real 3 lon lat time na Overall bulk density of the snow pack, including ice and liquid water 51 6 15 Yes SnowDepth m Total snow depth 1 Physical Variables No Yes real 3 lon lat time na Total snow depth 52 7 1 Yes CO2air micromol mol-1 Near surface CO2 concentration 1 Driver No Yes real 3 lon lat time na Near surface dry air CO2 mole fraction 53 7 2 Yes LWdown W/m2 Surface incident longwave radiation 1 Driver No Yes real 3 lon lat time na Surface incident longwave radiation 54 7 3 Yes Psurf Pa Surface pressure 1 Driver No Yes real 3 lon lat time na Surface pressure 55 7 4 Yes Qair kg kg-1 Near surface specific humidity 1 Driver No Yes real 3 lon lat time na Near surface specific humidity 56 7 5 Yes Rainf kg m-2 s-1 Rainfall rate 1 Driver No Yes real 3 lon lat time na Rainfall rate 57 7 6 Yes SWdown W m-2 Surface incident shortwave radiation 1 Driver No Yes real 3 lon lat time na Surface incident shortwave radiation 58 7 7 Yes Tair K Near surface air temperature 1 Driver No Yes real 3 lon lat time na Near surface air temperature 59 7 8 Yes Wind m s-1 Near surface module of the wind 1 Driver No Yes real 3 lon lat time na Near surface wind magnitude "],
["available-meteorological-drivers.html", "21 Available Meteorological Drivers 21.1 Ameriflux 21.2 AmerifluxLBL 21.3 Fluxnet2015 21.4 NARR 21.5 CRUNCEP 21.6 CMIP5 21.7 NLDAS 21.8 GLDAS 21.9 PalEON 21.10 FluxnetLaThuile", " 21 Available Meteorological Drivers 21.1 Ameriflux Scale: site Resolution: 30 or 60 min Availability: varies by site http:\\/\\/ameriflux.lbl.gov\\/data\\/data-availability\\/ Notes: Old ORNL server, use is deprecated 21.2 AmerifluxLBL Scale: site Resolution: 30 or 60 min Availability: varies by site http:\\/\\/ameriflux.lbl.gov\\/data\\/data-availability\\/ Notes: new Lawrence Berkeley Lab server 21.3 Fluxnet2015 Scale: site Resolution: 30 or 60 min Availability: varies by site http://fluxnet.fluxdata.org/sites/site-list-and-pages Notes: Fluxnet 2015 synthesis product. Does not cover all FLUXNET sites 21.4 NARR Scale: North America Resolution: 3 hr, approx. 32km \\(Lambert conical projection\\) Availability: 1979-present 21.5 CRUNCEP Scale: global Resolution: 6hr, 0.5 degree Availability: 1901-2010 21.6 CMIP5 Scale: varies by model Resolution: 3 hr Availability: 2000-2100 Currently only GFDL available. Different scenerios and ensemble members can be set via Advanced Edit. 21.7 NLDAS Scale: Lower 48 + buffer, Resolution: 1 hour, .125 degree Availability: 1980-present 21.8 GLDAS Scale: Global Resolution: 3hr, 1 degree Availability: 1948-2010 21.9 PalEON Scale: -100 to -60 W Lon, 35 to 50 N Latitude \\(US northern hardwoods + buffer\\) Resolution: 6hr, 0.5 degree Availability: 850-2010 21.10 FluxnetLaThuile Scale: site Resolution: 30 or 60 min Availability: varies by site http:\\/\\/www.fluxdata.org\\/DataInfo\\/Dataset%20Doc%20Lib\\/SynthDataSummary.aspx Notes: 2007 synthesis. Fluxnet2015 supercedes this for sites that have been updated "],
["workflow.html", "22 Workflow", " 22 Workflow "],
["the-pecan-bety-connection.html", "23 The PEcAN BETY connection", " 23 The PEcAN BETY connection "],
["overview.html", "24 Overview 24.1 Bugs, Issues, Features, etc. 24.2 Requesting a feature 24.3 Closing an issue 24.4 When to submit an issue?", " 24 Overview We use GitHub to track development. To learn about GitHub, it is worth taking some time to read through the FAQ. When in doubt, the first step is to click the “Help” button at the top of the page. To address specific people, use a github feature called "@mentions" e.g. write @dlebauer, @robkooper, @mdietze, or @serbinsh … in the issue to alert the user as described in the GitHub documentation on notifications 24.1 Bugs, Issues, Features, etc. 24.1.1 Reporting a bug (For developers) work through debugging (see debugging wiki). Once you have identified a problem, that you can not resolve, you can write a bug report Write a bug report submit the bug report If you do find the answer, explain the resolution (in the issue) and close the issue 24.1.2 Required content Note: a bug is only a bug if it is reproducible clear bug reports save time Clear, specific title Description - What you did What you expected to happen What actually happened What does work, under what conditions does it fail? Reproduction steps - minimum steps required to reproduce the bug additional materials that could help identify the cause: screen shots stack traces, logs, scripts, output specific code and data / settings / configuration files required to reproduce the bug environment (operating system, browser, hardware) 24.2 Requesting a feature (from The Pragmatic Programmer, available as ebook through UI libraries, hardcopy on David’s bookshelf) * focus on “user stories”, e.g. specific use cases * Be as specific as possible, Here is an example: Bob is at www.mysite.edu/maps map of the the region (based on user location, e.g. US, Asia, etc) option to “use current location” is provided, if clicked, map zooms in to, e.g. state or county level for site run: option to select existing site or specify point by lat/lon option to specify a bounding box and grid resolution in either lat/lon or polar stereographic. asked to specify start and end times in terms of year, month, day, hour, minute. Time is recorded in UTC not local time, this should be indicated 24.3 Closing an issue Definition of “Done” test documentation when issue is resolved: status is changed to “resolved” assignee is changed to original author if original author agrees that issue has been resolved original author changes status to “closed” except for trivial issues, issues are only closed by the author 24.4 When to submit an issue? 24.4.1 Ideally, non-trivial code changes will be linked to an issue and a commit. This requires creating issues for each task, making small commits, and referencing the issue within your commit message. Issues can be created on GitHub. These issues can be linked to commits by adding text such as fixes gh-5 (see [[Using Git During Development|Using-Git#during-development]]). Rationale: This workflow is a small upfront investment that reduces error and time spent re-creating and debugging errors. Associating issues and commits, makes it easier to identify why a change was made, and potential bugs that could arise when the code is changed. In addition, knowing which issue you are working on clarifies the scope and objectives of your current task. "],
["using-git.html", "25 Using Git 25.1 Git 25.2 Recommended Git Workflow 25.3 During development: 25.4 For PEcAn 25.5 Git + Rstudio 25.6 For development: 25.7 References:", " 25 Using Git This document describes the steps required to download PEcAn, make changes to code, and submit your changes. For asking questions, reporting bugs, and requesting features, see our documentation for reporting issues on Redmine and GitHub If you are new to GitHub or to PEcAn, start with the one-time set-up instructions under Before any work is done. Also see the excellent tutorials and references in the References section at the the bottom of this page. To make trivial changes, see Quick and Easy To make changes to the code, start with the basic workflow. If you want to submit changes that you’ve made to be part of PEcAn you’ll want to follow Committing Changes Using Pull Requests To update your local branch cd pecan git pull upstream master ./scripts/build.sh 25.1 Git Git is a version control software; GitHub is a project-hosting website that is similar to Redmine but easier to use for open and collaborative development. Git is a free & open source, distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Every Git clone is a full-fledged repository with complete history and full revision tracking capabilities, not dependent on network access or a central server. Branching and merging are fast and easy to do. A good place to start is the GitHub 5 minute illustrated tutorial. In addition, there are three fun tutorials for learning git: Learn Git is a great web-based interactive tutorial. LearnGitBranching TryGit. For additional tutorials and background see References URLs In the rest of the document will use specific URL’s to clone the code. There a few URL’s you can use to clone a project, using https, ssh and git. You can use either https or git to clone a repository and write to it. The git protocol is read-only. 25.1.1 PEcAnProject on GitHub: https://github.com/organizations/PecanProject PEcAn source code: https://github.com/PecanProject/pecan.git git@github.com:PecanProject/pecan.git BETYdb source code: https://github.com/PecanProject/bety.git git@github.com:PecanProject/bety.git These instructions apply to other repositories too. 25.1.2 Milestones, Issues, Tasks The Milestones, issues, and tasks can be used to organize specific features or research projects. In general, there is a heirarchy: milestones (Big picture, “Epic”): contains many issues issues (Specific features / bugs, “Story”): may contain a list of tasks; represent task list (to do list, “Tasks”): list of steps required to close an issue, e.g.: ``` [ ] first do this [ ] then this [ ] completed when x and y ``` 25.1.3 Quick and Easy The easiest approach is to use GitHub’s browser based workflow. This is useful when your change is a few lines, if you are editing a wiki, or if the edit is trivial (and won’t break the code). The GitHub documentation is here but it is simple: finding the page or file you want to edit, click “edit” and then the GitHub web application will automatically forking and branch, then allow you to submit a pull request. However, it should be noted that unless you are a member of the PEcAn project that the “edit” button will not be active and you’ll want to follow the workflow described below for forking and then submitting a pull request. 25.2 Recommended Git Workflow Each feature should be in it’s own branch (for example each redmine issue is a branch, names of branches are often the issue in a bug tracking system). Commit and Push Frequency On your branch, commit at minimum once a day before you push changes: even better: every time you reach a stopping point and move to a new issue. best: any time that you have done work that you do not want to re-do. Remember, pushing changes to your branch is like saving a draft. Submit a pull request when you are done. 25.2.1 Before any work is done: The first step below only needs to be done once when you first start working on the PEcAn code. The steps below that need to be done to set up PEcAn on your computer, and would need to be repeated if you move to a new computer. If you are working from the PEcAn VM, you can skip the “git clone” since the PEcAn code is already installed. Most people will not be able to work in the PEcAn repository directly and will need to create a fork of the PEcAn source code in their own folder. To fork PEcAn into your own github space (github help: “fork a repo”). This forked repository will allow you to create branches and commit changes back to GitHub and create pull requests to the master of PEcAn. The forked repository is the only way for external people to commit code back to PEcAn and BETY. The pull request will start a review process that will eventually result in the code being merged into the main copy of the codebase. See https://help.github.com/articles/fork-a-repo for more information, especially on how to keep your fork up to date with respect to the original. (Rstudio users should also see Git + Rstudio, below) You can setup SSH keys to make it easier to commit cod back to GitHub. This might especially be true if you are working from a cluster, see set up ssh keys Introduce yourself to GIT git config --global user.name "FULLNAME" git config --global user.email you@yourdomain.example.com Fork PEcAn on GitHub. Goto the PEcAn source code and click on the Fork button in the upper right. This will create a copy of PEcAn in your personal space. Clone to your local machine git clone git@github.com:<username>/pecan.git Define upstream repository cd pecan git remote add upstream git@github.com:PecanProject/pecan.git 25.3 During development: commit often; each commit can address 0 or 1 issue; many commits can reference an issue (see [[Link commits to issue|Using-Git#link-commits-to-issues]]) ensure that all tests are passing before anything is pushed into master. 25.3.1 Basic Workflow PLEASE DO NOT USE THIS, SEE ADVANCED WORKFLOW! Get the latest code from the main repository git pull upstream master Do some coding Commit after each chunk of code (multiple times a day) git commit -m "<some descriptive information about what was done; references/fixes gh-X>" Push to YOUR Github (when a feature is working, a set of bugs are fixed, or you need to share progress with others) git push origin <branchname> Before submitting code back to the main repository, make sure that code compiles ./scripts/build.sh -c submit pull request with a reference to related issue (see [[Link commits to issue|Using-Git#link-commits-to-issues]]); also see github documentation 25.3.2 Advanced Workflow: A new branch for each change Make sure you start in master git checkout master Make sure master is up to date git pull upstream master Run any tests / make sure that code compiles For PEcAn: Build most recent versions of R packages (./scripts/build.sh -h for help)) ./scripts/build.sh For BETYdb (see BETY wiki) rspec Create a branch and switch to it git checkout -b <branchname> work/commit/etc git commit -m "<some descriptive information about what was done>" Run any tests / make sure that code compiles For PEcAn: ./scripts/build.sh -c Push this branch to your github space git push origin <branchname> submit pull request with [[link commits to issues|Using-Git#link-commits-to-issuess]]; also see explanation in this PecanProject/bety issue and github documentation 25.3.3 After pull request is merged Make sure you start in master git checkout master delete branch remotely git push origin --delete <branchname> delete branch locally git branch -D <branchname> 25.3.4 Link commits to issues You can reference and close issues from comments, pull requests, and commit messages. This should be done when you commit code that is related to or will close/fix an existing issue. There are two ways to do this. One easy way is to include the following text in your commit message: Github to close: “closes gh-xxx” (or syn. close, closed, fixes, fix, fixed) to reference: just the issue number (e.g. “gh-xxx”) avoid “closes #xxx” which will cross-reference Redmine issues Redmine: to close: “fixes redmine #xxx” (or closes etc.) to reference: “redmine #xxx” Bitbucket: to close: reference and use web interface! to reference: “re #xxx” Another way is to just add the url to the issue that you are updating. For example, if you are resolving an issue in Redmine, you can simply write the text “resolved by pull request https://github.com/PecanProject/pecan/pull/1” in the comments. 25.4 For PEcAn ./scripts/build.sh -c 25.4.1 Committing Changes Using Pull Requests GitHub provides a useful overview of how to submit changes to a project, Using Pull Requests. Once you have added a feature on your local fork of the project that you would like to contribute, these are the steps: Submit a Pull Request Pull request is reviewed by a member of the PEcAn core group Any comments should be addressed Additional commits are added to the pull request When ready, changes are merged 25.4.2 Other Useful Git Commands: GIT encourages branching “early and often” First pull from master Branch before working on feature One branch per feature You can switch easily between branches Merge feature into main line when branch done git branch <name of branch> git checkout <name of branch> repeat write some code commit until done git checkout master git merge <name of brach> git push If during above process you want to work on something else, commit all your code, create a new branch, and work on new branch. Delete a branch: git branch -d <name of branch> To push a branch git: push -u origin` To check out a branch: git fetch origin git checkout --track origin/<name of branch> Get the remote repository locally: git clone URL To update local repository to latest: git pull To add new files to the local repository: git add <file> To commit changes git commit <file|folder> To update remote repository: git push Show graph of commits: git log --graph --oneline --all 25.4.3 Tags Git supports two types of tags: lightweight and annotated. For more information see the Tagging Chapter in the Git documentation. Lightweight tags are useful, but here we discuss the annotated tags that are used for marking stable versions, major releases, and versions associated with published results. The basic command is git tag. The -a flag means ‘annotated’ and -m is used before a message. Here is an example: git tag -a v0.6 -m "stable version with foo and bar features, used in the foobar publication by Bob" Adding a tag to the a remote repository must be done explicitly with a push, e.g. git push v0.6 To use a tagged version, just checkout: git checkout v0.6 To tag an earlier commit, just append the commit SHA to the command, e.g. git tag -a v0.99 -m "last version before 1.0" 9fceb02 Using GitHub The easiest way to get working with GitHub is by installing the GitHub client. For instructions for your specific OS and download of the GitHub client, see https://help.github.com/articles/set-up-git. This will help you set up an SSH key to push code back to GitHub. To check out a project you do not need to have an ssh key and you can use the https or git url to check out the code. 25.5 Git + Rstudio Rstudio is nicely integrated with many development tools, including git and GitHub. It is quite easy to check out source code from within the Rstudio program or browser. The Rstudio documentation includes useful overviews of version control and R package development. Once you have git installed on your computer (see the Rstudio version control documentation for instructions), you can use the following steps to install the PEcAn source code in Rstudio. 25.5.1 Creating a Read-only version: This is a fast way to clone the repository that does not support contributing new changes (this can be done with further modification). install Rstudio (www.rstudio.com) click (upper right) project create project version control Git - clone a project from a Git Repository paste https://www.github.com/PecanProject/pecan choose working dir. for repo 25.6 For development: create account on github create a fork of the PEcAn repository to your own account https://www.github.com/pecanproject/pecan install Rstudio (www.rstudio.com) generate an ssh key in Rstudio: Tools -> Options -> Git/SVN -> "create RSA key" View public key -> ctrl+C to copy in GitHub go to ssh settings -> 'add ssh key' -> ctrl+V to paste -> 'add key' Create project in Rstudio project (upper right) -> create project -> version control -> Git - clone a project from a Git Repository paste repository url git@github.com:<username>/pecan.git> choose working dir. for repository 25.7 References: 25.7.0.1 Git Documentation Scott Chacon, ‘Pro Git book’, http://git-scm.com/book GitHub help pages, https://help.github.com/ Main GIT page, http://git-scm.com/documentation Information about branches, http://nvie.com/posts/a-successful-git-branching-model/ Another set of pages about branching, http://sandofsky.com/blog/git-workflow.html Stackoverflow highest voted questions tagged “git” 25.7.0.2 GitHub Documentation When in doubt, the first step is to click the “Help” button at the top of the page. GitHub Flow by Scott Chacon (Git evangelist and Ruby developer working on GitHub.com) GitHub FAQ Using Pull Requests SSH Keys "],
["directory-structure.html", "26 Directory structure 26.1 Overview of PEcAn repository as of PEcAn 1.2 26.2 Generic R package structure:", " 26 Directory structure 26.1 Overview of PEcAn repository as of PEcAn 1.2 trunk/ # Overall repository +- all # Dummy Package to load all PEcAn R packages +- common # Core functions +- db # Modules for querying the database +- depreciated # code from PEcAn < 1.1 for reference or merging into new packages +- models # Directory of wrapper scripts for each ecosystem model +- ed # Wrapper scripts for running ED within PEcAn +- sipnet # Wrapper scripts for running SIPNET within PEcAn +- ... # Wrapper scripts for running [MODELX] within PEcAn +- modules # Core modules +- allometry +- data.atmosphere +- data.hydrology +- data.land +- meta.analysis +- priors +- rtm +- uncertainty +- qaqc # Database and model QAQC scripts +- scripts # R and Shell scripts for use with PEcAn +- shell # Shell scripts, e.g.. install.all.sh, check.all.sh +- R # R scripts e.g. pft.add.spp.R, workflow.R, install.dependencies.R +- fia # FIA module +- utils # Misc. utility functions +- visualization # Advanced PEcAn visualization module +- web # Main PEcAn website files +- documentation # index_vm.html, references, other misc. 26.2 Generic R package structure: see the R development wiki for more information on writing code and adding data. +- DESCRIPTION # short description of the PEcAn library +- R/ # location of R source code +- man/ # Documentation +- tests/ # runall.R script to call all R test scripts +- inst/ # location of install scripts and misc. +- tests/ # location of PEcAn testing scripts +- extdata/ # misc. data files (in misc. formats) +- data/ # data used in testing and examples (saved as *.RData or *.rda files) "],
["adding-an-ecosystem-model.html", "27 Adding An Ecosystem Model 27.1 PEcAn Database 27.2 Define MODEL_TYPE 27.3 MACHINE 27.4 MODEL 27.5 FORMATS 27.6 INPUTS 27.7 PFTS (Plant Functional Types) 27.8 PRIORS 27.9 Interface Modules", " 27 Adding An Ecosystem Model Adding a model to PEcAn involves two activities: Updating the PEcAn database to register the model Writing the interface modules between the model and PEcAn Note that coupling a model to PEcAn should not require any changes to the model code itself. A key aspect of our design philosophy is that we want it to be easy to add models to the system and we want to using the working version of the code that is used by all other model users, not a special branch (which would rapidly end up out-of-date). 27.1 PEcAn Database To run a model within PEcAn requires that the PEcAn database know about the model – this includes a MODEL_TYPE designation, the types of inputs the model requires, the location of the model executable, and the plant functional types used by the model. The instructions below assume that you will be specifying this information using the BETYdb web-based interface. This can be done either on your local VM (localhost:3280/bety or localhost:6480/bety) or on a server installation of BETYdb, though in either case we’d encourage you to set up your PEcAn instance to support database syncs so that these changes can be shared and backed-up across the PEcAn network. The figure below summarizes the relevant database tables that need to be updated to add a new model and the primary variables that define each table. 27.2 Define MODEL_TYPE The first step to adding a model is to create a new MODEL_TYPE, which defines the abstract model class which we will then use to specify input requirements, define plant functional types, and keep track of different model versions. A MODEL_TYPE is created by selecting Runs > Model Type and then clicking on New Model Type. The MODEL_TYPE name should be identical to the MODEL package name (see Interface Module below) and is case sensitive. 27.3 MACHINE The PEcAn design acknowledges that the same model executables and input files may exist on multiple computers. Therefore, we need to define the machine that that we are using. If you are running on the VM then the local machine is already defined as pecan32 or pecan64 for the 32-bit and 64-bit versions respectively. Otherwise, you will need to select Runs > Machines, click New Machine, and enter the URL of your server (e.g. pecan2.bu.edu). 27.4 MODEL Next we are going to tell PEcAn where the model executable is. Select Runs > Files, and click ADD. Use the pull down menu to specify the machine you just defined above and fill in the path and name for the executable. For example, if SIPNET is installed at /usr/local/bin/sipnet then the path is /usr/local/bin/ and the file (executable) is sipnet. Now we will create the model record and associate this with the File we just registered. The first time you do this select Runs > Models and click New Model. Specify a descriptive name of the model (which doesn’t have to be the same as MODEL_TYPE), select the MODEL_TYPE from the pull down, and provide a revision identifier for the model (e.g. v3.2.1). Once the record is created select it from the Models table and click EDIT RECORD. Click on “View Related Files” and when the search window appears search for the model executable you just added (if you are unsure which file to choose you can go back to the Files menu and look up the unique ID number). You can then associate this Model record with the File by clicking on the +/- symbol. By contrast, clicking on the name itself will take you to the File record. In the future, if you set up the SAME MODEL VERSION on a different computer you can add that Machine and File to PEcAn and then associate this new File with this same Model record. A single version of a model should only be entered into PEcAn once. If a new version of the model is developed that is derived from the current version you should add this as a new Model record but with the same MODEL_TYPE as the original. Furthermore, you should set the previous version of the model as Parent of this new version. 27.5 FORMATS The PEcAn database keep track of all the input files passed to models, as well as any data used in model validation or data assimilation. Before we start to register these files with PEcAn we need to define the format these files will be in. To create a new format click on Runs > Formats and click New Format. Lets start with the input meteorology used by your model. The first thing you need to do is specify the mime type of your met file (e.g. CSV, netCDF, etc) and then give a unique, descriptive name to your file format (e.g. SIPNET clim). If your met file is text-based you can also specify whether the file has a header or not (TRUE / FALSE) and if any lines need to be skipped before the header or start of file (skip = INTEGER). You can also enter a more descriptive Note that explains the format in more detail. 27.5.1 Formats -> Variables After a Format entry has been created, you are encouraged to edit the entry to add relationships between the file’s variables and the Variables table in PEcAn. Not only do these relationships provide meta-data describing the file format, but they also allow PEcAn to search and (for some mime types) read files. To enter this data Edit the existing record and click on “View related variables”. For each variable in the file you will want at a minimum to specify the NAME of the variable within your file and match that to the equivalent Variable in the pulldown. If there is no equivalent, new Variables can be searched and added under Data > Variables. If your input file and PEcAn already have the variable in the same units then the Unit column can be left blank, otherwise units should be specified in a format that can be parsed by the UDUNITS library. Storage Type only needs to be specified if the variable is stored in a format other than what would be expected (e.g. if numeric values are stored as quoted character strings). Finally, if your data is in text format with variables in a standard order then you can specify the Column Number for the variable. This is particularly important for text files that lack headers. 27.5.2 MODEL_TYPE -> Formats For each of the input formats you specify for your model, you will need to edit your MODEL_TYPE record to add an association between the format and the MODEL_TYPE. Go to Runs > Model Type, select your record and click on the Edit button. Next, click on “Edit Associated Formats” and choose the Format you just defined from the pull down menu. If the Input box is checked then all matching Input records will be displayed in the PEcAn site run selection page when you are defining a model run. In other words, the set of model inputs available through the PEcAn web interface is model-specific and dynamically generated from the associations between MODEL_TYPEs and Formats. If you also check the Required box, then the Input will be treated as required and PEcAn will not run the model if that input is not available. Furthermore, on the site selection webpage, PEcAn will filter the available sites and only display pins on the Google Map for sites that have a full set of required inputs (or where those inputs could be generated using PEcAn’s workflows). Similarly, to make a site appear on the Google Map, all you need to do is specify Inputs, as described in the next section, and the point should automatically appear on the map. 27.6 INPUTS After a file Format has been created then input files can be registered with the database. The first step for this is to create a new File entry for the file (Runs > Files; New File) same as you did when you added the model executable. Next an Input record is created (Runs > Inputs; New Input) by selecting the Site the data are from, the data Format, the date range, and specifying a Name for the Input. If you are adding an input for a Site that doesn’t exist in the database you will want to create a new site (Data > Sites; New Site) and give it a name, latitude, and longitude. After the Input record is created you Edit the record and use “View related files” to associate the File with the Input. As with Models, if the exact same Input file exists in multiple place or over multiple machines then you can create multiple File entries for each of those files but associate each with the same Input. Doing so makes it simple for PEcAn to move between machines since the Input record stays the same for a give model run and the system just looks up where the local copy of the file is stored. Additional detail for creating Inputs can be found under [How to insert new Input data]. 27.7 PFTS (Plant Functional Types) Since many of the PEcAn tools are designed to keep track of parameter uncertainties and assimilate data into models, to use PEcAn with a model it is important to define Plant Functional Types for the sites or regions that you will be running the model. PFTs are MODEL_TYPE specific, so when you create a new PFT entry (Data > PFTs; New PFT) you will want to choose your MODEL_TYPE from the pull down and then give the PFT a descriptive name (e.g. temperate deciduous). 27.7.1 Species Within PEcAn there are no predefined PFTs and user can create new PFTs very easily at whatever taxonomic level is most appropriate, from PFTs for individual species up to one PFT for all plants globally. To allow PEcAn to query its trait database for information about a PFT, you will want to associate species with the PFT record by choosing Edit and then “View Related Species”. Species can be searched for by common or scientific name and then added to a PFT using the +/- button. 27.8 PRIORS In addition to adding species, a PFT is defined in PEcAn by the list of variables associated with the PFT. PEcAn takes a fundamentally Bayesian approach to representing model parameters, so variables are not entered as fixed constants but as Prior probability distributions (see below). Once Priors are defined for each model variable then you Edit the PFT and use “View Related Priors” to search for and add Prior distributions for each model parameter. It is important to note that the priors are defined for the variable name and units as specified in the Variables table. If the variable name or units is different within the model it is the responsibility of write.configs.MODEL function to handle name and unit conversions (see Interface Modules below). This can also include common but nonlinear transformations, such as converting SLA to LMA or changing the reference temperature for respiration rates. There are a wide variety of priors already defined in the PEcAn database that often range from very diffuse and generic to very informative priors for specific PFTs. If the current set of Priors for a variable are inadequate, or if a prior needs to be specified for a new variable, this can be done under Data > Priors then “New Prior”. After using the pull-down menu to select the Variable you want to generate a prior for, the prior is defined by choosing a probability distribution and specifying values for that distribution’s parameters. These are labeled Parameter a & b but their exact meaning depends upon the distribution chosen. For example, for the Normal distribution a and b are the mean and standard deviation while for the Uniform they are the minimum and maximum. All parameters are defined based on their standard parameterization in the R language. If the prior is based on observed data (independent of data in the PEcAn database) then you can also specify the prior sample size, N. The Phylogeny variable allows one to specify what taxonomic grouping the prior is defined for, at it is important to note that this is just for reference and doesn’t have to be specified in any standard way nor does it have to be monophyletic (i.e. it can be a functional grouping). Finally, the Citation is a required variable that provides a reference for how the prior was defined. That said, there are a number of unpublished Citations in current use that simply state the expert opinion of an individual. Additional information on adding PFTs, Species, and Priors can be found under [[Choosing PFTs]] 27.9 Interface Modules 27.9.1 Setting up the module directory (required) PEcAn assumes that the interface modules are available as an R package in the models directory named after the model in question. The simplest way to get started on that R package is to make a copy the template directory in the pecan/models folder and re-name it to the name of your model. In the code, filenames, and examples below you will want to substitute the word MODEL for the name of your model (note: R is case-sensitive). If you do not want to write the interface modules in R then it is fairly simple to set up the R functions describe below to just call the script you want to run using R’s system command. Scripts that are not R functions should be placed in the inst folder and R can look up the location of these files using the function system.file which takes as arguments the local path of the file within the package folder and the name of the package (typically PEcAn.MODEL). For example ## Example met conversion wrapper function met2model.MODEL <- function(in.path, in.prefix, outfolder, start_date, end_date){ myMetScript <- system.file("inst/met2model.MODEL.sh", "PEcAn.MODEL") system(paste(myMetScript, file.path(in.path, in.prefix), outfolder, start_date, end_date)) } would execute the following at the Linux command line inst/met2model.MODEL.sh in.path/in.prefix outfolder start_date end_date ` 27.9.2 DESCRIPTION Within the module folder open the DESCRIPTION file and change the package name to PEcAn.MODEL. Fill out other fields such as Title, Author, Maintainer, and Date. 27.9.3 NAMESPACE Open the NAMESPACE file and change all instances of MODEL to the name of your model. If you are not going to implement one of the optional modules (described below) at this time then you will want to comment those out using the pound sign #. For a complete description of R NAMESPACE files see here. If you create additional functions in your R package that you want to be used make sure you include them in the NAMESPACE as well (internal functions don’t need to be declared) 27.9.4 Building the package Once the package is defined you will then need to add it to the PEcAn build scripts. From the root of the pecan directory, go into the scripts folder and open the file build.sh. Within the section of code that includes PACKAGES= add model/MODEL to the list of packages to compile. If, in writing your module, you add any other R packages to the system you will want to make sure those are listed in the DESCRIPTION and in the script scripts/install.dependencies.R. Next, from the root pecan directory open all/DESCRIPTION and add your model package to the Suggests: list. At any point, if you want to check if PEcAn can build your MODEL package successfully, just go to the linux command prompt and run scripts/build.sh. You will need to do this before the system can use these packages. 27.9.5 write.config.MODEL (required) This module performs two primary tasks. The first is to take the list of parameter values and model input files that it receives as inputs and write those out in whatever format(s) the MODEL reads (e.g. a settings file). The second is to write out a shell script, jobs.sh, which, when run, will start your model run and convert its output to the PEcAn standard (netCDF with metadata currently equivalent to the MsTMIP standard). Within the MODEL directory take a close look at inst/template.job and the example write.config.MODEL to see an example of how this is done. It is important that this script writes or moves outputs to the correct location so that PEcAn can find them. The example function also shows an example of writing a model-specific settings/config file, also by using a template. You are encouraged to read the section above on defining PFTs before writing write.config.MODEL so that you understand what model parameters PEcAn will be passing you, how they will be named, and what units they will be in. Also note that the (optional) PEcAn input/driver processing scripts are called by separate workflows, so the paths to any required inputs (e.g. meteorology) will already be in the model-specific format by the time write.config.MODEL receives that info. 27.9.6 Output Conversions The module model2netcdf.MODEL converts model output into the PEcAn standard (netCDF with metadata currently equivalent to the MsTMIP standard). This function was previously required, but now that the conversion is called within jobs.sh it may be easier for you to convert outputs using other approaches (or to just directly write outputs in the standard). Whether you implement this function or convert outputs some other way, please note that PEcAn expects all outputs to be broken up into ANNUAL files with the year number as the file name (i.e. YEAR.nc), though these files may contain any number of scalars, vectors, matrices, or arrays of model outputs, such as time-series of each output variable at the model’s native timestep. Note: PEcAn reads all variable names from the files themselves so it is possible to add additional variables that are not part of the MsTMIP standard. Similarly, there are no REQUIRED output variables, though time is highly encouraged. We are shortly going establish a canonical list of PEcAn variables so that if users add additional output variables they become part of the standard. We don’t want two different models to call the same output with two different names or different units as this would prohibit the multi-model syntheses and comparisons that PEcAn is designed to facilitate. 27.9.7 met2model.MODEL met2model.MODEL(in.path, in.prefix, outfolder, start_date, end_date) Converts meteorology input files from the PEcAn standard (netCDF, CF metadata) to the format required by the model. This file is optional if you want to load all of your met files into the Inputs table as described in [How to insert new Input data], which is often the easiest way to get up and running quickly. However, this function is required if you want to benefit from PEcAn’s meteorology workflows and model run cloning. You’ll want to take a close look at [Adding-an-Input-Converter] to see the exact variable names and units that PEcAn will be providing. Also note that PEcAn splits all meteorology up into ANNUAL files, with the year number explicitly included in the file name, and thus what PEcAn will actually be providing is in.path, the input path to the folder where multiple met files may stored, and in.prefix, the start of the filename that precedes the year (i.e. an individual file will be named <in.prefix>.YEAR.nc). It is valid for in.prefix to be blank. The additional REQUIRED arguments to met2model.MODEL are outfolder, the output folder where PEcAn wants you to write your meteorology, and start_date and end_date, the time range the user has asked the meteorology to be processed for. 27.9.8 Commit changes Once the MODEL modules are written, you should follow the Using-Git instructions on how to commit your changes to your local git repository, verify that PEcAn compiles using scripts/build.sh, push these changes to Github, and submit a pull request so that your model module is added to the PEcAn system. It is important to note that while we encourage users to make their models open, adding the PEcAn interface module to the Github repository in no way requires that the model code itself be made public. It does, however, allow anyone who already has a copy of the model code to use PEcAn so we strongly encourage that any new model modules be committed to Github. General Note: dates in the database should be datatime (preferably with timezone), and datetime passed around in PEcAn should be of type POSIXlt. "],
["met-data.html", "28 Met Data 28.1 Dimensions: 28.2 The variable names should be standard_name", " 28 Met Data The standard met data inputs should be of the form: Converters from a raw to standard format go in /modules/data.atmosphere/R; converters from standard to model-specific go in models/<mymodel>/R. For a number of common gridded products (NARR, CRUNCEP, ISIMIP), there are bash scripts for converting and rechunking in this repository: https://github.com/ebimodeling/model-drivers; these are not generalized, but may be merged into PEcAn at some point. Examples: * NARR: * CRUNCEP: * ISIMIP: Names should be met2CF.<sourcename> and met2model.<modelname>. 28.1 Dimensions: CF standard-name units time days since 1700-01-01 00:00:00 UTC longitude degrees_east latitude degrees_north 28.2 The variable names should be standard_name CF standard-name units bety isimip cruncep narr ameriflux air_temperature K airT tasAdjust tair air TA (C) air_temperature_max K tasmaxAdjust NA tmax air_temperature_min K tasminAdjust NA tmin air_pressure Pa air_pressure PRESS (KPa) mole_fraction_of_carbon_dioxide_in_air mol/mol CO2 moisture_content_of_soil_layer kg m-2 soil_temperature K soilT TS1 (NOT DONE) relative_humidity % relative_humidity rhurs NA rhum RH specific_humidity 1 specific_humidity NA qair shum CALC(RH) water_vapor_saturation_deficit Pa VPD VPD (NOT DONE) surface_downwelling_longwave_flux_in_air W m-2 same rldsAdjust lwdown dlwrf Rgl surface_downwelling_shortwave_flux_in_air W m-2 solar_radiation rsdsAdjust swdown dswrf Rg surface_downwelling_photosynthetic_photon_flux_in_air mol m-2 s-1 PAR PAR (NOT DONE) precipitation_flux kg m-2 s-1 cccc prAdjust rain acpc PREC (mm/s) degrees wind_direction WD wind_speed m/s Wspd WS eastward_wind m/s eastward_wind CALC(WS+WD) northward_wind m/s northward_wind CALC(WS+WD) preferred variables indicated in bold wind_direction has no CF equivalent and should not be converted, instead the met2CF functions should convert wind_direction and wind_speed to eastward_wind and northward_wind variable names are from MsTMIP, but lowercase to be consistent with the MsTMIP drivers. standard_name is CF-convention standard names units can be converted by udunits, so these can vary (e.g. the time denominator may change with time frequency of inputs) soil moisture for the full column, rather than a layer, is soil_moisture_content For the standardized files, we are using CF standard names as variable names. For example, in the MsTMIP-CRUNCEP data, the variable rain should be precipitation_rate. We want to standardize the units as well as part of the met2CF.<product> step. I believe we want to use the CF “canonical” units but retain the MsTMIP units any time CF is ambiguous about the units. The key is to process each type of met data (site, reanalysis, forecast, climate scenario, etc) to the exact same standard. This way every operation after that (extract, gap fill, downscale, convert to a model, etc) will always have the exact same inputs. This will make everything else much simpler to code and allow us to avoid a lot of unnecessary data checking, tests, etc being repeated in every downstream function. In addition to the state assimilation routines found in the assim.sequential module, another approach for state data assimilation in PEcAn is through the DART workflow created by the DARES group in NCAR. This section gives a straight-forward explanation how to implement DART, focused on the technical aspects of the implementation. If there are any questions, feel free to send @Viskari an email (tt.viskari@gmail.com) or contacting DART support as they are quite awesome in helping people with problems. Also, if there are any suggestions on how to improve the wiki, please let me know. Running with current folders in PEcAn Currently the DART folders in PEcAn are that you can simply copy the structure there over a downloaded DART workflow and it should replace/add relevant files and folders. The most important step after that is to check and change the run paths in the following files: Path_name files in the work folders T_ED2IN file, as it indicates where the run results be written. advance_model.csh, as it indicates where to copy files from/to. Second thing is setting the state vector size. This is explained in more detail below, but essentially this is governed by the variable model_size in model_mod.f90. In addition to this, it should be changed in utils/F2R.f90 and R2F.f90 programs, which are responsible for reading and writing state variable information for the different ensembles. This also will be expanded below. Finally, the new state vector size should be updated for any other executable that runs it. Third thing needed are the initial condition and observation sequence files. They will always follow the same format and are explained in more detail below. Finally the ensemble size, which is the easiest to change. In the work subfolder, there is a file named input.nml. Simply changing the ensemble size there will set it for the run itself. Also remember that initial conditions file should have the equal amount of state vectors as there are ensemble members. Adjusting the workflow The central file for the actual workflow is advance_model.csh. It is a script DART calls to determine how the state vector changes between the two observation times and is essentially the only file one needs to change when changing state models or observations operators. The file itself should be commented to give a good idea of the flow, but beneath is a crude order of events. 1. Create a temporary folder to run the model in and copy/link required files in to it. 2. Read in the state vector values and times from DART. Here it is important to note that the values will be in binary format, which need to be read in by a Fortran program. In my system, there is a program called F2R which reads in the binary values and writes out in ascii form the state vector values as well as which ED2 history files it needs to copy based on the time stamps. 3. Run the observation operator, which writes the state vector state in to the history files and adjusts them if necessary. 4. Run the program. 5. Read the new state vector values from output files. 6. Convert the state vector values to the binary. In my system, this is done by the program R2F. Initial conditions file The initial conditions file, commonly named filter_ics although you can set it to something else in input.nml, is relatively simple in structure. It has one sequence repeating over the number of ensemble members. First line contains two times: Seconds and days. Just use one of them in this situation, but it has to match the starting time given in input.nml. After that each line should contain a value from the state vector in the order you want to treat them. R functions filter_ics.R and B_filter_ics.R in the R folder give good examples of how to create these. Observations files The file which contains the observations is commonly known as obs_seq.out, although again the name of the file can be changed in input.nml. The structure of the file is relatively straight-forward and the R function ObsSeq.R in the R subfolder has the write structure for this. Instead of writing it out here, I want to focus on a few really important details in this file. Each observations will have a time, a value, an uncertainty, a location and a kind. The first four are self-explanatory, but the kind is really important, but also unfortunately really easy to misunderstand. In this file, the kind does not refer to a unit or a type of observation, but which member of the state vector is this observation of. So if the kind was, for example, 5, it would mean that it was of the fifth member of the state vector. However, if the kind value is positive, the system assumes that there is some sort of an operator change in comparing the observation and state vector value which is specified in a subprogram in model_mod.f90. So for an direct identity comparison between the observation and the state vector value, the kind needs to be negative number of the state vector component. Thus, again if the observation is of the fifth state vector value, the kind should be set as -5. Thus it is recommendable that the state vector values have already been altered to be comparable with the observations. As for location, there are many ways to set in DART and the method needs to be chosen when compiling the code by giving the program which of the location mods it is to use. In our examples we used a 1-dimensional location vector with scaled values between 0 and 1. For future it makes sense to switch to a 2 dimensional long- and lat-scale, but for the time being the location does not impact the system a lot. The main impact will be if the covariances will be localized, as that will be decided on their locations. State variable vector in DART Creating/adjusting a state variable vector in DART is relatively straight-forward. Below are listed the steps to specify a state variable vector in DART. I. For each specific model, there should be an own folder within the DART root models folder. In this folder there is a model_mod.f90, which contains the model specific subroutines necessary for a DART run. At the beginning of this file there should be the following line: integer, parameter :: model_size = [number] The number here should be the number of variables in the vector. So for example if there were three state variables, then the line should look like this: integer, parameter :: model_size = 3 This number should also be changed to match with any of the other executables called during the run as indicated by the list above. In the DART root, there should be a folder named obs_kind, which contains a file called DEFAULT_obs_kind_mod.F90. It is important to note that all changes should be done to this file instead of obs_kind_mod.f90, as during compilation DART creates obs_kind_mod.f90 from DEFAULT_obs_kind_mod.F90. This program file contains all the defined observation types used by DART and numbers them for easier reference later. Different types are classified according to observation instrument or relevant observation phenomenon. Adding a new type only requires finding an unused number and starting a new identifying line with the following: integer, parameter, public :: & KIND_… Note that the observation kind should always be easy to understand, so avoid using unnecessary acronyms. For example, when adding an observation type for Leaf Area Index, it would look like below: integer, parameter, public :: & KIND_LEAF_AREA_INDEX = [number] In the DART root, there should be a folder named obs_def, which contains several files starting with obs_def_. There files contain the different available observation kinds classified either according to observation instrument or observable system. Each file starts with the line ! BEGIN DART PREPROCESS KIND LIST And end with line ! END DART PREPROCESS KIND LIST The lines between these two should contain ! The desired observation reference, the observation type, COMMON_CODE. For example, for observations relating to phenology, I have created a file called obs_def_phen_mod.f90. In this file I define the Leaf Area Index observations in the following way. ! BEGIN DART PREPROCESS KIND LIST ! LAI, TYPE_LEAF_AREA_INDEX, COMMON_CODE ! END DART PREPROCESS KIND LIST Note that the exclamation marks are necessary for the file. In the model specific folder, in the work subfolder there is a namelist file input.nml. This contains all the run specific information for DART. In it, there is a subtitle &preprocess, under which there is a line input_files = ‘….’ This input_files sections must be set to refer to the obs_def file created in step III. The input files can contain references to multiple obs_def files if necessary. As an example, the reference to the obs_def_phen_mod.f90 would look like input_files = ‘../../../obs_def/obs_def_phen_mod.f90’ V. Finally, as an optional step, the different values in state vector can be typed. In model_mod, referred to in step I, there is a subroutine get_state_meta_data. In it, there is an input variable index_in, which refers to the vector component. So for instance for the second component of the vector index_in would be 2. If this is done, the variable kind has to be also included at the beginning of the model_mod.f90 file, at the section which begins use obs_kind_mod, only :: The location of the variable can be set, but for a 0-dimensional model we are discussing here, this is not necessary. Here, though, it is possible to set the variable types by including the following line if(index_in .eq. [number]) var_type = [One of the variable kinds set in step II] If the length of the state vector is changed, it is important that the script ran with DART produces a vector of that length. Change appropriately if necessary. After these steps, DART should be able to run with the state vector of interest. PEcAn has a single xml configuration file. This fill will dictate what models to run, where, with what input files and for how long. Each model has its own configuration file as well. 28.2.1 PEcAn Configuration Files This file is often called pecan.xml. The system will try to find this file in the following locations: * Passed in when calling R with –settings * Passed in to read.settings() function * PECAN_SETTINGS environment variable * File called pecan.xml in the current working folder Once any of these are loaded it will try to locate a pecan.xml in the outdir specified in the loaded file. 28.2.2 Model-specific configuration files ED2 SIPNET BIOCRO "],
["coding-style.html", "29 Coding Style 29.1 Use Roxygen2 documentation 29.2 Write your name at the top 29.3 Use testthat testing package 29.4 Don’t use shortcuts 29.5 Package Dependencies:", " 29 Coding Style Consistent coding style improves readability and reduces errors in shared code. R does not have an official style guide, but Hadley Wickham provides one that is well thought out and widely adopted. Advanced R: Coding Style. Both the Wickham text and this page are derived from Google’s R Style Guide. 29.1 Use Roxygen2 documentation This is the standard method of documentation used in PEcAn development, it provides inline documentation similar to doxygen. Even trivial functions should be documented. See Roxygen2 Wiki page 29.2 Write your name at the top Any function that you create or make a meaningful contribution to should have your name listed after the author tag in the function documentation. 29.3 Use testthat testing package See Unit_Testing wiki for instructions, and Advanced R: Tests. tests provide support for documentation - they define what a function is (and is not) expected to do all functions need tests to ensure basic functionality is maintained during development. all bugs should have a test that reproduces the bug, and the test should pass before bug is closed 29.4 Don’t use shortcuts R provides many shortcuts that are useful when coding interactively, or for writing scripts. However, these can make code more difficult to read and can cause problems when written into packages. 29.4.1 Function Names (verb.noun) Following convention established in PEcAn 0.1, we use the all lowercase with periods to separate words. They should generally have a verb.noun format, such as query.traits, get.samples, etc. 29.4.2 File Names File names should end in .R, .Rdata, .Rscript and should be meaningful, e.g. named after the primary functions that they contain. There should be a separate file for each major high-level function to aid in identifying the contents of files in a directory. 29.4.3 Use “<-” as an assignment operator Because most R code uses <- (except where = is required), we will use <- “=” is used for function arguments 29.4.4 Use Spaces around all binary operators (=, +, -, <-, etc.). after but not before a comma 29.4.5 Use curly braces The option to omit curly braces is another shortcut that makes code easier to write but harder to read and more prone to error. 29.5 Package Dependencies: 29.5.1 library vs require When another package is required by a function or script, it can be called in the following ways: (As a package dependency loads with the package, these should be the default approaches when writing functions in a package. There can be some exceptions, such as when a rarely-used or non-essential function requires an esoteric package.) 1. When using library, if dependency is not met, it will print an error and stop 2. When using require, it will print a warning and continue (but will throw an error when a function from the required package is called) Reference: Stack Overflow “What is the difference between require and library?” 29.5.2 DEPENDS, SUGGESTS, IMPORTS It is considered best practice to use DEPENDS and SUGGESTS in DESCRIPTION; SUGGESTS should be used for packages that are called infrequently, or only in examples and vignettes; suggested packages are called by require inside a function. Consider using IMPORTS instead of depends in the DESCRIPTION files. This will make loading packages faster by allowing it to have functions available without loading the hierarchy of dependencies, dependencies of dependencies, ad infinitum … From p. 6 of the “R extensions manual”:http://cran.r-project.org/doc/manuals/R-exts.html The Suggests field uses the same syntax as Depends and lists packages that are not necessarily needed. This includes packages used only in examples, tests or vignettes (see Section 1.4 [Writing package vignettes], page 26), and packages loaded in the body of functions. E.g., suppose an example from package foo uses a dataset from package bar. Then it is not necessary to have bar use foo unless one wants to execute all the examples/tests/vignettes: it is useful to have bar, but not necessary. Version requirements can be specified, and will be used by R CMD check. "],
["roxygen2.html", "30 Roxygen2 30.1 Canonical references: 30.2 Basic Roxygen2 instructions: 30.3 Example 30.4 Updating documentation", " 30 Roxygen2 This is the standard method of documentation used in PEcAn development, it provides inline documentation similar to doxygen. 30.1 Canonical references: Must Read: R package development by Hadley Wickham: Object Documentation Package Metadata Roxygen2 Documentation Roxygen2 Package Documentation GitHub 30.2 Basic Roxygen2 instructions: Section headers link to “Writing R extensions” which provides in-depth documentation. This is provided as an overview and quick reference. 30.2.1 Tags tags are preceeded by ##' tags required by R: ** title tag is required, along with actual title ** param one for each parameter, should be defined ** return must state what function returns (or nothing, if something occurs as a side effect tags strongly suggested for most functions: ** author ** examples can be similar to test cases. optional tags: ** export required if function is used by another package ** import can import a required function from another package (if package is not loaded or other function is not exported) ** seealso suggests related functions. These can be linked using \\code{link{}} 30.2.2 Text markup 30.2.2.1 Formatting \\bold{} \\emph{} italics 30.2.2.2 Links \\url{www.url.com} or \\href{url}{text} for links \\code{\\link{thisfn}} links to function “thisfn” in the same package \\code{\\link{foo::thatfn}} links to function “thatfn” in package “foo” \\pkg{package_name} 30.2.2.3 Math \\eqn{a+b=c} uses LaTex to format an inline equation \\deqn{a+b=c} uses LaTex to format displayed equation \\deqn{latex}{ascii} and \\eqn{latex}{ascii} can be used to provide different versions in latex and ascii. 30.2.2.4 Lists \\enumerate{ \\item A database consists of one or more records, each with one or more named fields. \\item Regular lines start with a non-whitespace character. \\item Records are separated by one or more empty lines. } \\itemize and \\enumerate commands may be nested. 30.2.2.5 “Tables”:http://cran.r-project.org/doc/manuals/R-exts.html#Lists-and-tables \\tabular{rlll}{ [,1] \\tab Ozone \\tab numeric \\tab Ozone (ppb)\\cr [,2] \\tab Solar.R \\tab numeric \\tab Solar R (lang)\\cr [,3] \\tab Wind \\tab numeric \\tab Wind (mph)\\cr [,4] \\tab Temp \\tab numeric \\tab Temperature (degrees F)\\cr [,5] \\tab Month \\tab numeric \\tab Month (1--12)\\cr [,6] \\tab Day \\tab numeric \\tab Day of month (1--31) } 30.3 Example Here is an example documented function, myfun ##' My function adds three numbers ##' ##' A great function for demonstrating Roxygen documentation ##' @param a numeric ##' @param b numeric ##' @param c numeric ##' @return d, numeric sum of a + b + c ##' @export ##' @author David LeBauer ##' @examples ##' myfun(1,2,3) ##' \\dontrun{myfun(NULL)} myfun <- function(a, b, c){ d <- a + b + c return(d) } In emacs, with the cursor inside the function, the keybinding C-x O will generate an outline or update the Roxygen2 documentation. 30.4 Updating documentation After adding documentation run the following command (replacing common with the name of the folder you want to update): ** In R using devtools to call roxygenize: require(devtools) document("common") "],
["testing.html", "31 Testing 31.1 Rationale 31.2 Tests makes development easier and less error prone 31.3 Quick Start: 31.4 Test files", " 31 Testing PEcAn uses the testthat package developed by Hadley Wickham. Hadley has written instructions for using this pakcage in his Testing chapter. 31.1 Rationale makes development easier provides working documentation of expected functionality saves time by allowing computer to take over error checking once a test has been made improves code quality Further reading: Aruliah et al 2012 Best Practices for Scientific Computing 31.2 Tests makes development easier and less error prone Testing makes it easier to develop by organizing everything you are already doing anyway - but integrating it into the testing and documentation. With a codebase like PEcAn, it is often difficult to get started. You have to figure out what was I doing yesterday? what do I want to do today? what existing functions do I need to edit? what are the arguements to these functions (and what are examples of valid arguments) what packages are affected where is a logical place to put files used in testing 31.3 Quick Start: decide what you want to do today identify the issue in redmine or github (if none exists, create one) to work on issue 99, create a new branch called “redmine99”, “github99” or some descriptive name… Today we will add enable an existing function, make.cheas to make goat.cheddar. We will know that we are done by the color and taste. git branch goat-cheddar git checkout goat-cheddar * open existing (or create new) file in `inst/tests/`. If working on code in "myfunction" or a set of functions in "R/myfile.R", the file should be named accordingly, e.g. "inst/tests/test.myfile.R" * if you are lucky, the function has already been tested and has some examples. * if not, you may need to create a minimal example, often requiring a settings file. The default settings file can be obtained in this way: settings <- read.settings(system.file("extdata/test.settings.xml", package = "PEcAn.utils")) * write what you want to do test_that("make.cheas can make cheese",{ goat.cheddar <- make.cheas(source = 'goat', style = 'cheddar') expect_equal(color(goat.cheddar), "orange") expect_is(object = goat.cheddar, class = "cheese") expect_true(all(c("sharp", "creamy") %in% taste(goat.cheddar))) } * now edit the goat.cheddar function until it makes savory, creamy, orange cheese. * commit often * update documentation and test ```r library(devtools) document("mypkg") test("mypkg") ``` * commit again * when complete, merge, and push ```bash git commit -m "make.cheas makes goat.cheddar now" git checkout master git merge goat-cheddar git push ``` 31.4 Test files Many of PEcAn’s functions require inputs that are provided as data. These can be in the /data or the /inst/extdata folders of a package. Data that are not package specific should be placed in the PEcAn.all or PEcAn.utils files. Some useful conventions: 31.4.1 Settings A generic settings can be found in the PEcAn.all package settings.xml <- system.file("pecan.biocro.xml", package = "PEcAn.BIOCRO") settings <- read.settings(settings.xml) database settings can be specified, and tests run only if a connection is available We currently use the following database to run tests against; tests that require access to a database should be enclosed by if(db.exists()) to avoid failed tests on systems that do not have the database installed. settings$database <- list(userid = "bety", passwd = "bety", name = "bety", # database name host = "localhost" # server name) if(db.exists()){ test_that(... ## write tests here } instructions for installing this are available on the VM creation wiki examples can be found in the PEcAn.DB package (db/inst/tests/). Model specific settings can go in the model-specific module, for example: settings.xml <- system.file("pecan.biocro.xml", package = "PEcAn.BIOCRO") settings <- read.settings(settings.xml) test-specific settings: settings text can be specified inline: settings.text <- " <pecan> <nocheck>nope</nocheck> ## allows bypass of checks in the read.settings functions <pfts> <pft> <name>ebifarm.pavi</name> <outdir>test/</outdir> </pft> </pfts> <outdir>test/</outdir> <database> <userid>bety</userid> <passwd>bety</passwd> <location>localhost</location> <name>bety</name> </database> </pecan>" settings <- read.settings(settings.text) values in settings can be updated: settings <- read.settings(settings.text) settings$outdir <- "/tmp" ## or any other settings 31.4.2 Helper functions created to make testing easier tryl returns FALSE if function gives error temp.settings creates temporary settings file test.remote returns TRUE if remote connection is available db.exists returns TRUE if connection to database is available 31.4.3 When should I test? A test should be written for each of the following situations: Each bug should get a regression test. The first step in handling a bug is to write code that reproduces the error This code becomes the test most important when error could re-appear essential when error silently produces invalid results Every time a (non-trivial) function is created or edited Write tests that indicate how the function should perform example: expect_equal(sum(1,1), 2) indicates that the sum function should take the sum of its arguments Write tests for cases under which the function should throw an error example: expect_error(sum("foo") better : expect_error(sum("foo"), "invalid 'type' (character)") 31.4.4 What types of testing are important to understand? 31.4.5 Unit Testing / Test Driven Development Tests are only as good as the test write test write code 31.4.6 Regression Testing When a bug is found, write a test that finds the bug (the minimum test required to make the test fail) fix the bug bug is fixed when test passes 31.4.7 How should I test in R? The testthat package. tests are found in ~/pecan/<packagename>/inst/tests, for example utils/inst/tests/ See attached file and https://github.com/hadley/devtools/wiki/Testing for details on how to use the testthat package. 31.4.7.1 List of Expectations Full Abbreviation expect_that(x, is_true()) expect_true(x) expect_that(x, is_false()) expect_false(x) expect_that(x, is_a(y)) expect_is(x, y) expect_that(x, equals(y)) expect_equal(x, y) expect_that(x, is_equivalent_to(y)) expect_equivalent(x, y) expect_that(x, is_identical_to(y)) expect_identical(x, y) expect_that(x, matches(y)) expect_matches(x, y) expect_that(x, prints_text(y)) expect_output(x, y) expect_that(x, shows_message(y)) expect_message(x, y) expect_that(x, gives_warning(y)) expect_warning(x, y) expect_that(x, throws_error(y)) expect_error(x, y) 31.4.7.2 How to run tests add the following to “pecan/tests/run.all.R” library(testthat) library(mypackage) test_package("mypackage") 31.4.8 basic use of the testthat package Here is an example of tests (these should be placed in <packagename>/inst/tests/test_<sourcefilename>.R: test_that("mathematical operators plus and minus work as expected",{ expect_equal(sum(1,1), 2) expect_equal(sum(-1,-1), -2) expect_equal(sum(1,NA), NA) expect_error(sum("cat")) set.seed(0) expect_equal(sum(matrix(1:100)), sum(data.frame(1:100))) }) test_that("different testing functions work, giving excuse to demonstrate",{ expect_identical(1, 1) expect_identical(numeric(1), integer(1)) expect_equivalent(numeric(1), integer(1)) expect_warning(mean('1')) expect_that(mean('1'), gives_warning("argument is not numeric or logical: returning NA")) expect_warning(mean('1'), "argument is not numeric or logical: returning NA") expect_message(message("a"), "a") }) 31.4.8.1 Script testing It is useful to add tests to a script during development. This allows you to test that the code is doing what you expect it to do. * here is a fake script using the iris data set test_that("the iris data set has the same basic features as before",{ expect_equal(dim(iris), c(150,5)) expect_that(iris$Sepal.Length, is_a("numeric")) expect_is(iris$Sepal.Length, "numeric")#equivalent to prev. line expect_is(iris$Species, "factor") }) iris.color <- data.frame(Species = c("setosa", "versicolor", "virginica"), color = c("pink", "blue", "orange")) newiris <- merge(iris, iris.color) iris.model <- lm(Petal.Length ~ color, data = newiris) test_that("changes to Iris code occurred as expected",{ expect_that(dim(newiris), equals(c(150, 6))) expect_that(unique(newiris$color), is_identical_to(unique(iris.color$color))) expect_equivalent(iris.model$coefficients["(Intercept)"], 4.26) }) 31.4.8.2 Function testing Testing of a new function, as.sequence. The function and documentation are in source:R/utils.R and the tests are in source:tests/test.utils.R. Recently, I made the function as.sequence to turn any vector into a sequence, with custom handling of NA’s: function(x, na.rm = TRUE){ x2 <- as.integer(factor(x, unique(x))) if(all(is.na(x2))){ x2 <- rep(1, length(x2)) } if(na.rm == TRUE){ x2[is.na(x2)] <- max(x2, na.rm = TRUE) + 1 } return(x2) } The next step was to add documentation and test. Many people find it more efficient to write tests before writing the function. This is true, but it also requires more discipline. I wrote these tests to handle the variety of cases that I had observed. As currently used, the function is exposed to a fairly restricted set of options - results of downloads from the database and transformations. test_that(“as.sequence works”;{ expect_identical(as.sequence(c(“a”, “b”)), 1:2) expect_identical(as.sequence(c(“a”, NA)), 1:2) expect_equal(as.sequence(c(“a”, NA), na.rm = FALSE), c(1,NA)) expect_equal(as.sequence(c(NA,NA)), c(1,1)) }) "],
["package-data.html", "32 Package Data 32.1 Summary: 32.2 Accessing data 32.3 Packages used in development", " 32 Package Data 32.1 Summary: Files with the following extensions will be read by R as data: plain R code in .R and .r files are sourced using source() text tables in .tab, .txt, .csv files are read using read() ** objects in R image files: .RData, .rda are loaded using load() capitalization matters all objects in foo.RData are loaded into environment pro: easiset way to store objects in R format con: format is application (R) specific (discussed in #318) Details are in ?data, which is mostly a copy of Data section of Writing R Extensions. 32.2 Accessing data Data in the [data] directory will be accessed in the following ways, efficient way: (especially for large data sets) using the data function: data(foo) # accesses data with, e.g. load(foo.RData), read(foo.csv), or source(foo.R) easy way: by adding the following line to the package DESCRIPTION: note: this should be used with caution or it can cause difficulty as discussed in redmine issue #1118 LazyData: TRUE From the R help page: Currently, a limited number of data formats can be accessed using the data function by placing one of the following filetypes in a packages’ data directory: * files ending .R or .r are source()d in, with the R working directory changed temporarily to the directory containing the respective file. (data ensures that the utils package is attached, in case it had been run via utils::data.) * files ending .RData or .rda are load()ed. * files ending .tab, .txt or .TXT are read using read.table(..., header = TRUE), and hence result in a data frame. * files ending .csv or .CSV are read using read.table(..., header = TRUE, sep = ';'), and also result in a data frame. If your data does not fall in those 4 categories, or you can use the system.file function to get access to the data: system.file("data", "ed.trait.dictionary.csv", package="PEcAn.utils") [1] "/home/kooper/R/x86_64-pc-linux-gnu-library/2.15/PEcAn.utils/data/ed.trait.dictionary.csv" The arguments are folder, filename(s) and then package. It will return the fully qualified path name to a file in a package, in this case it points to the trait data. This is almost the same as the data function, however we can now use any function to read the file, such as read.csv instead of read.csv2 which seems to be the default of data. This also allows us to store arbitrary files in the data folder, such as the the bug file and load it when we need it. 32.2.0.1 Examples of data in PEcAn packages Redmine issue #1060 added time constants in source:utils/data/time.constants.RData outputs: [/modules/uncertainties/data/output.RData] parameter samples [/modules/uncertainties/data/samples.RData] 32.3 Packages used in development 32.3.1 roxygen2 Used to document code. See instructions under [[R#Coding_Style|Coding Style]] 32.3.2 devtools Provides functions to simplify development Documentation: The R devtools packate load_all("pkg") document("pkg") test("pkg") install("pkg") build("pkg") other tips for devtools (from the documentation): Adding the following to your ~/.Rprofile will load devtools when running R in interactive mode: # load devtools by default if (interactive()) { suppressMessages(require(devtools)) } Adding the following to your .Rpackages will allow devtools to recognize package by folder name, rather than directory path # in this example, devhome is the pecan trunk directory devhome <- "/home/dlebauer/R-dev/pecandev/" list( default = function(x) { file.path(devhome, x, x) }, "utils" = paste(devhome, "pecandev/utils", sep = "") "common" = paste(devhome, "pecandev/common", sep = "") "all" = paste(devhome, "pecandev/all", sep = "") "ed" = paste(devhome, "pecandev/models/ed", sep = "") "uncertainty" = paste(devhome, "modules/uncertainty", sep = "") "meta.analysis" = paste(devhome, "modules/meta.analysis", sep = "") "db" = paste(devhome, "db", sep = "") ) Now, devtools can take pkg as an argument instead of /path/to/pkg/, e.g. so you can use build("pkg") instead of build("/path/to/pkg/") "],
["logging.html", "33 Logging 33.1 PEcAn logging functions 33.2 Other R logging packages 33.3 Example Usage 33.4 Inputs 33.5 Outputs", " 33 Logging During development we often add many print statements to check to see how the code is doing, what is happening, what intermediate results there are etc. When done with the development it would be nice to turn this additional code off, but have the ability to quickly turn it back on if we discover a problem. This is where logging comes into play. Logging allows us to use “rules” to say what information should be shown. For example when I am working on the code to create graphs, I do not have to see any debugging information about the SQL command being sent, however trying to figure out what goes wrong during a SQL statement it would be nice to show the SQL statements without adding any additional code. 33.1 PEcAn logging functions These logger family of functions are more sophisticated, and can be used in place of stop, warn, print, and similar functions. The logger functions make it easier to print to a system log file. 33.1.1 Examples The file test.logger.R provides descriptive examples This query provides an current overview of functions that use logging logger functions (in order of increasing level): logger.debug logger.info logger.warn logger.error the logger.setLevel function sets the level at which a message will be printed logger.setLevel("DEBUG") will print messages from all logger functions logger.setLevel("ERROR") will only print messages from logger.error logger.setLevel("INFO") and logger.setLevel("WARN") shows messages fromlogger.and higher functions, e.g.logger.setLevel(“WARN”)shows messages fromlogger.warnandlogger.error` logger.setLevel("OFF") suppresses all logger messages To print all messages to console, use logger.setUseConsole(TRUE) 33.1.2 Related Issues (requires Redmine developer account) #1071 How to handle errors? #1222 Ignore warnings You can use @logger.setLevel(“ERROR”)@ to only show error messages. All the code that does not use logger will not be filtered. 33.2 Other R logging packages This section is for reference - these functions should not be used in PEcAn, as they are redundant with the logger.* functions described above R does provide a basic logging capability using stop, warning and message. These allow to print message (and stop execution in case of stop). However there is not an easy method to redirect the logging information to a file, or turn the logging information on and off. This is where one of the following packages comes into play. The packages themselves are very similar since they try to emulate log4j. Both of the following packages use a hierarchic loggers, meaning that if you change the level of displayed level of logging at one level all levels below it will update their logging. 33.2.1 logging The logging development is done at http://logging.r-forge.r-project.org/ and more information is located at http://cran.r-project.org/web/packages/logging/index.html . To install use the following command: install.packages(\"logging\", repos=\"http://R-Forge.R-project.org\") This has my preference pure based on documentation. 33.2.2 futile The second logging package is http://cran.r-project.org/web/packages/futile.logger/ and is eerily similar to logging (as a matter of fact logging is based on futile). 33.3 Example Usage To be able to use the loggers there needs to be some initialization done. Neither package allows to read it from a configuration file, so we might want to use the pecan.xml file to set it up. The setup will always be somewhat the same: # load library library(logging) logReset() # add handlers, responsible for actually printing/saving the messages addHandler(writeToConsole) addHandler(writeToFile, file="file.log") # setup root logger with INFO setLevel('INFO') # make all of PEcAn print debug messages setLevel('DEBUG', getLogger('PEcAn')) # only print info and above for the SQL part of PEcAn setLevel('INFO', getLogger('PEcAn.SQL')) To now use logging in the code you can use the following code: pl <- getLogger('PEcAn.MetaAnalysis.function1') pl$info("This is an INFO message.") pl$debug("The value for x=%d", x) pl$error("Something bad happened and I am scared now.") or loginfo("This is an INFO message.", logger="PEcAn.MetaAnalysis.function1") logdebug("The value for x=%d", x, logger="PEcAn.MetaAnalysis.function1") logerror("Something bad happened and I am scared now.", logger="PEcAn.MetaAnalysis.function1") 33.4 Inputs see Adding-an-Input-Converter 33.4.1 Defining new input formats New formats can be defined on the ‘formats’ page of BETYdb After creating a new format, the contents should be defined by specifying the BETYdb variable name and the name used in the file/ 33.5 Outputs created by model2netcdf functions based on format used by MsTMIP "],
["installing-pecan.html", "34 Installing PEcAn 34.1 Installing Prerequisites 34.2 Install models 34.3 CLM-FATES 34.4 PEcAn Installation 34.5 Installing PEcAn 34.6 Installing BETY", " 34 Installing PEcAn We recommend that new users download the PEcAn VM instead of creating from one from scratch. These instructions are provided to document how to install PEcAn on different Operating Systems. 34.1 Installing Prerequisites Check specific notes in: VM Creation Installing PEcAn Data Enabling Remote Execution OS specific installation instructions: Installing PEcAn Ubuntu Installing PEcAn OSX Installing PEcAn CentOS Installing PEcAn RedHat deprecated To update PEcAn see: Updating PEcAn Updating BETY The rest of the instructions assumes you have done the appropriate steps in each of the above guides. 34.1.1 Set R_LIBS_USER CRAN Reference # point R to personal lib folder echo 'export R_LIBS_USER=${HOME}/R/library' >> ~/.profile source ~/.profile mkdir -p ${R_LIBS_USER} 34.2 Install models 34.2.1 ED2 34.2.1.1 ED2.2 r46 (used in PEcAn manuscript) # ---------------------------------------------------------------------- # Get version r46 with a few patches for ubuntu cd curl -o ED.r46.tgz http://isda.ncsa.illinois.edu/~kooper/EBI/ED.r46.tgz tar zxf ED.r46.tgz rm ED.r46.tgz # ---------------------------------------------------------------------- # configure and compile ed cd ~/ED.r46/ED/build/bin curl -o include.mk.VM http://isda.ncsa.illinois.edu/~kooper/EBI/include.mk.opt.`uname -s` make OPT=VM sudo cp ../ed_2.1-VM /usr/local/bin/ed2.r46 Perform a test run using pre configured ED settings for ED2.2 r46 # ---------------------------------------------------------------------- # Create sample run cd mkdir testrun.ed.r46 cd testrun.ed.r46 curl -o ED2IN http://isda.ncsa.illinois.edu/~kooper/EBI/ED2IN.r46 sed -i -e "s#\\$HOME#$HOME#" ED2IN curl -o config.xml http://isda.ncsa.illinois.edu/~kooper/EBI/config.r46.xml # execute test run time ed2.r46 34.2.1.2 ED 2.2 r82 cd curl -o ED.r82.tgz http://isda.ncsa.illinois.edu/~kooper/EBI/ED.r82.tgz tar zxf ED.r82.tgz rm ED.r82.tgz cd ED.r82 curl -o ED.r82.patch http://isda.ncsa.illinois.edu/~kooper/EBI/ED.r82.patch patch -p1 < ED.r82.patch cd ED/build/bin curl -o include.mk.VM http://isda.ncsa.illinois.edu/~kooper/EBI/include.mk.opt.`uname -s` make OPT=VM sudo cp ../ed_2.1-VM /usr/local/bin/ed2.r82 Perform a test run using pre configured ED settings for ED2.2 r82 cd mkdir testrun.ed.r82 cd testrun.ed.r82 curl -o ED2IN http://isda.ncsa.illinois.edu/~kooper/EBI/ED2IN.r82 sed -i -e "s#\\$HOME#$HOME#" ED2IN curl -o config.xml http://isda.ncsa.illinois.edu/~kooper/EBI/config.r82.xml # execute test run time ed2.r82 34.2.1.3 ED 2.2 bleeding edge cd git clone https://github.com/EDmodel/ED2.git cd ED2/ED/build/bin curl -o include.mk.VM http://isda.ncsa.illinois.edu/~kooper/EBI/include.mk.opt.`uname -s` ./generate_deps.sh make OPT=VM sudo cp ../ed_2.1-VM /usr/local/bin/ed2.git 34.2.2 SIPNET 34.2.2.1 Sipnet Installation cd curl -o sipnet_unk.tar.gz http://isda.ncsa.illinois.edu/~kooper/EBI/sipnet_unk.tar.gz tar zxf sipnet_unk.tar.gz rm sipnet_unk.tar.gz cd sipnet_unk make sudo cp sipnet /usr/local/bin/sipnet.runk 34.2.2.2 SIPNET testrun cd curl -o testrun.sipnet.tar.gz http://isda.ncsa.illinois.edu/~kooper/EBI/testrun.sipnet.tar.gz tar zxf testrun.sipnet.tar.gz rm testrun.sipnet.tar.gz cd testrun.sipnet sipnet.runk 34.2.3 BioCro 34.2.3.1 Installation # Public echo 'devtools::install_github("ebimodeling/biocro")' | R --vanilla # Development: echo 'devtools::install_github("ebimodeling/biocro-dev")' | R --vanilla BioCro Developers: request from [@dlebauer on GitHub](https://github.com/dlebauer) 34.2.4 Linkages 34.2.4.1 Installation # Public echo 'devtools::install_github("araiho/linkages_package")' | R --vanilla 34.2.5 DALEC 34.2.5.1 Installation cd curl -o dalec_EnKF_pub.tgz http://isda.ncsa.illinois.edu/~kooper/EBI/dalec_EnKF_pub.tgz tar zxf dalec_EnKF_pub.tgz rm dalec_EnKF_pub.tgz cd dalec_EnKF_pub make dalec_EnKF make dalec_seqMH sudo cp dalec_EnKF dalec_seqMH /usr/local/bin 34.2.6 LINKAGES 34.2.6.1 Installation #FORTRAN VERSION cd git clone https://github.com/araiho/Linkages.git cd Linkages gfortran -o linkages linkages.f sudo cp linkages /usr/local/bin/linkages.git #R Version git clone https://github.com/araiho/linkages_package.git R CMD INSTALL --no-multiarch --with-keep.source linkages_package 34.2.7 CLM 4.5 The version of CLM installed on PEcAn is the ORNL branch provided by Dan Ricciuto. This version includes Dan’s point-level CLM processing scripts Download the code (~300M compressed), input data (1.7GB compressed and expands to 14 GB), and a few misc inputs. mkdir models cd models wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clm4_5_1_r085.tar.gz wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clm/ccsm_inputdata.tar.gz tar -xvzf clm4_5* tar -xvzf ccsm_inputdata.tar.gz #Parameter file: mkdir /home/carya/models/ccsm_inputdata/lnd/clm2/paramdata cd /home/carya/models/ccsm_inputdata/lnd/clm2/paramdata wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clm_params.c130821.nc wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clm_params.c140423.nc #Domain file: cd /home/carya/models/ccsm_inputdata/share/domains/domain.clm/ wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/domain.lnd.1x1pt_US-UMB_navy.nc #Aggregated met data file: cd /home/carya/models/ccsm_inputdata/atm/datm7/CLM1PT_data/1x1pt_US-UMB wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/all_hourly.nc ## lightning database cd /home/carya/models/ccsm_inputdata/atm/datm7/NASA_LIS/ wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clmforc.Li_2012_climo1995-2011.T62.lnfm_Total_c140423.nc ## surface data cd /home/carya/models/ccsm_inputdata/lnd/clm2/surfdata wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clm/surfdata_360x720cru_simyr1850_c130927.nc cd /home/carya/models/ccsm_inputdata/lnd/clm2/surfdata_map wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clm/surfdata_1x1pt_US-UMB_I1850CLM45CN_simyr1850.nc_new mv surfdata_1x1pt_US-UMB_I1850CLM45CN_simyr1850.nc_new surfdata_1x1pt_US-UMB_I1850CLM45CN_simyr1850.nc Required libraries sudo apt-get install mercurial csh tcsh subversion cmake sudo ln -s /usr/bin/make /usr/bin/gmake Compile and build default inputs cd ~/carya/models/clm4_5_1_r085/scripts python runCLM.py --site US-UMB ––compset I1850CLM45CN --mach ubuntu --ccsm_input /home/carya/models/ccsm_inputdata --tstep 1 --nopointdata --coldstart --cpl_bypass --clean_build 34.2.7.1 CLM Test Run You will see a new directory in scripts: US-UMB_I1850CLM45CN Enter this directory and run (you shouldn’t have to do this normally, but there is a bug with the python script and doing this ensures all files get to the right place): ./US-UMB_I1850CLM45CN.build Next you are ready to go to the run directory: /home/carya/models/clm4_5_1_r085/run/US-UMB_I1850CLM45CN/run Open to edit file: datm.streams.txt.CLM1PT.CLM_USRDAT and check file paths such that all paths start with /home/carya/models/ccsm_inputdata From this directory, launch the executable that resides in the bld directory: /home/carya/clm4_5_1_r085/run/US-UMB_I1850CLM45CN/bld/cesm.exe not sure this was the right location, but wherever the executable is You should begin to see output files that look like this: US-UMB_I1850CLM45CN.clm2.h0.yyyy-mm.nc (yyyy is year, mm is month) These are netcdf files containing monthly averages of lots of variables. The lnd_in file in the run directory can be modified to change the output file frequency and variables. 34.3 CLM-FATES Prerequisites sudo apt-get upgrade libnetcdf-dev sudo apt-get install subversion sudo apt-get install csh sudo apt-get install cmake sudo ln -s /usr/bin/make /usr/bin/gmake sudo rm /bin/sh sudo ln -s /bin/bash /bin/sh wget https://github.com/Unidata/netcdf-fortran/archive/v4.4.4.tar.gz cd netcdf-4.4.4 ./configure make sudo make install you might need to mess around with installing netcdf and netcdf-fortran to get a version FATES likes… Get code from Github (currently private) and go to cime/scripts directory git clone git@github.com:NGEET/ed-clm.git cd ed-clm/cime/scripts/ Within CLM-FATES, to be able to build an executable we need to create a reference run. We’ll also use this reference run to grab defaults from, so we’ll be registering the location of both the reference case (location of executable, scripts, etc) and the reference inputs with the PEcAn database. To begin, copy reference run script from pecan cp ~/pecan/models/fates/inst/create_1x1_ref_case.sh . Edit reference case script to set NETCDF_HOME, CROOT (reference run case), DIN_LOC_ROOT (reference run inputs). Also, make sure DIN_LOC_ROOT exists as FATES will not create it itself. Then run the script ./create_1x1_ref_case.sh Be aware that this script WILL ask you for your password on the NCAR server to download the reference case input data (the guest password may work, haven’t tried this). If it gives an error at the pio stage check the log, but the most likely error is it being unable to find a version of netcdf it likes. Once FATES is installed, set the whole reference case directory as the Model path (leave filename blank) and set the whole inputs directory as an Input with format clm_defaults. 34.4 PEcAn Installation PEcAn is the software package that ties all the pieces together. 34.5 Installing PEcAn Download, compile and install PEcAn # download pecan cd git clone https://github.com/PecanProject/pecan.git # compile pecan cd pecan ./scripts/build.sh --dependencies Following will run a small script to setup some hooks to prevent people from using the pecan demo user account to check in any code. # prevent pecan user from checking in code ./scripts/create-hooks.sh 34.6 Installing BETY 34.6.1 Install Postgres See OS-specific instructions for installing Postgres + PostGIS * Ubuntu * OSX * RedHat / CentOS 34.6.2 Install Database + Data note To install BETYdb without PEcAn, first download the load.bety.sh script # install database (code assumes password is bety) sudo -u postgres createuser -d -l -P -R -S bety sudo -u postgres createdb -O bety bety sudo -u postgres ./scripts/load.bety.sh -c YES -u YES -r 0 sudo -u postgres ./scripts/load.bety.sh -r 1 sudo -u postgres ./scripts/load.bety.sh -r 2 # configure for PEcAn web app (change password if needed) cp web/config.example.php web/config.php # add models to database (VM only) ./scripts/add.models.sh # add data to database ./scripts/add.data.sh # create outputs folder mkdir ~/output chmod 777 ~/output 34.6.3 Installing BETYdb Web Application There are two flavors of BETY, PHP and RUBY. The PHP version allows for a minimal interaction with the database while the RUBY version allows for full interaction with the database. 34.6.3.1 PHP version The php version comes with PEcAn and is already configured. 34.6.3.2 RUBY version The RUBY version requires a few extra packages to be installed first. Next we install the web app. # install bety cd git clone https://github.com/PecanProject/bety.git # install gems cd bety sudo gem2.0 install bundler bundle install --without development:test:javascript_testing:debug and configure BETY # create folders for upload folders mkdir paperclip/files paperclip/file_names chmod 777 paperclip/files paperclip/file_names # create folder for log files mkdir log touch log/production.log chmod 0666 log/production.log # fix configuration for vm cp config/additional_environment_vm.rb config/additional_environment.rb chmod go+w public/javascripts/cache/ # setup bety database configuration cat > config/database.yml << EOF production: adapter: postgis encoding: utf-8 reconnect: false database: bety pool: 5 username: bety password: bety EOF # setup login tokens cat > config/initializers/site_keys.rb << EOF REST_AUTH_SITE_KEY = 'thisisnotasecret' REST_AUTH_DIGEST_STRETCHES = 10 EOF 34.6.4 PEcAn Testrun Do the run, this assumes you have installed the BETY database, sites tar file and sipnet. # create folder cd mkdir testrun.pecan cd testrun.pecan # copy example of pecan workflow and configuration file cp ../pecan/tests/pecan32.sipnet.xml pecan.xml cp ../pecan/scripts/workflow.R workflow.R # exectute workflow rm -rf pecan ./workflow.R pecan.xml NB: pecan.xml is configured for the virtual machine, you will need to change the field from ‘/home/carya/’ to wherever you installed your ‘sites’, usually $HOME "],
["creating-a-virtual-machine.html", "35 Creating a Virtual Machine", " 35 Creating a Virtual Machine First create virtual machine # ---------------------------------------------------------------------- # CREATE VM USING FOLLOWING: # - VM NAME = PEcAn 32bit | PEcAn 64bit # - CPU = 1 # - MEMORY = 2GB # - DISK = 100GB # - HOSTNAME = pecan32/64 # - FULLNAME = PEcAn Demo User # - USERNAME = xxxxxxx # - PASSWORD = yyyyyyy # - PACKAGE = openssh # ---------------------------------------------------------------------- To enable tunnels run the following on the host machine, I used xx=64 for the 64 bit version of the VM and xx=32 for the 32 bit version of the VM: VBoxManage modifyvm "<virtual machine name>" --natpf1 "ssh,tcp,,xx22,,22" VBoxManage modifyvm "<virtual machine name>" --natpf1 "www,tcp,,xx80,,80" For the virtual machines created the following routes are created: VBoxManage modifyvm "PEcAn 32bit" --natpf1 "ssh,tcp,,3222,,22" VBoxManage modifyvm "PEcAn 32bit" --natpf1 "www,tcp,,3280,,80" VBoxManage modifyvm "PEcAn 64bit" --natpf1 "ssh,tcp,,6422,,22" VBoxManage modifyvm "PEcAn 64bit" --natpf1 "www,tcp,,6480,,80" Make sure machine is up to date. UBUNTU sudo apt-get update sudo apt-get -y dist-upgrade sudo reboot CENTOS/REDHAT sudo yum -y update sudo reboot Install compiler and other packages needed and install the tools. UBUNTU sudo apt-get -y install build-essential linux-headers-server dkms CENTOS/REDHAT sudo yum -y groupinstall "Development Tools" sudo yum -y install wget Install Virtual Box additions for better integration sudo mount /dev/cdrom /mnt sudo /mnt/VBoxLinuxAdditions.run sudo umount /mnt sudo usermod -a -G vboxsf carya Finishing up the machine Add a message to the login: sudo -s export PORT=$( hostname | sed 's/pecan//' ) cat > /etc/motd << EOF PEcAn version 1.4.3 For more information about: Pecan - http://pecanproject.org BETY - http://www.betydb.org For a list of all models currently navigate [here](../users_guide/basic_users_guide/models_table.md) You can access this system using a webbrowser at http://<hosting machine>:${PORT}80/ or using SSH at ssh -l carya -p ${PORT}22 <hosting machine> where <hosting machine> is the machine where the VM runs on. EOF exit Finishing up Script to clean the VM and remove as much as possible history cleanvm.sh wget -O ~/cleanvm.sh http://isda.ncsa.uiuc.edu/~kooper/EBI/cleanvm.sh chmod 755 ~/cleanvm.sh Make sure machine has SSH keys rc.local sudo wget -O /etc/rc.local http://isda.ncsa.illinois.edu/~kooper/EBI/rc.local Change the resolution of the console sudo sed -i -e 's/#GRUB_GFXMODE=640x480/GRUB_GFXMODE=1024x768/' /etc/default/grub sudo update-grub Once all done, stop the virtual machine history -c && ${HOME}/cleanvm.sh "],
["centosredhat.html", "36 CentOS/RedHat 36.1 Install build environment 36.2 Install Postgres 36.3 Apache Configuration PEcAn 36.4 Apache Configuration BETY 36.5 Rstudio-server 36.6 Additional packages", " 36 CentOS/RedHat These are specific notes for installing PEcAn on CentOS (7) and will be referenced from the main installing PEcAn page. You will at least need to install the build environment and Postgres sections. If you want to access the database/PEcAn using a web browser you will need to install Apache. To access the database using the BETY interface, you will need to have Ruby installed. This document also contains information on how to install the Rstudio server edition as well as any other packages that can be helpful. 36.1 Install build environment sudo -s # install packages needed for PEcAn yum -y groupinstall 'Development Tools' yum -y install git netcdf-fortran-openmpi-devel R bc curl libxml2-devel openssl-devel ed udunits2 udunits2-devel netcdf netcdf-devel gmp-devel python-devel gdal-devel proj-devel proj-epsg expect # jags yum -y install http://download.opensuse.org/repositories/home:/cornell_vrdc/CentOS_7/x86_64/jags3-3.4.0-54.1.x86_64.rpm yum -y install http://download.opensuse.org/repositories/home:/cornell_vrdc/CentOS_7/x86_64/jags3-devel-3.4.0-54.1.x86_64.rpm # fix include folder for udunits2 ln -s /usr/include/udunits2/* /usr/include/ # install packages needed for ED2 yum -y install environment-modules openmpi-bin libopenmpi-dev # install requirements for DALEC yum -y install gsl-devel # install packages for webserver yum -y install httpd php systemctl enable httpd systemctl start httpd firewall-cmd --zone=public --add-port=80/tcp --permanent firewall-cmd --reload # install packages to compile docs #apt-get -y install texinfo texlive-latex-base texlive-latex-extra texlive-fonts-recommended # install devtools echo 'install.packages("devtools", repos="http://cran.rstudio.com/")' | R --vanilla # done as root exit echo "module load mpi" >> ~/.bashrc module load mpi 36.2 Install Postgres sudo -s # point to latest PostgreSQL yum install -y epel-release yum -y install http://yum.postgresql.org/9.4/redhat/rhel-7-x86_64/pgdg-centos94-9.4-1.noarch.rpm # install packages for postgresql (using a newer version than default) yum -y install postgresql94-server postgresql94-contrib postgis2_94 postgresql94-devel # install following if you want to run pecan through the web yum -y install php-pgsql # enable bety user to login with trust by adding the following lines after # the ability of postgres user to login in /var/lib/pgsql/9.4/data/pg_hba.conf local all bety trust host all bety 127.0.0.1/32 trust host all bety ::1/128 trust # Create database /usr/pgsql-9.4/bin/postgresql94-setup initdb # Enable postgres systemctl enable postgresql-9.4 systemctl start postgresql-9.4 exit 36.3 Apache Configuration PEcAn # become root sudo -s # get index page rm /var/www/html/index.html ln -s /home/carya/pecan/documentation/index_vm.html /var/www/html/index.html # fix selinux context (does this need to be done after PEcAn is installed?) chcon -R -t httpd_sys_content_t /home/carya/pecan /home/carya/output # setup a redirect cat > /etc/httpd/conf.d/pecan.conf << EOF Alias /pecan /home/carya/pecan/web <Directory /home/carya/pecan/web> DirectoryIndex index.php Options +ExecCGI Require all granted </Directory> EOF a2enconf pecan /etc/init.d/apache2 restart # done as root exit 36.4 Apache Configuration BETY sudo -s # install all ruby related packages sudo curl --fail -sSLo /etc/yum.repos.d/passenger.repo https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger.repo yum -y install ruby ruby-devel mod_passenger # link static content ln -s /home/carya/bety/public /var/www/html/bety # fix GemFile echo 'gem "test-unit"' >> bety/Gemlile # fix selinux context (does this need to be done after bety is installed?) chcon -R -t httpd_sys_content_t /home/carya/bety # setup a redirect cat > /etc/httpd/conf.d/bety.conf << EOF RailsEnv production RailsBaseURI /bety PassengerRuby /usr/bin/ruby <Directory /var/www/html/bety> Options +FollowSymLinks Require all granted </Directory> EOF systemctl restart httpd 36.5 Rstudio-server NEED FIXING NOTE This will allow anybody to login to the machine through the rstudio interface and run any arbitrary code. The login used however is the same as the system login/password. Based on version of ubuntu 32/64 use either of the following 32bit only wget http://download2.rstudio.org/rstudio-server-0.98.1103-i386.deb 64bit only wget http://download2.rstudio.org/rstudio-server-0.98.1103-amd64.deb # bceome root sudo -s # install required packages apt-get -y install libapparmor1 apparmor-utils libssl0.9.8 # install rstudio dpkg -i rstudio-server-* rm rstudio-server-* echo "www-address=127.0.0.1" >> /etc/rstudio/rserver.conf echo "r-libs-user=~/R/library" >> /etc/rstudio/rsession.conf rstudio-server restart # setup rstudio forwarding in apache a2enmod proxy_http cat > /etc/apache2/conf-available/rstudio.conf << EOF ProxyPass /rstudio/ http://localhost:8787/ ProxyPassReverse /rstudio/ http://localhost:8787/ RedirectMatch permanent ^/rstudio$ /rstudio/ EOF a2enconf rstudio /etc/init.d/apache2 restart # all done, exit root exit 36.6 Additional packages NEED FIXING HDF5 Tools, netcdf, GDB and emacs sudo apt-get -y install hdf5-tools cdo nco netcdf-bin ncview gdb emacs ess nedit "],
["centos-rhel.html", "37 CentOS / RHEL 37.1 Install build environment 37.2 Apache Configuration 37.3 Install and configure Rstudio-server 37.4 Install Postgres", " 37 CentOS / RHEL These are specific notes for installing PEcAn on RedHat/CentOS and will be referenced from the main installing PEcAn page. You will at least need to install the build environment and Postgres sections. If you want to access the database/PEcAn using a web browser you will need to install Apache. To access the database using the BETY interface, you will need to have Ruby installed. This document also contains information on how to install the Rstudio server edition as well as any other packages that can be helpful. 37.1 Install build environment 37.1.0.1 Install and configure PostgreSQL, udunits2, NetCDF Reference: centoshelp.org yum install -y epel-release yum -y install http://yum.postgresql.org/9.4/redhat/rhel-7-x86_64/pgdg-centos94-9.4-1.noarch.rpm yum -y install git R postgresql94-server postgresql94-contrib postgis2_94 udunits2 netcdf /usr/pgsql-9.4/bin/postgresql94-setup initdb systemctl enable postgresql-9.4 systemctl start postgresql-9.4 37.1.0.2 Install and start Apache yum -y install httpd systemctl enable httpd systemctl start httpd 37.1.0.3 Install PHP sudo yum -y install php php-pgsql 37.1.0.4 Install ruby-netcdf gem cd $RUBY_APPLICATION_HOME export $NETCDF_URL=http://www.gfd-dennou.org/arch/ruby/products/ruby-netcdf/release/ruby-netcdf-0.6.6.tar.gz export $NETCDF_DIR=/usr/local/netcdf gem install narray export NARRAY_DIR="$(ls $GEM_HOME/gems | grep 'narray-')" export NARRAY_PATH="$GEM_HOME/gems/$NARRAY_DIR" cd $MY_RUBY_HOME/bin wget $NETCDF_URL -O ruby-netcdf.tgz tar zxf ruby-netcdf.tgz && cd ruby-netcdf-0.6.6/ ruby -rubygems extconf.rb --with-narray-include=$NARRAY_PATH --with-netcdf-dir=/usr/local/netcdf-4.3.0 sed -i 's|rb/$|rb|' Makefile make make install cd ../ && sudo rm -rf ruby-netcdf* cd $RUBY_APPLICATION bundle install --without development 37.2 Apache Configuration 37.3 Install and configure Rstudio-server based on Rstudio Server documentation add PATH=$PATH:/usr/sbin:/sbin to /etc/profile cat "PATH=$PATH:/usr/sbin:/sbin; export PATH" >> /etc/profile add rstudio.conf to /etc/httpd/conf.d/ wget https://gist.github.com/dlebauer/6921889/raw/d1e0f945228e5519afa6223d6f49d6e0617262bd/rstudio.conf sudo mv rstudio.conf /httpd/conf.d/ download and install server: wget http://download2.rstudio.org/rstudio-server-0.97.551-i686.rpm sudo yum install --nogpgcheck rstudio-server-0.97.551-i686.rpm restart server sudo httpd restart now you should be able to access http://<server>/rstudio 37.4 Install Postgres See documentation under the BETYdb Wiki "],
["mac-osx.html", "38 Mac OSX 38.1 Install build environment 38.2 Install Postgres 38.3 Additional installs 38.4 Apache Configuration 38.5 Ruby 38.6 Rstudio Server", " 38 Mac OSX These are specific notes for installing PEcAn on Mac OSX and will be referenced from the main installing PEcAn page. You will at least need to install the build environment and Postgres sections. If you want to access the database/PEcAn using a web browser you will need to install Apache. To access the database using the BETY interface, you will need to have Ruby installed. This document also contains information on how to install the Rstudio server edition as well as any other packages that can be helpful. 38.1 Install build environment # install R # download from http://cran.r-project.org/bin/macosx/ # install gfortran # download from http://cran.r-project.org/bin/macosx/tools/ # install OpenMPI curl -o openmpi-1.6.3.tar.gz http://www.open-mpi.org/software/ompi/v1.6/downloads/openmpi-1.6.3.tar.gz tar zxf openmpi-1.6.3.tar.gz cd openmpi-1.6.3 ./configure --prefix=/usr/local make all sudo make install cd .. # install szip curl -o szip-2.1-MacOSX-intel.tar.gz ftp://ftp.hdfgroup.org/lib-external/szip/2.1/bin/szip-2.1-MacOSX-intel.tar.gz tar zxf szip-2.1-MacOSX-intel.tar.gz sudo mv szip-2.1-MacOSX-intel /usr/local/szip # install HDF5 curl -o hdf5-1.8.11.tar.gz http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.11.tar.gz tar zxf hdf5-1.8.11.tar.gz cd hdf5-1.8.11 sed -i -e 's/-O3/-O0/g' config/gnu-flags ./configure --prefix=/usr/local/hdf5 --enable-fortran --enable-cxx --with-szlib=/usr/local/szip make # make check sudo make install # sudo make check-install cd .. 38.2 Install Postgres For those on a Mac I use the following app for postgresql which has postgis already installed (http://postgresapp.com/) To get postgis run the following commands in psql: ## Enable PostGIS (includes raster) CREATE EXTENSION postgis; ## Enable Topology CREATE EXTENSION postgis_topology; ## fuzzy matching needed for Tiger CREATE EXTENSION fuzzystrmatch; ## Enable US Tiger Geocoder CREATE EXTENSION postgis_tiger_geocoder; To check your postgis run the following command again in psql: SELECT PostGIS_full_version(); 38.3 Additional installs 38.3.1 Install JAGS Download JAGS from http://sourceforge.net/projects/mcmc-jags/files/JAGS/3.x/Mac%20OS%20X/JAGS-Mavericks-3.4.0.dmg/download 38.3.2 Install udunits Installing udunits-2 on MacOSX is done from source. download most recent version of Udunits here instructions for compiling from source curl -o udunits-2.1.24.tar.gz ftp://ftp.unidata.ucar.edu/pub/udunits/udunits-2.1.24.tar.gz tar zxf udunits-2.1.24.tar.gz cd udunits-2.1.24 ./configure make sudo make install 38.4 Apache Configuration Mac does not support pdo/postgresql by default. The easiest way to install is use: http://php-osx.liip.ch/ To enable pecan to run from your webserver. cat > /etc/apache2/others/pecan.conf << EOF Alias /pecan ${PWD}/pecan/web <Directory ${PWD}/pecan/web> DirectoryIndex index.php Options +All Require all granted </Directory> EOF 38.5 Ruby The default version of ruby should work. Or use JewelryBox. 38.6 Rstudio Server For the mac you can download Rstudio Desktop. "],
["ubuntu.html", "39 Ubuntu 39.1 Install build environment 39.2 Install Postgres 39.3 Apache Configuration BETY 39.4 Rstudio-server 39.5 Additional packages 39.6 Download and Install Models 39.7 Install instructions", " 39 Ubuntu These are specific notes for installing PEcAn on Ubuntu (14.04) and will be referenced from the main installing PEcAn page. You will at least need to install the build environment and Postgres sections. If you want to access the database/PEcAn using a web browser you will need to install Apache. To access the database using the BETY interface, you will need to have Ruby installed. This document also contains information on how to install the Rstudio server edition as well as any other packages that can be helpful. 39.1 Install build environment sudo -s # point to latest R echo "deb http://cran.rstudio.com/bin/linux/ubuntu `lsb_release -s -c`/" > /etc/apt/sources.list.d/R.list apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9 # update package list apt-get -y update # install packages needed for PEcAn apt-get -y install build-essential gfortran git r-base-core r-base-dev jags liblapack-dev libnetcdf-dev netcdf-bin bc libcurl4-gnutls-dev curl udunits-bin libudunits2-dev libgmp-dev python-dev libgdal1-dev libproj-dev expect # install packages needed for ED2 apt-get -y install openmpi-bin libopenmpi-dev # install requirements for DALEC apt-get -y install libgsl0-dev # install packages for webserver apt-get -y install apache2 libapache2-mod-php5 php5 # install packages to compile docs apt-get -y install texinfo texlive-latex-base texlive-latex-extra texlive-fonts-recommended # install devtools echo 'install.packages("devtools", repos="http://cran.rstudio.com/")' | R --vanilla # done as root exit 39.2 Install Postgres Documentation: http://trac.osgeo.org/postgis/wiki/UsersWikiPostGIS21UbuntuPGSQL93Apt sudo -s # point to latest PostgreSQL echo "deb http://apt.postgresql.org/pub/repos/apt `lsb_release -s -c`-pgdg main" > /etc/apt/sources.list.d/pgdg.list wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - # update package list apt-get -y update # install packages for postgresql (using a newer version than default) apt-get -y install libdbd-pgsql postgresql postgresql-client libpq-dev postgresql-9.4-postgis-2.1 postgresql-9.4-postgis-2.1-scripts # install following if you want to run pecan through the web apt-get -y install php5-pgsql # enable bety user to login with trust by adding the following lines after # the ability of postgres user to login in /etc/postgresql/9.4/main/pg_hba.conf local   all             bety                                    trust host    all             bety            127.0.0.1/32            trust host    all             bety            ::1/128                 trust # Once done restart postgresql /etc/init.d/postgresql restart exit To install the BETYdb database .. ## Apache Configuration PEcAn # become root sudo -s # get index page rm /var/www/html/index.html ln -s ${HOME}/pecan/documentation/index_vm.html /var/www/html/index.html # setup a redirect cat > /etc/apache2/conf-available/pecan.conf << EOF Alias /pecan ${HOME}/pecan/web <Directory ${HOME}/pecan/web> DirectoryIndex index.php Options +ExecCGI Require all granted </Directory> EOF a2enconf pecan /etc/init.d/apache2 restart # done as root exit 39.3 Apache Configuration BETY sudo -s # install all ruby related packages apt-get -y install ruby2.0 ruby2.0-dev libapache2-mod-passenger # link static content ln -s ${HOME}/bety/public /var/www/html/bety # setup a redirect cat > /etc/apache2/conf-available/bety.conf << EOF RailsEnv production RailsBaseURI /bety PassengerRuby /usr/bin/ruby2.0 <Directory /var/www/html/bety> Options +FollowSymLinks Require all granted </Directory> EOF a2enconf bety /etc/init.d/apache2 restart 39.4 Rstudio-server NOTE This will allow anybody to login to the machine through the rstudio interface and run any arbitrary code. The login used however is the same as the system login/password. Based on version of ubuntu 32/64 use either of the following 32bit only wget http://download2.rstudio.org/rstudio-server-0.98.1103-i386.deb 64bit only wget http://download2.rstudio.org/rstudio-server-0.98.1103-amd64.deb # bceome root sudo -s # install required packages apt-get -y install libapparmor1 apparmor-utils libssl0.9.8 # install rstudio dpkg -i rstudio-server-* rm rstudio-server-* echo "www-address=127.0.0.1" >> /etc/rstudio/rserver.conf echo "r-libs-user=~/R/library" >> /etc/rstudio/rsession.conf rstudio-server restart # setup rstudio forwarding in apache a2enmod proxy_http cat > /etc/apache2/conf-available/rstudio.conf << EOF ProxyPass /rstudio/ http://localhost:8787/ ProxyPassReverse /rstudio/ http://localhost:8787/ RedirectMatch permanent ^/rstudio$ /rstudio/ EOF a2enconf rstudio /etc/init.d/apache2 restart # all done, exit root exit 39.5 Additional packages HDF5 Tools, netcdf, GDB and emacs sudo apt-get -y install hdf5-tools cdo nco netcdf-bin ncview gdb emacs ess nedit 39.6 Download and Install Models This page contains instructions on how to download and install ecosystem models that have been or are being coupled to PEcAn. These instructions have been tested on the PEcAn unbuntu VM. Commands may vary on other operating systems. Also, some model downloads require permissions before downloading, making them unavailable to the general public. Please contact the PEcAn team if you would like access to a model that is not already installed on the default PEcAn VM. BioCro CLM 4.5 DALEC ED2 GDAY JULES LINKAGES LPJ-GUESS MAESPA SIPNET 39.7 Install instructions 39.7.1 ED2 39.7.1.1 ED2.2 r46 (used in PEcAn manuscript) # ---------------------------------------------------------------------- # Get version r46 with a few patches for ubuntu cd curl -o ED.r46.tgz http://isda.ncsa.illinois.edu/~kooper/EBI/ED.r46.tgz tar zxf ED.r46.tgz rm ED.r46.tgz # ---------------------------------------------------------------------- # configure and compile ed cd ~/ED.r46/ED/build/bin curl -o include.mk.VM http://isda.ncsa.illinois.edu/~kooper/EBI/include.mk.opt.`uname -s` make OPT=VM sudo cp ../ed_2.1-VM /usr/local/bin/ed2.r46 Perform a test run using pre configured ED settings for ED2.2 r46 # ---------------------------------------------------------------------- # Create sample run cd mkdir testrun.ed.r46 cd testrun.ed.r46 curl -o ED2IN http://isda.ncsa.illinois.edu/~kooper/EBI/ED2IN.r46 sed -i -e "s#\\$HOME#$HOME#" ED2IN curl -o config.xml http://isda.ncsa.illinois.edu/~kooper/EBI/config.r46.xml # execute test run time ed2.r46 39.7.1.2 ED 2.2 r82 cd curl -o ED.r82.tgz http://isda.ncsa.illinois.edu/~kooper/EBI/ED.r82.tgz tar zxf ED.r82.tgz rm ED.r82.tgz cd ED.r82 curl -o ED.r82.patch http://isda.ncsa.illinois.edu/~kooper/EBI/ED.r82.patch patch -p1 < ED.r82.patch cd ED/build/bin curl -o include.mk.VM http://isda.ncsa.illinois.edu/~kooper/EBI/include.mk.opt.`uname -s` make OPT=VM sudo cp ../ed_2.1-VM /usr/local/bin/ed2.r82 Perform a test run using pre configured ED settings for ED2.2 r82 cd mkdir testrun.ed.r82 cd testrun.ed.r82 curl -o ED2IN http://isda.ncsa.illinois.edu/~kooper/EBI/ED2IN.r82 sed -i -e "s#\\$HOME#$HOME#" ED2IN curl -o config.xml http://isda.ncsa.illinois.edu/~kooper/EBI/config.r82.xml # execute test run time ed2.r82 39.7.1.3 ED 2.2 bleeding edge cd git clone https://github.com/EDmodel/ED2.git cd ED2/ED/build/bin curl -o include.mk.VM http://isda.ncsa.illinois.edu/~kooper/EBI/include.mk.opt.`uname -s` ./generate_deps.sh make OPT=VM sudo cp ../ed_2.1-VM /usr/local/bin/ed2.git 39.7.2 SIPNET 39.7.2.1 Sipnet Installation cd curl -o sipnet_unk.tar.gz http://isda.ncsa.illinois.edu/~kooper/EBI/sipnet_unk.tar.gz tar zxf sipnet_unk.tar.gz rm sipnet_unk.tar.gz cd sipnet_unk make sudo cp sipnet /usr/local/bin/sipnet.runk 39.7.2.2 SIPNET testrun cd curl -o testrun.sipnet.tar.gz http://isda.ncsa.illinois.edu/~kooper/EBI/testrun.sipnet.tar.gz tar zxf testrun.sipnet.tar.gz rm testrun.sipnet.tar.gz cd testrun.sipnet sipnet.runk 39.7.3 BioCro 39.7.3.1 Installation # Public echo 'devtools::install_github("ebimodeling/biocro")' | R --vanilla # Development: echo 'devtools::install_github("ebimodeling/biocro-dev")' | R --vanilla BioCro Developers: request from [@dlebauer on GitHub](https://github.com/dlebauer) 39.7.4 Linkages 39.7.4.1 Installation # Public echo 'devtools::install_github("araiho/linkages_package")' | R --vanilla 39.7.5 DALEC 39.7.5.1 Installation cd curl -o dalec_EnKF_pub.tgz http://isda.ncsa.illinois.edu/~kooper/EBI/dalec_EnKF_pub.tgz tar zxf dalec_EnKF_pub.tgz rm dalec_EnKF_pub.tgz cd dalec_EnKF_pub make dalec_EnKF make dalec_seqMH sudo cp dalec_EnKF dalec_seqMH /usr/local/bin 39.7.6 LINKAGES 39.7.6.1 Installation #FORTRAN VERSION cd git clone https://github.com/araiho/Linkages.git cd Linkages gfortran -o linkages linkages.f sudo cp linkages /usr/local/bin/linkages.git #R Version git clone https://github.com/araiho/linkages_package.git R CMD INSTALL --no-multiarch --with-keep.source linkages_package 39.7.7 CLM 4.5 The version of CLM installed on PEcAn is the ORNL branch provided by Dan Ricciuto. This version includes Dan’s point-level CLM processing scripts Download the code (~300M compressed), input data (1.7GB compressed and expands to 14 GB), and a few misc inputs. mkdir models cd models wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clm4_5_1_r085.tar.gz wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clm/ccsm_inputdata.tar.gz tar -xvzf clm4_5* tar -xvzf ccsm_inputdata.tar.gz #Parameter file: mkdir /home/carya/models/ccsm_inputdata/lnd/clm2/paramdata cd /home/carya/models/ccsm_inputdata/lnd/clm2/paramdata wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clm_params.c130821.nc wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clm_params.c140423.nc #Domain file: cd /home/carya/models/ccsm_inputdata/share/domains/domain.clm/ wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/domain.lnd.1x1pt_US-UMB_navy.nc #Aggregated met data file: cd /home/carya/models/ccsm_inputdata/atm/datm7/CLM1PT_data/1x1pt_US-UMB wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/all_hourly.nc ## lightning database cd /home/carya/models/ccsm_inputdata/atm/datm7/NASA_LIS/ wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clmforc.Li_2012_climo1995-2011.T62.lnfm_Total_c140423.nc ## surface data cd /home/carya/models/ccsm_inputdata/lnd/clm2/surfdata wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clm/surfdata_360x720cru_simyr1850_c130927.nc cd /home/carya/models/ccsm_inputdata/lnd/clm2/surfdata_map wget ftp://nacp.ornl.gov/synthesis/2008/firenze/site/clm/surfdata_1x1pt_US-UMB_I1850CLM45CN_simyr1850.nc_new mv surfdata_1x1pt_US-UMB_I1850CLM45CN_simyr1850.nc_new surfdata_1x1pt_US-UMB_I1850CLM45CN_simyr1850.nc Required libraries sudo apt-get install mercurial csh tcsh subversion cmake sudo ln -s /usr/bin/make /usr/bin/gmake Compile and build default inputs cd ~/carya/models/clm4_5_1_r085/scripts python runCLM.py --site US-UMB ––compset I1850CLM45CN --mach ubuntu --ccsm_input /home/carya/models/ccsm_inputdata --tstep 1 --nopointdata --coldstart --cpl_bypass --clean_build 39.7.7.1 CLM Test Run You will see a new directory in scripts: US-UMB_I1850CLM45CN Enter this directory and run (you shouldn’t have to do this normally, but there is a bug with the python script and doing this ensures all files get to the right place): ./US-UMB_I1850CLM45CN.build Next you are ready to go to the run directory: /home/carya/models/clm4_5_1_r085/run/US-UMB_I1850CLM45CN/run Open to edit file: datm.streams.txt.CLM1PT.CLM_USRDAT and check file paths such that all paths start with /home/carya/models/ccsm_inputdata From this directory, launch the executable that resides in the bld directory: /home/carya/clm4_5_1_r085/run/US-UMB_I1850CLM45CN/bld/cesm.exe not sure this was the right location, but wherever the executable is You should begin to see output files that look like this: US-UMB_I1850CLM45CN.clm2.h0.yyyy-mm.nc (yyyy is year, mm is month) These are netcdf files containing monthly averages of lots of variables. The lnd_in file in the run directory can be modified to change the output file frequency and variables. 39.7.8 JULES INSTALL STEPS: 1) Download JULES and FCM JULES: Model requires registration to download. Not to be put on PEcAn VM Registration: https://jules.jchmr.org/software-and-documentation Documentation: http://jules-lsm.github.io/vn4.2/index.html FCM: https://github.com/metomi/fcm/ wget https://github.com/metomi/fcm/archive/2015.05.0.tar.gz edit makefile ```bash open etc/fcm-make/make.cfg set JULES_NETCDF = actual instead of dummy set path (e.g. /usr/) and lib_path /lib64 to netCDF libraries ``` compile JULES cd etc/fcm-make/ {path.to.fcm}/fcm make -f etc/fcm-make/make.cfg --new UBUNTU VERSION: installed without having to add any perl libraries #perl stuff that I had to install on pecan2 not PEcAN VM sudo yum install perl-Digest-SHA sudo yum install perl-Time-modules sudo yum install cpan curl -L http://cpanmin.us | perl - --sudo App::cpanminus sudo cpanm Time/Piece.pm sudo cpanm IO/Uncompress/Gunzip.pm Executable is under build/bin/jules.exe Example rundir: examples/point_loobos 39.7.9 MAESPA Navigate to a directory you would like store MAESPA and run the following: git clone https://bitbucket.org/remkoduursma/maespa.git cd maespa make maespa.out is your executable. Example input files can be found in the inpufiles directory. Executing measpa.out from within one of the example directories will produce output. MAESPA developers have also developed a wrapper package called Maeswrap. The usual R package installation method install.packages may present issues with downloading an unpacking a dependency package called rgl. Here are a couple of solutions: 39.7.9.1 Solution 1 ### From the Command Line sudo apt-get install r-cran-rgl then from within R install.packages("Maeswrap") 39.7.9.2 Solution 2 ### From the Command line sudo apt-get install libglu1-mesa-dev then from within R install.packages("Maeswrap") 39.7.10 GDAY Navigate to a directory you would like to store GDAY and run the following: git clone https://github.com/mdekauwe/GDAY.git cd GDAY cd src make gday is your executable. 39.7.11 LPJ-GUESS Instructions to download source code wget http://stormbringerii.nateko.lu.se/public/guess_download/guess_3.1.tar.gz tar -xvzf guess_3.1.tar.gz "],
["porting-vm-to-aws.html", "40 Porting VM to AWS", " 40 Porting VM to AWS The following are Mike’s rough notes from a first attempt to port the PEcAn VM to the AWS. This was done on a Mac These notes are based on following the instructions here 40.0.1 Convert PEcAn VM AWS allows upload of files as VMDK but the default PEcAn VM is in OVA format If you haven’t done so already, download the PEcAn VM Split the OVA file into OVF and VMDK files tar xf <ovafile> 40.0.2 Set up an account on AWS After you have an account you need to set up a user and save your access key and secret key In my case I created a user named ‘carya’ Note: the key that ended up working had to be made at https://console.aws.amazon.com/iam/home#security_credential, not the link above. 40.0.3 Install EC2 command line tools wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip sudo mkdir /usr/local/ec2 sudo unzip ec2-api-tools.zip -d /usr/local/ec2 If need be, download and install JDK export JAVA_HOME=$(/usr/libexec/java_home) export EC2_HOME=/usr/local/ec2/ec2-api-tools-<version> export PATH=$PATH:$EC2_HOME/bin Then set your user credentials as environment variables: export AWS_ACCESS_KEY=xxxxxxxxxxxxxx export AWS_SECRET_KEY=xxxxxxxxxxxxxxxxxxxxxx Note: you may want to add all the variables set in the above EXPORT commands above into your .bashrc or equivalent. 40.0.4 Create an AWS S3 ‘bucket’ to upload VM to Go to https://console.aws.amazon.com/s3 and click “Create Bucket” In my case I named the bucket ‘pecan’ 40.0.5 Upload In the code below, make sure to change the PEcAn version, the name of the bucket, and the name of the region. Make sure that the architecture matches the version of PEcAn you downloaded (i386 for 32 bit, x86_64 for 64 bit). Also, you may want to choose a considerably larger instance type. The one chosen below is that corresponding to the AWS Free Tier ec2-import-instance PEcAn32bit_1.2.6-disk1.vmdk --instance-type t2.micro --format VMDK --architecture i386 --platform Linux --bucket pecan --region us-east-1 --owner-akid $AWS_ACCESS_KEY --owner-sak $AWS_SECRET_KEY Make sure to note the ID of the image since you’ll need it to check the VM status. Once the image is uploaded it will take a while (typically about an hour) for Amazon to convert the image to one it can run. You can check on this progress by running ec2-describe-conversion-tasks <image.ID> 40.0.6 Configuring the VM On the EC2 management webpage, https://console.aws.amazon.com/ec2, if you select Instances on the left hand side (LHS) you should be able to see your new PEcAn image as an option under Launch Instance. Before launching, you will want to update the firewall to open up additional ports that PEcAn needs – specifically port 80 for the webpage. Port 22 (ssh/sftp) should be open by default. Under “Security Groups” select “Inbound” then “Edit” and then add “HTTP”. Select “Elastic IPs” on the LHS, and “Allocate New Address” in order to create a public IP for your VM. Next, select “Network Interfaces” on the LHS and then under Actions select “Associate Addresses” then choose the Elastic IP you just created. See also http://docs.aws.amazon.com/AmazonVPC/latest/GettingStartedGuide/GetStarted.html 40.0.7 Set up multiple instances (optional) For info on setting up multiple instances with load balancing see: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/gs-ec2VPC.html Select “Load Balancers” on the LHS, click on “Create Load Balancer”, follow Wizard keeping defaults. To be able to launch multiple VMs: Under “Instances” convert VM to an Image. When done, select Launch, enable multiple instances, and associate with the previous security group. Once running, go back to “Load Balancers” and add the instances to the load balancer. Each instance can be accessed individually by it’s own public IP, but external users should access the system more generally via the Load Balancers DNS. 40.0.8 Booting the VM Return to “Instances” using the menu on the LHS. To boot the VM select “Actions” then “Instance State” then “Start”. In the future, once you have the VM loaded and configured this last step is the only one you will need to repeat to turn your VM on and off. The menu provided should specify the Public IP where the VM has launched "],
["installing-and-configuring-shiny-for-pecan.html", "41 Installing and configuring Shiny for PEcAn 41.1 Install the Shiny R package and Shiny server 41.2 CentOS 41.3 Ubuntu 41.4 Modify the shiny configuration file 41.5 Set the httpd proxy 41.6 Create a symbolic link to the shiny server service 41.7 Start the shiny server and restart httpd 41.8 Troubleshooting 41.9 Further reading", " 41 Installing and configuring Shiny for PEcAn authors - Alexey Shiklomanov - Rob Kooper NOTE: Instructions are only tested for CentOS 6.5. NOTE: Pretty much every step here requires root access. 41.1 Install the Shiny R package and Shiny server The shiny R package can be installed directly in R via install.packages("shiny"). Download and install the Shiny server binary (link). 41.2 CentOS wget https://download3.rstudio.org/centos5.9/x86_64/shiny-server-1.4.2.786-rh5-x86_64.rpm sudo yum install --nogpgcheck shiny-server-1.4.2.786-rh5-x86_64.rpm 41.3 Ubuntu NOTE: The additional gdebi dependence may be optional and dpkg may work just fine, but I haven’t tested this out. $ sudo apt-get install gdebi-core $ wget https://download3.rstudio.org/ubuntu-12.04/x86_64/shiny-server-1.4.2.786-amd64.deb $ sudo gdebi shiny-server-1.4.2.786-amd64.deb 41.4 Modify the shiny configuration file The Shiny configuration file is located in /etc/shiny-server/shiny-server.conf. Comment out the entire file and add the following, replacing <username> with your user name and <location> with the URL location you want for your app. This will allow you to run Shiny apps from your web browser at https://your.server.edu/shiny/your-location run as shiny; server { listen 3838; location /<location>/ { run as <username>; site_dir /path/to/your/shiny/app; log_dir /var/log/shiny-server; directory_index on; } } For example, my configuration on the old test-pecan looks like this. run as shiny; server { listen 3838; location /ashiklom/ { run as ashiklom; site_dir /home/ashiklom/fs-data/pecan/shiny/; log_dir /var/log/shiny-server; directory_index on; } } …and I can access my Shiny apps at, for instance, https://test-pecan.bu.edu/shiny/ashiklom/workflowPlots. You can add as many location <loc> { ... } fields as you would like. run as shiny; server { listen 3838; location /ashiklom/ { ... } location /bety/ { ... } } 41.5 Set the httpd proxy Create a file /etc/httpd/conf.d/shiny.conf containing the following proxy settings: ProxyPass /shiny/ http://localhost:3838/ ProxyPassReverse /shiny/ http://localhost:3838/ RedirectMatch permanent ^/shiny$ /shiny/ 41.6 Create a symbolic link to the shiny server service sudo ln -s /opt/shiny-server/config/init.d/redhat/shiny-server /etc/init.d 41.7 Start the shiny server and restart httpd sudo service shiny-server stop sudo service shiny-server start sudo service httpd restart You can check that Shiny is running with service shiny-server status. 41.8 Troubleshooting Refer to the log files for shiny (/var/log/shiny-server.log) and httpd (/var/log/httpd/error-log). 41.9 Further reading Shiny server configuration reference "],
["faq.html", "42 FAQ", " 42 FAQ "]
]