PEcAn - PEcAn v1.8.0

Added

• Created a new soilgrids function to extract the mean soil organic carbon profile with associated undertainty values at each depth for any lat/long location (#3040). Function was created for the CMS SDA workflow

• PEcAn.all gains new function pecan_version, which reports for each PEcAn
  package the version that was provided with this release and the version that
  is currently installed. Use it in scripts to record your system state for
  reproducibility, or for debugging.

• Added a new function unit_is_parseable in PEcAn.utils to replace udunits2::ud.is.parseable.
  (#3002; @nanu1605)

• Initial LDNDC model coupling

• Added optional process_partial argument to model2netcdf.ED2() to allow it to process existing output from failed runs.

• Added litter_mass_content_of_water to standard_vars table and model2netcdf.SIPNET

• Added all SIPNET state variables to read_restart and write_restart

• Added Observation preparation functions into the SDA workflow, which supports AGB, LAI, Soil Carbon, and Soil moisture.

• Added an optional pfts argument to PEcAn.uncertainty::run.sensitivity.analysis() so that sensitivity analysis and variance decomposition can be run on a subset of PFTs defined in settings if desired (#3155).

• Added new features of the SDA function including: 1) allow user-defined free-run mode;

2. allow user-defined parallel mode for the qsub submission; 3) allow user-defined email option to report the progress.

• PEcAnAssimSequential::GET.MultiSite() now supports the parallelization of multi-chain MCMC sampling with the fully randomized inits function.
• Added the new feature of the block-based SDA workflow, which supports the parallel computation.
• Added new SDA workflow for the 342 North America anchor sites.
• Added new feature of preparing initial conditions for MODIS LAI, AGB, ISCN SOC, and soil moisture across NA anchor sites.
• Added GEDI AGB preparation workflow.
• Added new feature of downloading datasets from the NASA DAAC ORNL database.

Fixed

• Replaced the broken ameriflux download pipeline with the new amerifluxr package's function.
• Fix broken build caused by BioCro updates #2925
• rstudio was not working behind traefik.
• plots now work in docker containers
• when specifying diferent rstudio user, dev setup would mount pecan folder in wrong path.
• fixed the pagination error in the next_page field of the workflows, inputs and runs Rest API.
• bugs in model2ncdf.ED2() that were causing it to both error and also only
  convert data for a single PFT fixed (#1329, #2974, #2981)
• Code cleanup in PEcAn.MA to protect against global namespace pollution (#2965, #2973; @nanu1605)
• Fixed vignettes and cleaned up internal warnings in PEcAn.DB (#2966, #2971; @nanu1605).
• Updated unit conversions throughout PEcAn to use the units R package
  instead of the unmaintained udunits2.
  Note that both units and udunits2 interface with the same underlying
  compiled code, so the udunits2 system library is still required.
  (#2989; @nanu1605)
• Occasionally some run directories were not getting copied over to remote hosts. This should be fixed now (#3025)
• Fixed a bug with ED2 where ED2IN tags supplied in settings that were not in the ED2IN template file were not getting added to ED2IN config files (#3034, #3033)
• Fixed a bug where warnings were printed for file paths on remote servers even when they did exist (#3020)
• Fixed bug in model2netcdf.SIPNET that caused LE to be overestimaed 10^3 (#3036)
• Added an updated ED2IN template file, models/ed/inst/ED2IN.r2.2.0.github, to reflect new variables in the development version of ED2
• PEcAn.data.land::gSSURGO.Query has been updated to work again after changes to the gSSURGO API.
• PEcAn.settings::read.settings() now prints a warning when falling back on default "pecan.xml" if the named inputfile doesn't exist.
• fqdn() can access hostname on Windows (#3044 fixed by #3058)
• write.config.xml.ED2() wasn't using the tag in settings correctly (#3080)
• runModule.get.trait.data() now correctly respects the settings$database$bety$write setting (#2968)
• Fixed a bug in model2netcdf.ed2() where .nc file connections were being closed multiple times, printing warnings (#3078)
• Fixed a bug causing the model2netcdf.ED2() step in jobs.sh to be incorrectly written (#3075)
• Fixed a bug where plant_min_temp trait value wasn't being converted from ºC to K when writing config file for ED2 (#3110)
• Fixed a bug in PEcAn.ED2::read_E_files() affecting PEcAn.ED2::model2netcdf.ED2() that resulted in incorrect calculations (#3126)
• Fixed a bug in PEcAn.utils::ud_convert() where it failed with objects of class "difftime" introduced by refactoring to use the units package instead of udunits (#3012)
• The propagation of aqq and bqq for the SINGLE Q type has been corrected.
• The issue where the indirect constraints will be increase with the increase of the covariance sizes.
• Updated URL for MERRA downloads (#2888)

Changed

• The default build of PEcAn now uses R 4.1. PEcAn is also tested daily on R 4.2 and weekly on R-devel. R 4.0 and older are no longer tested and will probably not work.
• Some changes have been made to the docker-compose file. Check your .env file to
  see if you need to change any of these:
  • TRAEFIK_HOST is now TRAEFIK_HOSTNAME without the Host: and ;, for example if you have
    TRAEFIK_HOST=Host:pecan.example.com; you will need to change this to
    TRAEFIK_HOST=pecan.example.com.
  • TRAEFIK_IPFILTER is no longer used, and should be removed.
  • TRAEFIK_HTTP_PORT now defaults to port 80
  • TRAEFIK_HTTPS_PORT now defaults to port 443
  • TRAEFIK_ACME_ENABLE is no longer used. If you want to use https you will need to add
    docker-compose.https.yml to your command line. You can use the script compose.sh
    to start PEcAn with https.
  • TRAEFIK_HTTPS_OPTIONS is no longer used, this is the default when you use https.
  • TRAEFIK_HTTP_REDIRECT is no longer used, this is the default when you use https.
• Updated Dockerfile for sipnet/maespa/template to use pecan/models:tag to build.
• The yr and yfiles arguments of PEcAn.ED2::read_E_files() are no longer used
  and the simulation date is extracted from the names of the .h5 files output by ED2.
• DDBH (change in DBH over time) is no longer extracted and summarized from
  monthly -E- files by PEcAn.ED2::model2netcdf.ED2(). We are not sure it makes
  sense to summarize this variable across cohorts of different sizes.
• PEcAn.SIPNET::model2netcdf.SIPNET can now export full year nc files by using
  the cdo_setup argument in the template job file. In detail, people will need
  to specify `cdosetup = "module load cdo/2.0.6"`` in the host section. More details
  are in the Create_Multi_settings.R script. (#3052)
• PEcAn.settings::read.settings() now strips comments so HTML style comments (e.g. <!-- a comment -->) are now allowed in pecan.xml files
• PEcAn.logger::setLevel() now invisibly returns the previously set logger level
• Warning messages for model2netcdf.ed2() coming from ncdf4::ncvar_put() now are prepended with the variable name for easier debugging (#3078)
• Database connections consistently use DBI::dbConnect instead of the deprecated dplyr::src_postgres (#2881). This change should be invisible to most users, but it involved converting a lot of internal variables from bety$con to con. If you see errors involving these symbols it means we missed a place, so please report them as bugs.
• PEcAn.utils::download.url argument retry404 is now renamed to retry and
  now functions as intended (it was being ignored completely before).
• We have begun the process of relicensing the PEcAn packages from the previous NCSA license to BSD-3, with the consent of all contributors. PEcAn.logger is now distributed as BSD-3; others will be changed as their authors sign off.
• Skipped ICOS and MERRA download tests when running in github actions
• Converted .zenodo.json to CITATION.cff
• Using traefik 2.5 instead of 1.7
• api is now open by default (was auth required in the past)
• Installation instructions updated in documentation
• PEcAn.assim.sequential is renamed to PEcAnAssimSequential
• convert.input is moved from PEcAn.utils to PEcAn.DB and renamed as convert_input.
  This was needed to resolve a cyclic dependency between PEcAn.DB and PEcAn.utils.
  (#3026; @nanu1605)
• Internal changes to keep up to date with tidyselect v1.2.0
• The PEcAn.utils::download.file() function has now been renamed to PEcAn.utils::download_file()
• The regrid() and grid2netcdf() function from PEcAn.utils have been moved to PEcAn.data.remote package.
• Web is now build using PHP 8 since PHP 7 is EOL.

Removed

• the check image (used in kubernetes) is removed, please use ncsa/checks instead.
• Unused (and apparently long-broken) function PEcAn.data.land::find.land has been removed.
• No longer building r136 sipnet docker image.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper 10 months ago

PEcAn - PEcAn 1.7.2 Release

Due to dependencies, PEcAn is now using R 4.0.3 for Docker images.

This is a major change:

• Newer version of R
• Ubuntu 20.04 instead of Debian.

Fixed

• Removed sender.py and now allow the submission of workflows from inside the rstudio container.
• Use TRAEFIK_FRONTEND_RULE in compose file and TRAEFIK_HOST in env.example, using TRAEFIK_HOST everywhere now. Make sure TRAEFIK_HOST is used in .env
• Use initial biomass pools for Sorghum and Setaria #2495, #2496
• PEcAn.DB::betyConnect() is now smarter, and will try to use either config.php or environment variables to create a connection. It has switched to use db.open helper function (#2632).
• PEcAn.utils::tranformstats() assumed the statistic names column of its input was a factor. It now accepts character too, and returns the same class given as input (#2545).
• fixed and added tests for get.rh function in PEcAn.data.atmosphere
• Invalid .zenodo.json that broke automatic archiving on Zenodo (b56ef53)
• Fixed a filehandle leak in multi-year runs of PEcAn.BIOCRO::met2model.BIOCRO: It was only closing the last input file it processed (#2485).
• Fix issue with cruncep download: use netcdf subset (ncss) method instead of opendap (#2424).
• The parse option to PEcAn.utils::read_web_config had no effect when expand was TRUE (#2421).
• Fixed a typo that made PEcAn.DB::symmetric_setdiff falsely report no differences (#2428).
• sipnet2netcdf will now only extract the data for the year requested (#2187)
• Fixed Priors vignette (#2439).
• When building sipnet model would not set correct model version
• Update pecan/depends docker image to have latest Roxygen and devtools.
• Update ED docker build, will now build version 2.2.0 and git
• Do not override meta-analysis settings random-effects = FALSE https://github.com/PecanProject/pecan/pull/2625
• model2netcdf.ED2 no longer detecting which varibles names -T- files have based on ED2 version (#2623)
• Changed docker-compose.yml to use user & group IDs of the operating system user (#2572)
• gSSURGO file download now added as inputs into BETY through extract_soil_gssurgo (#2666)
• ensure Tleaf converted to K for temperature corrections in PEcAn.photosynthesis::fitA (#2726)
• fix bug in summarize.result to output stat, which is needed to turn on RE in the meta-analysis (#2753)
• ensure that control treatments always receives the random effect index of 1; rename madata.Rdata to jagged.data.Rdata and include database ids and names useful for calculating parameter estimates by treatment (#2756)
• ensure that existing meta-analysis results can be used for pfts with cultivars (#2761)
• Major code cleanup by GSoC student @moki1202, fixing many check warnings across 10 packages
  (#2771, #2773, #2774, #2775, #2805, #2815, #2826, #2830, #2857)

Changed

• Removed deprecated mstmip_vars and mstmip_local; now all functions use the combined standard_vars.csv
• RabbitMQ is set to be 3.8 since the 3.9 version can no longer be configured with environment variables.
• Removed old api, now split into rpecanapi and apps/api.
• Replaced tmvtnorm package with TruncatedNormal package for speed up per #2621.
• Continuous integration changes: Added experimental GitHub Actions CI builds (#2544), streamlined Travis CI builds, added a fourth R version (second-newest old release; currently R 3.5) to Travis test matrix (#2592).
• Functions that update database entries no longer pass created_at or updated_at timestamps. The database now updates these itself and ensures they are consistently in UTC (#1083).
• PEcAn.DB::insert_table now uses DBI::dbAppendTable internally instead of manually constructed SQL (#2552).
• Rebuilt documentation using Roxygen 7. Readers get nicer formatting of usage sections, writers get more flexible behavior when inheriting parameters and less hassle when maintaining namespaces (#2524).
• Renamed functions that looked like S3 methods but were not:
  • PEcAn.priors: plot.posterior.density->plot_posterior.density, plot.prior.density->plot_prior.density, plot.trait->plot_trait (#2439).
  • PEcAn.visualization: plot.netcdf->plot_netcdf (#2526).
  • PEcAn.assim.sequential: Remote.Sync.launcher -> Remote_Sync_launcher (#2652)
• Stricter package checking: make check and CI builds will now fail if R CMD check returns any ERRORs or any "newly-added" WARNINGs or NOTEs. "Newly-added" is determined by strict string comparison against a check result saved 2019-09-03; messages that exist in the reference result do not break the build but will be fixed as time allows in future refactorings (#2404).
• No longer writing an arbitrary num for each PFT, this was breaking ED runs potentially.
• The pecan/data container has no longer hardcoded path for postgres
• PEcAn.JULES: Removed dependency on ncdf4.helpers package, which has been removed from CRAN (#2511).
• data.remote: Arguments to the function call_MODIS() have been changed (issue #2519).
• Changed precipitaion downscale in PEcAn.data.atmosphere::download.NOAA_GEFS_downscale. Precipitation was being downscaled via a spline which was causing fake rain events. Instead the 6 hr precipitation flux values from GEFS are preserved with 0's filling in the hours between.
• Changed dbfile.input.insert to work with inputs (i.e soils) that don't have start and end dates associated with them
• Default behavior for stop_on_error is now TRUE for non-ensemble runs; i.e., workflows that run only one model simulation (or omit the ensemble XML group altogether) will fail if the model run fails. For ensemble runs, the old behavior is preserved; i.e., workflows will continue even if one of the model runs failed. This behavior can also be manually controlled by setting the new run -> stop_on_error XML tag to TRUE or FALSE.
• Several functions have been moved out of PEcAn.utils into other packages (#2830, #2857):
  • ensemble.filename, get.results, runModule.get.results, read.sa.output, sensitivity.filename,
    and write.sa.configs have been moved to PEcAn.uncertainty.
  • create.base.plot, dhist, plot_data and theme_border have been moved to PEcAn.visualizaton.

Added

• BioCro can export Aboveground Biomass (#2790)
• Functionality for generating the same ensemble parameter sets with randtoolbox functions.
• Functionality for joint sampling from the posteriors using randtoolbox functions.
• BASGRA-SDA couplers.
• Now creates docker images during a PR, when merged it will push them to docker hub and github packages
• New functionality to the PEcAn API to GET information about PFTs, formats & sites, submit workflows in XML or JSON formats & download relevant inputs/outputs/files related to runs & workflows (#2674 #2665 #2662 #2655)
• Functions to send/receive messages to/from rabbitmq.
• Documentation in DEV-INTRO.md on development in a docker environment (#2553)
• PEcAn API that can be used to talk to PEcAn servers. Endpoints to GET the details about the server that user is talking to, PEcAn models, workflows & runs. Authetication enabled. (#2631)
• New versioned ED2IN template: ED2IN.2.2.0 (#2143) (replaces ED2IN.git)
• model_info.json and Dockerfile to template (#2567)
• Dockerize BASGRA_N model.
• Basic coupling for models BASGRA_N and STICS.
• PEcAn.priors now exports functions priorfig and plot_densities (#2439).
• Models monitoring container for Docker now shows a webpage with models it has seen
• Added small container to check if certain services are up, used as initi container for kubernetes
• Documentation how to run ED using singularity
• PEcAn.DB gains new function get_postgres_envvars, which tries to look up connection parameters from Postgres environment variables (if they are set) and return them as a list ready to be passed to db.open. It should be especially useful when writing tests that need to run on systems with many different database configurations (#2541).
• New shiny application to show database synchronization status (shiny/dbsync)
• Ability to run with MERRA-2 meteorology (reanalysis product based on GEOS-5 model)
• Ability to run with ICOS Ecosystem products

Removed

• Removed travis integration
• Removed the sugarcane and db folders from web, this removes the simple DB editor in the web folder. (#2532)
• Removed ED2IN.git (#2599) 'definitely going to break things for people' - but they can still use PEcAn <=1.7.1
• Database maintenance scripts vacuum.bety.sh and reindex.bety.sh have been moved to the BeTY database repository (#2563).
• Scripts dump.pgsql.sh and dump.mysql.sh have been deleted. See the "BeTY database administration" chapter of the PEcAn documentation for current recommendations (#2563).
• Old dependency management scripts check.dependencies.sh, update.dependencies.sh, and install_deps.R have been deleted. Use generate_dependencies.R and the automatic dependency handling built into make install instead (#2563).
• Deprecated copies of functions previously moved to other packages have been removed from PEcAn.utils (#2830):
  • do_conversions and runModule.run.write.configs, run.write.configs. These are now in PEcAn.workflow
  • get.ensemble.samples, read.ensemble.output, write.ensemble.configs. These are now in PEcAn.uncertainty
  • logger.debug, logger.error, logger.getLevel, logger.info, logger.setLevel,
    logger.setOutputFile, logger.setQuitOnSevere, logger.setWidth, logger.severe, logger.warn.
    These are now in PEcAn.logger

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper over 3 years ago

PEcAn - PEcAn 1.7.1 release

Fixed

• Replace deprecated rlang::UQ syntax with the recommended !!
• Explicitly use PEcAn.uncertainty::read.ensemble.output in PEcAn.utils::get.results. Otherwise, it would sometimes use the deprecated PEcAn.utils::read.ensemble.output version.
• PEcAn.ED2::met2model.ED2 now skips processing of years for which all output files are already present (unless overwrite = TRUE). This prevents a lot of unnecessary work when extending an existing ED met record.
• Fixed issue that prevented modellauncher from working properly #2262
• Use explicit namespacing (package::function) throughout PEcAn.meta.analysis. Otherwise, many of these functions would fail when trying to run a meta-analysis outside of the PEcAn workflow (i.e. without having loaded the packages first) (#2351).
• Standardize how PEcAn.DB tests create database connections, and make sure tests work with both the newer Postgres and older PostgreSQL drivers (#2351).
• Meta-analysis = "AUTO" now correctly skips the meta analysis if the PFT definition has not changed (#1217).
• Replace deprecated rlang::UQ syntax with the recommended !!
• Explicitly use PEcAn.uncertainty::read.ensemble.output in PEcAn.utils::get.results. Otherwise, it would sometimes use the deprecated PEcAn.utils::read.ensemble.output version.
• History page would not pass the hostname parameter when showing a running workflow, this would result in the running page showing an error.

Changed

• Updated modules/rtm PROSPECT docs
• Updated models/sipnet/R/model2netcdf.SIPNET.R to address issues in PR #2254
• Improved testing (#2281). Automatic Travis CI builds of PEcAn on are now run using three versions of R in parallel. This should mean fewer issues with new releases and better backwards compatibility, but note that we still only guarantee full compatibility with the current release version of R. The tested versions are:
  • release, the current public release of R (currently R 3.5). Build failures in this version are fixed before merging the change that caused them. When we say PEcAn is fully tested and working, this is the build we mean.
  • devel, the newest available development build of R. We will fix issues with this version before the next major R release.
  • oldrel, the previous major release of R (currently R 3.4). We will fix issues with this version as time allows, but we do not guarantee that it will stay compatible.
• Reverting back from PR (#2137) to fix issues with MAAT wrappers.
• Moved docker files for models into model specific folder, for example Dockerfile for sipnet now is in models/sipnet/Dockerfile.
• PEcAn.utils:
  • Remove, or make "Suggests", a bunch of relatively underutilized R package dependencies.
• Add template for documentation issues and add button to edit book.
• Conditionally skip unit tests for downloading raw met data or querying the database when the required remote connection is not available.
• Reorganization of docker folder
  • All dockerfiles now live in their own folder
  • scripts/generate_dependencies.R is now used to generate dependencies for make and docker
• In PEcAn.DB::get.trait.data, if trait.names is NULL or missing, use the traits for which at least one prior is available among the input list of PFTs. (Previously, we were getting this from the PEcAn.utils::trait.dictionary, which we are trying to deprecate #1747). (#2351)
• Cleanup and improve logging and code readability in parts of PEcAn.DB related to getting trait data, including replacing many manual database queries with dplyr calls.
• Reorganization of PEcAn documentation in accordance with isue #2253.
• SIPNET now is installed from the source code managed in git

Added

• Meta analysis functionality to not use greenhouse data.
• Dockerize the BioCro model.
• Added PRO4SAIL-D model, using existing 4SAIL src and coupling with PROSPECT-D Fortran code
• Models will not advertise themselvs, so no need to register them a-priori with the database #2158
• Added simple Docker container to show all containers that are available (http://localhost:8000/monitor/). This will also take care of registering the models with the BETY database.
• Added unit tests for met2model.<MODEL> functions for most models.
• Added MAESPA model to docker build
• PEcAn has more robust support for RPostgres::Postgres backend. The backend is officially supported by db.query, and basic workflows run top-to-bottom with the Postgres backend. However, RPostgreSQL is still the default until we do more robust testing of all modules.
• PEcAn.DB::db.query now optionally supports prepared statements (#395).
• New function PEcAn.DB::query_priors that expands the functionality of query.priors by (1) accepting PFTs by name or ID and (2) allowing the user to request all possible combinations of the input PFTs and traits (i.e. expand.grid(pfts, traits)) or just the pairwise combinations (i.e. pft[1]-trait[1], pft[2]-trait[2]). This function also comes with more robust error handling and a set of unit tests (#2351).
• New function PEcAn.DB::query_pfts for finding PFT IDs and types from the PFT name and (optionally) model type (#2351).
• Run Travis integration tests with both Postgres and PostgreSQL drivers (#2351).
• New function PEcAn.utils::load_local reads Rdata files into a named list (instead of into the current environment).

Removed

• Removed unused function PEcAn.visualization::points2county, thus removing many indirect dependencies by no longer importing the earth package.
• Removed package PEcAn.data.mining from the Make build. It can still be installed directly from R if desired, but is skipped by default because it is in early development, does not yet export any functions, and creates a dependency on the (large, often annoying to install) ImageMagick library.
• Fully deprecate support for MySQL database driver. Now, only PostgreSQL (and, experimentally, RPostgres) are supported. With this, remove RMySQL dependency in several places.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper over 5 years ago

PEcAn - PEcAn 1.7.0

Many updates to docker. Docker is now becoming more stable and can be used for new installations. Please check on the PEcAn slack channel if you want to start to use the docker installation.

This version replaces google maps in the web interface with leaflet. During this conversion we also sped up the rendering of the markers on the map. They are now grouped together, clicking a group marker will zoom into that area. You can always search in the box on the left to filter the markers if you know the sitename.

Below is a full list of all changes in this release.

Fixes

• Removed google maps and replaced with leaflet #2105
• Added ability to add a new site from web interface
• Small updated to models/ed/R/model2netcdf.ED2.R to fix issue realted to writing the time_bounds time attribute. Needed to add a check for which file types exitst (e.g. -E-, -T-, etc) and only write the appropriate attribute(s).
• Fixed error in read_web_config which would filter out all variables.
• Docker:
  • Make sure web interface posts RabbitMQ messages even after editing files (fixes #2151)
  • Can specify name of docker cluster using PECAN_FQDN and PECAN_NAME (fixes #2128)
  • Fixed issue where setting username/password for rabbitmq would break web submit (fixes #2185)
  • data image only registers sipnet and ed, has all data pre-downloaded
• ED2:
  • Fix processing of ed2in_tags from XML. Now numbers (e.g. <TRAIT_PLASTICITY_SCHEME>0</TRAIT_PLASTICITY_SCHEME>) and numeric vectors (e.g. <INCLUDE_THESE_PFT>9,10,11,12</INCLUDE_THESE_PFT>) are correctly written to ED2IN without quotes.

Added

• NEW FEATURE: PEcAn R API (PR #2192). Features include:
  • Modified docker/receiver.py to accept a pecan_json object containing a JSON version of the PEcAn settings. Can now Use RabbitMQ HTTP API (called from R with httr) to send a settings list (function submit_workflow)
  • Helper functions to make it easier to build the settings object, and to register a new workflow.
  • Helper functions for tracking workflow status
  • Helper functions for accessing workflow output files through THREDDS. All files are accessible through fileServer (basically, direct download), and NetCDF files are also readable through OpenDAP.
    • THREDDS catalog filter has been removed, so that all workflow outputs are available to THREDDS.
    • Added another datasetScan to the THREDDS catalog to search for dbfiles. Now, model inputs (e.g. meteorology files) are accessible via THREDDS as well.
• Lots of new documentation for running PEcAn using Docker
• Added Docker container with documentation #2160
• Download method (method) argument for data.atmosphere::download.CRUNCEP, which defaults to opendap (as it was), but can be switched to the slower but more robust NetCDF subset (ncss).
• In download.CRUNCEP, check target coordinate against the land-sea mask. If sea, pick the nearest land pixel within 1 degree of target. This facilitates doing runs at coastal sites that may get masked out.
• Added a prototype of the THREDDS data server (TDS) to the PEcAn Docker stack.
• Added portainer to the PEcAn Docker stack to easily look at running containers.
• Added ability to specify short name for a host (hostlist->displayname)
• ED2:
  • Add ability to pass arbitrary arguments to the ED binary through the pecan.xml (#2183; fixes #2146).
  • Add new model tag <all_pfts>. If "false" (default), set ED2IN's INCLUDE_THESE_PFT to only PFTs explicitly configured through PEcAn. If "true", use all 17 of ED2's PFTs.
  • Add new model tag <barebones_ed2in>. If "true", only write ED2IN tags, and do not include comment annotations. If "false" (default), try to transfer comments from ED2IN template to target ED2IN. Because of comments are written by matching line numbers, leaving this as "false" can lead to unexpected results whenever <ed2in_tags> contains tags missing from the ED2IN template.
  • Add some additional documentation for ED2 pecan.xml tags.

Removed

Changed

• PEcAn.utils::do_conversions has been moved to PEcAn.workflow::do_conversions.
  PEcAn.utils::do_conversions still works for now with a warning, but is deprecated and will be removed in the future.
• Docker:
  • Change base image for R code from r-base to rocker/tidyverse:3.5.1. This (1) saves build time (because many R packages and system dependencies are pre-installed), and (2) enhances reproducibility (because of the strict versioning rules of the rocker packages)
  • Re-factor web interface RabbitMQ create connections and post messages into their own PHP functions.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper over 6 years ago

PEcAn - Release v1.6.0

Fixes

• Fixed issue #2064 which sends one met path to write.sa.config.

• Fixed issue #1939 which corrects output time vector for FATES output

• Update to read.output to look for and read only PEcAn formatted .nc output based on the pecan standard filename format of YYYY.nc. Solves issues with models such as FATES and dvm-dos-tem where the original model output is also in .nc file format and was not ignored by read.output, causing errors with output parsing and plotting with Shiny. Removed deprecated function convert.outputs

• PEcAn.data.atmosphere:

  • download.Geostreams is pickier about formatting start/end datess, for fewer surprises in result timestamps
  • Fixed swapped lat/lon in met2CF.Geostreams
  • download.GFDL now records reference date in time units field, as required by the CF met standard
  • Reduced download.GFDL network load by not preloading dimension data
  • Fixed spurious No geonamesUsername set warning by updating geonames package to development version

• ED:

  • Change all history parameter files to have zero storage respiration

• missing_port_bety

• dataone_download.R:

  • Added functionality that spoofs our user address to prevent authentication errors with downloading files via wget.

• Could not specify the port for BETY in config.php. Can now use db_bety_port to specify port.

  • Added functionality that spoofs our user address to prevent authentication errors with downloading files via wget.

• Data_Ingest_App:

  • use updateSelectizeInput to populate selectizeInput with choices from BETYdb. This instantly loads the inputfields where other methods take minutes to load.

Added

• PEcAn.undertainty gains one new function (input.ens.gen) and three functions moved from PEcAn.utils (see "Changed" below)

• IC workflow now has functionality to generate ensembles.

• You can now generate ensembles for parameters and met separatly and using different methods.

• Soil process is now capable of reading in soil data from gSSURGO databse.

• In modules/rtm new function foursail() to interface with the 4SAIL Fortran code. To enable the use of 4SAIL with any version of PROSPECT (i.e. 4, 5, 5b, D) and custom soil/background reflectance inputs

• Shiny/Dependency explorer

• Explore the interdependencies between pecan packages/functions.

• From history you can now select an old run and show the curl command to re-execute this run. This only works with runs submitted through web interface right now.

• Experimental support for docker (#1028)

• dataone_download.R:

  • Added progress messages to indicate that the function is working during longer downloads via PEcAn logger.
  • Store path to downloaded data as newdir_D1 so that the download app can call this path.

• shiny/Data-Ingest

  • Download data from DataONE to a temporary directory, display the contents of that directory, and then move files from the temporary directory to dbfiles
  • Upload files from local machines (via drag and drop on some browsers), display files, and then move these files to a directory in dbfiles.
  • Spinner displays when class "shiny-busy" is invoked during the dataONE download process. In this way, users can be sure that the app is live and is processing their request.
  • Users can now input the name of the destination directory that they wish to create within dbfiles.
  • Updated Travis.yml to include librdf0-dev so that it can download redland, datapack, and dataone.
  • Added Data-Ingest UI (inputs, dbfiles, and formats record UI and some basic server side functionality are online)
  • Modularized input record, format record, and dbfiles record into shiny modules. This allows the app to be greatly simplified to two, single-page workflows. These functions can also be used "plug-and-play" style elsewhere in PEcAn shiny apps to load in data.
  • Replaced modularized input, format and dbfiles records with static "Ingest Workflow" page. On this page, the user can select either importing from dataONE or Uploading from local files. If creating a new format is necessary, the user can click "Create New Format" and a dropdown menu will walk them through this process.
  • Selected files now autofill name value in input record workflow
  • Store inputs and formats in the global environment
  • "Test BETY" button allows users create a record in BETY with dbfile.input.insert
  • Added input.format.vars to query the BETYdb
  • New File: helper.R
  • New Function: auto.name.directory This function uses the format_name and the site_id for a given input to create a directory name in the style of other dbfiles names.
  • Next Step buttons progress workflow programmatically
  • New formats-variables UI allows user to create a table of formats-variable records before completing the ingest process
  • Two separate complete Ingest buttons are rendered at the end of the workflow to trigger actions specific to local upload or dataONE download workflows. These buttons are rendered programmatically depending on the state of the selectInputMethod radio button.
  • Converted time inputs to properly merge startDate and startTime with EndDate and EndTime so they can be inserted into the start_date and end_date columns in BETYdb.
  • Error handling introduced using shinytoastr package
  • DESCRIPTION: Depends: PEcAn.visualization, shinytoastr, shinyWidgets, shinyjs

• pecan/base/db

  • New File: input.format.vars.R. This function registers the format and the (optional) formats_variables record using db_merge_into.

• data.atmosphere

  • check_met_input_file -- Check that target met file conforms to PEcAn meteorology data standard.
  • get_cf_variables_table -- Retrieve CF variables table as a data.frame

• docker:

  • Added updated docker container builds
    • Use docker.sh to create docker images
    • Use release.sh to push released images to push to docker registry (hub.docker.com by default)
  • Create pecan/depends docker image that holds all PEcAn dependencies
    • Needs to build seperatly, not part of the docker.sh build process to speed things up
    • Build using (cd docker ; docker build -t pecan/depends:latest -f Dockerfile.depends .)
  • docker-compose.yml file to bring up full PEcAn stack using docker
    • First time to start requires to install BETY database (see documentation)
  • SIPNET docker image which works with PEcAn docker stack
  • Data container that will download and install demo data in /data folder

Removed

• pecan.worldmap function no longer used, dropped from visualization package
• shiny/Data-Ingest/DESCRIPTION no longer DEPENDS on shinyFiles or shinycssloaders

Changed

• Fixed Git instructions and remote execution instructions.
• Five functions from PEcAn.utils functions have been moved to other packages. The versions in PEcAn.utils are deprecated, will not be updated with any new features, and will be removed in a future release.
  • run.write.configs and runModule.run.write.configs have been moved to PEcAn.workflow
  • read.ensemble.output, get.ensemble.samples and write.ensemble.configs have been moved to PEcAn.uncertainty
• Change the way packages are checked for and called in SHINY apps. DESCRIPTION files in SHINY apps are not the place to declare pacakge dpendencies.

Biosphere - Ecological and Environmental Modeling - R
Published by tonygardella over 6 years ago

PEcAn - 1.5.3

Fixes

• read.output now accepts date-times for start.year and end.year argument (#1887)
• read.output no longer assumes timestamps are in days since the beginning of the year
• Fixed xss issue in setup folder, now require users to login before accessing the setup folder scripts.
• Fixed issue where in website not all sites are shown #1884
• Fixed status page, should now be run from cronjob, creates static page
• Fixed bug that overwrote remote met file paths with local file paths
• PEcAnRTM:
  • Remove non-portable extensions from src/Makevars. This should make the package Windows-compatible.
• Fixed BrownDog shiny issues of removing site without geometry; fixing sites.length==0; removing old map markers when change input$type; fixing agreement bug when change input$type

Added

• Functionality to read pft-specific outputs and to run sensitivity analysis on pft-specific outputs.
• Ability to allow for insecure sync using -k flag
• Added information on how to join slack
• PEcAn.BIOCRO now supports BioCro version 1.0. BioCro 0.9x models should still work as before, but note parameter and weather format changes in the Changed section below.
• Added new model package (PEcAn.dvmdostem) and initial wrappers for integration of the DVM-DOS-TEM model and tested.
• PEcAn now supports PFTs whose members are cultivars rather than species, and will automatically restrict the meta-analysis to matching records, e.g. runs with a PFT containing only Panicum virgatum 'Cave-In-Rock' will not use observations from Panicum virgatum 'Alamo', but a PFT containing the whole species will use observations from both. However, there is not yet any BETYdb interface to create cultivar-PFTs other than manual SQL.
• New base package PEcAn.workflow, for functions used to perform the each major step of the analysis. These were previously scattered in other base packages.
• Added PR review time estimate to PR template
• New set of PEcAn.logger functions similar to stopifnot to facilitate assertive programming: severeifnot, errorifnot, warnifnot, infoifnot, debugifnot
• PEcAnRTM:
  • Exposed PROSPECT absorption coefficients and gpm() function ("generalized plate model"), facilitating experimentation with different absorption coefficients
  • Added spectra S3 class and methods for subsetting (e.g. myspec[[400:700]]), plotting (plot() and matplot()), and combining spectra by wavelength.
  • Added resample functions for quickly resampling spectra (and, more generally, vectors and functions) to different dimensions.
  • EDR API has been revised. Setup has been refactored from EDR via new setup_edr function, which relies on the ED utilities (see PEcAn.ED2 below), and the EDR function now focuses only on execution. Also, added new params2edr function to make it easy to convert complex EDR parameters list to flat parameter vector required by invert_bt (or other optimization functions).
• PEcAn.ED2:
  • New set of utilities for working with ED meteorology and vegetation inputs, and the ED2IN file. Existing PEcAn code has been revised to use these utilities.
• PEcAn.data.atmosphere:
  • New utilities for efficiently downloading NARR time series using THREDDS/OpenDAP

Removed

• Removed deprecated copies of PEcAn.utils::SafeList, PEcAn.utils::listToXml (both moved to PEcAn.settings in v 1.5.2), and PEcAn.utils::fqdn (moved to PEcAn.remote in 1.5.2). This fixes the masses of deprecation warnings in otherwise normal run logs (#1719).

Changed

• Updated wrappers for FATES model to work with recent CLM5 release. Updated write.config, job.template, and other associated files to work with CLM5 inputs and met drivers
• Updated model2netcdf.MAAT to use ncdf4::ncvar_def to define netCDF variables
• Fixed an remote code execution discovered by NCSA security team.
• Column name changes for newly generated biocromet csvs: SolarR is now solar and WS is now windspeed. Previously generated met files with the old names will still work for BioCro 0.9 runs, but will need to be renamed before using them with BioCro 1.0.
• write.configs.BIOCRO now requires a model version specification so that it can format parameters to match your version of BioCro. Set it in your Bety model record or in the model$revision field of your pecan.xml.
• When using BioCro 1.0 and no parameter file is given, PEcAn will make an attempt to find default parameters for your genus in the datasets provided by the BioCro package. Note that the default parameter files provided in models/biocro/inst/extdata/defaults will not work when using BioCro 1.0.
• Added documentation how to submit a run from the command line
• Updated models/maat to provide support for latest model code updates
• PEcAn.DB function rename.jags.columns renamed to rename_jags_columns to avoid conflict with S3 method naming conventions
• Replaced rhdf5 library with hdf5r, a more modern alternative that is available on CRAN.
• PEcAn.DB function runModule.get.trait.data has been moved to the new PEcAn.workflow package to avoid a circular package dependency between PEcAn.DB and PEcAn.settings.
• Major documentation refactoring. The documentation names are now directly tied to the order in which they are rendered, and all Rmd files in all subdirectories of the documentation source are rendered by default. The overall structure of the documentation has been revised for clarity and cohesiveness.
• Edited met2model.ED2 to not enforce leap years.
• Integrate demo 1 into basic user guide

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper almost 7 years ago

PEcAn - AGU 2017

Fixes

• Updated models/ed/data/pftmapping.csv to include two new BETYdb PFTs
• Simple fix to models/ed/R/write.configs.ed.R to properly align pss and css file prefix
• Fixed issue #1752 by updating the site.lst() function to include site.id=site$id instead of site.id=site, as site is an object not just the id
• Update to PEcAn.ED2::met2model.ED2 to fix issue with rhdf5::h5write. Bug fix to #1742
• Fixed write.config.xml.ED2 parsing of data/history* files
• PEcAn.utils now lazy-loads data for faster execution of functions that consult lookup tables, especially to_ncvar.
• Fixed incorrect PEcAn.BIOCRO daily and yearly results: Was calculating every row from whole simulation instead of that day (#1738)

Added

• New Dockerfile to create PEcAN specific container for SIPNET.

Removed

• Removed PEcAn.utils::model2netcdf, which has been deprecated since PEcAn 1.3.7. Use model2netcdf.<YOURMODEL> in the appropriate model package instead.

Changed

• Major namespace cleanup in the PEcAn.utils package. It now loads more quietly and is much less likely to mask functions in a package you loaded earlier.
• Moved many functions from PEcAn.utils into other PEcAn packages. The PEcAn.utils versions still work with a deprecation warning, but will be removed in next release.
  • listToXml and SafeList moved to PEcAn.settings
  • fqdn moved to PEcAn.remote
• PEcAnRTM: Removed effective sample size normalization from likelihood calculation. It was giving weird results.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper over 7 years ago

PEcAn - Version 1.5.1 : cleanup

Fixes

• Fixed hyperparameter draws in PDA
• Show workflowid in the URL when run is finshed and user clicks results (#1659)
• PEcAn.BIOCRO now uses PEcAn-standard variable names. As a result, two output variables have been renamed but keep their existing units and definitions:
• StemBiom renamed to AbvGrndWood
• RootBiom renamed to root_carbon_content
• Improved make install logic (#1558)
• Fixed remote code execution #1545
• Added check for NA end/start year in read.output
• Fixed jagify bug for raw field data
• Fixed bug (order of dims in nc_create) introduced in model2netcdf.DALEC by standard_vars changes
• Cleaned up NAMESPACE and source code of PEcAn.DB (#1520)
• Debugged python script in call_MODIS in data.remote to allow MODIS downloads
• Fixed FATES build script to work on ubuntu SIPNET output netcdf now includes LAI; some variable names changed to match standard
• Cleanup of leap year logic, using new PEcAn.utils::days_in_year(year) function (#801).
• Replace many hard-coded unit conversions with udunits2::ud.convert for consistency, readability, and clarity
• Refactored extract_soil_nc to create soil2netcdf, which will write soil data out in PEcAn standard.
• Added a new retry.func() to base/utils to provide ability to re-try a function X times before stopping. Currently using this function in the download.CRUNCEP() function to handle slow responses from THREDDS.
• Reformatted call_MODIS netcdf output to reflect the orientation of the MODIS grid
• Remote execution is more robust to errors in the submission process, not just the actual model execution
• PRELES model run script bug fix

Added

• Functionality to pass hyperparameters via settings
• Created new (and very rudimentary) web interface for downloading data from the dataone federation into the PEcAn database. More updates to come.
• Expanded initial conditions workflow for pool-based models, including PEcAn.data.land::prepare_pools to calculate pools from IC file (to be coupled with write.configs)
• New PEcAn.utils::days_in_year(year) function that should make it easier to work with leap years.
• New PEcAn.data.atmosphere::solar_angle function that replaces math that occurs in some models.
• New PEcAn.benchmarking::align_pft fucntion that aligns data assosiated with two different plant functional types
• #1594 shiny/workflowPlot Adding interactiveness using ggploltly
• #1594 shiny/workflowPlot Load outputs from multiple runs of the model
• #1594 shiny/workflowPlot Ways to toggle geometries (e.g. geom_point vs. geom_line).
• #1594 shiny/workflowPlot Smoothing using geom_smooth (Slider for specifying moving window width)
• #1594 shiny/workflowPlot Comparing model output vs loaded data according to tutorial
• Allow SIPNET and DALEC met files and model2netcdf to start or end mid year
• A Pre-release database clean up script that deletes unused/unassosiated entries from the database

Changed

• Clean up directory structure:
  • Move base packages (utils, settings, db, visualizaton) to a base directory, for consistency with modules and models
  • Move logger.* functions out of the PEcAn.utils package and into the PEcAn.logger package
  • Move remote functions out of the PEcAn.utils package and into their own PEcAn.remote package.
• #1594 shiny/workflowPlot Refactoring of code. get_workflow_ids in db/R/query.dplyr.R changed with ensemble = FALSE. Also allowing to load all workflow IDs. load_data_single_run and var_names_all also moved from shiny/workflowPlot/server.R to query.dplyr.R
• PEcAn.remote::start.model.runs has been significantly refactored to be less redundant and more robust
  betyConnect function in query.dplyr.R is now refactored into read_web_config so that the the Data-Ingest app can leverage read_web_config and provide it with a machine specific filepath for .../dbfiles
• Rpreles and Maeswrap package moved to suggest checked for within package function.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper over 7 years ago

PEcAn - THREDDS, Flux Course 2017

Biosphere - Ecological and Environmental Modeling - R
Published by mdietze almost 8 years ago

PEcAn - Met Process bug fixes

Changed

• Bugfix in Fluxnet2015
• Update Git workflow in Documentation
• download.CRUNCEP now uses CF-compliant time units (days since start of year instead of "secs")
  Bugfixes in met.process

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper about 8 years ago

PEcAn - NACP

Documentation

Added

• Source Rmarkdown and deploy scripts for PEcAn documentation
• Autocorrelation correction in PDA and scaling factor in emulator

Changed

• now dumping/loading experiments, cultivars_pfts, current_posteriors, experiments_sites experiments_treatments, trait_covariate_associations BETY #403 BETY #491

Removed

• Ameriflux is no longer selectable from the web gui #1291

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper about 8 years ago

PEcAn - AGU 2016 Release

Security Issue

There was a security issue in all of the previous versions of PEcAn. All users are highly encouraged to upgrade to the latest version of PEcAn that fixes this issue. Feel free to contact us for more information about this issue and how to check your instance.

New Models

• FATES
• MAAT

Code Clean-up

• Massive rework of almost the entire code base to better adhere to coding standards and making far more explicit use of R namespaces. A large thanks goes to Ben Bond-Lamberty for initiating this!
• Code clean up uncovered (and in some places created) a wide range of small bugs in the code
• A long standing bug in PEcAn's met processing workflow was an inability to change the temporal range of processed data (i.e. subsequent runs at a site needed to be for the same years). This has now been generalizied across the workflow to update the union of current and new years. The workflow also now distinguishes models that can choose dates from a directory of met data versus models that have to be fed a met driver that is the exact years of the run.

Benchmarking

• Fully-functional workflow now exists that can run benchmarks in the PEcAn database through the workflow based just on settings XML.
• This required splitting read.settings in two and having reference runs inserted in from database
• Generation of FACE benchmarks
• Addition of additional metric functions
• TODO: make it easier to create benchmarks and leverage these tools for general visualization and validation

Parameter Data Assimilation (PDA)

• Fixed mismatch in data alignment due to time zone differences
• Saving covariance matrix for running the MCMC longer
• handling the priors better
• PDA with derived variables (as in SA and EA, although I didn't have a case I needed to use it)
• More likelihoods (Gaussian and multiplicative Laplacian) than Heteroskedastic Laplacian (normal was always there but never really functional I think)
• Building emulator on sufficient statistics surface allowing us to update variances (but we don't insert posteriors for variances to the DB yet)
• Ability to handle non-model parameters like bias terms and precision
• Building one emulator per constraining variable

State Data Assimilation (SDA)

• Generalized SDA code to work better across models with a more standardized interface design
• Also generalized SDA's date handling code so that SDA can be run at different temporal frequencies (previously was hard-coded to annual)
• Added ability to calculate autocorrelation of observation data products and inflate observation covariance
• Generalized from multivariate normal model to multivariate Tobit model to handle zero- and range-bound data (e.g. AGB, fractional composition) in a way that is zero-inflated while also accounting for covariances.
• Initial work (incomplete) toward adding ED2 and LPJ-GUESS couplers for read.restart and write.restart.

Misc

• make: Switched PEcAn build to a make based system. This makes rebuilding quicker and also allows R to more automatically handle downloading required packages.
• add site: Added ability to add new sites on the PEcAn map by right-clicking on the map
• RTM PDA: RTM module now has a system in place to do stand-level parameter data assimilation using the ED2 radiative transfer model.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper over 8 years ago

PEcAn - Camp PEON: Assimilation, multi-site, soil params, Maespa, LPJ-GUESS, improvements to web & tutorials

Virtual machine available for download

State Data Assimilation (SDA)

One of the main goals of this release was to re-generalize the EnKF and generalized ensemble filter that @araiho has been using with LINKAGES back to also working with SIPNET and to bring the design one step closer to working with many models. This is working (yeah!), but not yet fully generalized.

The revised SDA tutorial has been migrated to documentation/tutorials/StateAssimilation and the gitbook updated to explain the use of SDA tags in the settings file and the main functions used.

Multisite runs

The basic PEcAn workflow was substantially re-organized to allow PEcAn to more easily run multiple sites at once (see Site Groups below). The core of the design is to split PEcAn’s settings object (what’s read from the XML) into a list of settings objects, one for each run, and applied to existing functions. Both Settings and SettingsList are now R classes. Within the settings file itself, there is now a <runs> block that can include any number of individual <run>. Where appropriate, PEcAn gathers information across runs — for example, the meta-analysis is performed on the set of PFTs extracted across all runs, rather than being done run-by-run. This change is the first step in a large plan to re-organize the PEcAn workflow to be more modular and also support multi-model runs.

Note: Multisite/multimodel run SettingLists cannot yet be generated through the web interface.

Parameter data assimilation

• Generalized to multiple PFTs
• Emulator now supports a range of additional features that can be controlled through the XML settings:
  • multiple chains
  • extending chains (additional samples) starting from their latest state
  • adding additional knots
  • automated burnin detection

The revised PDA tutorial has been migrated to documentation/tutorials/ParameterAssimilation and the PDA vignette contains a more detailed explanation of how to use the new features.

Non-PFT parameter constraint

As an experiment in giving PEcAn control of non-PFT parameters, we've created a 'soil' PFT for SIPNET that contains soil biogeochemical parameters. If successful, we will extend this approach to other models. This change already improves model calibration and validation and has allowed us to assess the relative contribution of plant vs soil parameters to model output uncertainty.

Models

• Maespa has basic functionality (thanks @mdekauwe for all the help!)
• LPJ-GUESS running, but without PEcAn controlling parameters

Drivers

• Fluxnet LaThuile

Web Interface

• PEcAn landing page updated to be links to tools. Avoids confusion when users forget to add the /pecan, /bety, /rstudio, etc. to the web address.
• Site groups has been populated with a number of major research networks: Ameriflux, Fluxnet, Fluxnet 2015 Tier 1 (open), LTER, NEON, ForestGEO, PalEON
• Links to documentation, gitter, and bug reports from all pages
• Additional mouseover text and links to model and variables tables
• Legend in default plot, removed spline from Shiny plot

Camp PalEON

• This is a special release for the PalEON summer course, and comes pre-installed with extra libraries and data for the course

Tutorials

In addition to migrating the PDA and SDA tutorials to RMarkdown, the model-data comparison at the end of Demo 1 has also been migrated to documentation/tutorials/AnalyzeOutput

Benchmarking

While still in alpha development, @bcow contributed a large chunk of code bringing us much closer to automated benchmark execution and a first pass at a Shiny page for benchmark visualization.

Bugfixes

• C_stack bug in run.sensitivity.analysis: fixed on new VM but older users should upgrade gridExtra R package
• Units in figures and reading of model output #792 -- downstream applications/code should check for changes.
• Can skip meta-analysis in pecan settings. May require updating old pecan.xml, particularly for integration tests.
• Interface: met date checking
• Misc: #906 #916 #919 #923 #948 #950 #951

Known Issues

• This version of PEcAn is misconfigured and is using the wrong server for Brown Dog. The server that is used is the development server for Brown Dog and might result in no data being returned. During the Camp PEON the development server will be using the same code as the production server. If you want to use BrownDog after Camp PEON we highly encourage you to sign up with Brown Dog. We will at that point contact you and send you the instructions to modify your VM to use your new account and the production servers.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper over 8 years ago

PEcAn - CMIP5, Shiny, FLUXNET2015, Global Sensitivity

CMIP5:

We’ve add support for running models into the future under climate change scenarios! This release adds support for grabbing GFDL runs for all four RCP scenarios multiple experiments. RCP4.5 is the ‘default’, but Advanced Edit can be used to change the model (CM3, ESM2M, ESM2G), scenario (rcp26, rcp45, rcp60, rcp85), or ensemble member (r1i1p1, r3i1p1, r5i1p1). We are working with Earth System Grid to provide support for other climate models in future PEcAn releases.

Advanced Visualizations:

We’ve added support for R Shiny visualizations in a way that we hope will allow users to easily add new Shiny apps to the catalog of PEcAn visualizations. Simply drop a new app into the pecan/shiny folder and it should go live. We’ve initially populated our Shiny catalog with two visualizations:

workflowPlot:

Similar to our current visualizations, which plots model output timeseries, but now shows output over all years and without a default 1-day time averaging.

global-sensitivity:

Displays results from our new Monte Carlo global sensitivity analysis. Select outputs by model parameter (all model outputs), by output variable (all model parameters), or dig down into individual parameter-by-output relationships.

FLUXNET2015 and Ameriflux LBL:

At the site-level we’ve added support for the new FLUXNET2015 synthesis product, extending PEcAn’s support for flux tower sites globally. For Ameriflux, we’ve also added support for data on the new Lawrence Berkeley server. Past support for data on the ORNL server will also continue so long as that site remains active, but LBL is the preferred server and will contain the most up-do-date data.

Finally, in response to the above changes and feedback from users, we’ve tried to streamline the main PEcAn tutorial. The Parameter Data Assimilation (PDA) tutorial has also been updated to leverage emulator upgrades in the last release.

Known issues:

MAESPA, G’DAY, LPJ-GUESS not yet ready. JULES supported but not installed on VM, need to download and install yourself.

Met processing bugs: code for extending the date range from previous runs has a bug. Current workaround through advanced edit: set new met output directory (See PDA demo). For CRUNCEP and GFDL, workflow gets tripped up when extending to a new model. Currently requires running met process code by hand in R. Bugfixes will be in next release (Aug 12).

In the pipeline:

Next release will focus on PalEON summer course (SDA, tree ring & allometry modules, hopefully alpha support for PalEON met and NEOTOMA database). Also hope to bring Benchmarking up to a functional beta version, add GLDAS & NLDAS met.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper almost 9 years ago

PEcAn - Site Groups; Beta: SDA, emulator, PDA, MAESPA, LPJ-GUESS, TRY; Alpha: benchmarking

• Ability to aggregate collections of sites into groupings. This includes both well recognized networks (Amerflux, Fluxnet, LTER, NEON, etc) and user-defined collections of sites.
• Initial support for the MAESPA and LPJ-GUESS models. Neither is fully-operational via the web interface, but both should run with prescribed inputs. Alpha development on CLM, G'DAY, and MAAT also included.
• Generalization of the parameter data assimilation to be able to use a long list of alternative MCMC sampling schemes implemented in the BayesianTools R package (developed by the COST PROFOUND project) as well as a substantial revitalization and improvement of the emulator-based Bayesian calibration scheme.
• Beta release of a generalization of the Ensemble Kalman Filter state data assimilation code to dynamically estimate both states and process error. More general improvements and generalization in multivariate SDA
• Alpha development of benchmarking. These features require additional database schema changes and further development -- look for a working beta in the next release. Key new concepts include a generalization of data loading based on MIME-type and format, a generalization of model-data alignment, and the beginning of a catalog of skill scores and other statistical metrics.
• Code for merging TRY database into local BETY instance. Because TRY is not public, we are not loading it globally into the database but rather providing tools to allow users to easily upload their own TRY data dumps.
• Various bug fixes in meteorology
• Improvements in RTM package supporting stand-level inversion / calibration

VM available from NCSA

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper almost 9 years ago

PEcAn - PEcAn 1.4.5 : PRELES + CRUNCEP

This release requires an update to the config file in PEcAn web interface. Without this change the web interface will not work. See config.php changes.

Following are the major new features of this release. Besides these new features many bugs have been fixed, see milestone 1.4.5 list of issues or complete list of commits.

New Models

This release now adds support for the PRELES model.

PRELES

The PREdiction of Light use efficiency, Evapotranspiration, and Soil water (PRELES) model was developed by a team at the Natural Resources Institute Finland with the goal to "develop a semi-empirical, intermediate-complexity model of evapotranspiration and its coupling with canopy photosynthesis" (Peltoniemi et al 2015). Calibrated for boreal regions, it uses common meteorological drivers (temperature, pressure, precipitation, photosynthetically active radiation, and CO2) to predict GPP and water balance (soil water, evaporation, and transpiration). It offers comparatively quick analyses, due to its parsimonious code, making it a unique choice among our suite of model options.

New Met Data

This release adds support for CRUNCEP

CRUNCEP

The Climate Research Unit, National Centers for Environmental Prediction (CRUNCEP) Multi-Scale Synthesis and Terrestrial Model Intercomparison Project (MsTMIP) data set provides climatology data that have been standardized and aggregated for use as input to carbon cycle models at a global (0.5-degree resolution) scale. The data were integrated into gridded global collections of meteorological variables (precipitation, air temperature, air specific humidity, pressure, downward longwave radiation, downward shortwave radiation, and wind speed) and has 6-hour temporal resolution between the years 1901-2010. Using PEcAn’s CRUNCEP function, a user can download a dataset containing these variables. A full description of the data can be found here; https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=1220.

Web Interface

The web interface now allows the user to login using their credentials, event if the flag authorization required is not set. Any run started when logged in will be recorded as belonging to the logged in user. Only users logged in can now delete workflows.

Users can now filter the history page, by their own runs, those that are available locally, succeeded and by textual search (in notes, model, site).

Notes are no longer an advanced feature, but can now be added during the input section.

config.php changes

A major change has been done to the web interface configuration files. Please make sure to update your config.php, without these changes your web interface will not allow you to do any runs!

The variables $hostlist, $qsuboptions and $qsublist have been merged into a single variable called $hostlist. This list has a list of all hosts that are shown in the host pull down list, as well as any options for qsub. This now also allows you to specify a folder on the remote system to use. All runs will use this remote folder! Please see the example.config.php

The following variables are no longer used : $qsuboptions, $qsublist, $ed_soil, $ed_inputs and can be safely removed from you config.php (after fixing the above).

Additional changes

PEcAn now leverages of Travis CI for continuous integration. To use this in your own branches and forks of PEcAn make sure you have merged in the master branch from PEcAn and created an account on Travis CI. Once that is done you should be able to flip the switch for PEcAn and it will start to build automatically anytime you push changes to your branches/fork.

This patch also marks a change to documentation. We have migrated our github wiki to gitbook and will no longer update the wiki on github. It will allow for easier exportation of PecAn documentation (pdf, epub, mobi, web) as well as make it easier to gain feedback from users (inline commenting).

Known Bugs

The NARR met product will not be processed locally from raw global to model/site specific format and properly added to the database. Unless a global NARR met file exists and is added to BETY, selecting NARR as the met driver for your model will create an error. Those wishing to use NARR are strongly encouraged to select the "Brown Dog" option so that the met is processed remotely rather than locally.

Downloads

The latest version of the VM can be downloaded from NCSA

Citations

Peltoniemi, M., Pulkkinen, M., Aurela, M., Pumpanen, J.,Kolari, P., Makela, A.(2015)."A semi-empirical model of boreal-forest gross primary production , evapotranspiration , and soil water — calibration and sensitivity analysis". Boreal Environment Research, 20(2), 151-171.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper about 9 years ago

PEcAn - PEcAn 1.4.4: Remote execution from web interface

The major new feature of this release of PEcAn is the ability to launch workflows from the web interface that then run on remote machines (e.g. clusters, cloud services). For this to work you will need to have ssh access to that remote machine. PEcAn further supports both remote and local servers that use qsub to schedule job execution instead of just running the job, with PEcAn settings allowing the job submission to be tweaked to particular systems. Running PEcAn remote execution does not require a full PEcAn installation on the remote server; details on remote installation and set-up can be found at remote PEcAn setup

The web interface can now also be configured to control user access to visualizing and downloading existing outputs versus running new workflows. This allows you to provide broad access to model outputs and analyses on your server without risk that unauthorized users will burden compute resources with additional runs. User access is controlled using BETY to configure the users that can view and/or run new workflows.

changes to web interface

One new feature (besides the remote run capability) is the ability to leave notes with the workflow. This allows you to record information about the workflow. Right now this is part of the advanced features, but will be moved in the next release to a more prominent place (#679).

You will need to add the following to you config.php for the ability to control who can run a workflow.

# Require username/password, can set min level to 0 so nobody can
# run/delete.
# 4 = viewer
# 3 = creator
# 2 = manager
# 1 = administrator
$min_run_level=2;
$min_delete_level=2;

For the ability to remote launch instances you will need to add the following to your config.php:

# sshTunnel binary
$SSHtunnel=dirname(__FILE__) . DIRECTORY_SEPARATOR . "sshtunnel.sh";

# List of hosts that need qsub
$qsublist=array("geo.bu.edu");

# List of qsub options, this might only be needed for certain systems
# the list should be the server pointing to an array, the second array
# can contain 3 values (qsub, jobid and qstat).
$qsuboptions=array("geo.bu.edu" => 
                    array("qsub"   => "qsub -V -N @NAME@ -o @STDOUT@ -e @STDERR@ -S /bin/bash",
                          "jobid"  => "Your job ([0-9]+) .*",
                          "qstat"  => "qstat -j @JOBID@ || echo DONE",
                          "job.sh" => "module load udunits R/R-3.0.0_gnu-4.4.6",
                          "models" => array("ED2"    => "module load hdf5")),
                    "host2" => 
                    array( "job.sh" => "echo 'Hello Word'"));

Examples of both these settings are provided in the web/config.example.php file.

Database

This release now will have the ability to sync with the database hosted at the University of Wisconsin - Madison. We have also launched a status page for tracking the overall PEcAn network database synchronization: https://pecan2.bu.edu/pecan/status.php

Changes to PEcAn

As always bug fixes.

Updates to RTM package: additional documentation and vignette updates, support for sensor spectral response functions.

Photosynthesis package: added option for TPU limitation and temperature dependence on parameters

Coming down the pipeline for v1.4.5

The current release of PEcAn includes partial (incomplete) code adding additional models (CLM4.5, PRELES, a revised version of LINKAGES), utilities for benchmarking, support for global meteorology products, integration of remotely-sensed data, and additional generalizations in the state & parameter data assimilation schemes.

Downloads

A VM with this version of PEcAn can be downloaded from NCSA

Biosphere - Ecological and Environmental Modeling - R
Published by dlebauer over 9 years ago

PEcAn - PEcAn 1.4.3: Improved web accessibility, database sync, meteorology processing. New R vignettes.

Web

• Brown Dog support for processing NARR meteorology in the cloud. You can now run any model for any location in North America without having to download and process 1/2 TB of data!
• Ability to run sensitivity, uncertainty, and ensemble analyses 100% through web interface
• Ability to run MCMC model-data fusion through pecan.xml 'advanced edit', rather than R command line
• VM now comes with Brown Dog enabled
• output from web is now writeable (request by @ankurdesai)

Database

• load.bety.sh now uses transactions (#563) for more stable syncs
• TEST at Brookhaven National Lab now part of PEcAn database sync network
• Numerous updates to support new BETY database constraints

Analysis Modules

• New vignettes for the RTM, photosynthesis, allometry, and priors modules
• New documentation/tutorials
• Changed names of output files to be more informative (#534, #507)
• The RTM module can simulate from and perform a Bayesian (Metropolis-Hastings) inversion of PROSPECT versions 4, 5, and 5B and PROSAIL (single canopy layer). The inversion can also accommodate custom priors (so far, only normal and lognormal) and preset constants (i.e. inversions can be performed on all parameters, or only a subset of unknown parameters if some are known). The inversion process has been rewritten in Fortran to facilitate addition of new models. In addition, users can perform a much slower inversion with fully customizable models and priors (passed as function arguments).

Data Modules

• rh2qair now accepts pressure as an argument, rewritten to be consistent with qair2rh (#518, #528)
• fixes to NARR & PalEON conversions
• support for FACE MIP meteorology

Models

• DALEC: greatly expanded set of supported parameters, priors
• LINKAGES: updates to support code development
• ED2:
  • update ED2IN for r46 (#537)
  • update model2netcdf for latest ED2 code on Github
• CLM4.5: Initial stub for interface modules. Working with Dan Ricciuto to get CLM4.5 on the VM for PEcAn 1.4.4

Notes:

• Ubuntu users: OS v >= 14.04 is required to support nco with netcdf4

Biosphere - Ecological and Environmental Modeling - R
Published by dlebauer almost 10 years ago

PEcAn - PEcAn 1.4.2: MET Process and Brown Dog

PEcAn

Run any model at any location in North America! This version allows for the automated conversion and extraction of NARR and Ameriflux met data on both local and remote (HPC) servers. Processed meteorology is now saved to database for later reuse.

This version of PEcAn now can use BrownDog. BrownDog helps with the conversion of data from input files to model specific files using cloud-based web-services.

PEcAn WEB

• allow the user to search/filter Google Map of sites
• ability to use BrownDog if 'use BrownDog' enabled
• data use agreements displayed for Ameriflux and NARR
• ability to run 'bare ground' ED without vegetation initial conditions (PSS/CSS/Site)
• ability to run parameter data assimilation (MCMC) via advanced edit
• 'Conversion' toggles Google Map between sites with a full set of inputs already in the database vs. sites where inputs can be processed automatically using workflows.
• allow to download input files (even if run has failed)

Modules

• RTM module now support Bayesian inversion of PROSPECT leaf-level spectral model
• Allometry module now has a 'predict' function

As always there are also many bugfixes done

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper almost 10 years ago

PEcAn - PEcAn 1.4.1: Fetch NARR met; add LINKAGES model

PEcAn

Besides bug fixes, this new version of the VM supports fetching the MET data from NARR and converting it to model specific inputs. As well as a new model LINKAGES (https://github.com/araiho/Linkages).

BETY

Bety is now installed in /home/carya/bety and does not require root to upgrade. Users can simply call the following commands as non root to update the server.

git pull
bundle install
rake db:migrate

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper about 10 years ago

PEcAn - PEcAn 1.4.0 : Ameriflux Download

Overview

This version supports conversion of met files. The code can now download and convert the met data to CF standard and to model specific format. The met data will along the way be gap filled. This allows any researchers to pick any Ameriflux site and run the SIPNET model on this site.

As always the latest VM can be downloaded from http://isda.ncsa.illinois.edu/download/index.php?project=PEcAn&sort=version

pecan.xml Changes

The met tag can now support a special notation that lets it download and convert the met data. The met tag can have an input="Ameriflux" and output="sipnet" attribute. Currently these are the only supported conversion but more will be added in the next release of PEcAn.

web changes

When running in the web interface you now have the ability to edit the pecan.xml before the software runs. This allows you to change the number of ensemble runs done, as well as any other changes to the pecan.xml.

After the run finishes, you can now download the sipnet clim file used for a particular run. You can now also plot different variables against each other instead of time vs one of hte variables.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper over 10 years ago

PEcAn - PEcAn 1.3.7 : better model support

Overview

This version supports new model types better. BETY now allows to add new model types to the database. You can associate these model types with PFT's. Each model type will have a list of required and optional inputs associated. These inputs will be listed in <run><inputs> tag.

As always the latest VM can be downloaded from http://isda.ncsa.illinois.edu/download/index.php?project=PEcAn&sort=version

Support for DALEC

A new model has been added to this release, DALEC. BETY already has a modeltype DALEC in the database to support this model.

pecan.xml Changes

Using the new modeltypes we can now check the inputs specified in the pecan.xml file. All inputs should now be listed under the <run><inputs>. The code will try and update your pecan.xml file to this new format. Any missing inputs that are required will quit the run, allowing you to quickly fix your pecan.xml file. For updated information see https://github.com/PecanProject/pecan/wiki/PEcAn-Configuration.

Sites now store their information as geometry objects. If you have custom queries that use the site.lat and site.lon information you will need to change this to ST_X(ST_CENTROID(geometry)) AS lon, ST_Y(ST_CENTROID(geometry)) AS lat.

Database Changes

The database requires updates for the model types table. You will need to update BETY to be able to see this. This requires version 3.2 of BETYdb (https://github.com/PecanProject/bety/releases/tag/betydb_3.2).

Code Changes

One major change is to the function called get.trait.data(). This function now requires a model type as the second parameter. The model type is used to make sure we get the right PFT. This allows multiple PFTs to have the same name, but be distinguishable by the combination of name and model type.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper over 10 years ago

PEcAn - PEcAn 1.3.6: Updates to support changes in BETYdb and FIA databases

This version focuses on more database updates.

As always you can download the VM from:
http://isda.ncsa.illinois.edu/download/index.php?project=PEcAn&sort=category

PEcAn

add sites/data

• the script to add sites is now part of PEcAn. To add the sites data you can use pecan/scripts/add.data.sh
• the script to add models is updated and will setup the binaries and is called pecan/scripts/add.models.sh

Both scripts can be called multiple times, but will only add the information to the database once.

pecan.xml:

• In pecan.xml the database section now has 2 subsections, bety and fia. Any existing information that was in the database section in pecan.xml will need to be moved to the bety subsection, read.settings() will automatically update your pecan.xml to reflect this.
• The <bety><write> option in the pecan.xml is now part of the <database><bety> section, read.settings() will automatically update your pecan.xml to reflect this.
• The ED land use/soil/vegetation/thermal sums data files are now inputs in BETY that need to be specified as inputs.

Action Required: add ED data files to database, see https://github.com/PecanProject/pecan/issues/120

SipNET

• Support for meteorology conversion from netCDF CF standard to SipNET's .clim driver files

PEcAn Web App

• config.php now declares the fully qualified host name ($fqdn)
• the config.php in pecan/web will have new database configuration parameters, please see config.example.php.
• no longer needed to specify vet/soil/ed_inputs in config.php, these are now stored as inputs in BETY.
• if the fia database is specified in config.php, the user can now select fia2ed from psscss selector.

Action Required: update pecan/web/config.php, see pecan/web/config_example for latest settings.

BETY:

To update BETY database either download a new version or update the database.

• In the database there is now a link from a model to the dbfiles table. This allows you to have a single model, with binaries located in different places on different systems.
• Each PFT now stores the model type associated with it, the web application will use this to find the right pfts to display for the user, and read.settings will check to see if the PFT is compatible with the model.

Action Required: add binaries to models, for example using the pecan/scripts/addmodels.sh.
Action Required: the migration of the database will add the appropriate model_type to each PFT based on the prefix of the PFT.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper almost 11 years ago

PEcAn - PEcAn 1.3.5: Switch to PostgreSQL, enable distributed instances of BETYdb, maintaining local database changes

This version officially switches to postgreSQL. This allows for the script to enable to keep local changes to the PEcAn database.

This enables multiple instances to run BETYdb, and for local changes to be exchanged across databases. Each installation has a reserved set of foreign key ids (e.g. Boston University has 1,000,000,000-1,999,999,999 reserved. Edits to existing records added on another server will not be preserved.

Biosphere - Ecological and Environmental Modeling - R
Published by robkooper almost 11 years ago