Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

A curated list of open technology projects to sustain a stable climate, energy supply, biodiversity and natural resources.

Model My Watershed

A watershed-modeling web app for citizens, conservation practitioners, municipal decision-makers, educators, and students.
https://github.com/WikiWatershed/model-my-watershed

Category: Natural Resources
Sub Category: Water Supply and Quality

Keywords

civic-apps-team hydrology stormwater watersheds

Keywords from Contributors

earth-observation geotrellis geospatial-processing geospatial-analytics hydrology-modeling bicycle routes semantic-segmentation bikeshare opentripplanner

Last synced: about 16 hours ago
JSON representation

Repository metadata

The web application front end for Model My Watershed.

README.md

model-my-watershed

Local Development

A combination of Vagrant 2.2+ and Ansible 2.8 is used to setup the development environment for this project. The project consists of the following virtual machines:

  • app
  • services
  • tiler
  • worker

The app virtual machine contains an instance of the Django application, services contains:

  • PostgreSQL
  • Redis

tiler contains:

  • Windshaft
  • Mapnik

worker contains:

  • Celery
  • Docker

Getting Started

First, ensure that you have a set of Amazon Web Services (AWS) credentials with access to Azavea's pre-processed NLCD data set. This setup generally needs to happen on the virtual machine host using the AWS CLI:

$ aws configure --profile mmw-stg

You will also need to set the MMW Datahub AWS credential as your default. These are stored in lastpass under the name MMW Azavea DataHub AWS. Ensure that the AWS credentials file has universal read permissions.

Ensure you have the vagrant-disksize plugin installed:

$ vagrant plugin install vagrant-disksize

Starting with Virtualbox 6.1.28, host-only networks are restricted to 192.168.56.0/21 by default.

We will need to do the following to override this restriction:

sudo mkdir /etc/vbox

echo "
* 192.168.56.0/21
* 33.33.0.0/16" | sudo tee /etc/vbox/networks.conf

Next, use the following command to bring up a local development environment, ensuring you have the most recent version of the base box:

$ vagrant box update
$ vagrant up

The application will now be running at http://localhost:8000.

After significant changes, you may need to run the following two commands to apply database migrations and rebuild JavaScript assets:

$ ./scripts/manage.sh migrate
$ ./scripts/bundle.sh

To load or reload boundary data, from an app server, run (scripts is not mounted by default to the VM, you may need to copy the file over):

$ vagrant upload ./scripts/ app
$ vagrant ssh app
$ ./scripts/aws/setupdb.sh -b

The same script can be used to load the stream network data:

$ ./scripts/aws/setupdb.sh -s

and all the other data:

$ ./scripts/aws/setupdb.sh -d
$ ./scripts/aws/setupdb.sh -m
$ ./scripts/aws/setupdb.sh -p
$ ./scripts/aws/setupdb.sh -c
$ ./scripts/aws/setupdb.sh -q

Note that if you receive out of memory errors while loading the data, you may want to increase the RAM on your services VM (1512 MB may be all that is necessary).

See debug messages from the web app server:

$ ./scripts/debugserver.sh

Watch the JavaScript and SASS files for changes:

$ ./scripts/bundle.sh --debug --watch

When creating new JavaScript or SASS files, you may need to stop and restart the bundle script.

If you add a JS dependency and want it to be included in the vendor.js bundle, you will need to update the JS_DEPS array in bundle.sh accordingly.

If changes were made to the one of the VM's configuration or requirements since the last time you provisioned, you'll need to reprovision.

$ vagrant provision <VM name>

After provisioning is complete, you can execute Django management commands with:

$ ./scripts/manage.sh test

Note: If you get an error that resembles the following, try logging into the app virtual machine again for the group permissions changes to take effect:

envdir: fatal: unable to switch to directory /etc/mmw.d/env: access denied

Ports

The Vagrant configuration maps the following host ports to services running in the virtual machines.

Service Port URL
Django Web Application 8000 http://localhost:8000
PostgreSQL 5432 psql -h localhost
Redis 6379 redis-cli -h localhost 6379
Testem 7357 http://localhost:7357
Tiler 4000 http://localhost:4000

Caching

To speed up geoprocessing, those requests are cached in Redis. To disable this caching for development purposes, set the value of MMW_GEOPROCESSING_CACHE to 0:

$ vagrant ssh worker -c 'echo "0" | sudo tee /etc/mmw.d/env/MMW_GEOPROCESSING_CACHE'
$ vagrant ssh worker -c 'sudo service celeryd restart'

To enable the geoprocessing cache simply set it to 1 and restart the celeryd service.

In some cases, it may be necessary to remove all cached values. This can be done with:

$ vagrant ssh services -c 'redis-cli -n 1 --raw KEYS ":1:geop_*" | xargs redis-cli -n 1 DEL'

Test Mode

In order to run the app in test mode, which simulates the production static asset bundle, reprovision with VAGRANT_ENV=TEST vagrant provision.

Testing

Run all the tests:

$ ./scripts/test.sh
Python

To check for Python lint:

$ ./scripts/check.sh

To run all the tests on the Django app:

$ ./scripts/manage.sh test

Or just for a specific app:

$ ./scripts/manage.sh test apps.app_name.tests

More info here.

Run MapShed tests, which require MapShed tables installed in the local database
(using setupdb.sh):

$ ./scripts/manage.sh test_mapshed
JavaScript

To check for JavaScript lint:

$ ./scripts/yarn.sh run lint

When creating new tests or debugging old tests, it may be easier to open the testem page, which polls for changes to the test bundle and updates the test state dynamically.

First, start the testem process.

$ ./scripts/testem.sh

Then view the test runner page at http://localhost:7357.

To enable livereload, download the browser extension
and start the livereload server with the following command:

./scripts/yarn.sh run livereload

Bundling static assets

The bundle.sh script runs browserify, node-sass, and othe pre-processing
tasks to generate static assets.

The vendor bundle is not created until you run this command with the
--vendor flag. This bundle will be very large if combined with --debug.

Test bundles are not created unless the --tests flag is used.

In general, you should be able to combine --vendor, --tests, --debug,
and --watch and have it behave as you would expect.

You can also minify bundles by using the --minify flag. This operation is
not fast, and also disables source maps.

The --list flag displays module dependencies and does not actually generate
any bundles. It doesn't make sense to combine this with --watch.
This flag is for troubleshooting purposes only.

> bundle.sh -h
bundle.sh [OPTION]...

Bundle JS and CSS static assets.

 Options:
  --watch      Listen for file changes
  --debug      Generate source maps
  --minify     Minify bundles (**SLOW**); Disables source maps
  --tests      Generate test bundles
  --list       List browserify dependencies
  --vendor     Generate vendor bundle and copy assets
  -h, --help   Display this help text

Adding JS dependencies

To add a new JS dependency, use

$ ./scripts/yarn.sh add --exact <dependency>

this will download the dependency to node_modules, add to the package.json, and to yarn.lock.
Furthermore, it will be pinned to the current version.
Then, update the JS_DEPS array in bundle.sh.
Rebuild the vendor bundle using ./scripts/bundle.sh --vendor.
yarn commands can be run using ./scripts/yarn.sh.

Owner metadata

GitHub Events

Total
Last Year

Committers metadata

Last synced: 6 days ago

Total Commits: 3,308
Total Committers: 36
Avg Commits per committer: 91.889
Development Distribution Score (DDS): 0.531

Commits in past year: 95
Committers in past year: 2
Avg Commits per committer in past year: 47.5
Development Distribution Score (DDS) in past year: 0.053

Name Email Commits
Terence Tuhinanshu t****u@a****m 1552
Alice Rottersman a****n@a****m 343
Casey Thomas c****s@a****m 203
Kevin DeLoach k****h@a****m 186
Matthew McFarland m****d@a****m 170
Kelly Innes k****s@a****m 144
Laurence Liss l****s@a****m 124
Lewis Fishgold l****d@a****m 117
Hector Castro h****o@a****m 116
James McClain j****n@a****m 83
jfrankl j****l@a****m 47
Hector Castro h****o@g****m 43
Joe Tarricone j****e@a****m 36
Lewis Fishgold l****h@g****m 22
Christopher Taylor c****r@a****m 19
Justin Walgran j****n@a****m 16
Matthew McFarland m****d@g****m 14
lsetiawan l****s@u****u 14
lossyrob r****e@g****m 12
Rocky Breslow r****w@a****m 11
Emily Hu a****u@h****m 6
dependabot[bot] 4****] 5
Klaas Hoekema k****a@e****m 5
Taylor Nation t****4@g****m 4
Claude M. Schrader c****r@a****m 2
Anthony Aufdenkampe a****e@l****m 2
Emilio Mayorga e****a@g****m 2
steveskrip s****k@g****m 2
simonkassel k****n@g****m 1
Rachele Morino r****o@a****m 1
and 6 more...

Committer domains:

Issue and Pull Request metadata

Last synced: 1 day ago

Total issues: 2,213
Total pull requests: 1,470
Average time to close issues: 2 months
Average time to close pull requests: 3 days
Total issue authors: 42
Total pull request authors: 26
Average comments per issue: 1.72
Average comments per pull request: 4.29
Merged pull request: 1,385
Bot issues: 0
Bot pull requests: 5

Past year issues: 39
Past year pull requests: 22
Past year average time to close issues: about 2 months
Past year average time to close pull requests: 4 days
Past year issue authors: 4
Past year pull request authors: 2
Past year average comments per issue: 1.67
Past year average comments per pull request: 2.0
Past year merged pull request: 22
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/WikiWatershed/model-my-watershed

Top Issue Authors

  • rajadain (711)
  • mmcfarland (405)
  • ajrobbins (317)
  • caseycesari (150)
  • arottersman (140)
  • hectcastro (112)
  • kdeloach (73)
  • lewfish (55)
  • kellyi (40)
  • SRGDamia1 (34)
  • jfrankl (33)
  • lliss (20)
  • aufdenkampe (16)
  • jamesmcclain (15)
  • jwalgran (15)

Top Pull Request Authors

  • rajadain (559)
  • arottersman (135)
  • hectcastro (114)
  • mmcfarland (107)
  • caseycesari (106)
  • kellyi (99)
  • kdeloach (73)
  • lewfish (73)
  • jamesmcclain (64)
  • lliss (44)
  • jfrankl (19)
  • jtarricone (18)
  • ctaylo37 (14)
  • rbreslow (8)
  • jwalgran (8)

Top Issue Labels

  • WPF (380)
  • bug (339)
  • + (285)
  • tested/verified (183)
  • BigCZ (173)
  • MapShed (134)
  • NSF (117)
  • WPF 2-7 (103)
  • enhancement (102)
  • OSI (88)
  • design (84)
  • operations (72)
  • production (66)
  • PA DEP (57)
  • Sub-basin Modeling (57)
  • HydroShare Export (56)
  • Compare View Revamp (50)
  • 1 (47)
  • WPF 2-3 (46)
  • WPF 2-4 (45)
  • tech-debt (41)
  • size: 1 (39)
  • DRWI (38)
  • WPF 3-5 (36)
  • ITSI Portal (34)
  • AWS (34)
  • Monitor (33)
  • micro site (29)
  • Custom Weather Data (26)
  • wontfix (25)

Top Pull Request Labels

  • NSF (64)
  • BigCZ (62)
  • PA DEP (35)
  • OSI (31)
  • HydroShare Export (29)
  • WPF (23)
  • MapShed (20)
  • DRWI (19)
  • Monitor (19)
  • AWS (18)
  • Custom Weather Data (14)
  • Sub-basin Modeling (14)
  • Compare View Revamp (12)
  • GEN (10)
  • CitSci (10)
  • Unit Conversion (8)
  • Geoprocessing API (5)
  • dependencies (5)
  • in review (4)
  • javascript (3)
  • ITSI Portal (2)
  • bug (2)
  • tech-debt (2)
  • tested/verified (2)
  • in progress (2)
  • python (1)
  • operations (1)
  • WPF 2-7 (1)
  • WPF 2-4 (1)
  • User Accounts (1)

Dependencies

src/mmw/mmw/settings/data/scripts/package.json npm
  • turf-buffer 3.0.12
src/mmw/yarn.lock npm
  • 805 dependencies
src/mmw/package.json npm
  • backbone 1.1.2
  • backbone.marionette 2.4.1
  • backbone.paginator 2.0.5
  • blueimp-md5 1.1.0
  • bootstrap 3.4.1
  • bootstrap-datepicker 1.7.1
  • bootstrap-select 1.13.18
  • bootstrap-table 1.11.0
  • browserify 16.5.0
  • chai 1.10.0
  • clipboard 1.7.1
  • d3 3.5.5
  • font-awesome 4.3.0
  • iframe-phone https://github.com/concord-consortium/iframe-phone#v1.1.3
  • jquery 3.4.1
  • jshint 2.10.2
  • jstify 0.14.0
  • jszip 3.1.3
  • leaflet 0.7.3
  • leaflet-draw https://github.com/azavea/Leaflet.draw
  • leaflet-plugins https://github.com/azavea/leaflet-plugins#feature/browserify
  • leaflet.locatecontrol 0.43.0
  • livereload 0.8.2
  • lodash 4.17.19
  • minifyify 7.0.0
  • mocha 8.3.0
  • moment 2.24.0
  • node-sass 5.0.0
  • nunjucks 1.3.4
  • nunjucksify 0.2.2
  • reproject 1.1.1
  • retina.js https://github.com/imulus/retinajs#1.3.0
  • shapefile 0.6.2
  • sinon 1.14.1
  • testem 3.2.0
  • turf-area 1.1.1
  • turf-bbox-polygon 1.0.1
  • turf-centroid 3.0.12
  • turf-destination 1.2.1
  • turf-erase 1.3.2
  • turf-intersect 1.4.2
  • turf-kinks 3.0.12
  • underscore 1.8.3
  • watchify 3.11.1
  • wellknown 0.5.0
src/tiler/package-lock.json npm
  • 346 dependencies
src/tiler/package.json npm
  • supervisor ^0.6.0 development
  • @carto/mapnik ^3.6.2-carto.15
  • aws-sdk ^2.482.0
  • debug ^4.1.1
  • express ^4.17.1
  • lodash ^4.17.19
  • morgan ^1.9.1
  • pg ^7.11.0
  • redis ^2.8.0
  • redis-mpool ^0.7.0
  • rollbar ^2.7.1
  • step ^1.0.0
  • underscore ^1.9.1
  • windshaft 5.4.0
deployment/requirements.txt pypi
  • ansible ==2.9.27
  • awscli >=1.9.15
  • boto ==2.49.0
  • cryptography ==3.2.1
  • majorkirby ==1.0.0
  • troposphere ==2.4.9
scripts/data/climate/requirements.txt pypi
  • Pillow ==6.2.0
  • matplotlib ==2.0.2
  • rasterio ==1.0a9
src/mmw/requirements/base.txt pypi
  • BMPxlsx ==3.0.0
  • cryptography ==36.0.1
  • dictdiffer ==0.9.0
  • django <4
  • django-cors-headers ==3.10.1
  • django-extensions ==3.1.5
  • django-filter ==21.1
  • django-redis ==5.1.0
  • django-registration-redux ==2.9
  • django_celery_results ==2.2.0
  • djangorestframework ==3.13.1
  • djangorestframework-gis ==0.17.0
  • drf_yasg ==1.20.0
  • fiona ==1.8.20
  • gunicorn ==20.1.0
  • gwlf-e ==3.1.0
  • hiredis ==2.0.0
  • hs_restclient ==1.3.7
  • markdown ==3.3.6
  • numpy ==1.20.3
  • pandas ==1.3.5
  • pyOpenSSL ==21.0.0
  • python-dateutil ==2.8.2
  • python-omgeo ==6.1.0
  • rauth ==0.7.3
  • redis <4
  • requests ==2.26.0
  • retry ==0.9.2
  • rollbar ==0.16.2
  • suds-community ==1.0.0
  • tr55 ==1.3.0
  • ulmo ==0.8.8
src/mmw/requirements/development.txt pypi
  • curlify2 ==1.0.1 development
  • flake8 ==4.0.1 development
  • ipdb ==0.13.9 development
  • ipython ==7.17.0 development
  • jedi ==0.17.2 development
src/mmw/requirements/production.txt pypi
  • awscli ==1.22.5
  • boto ==2.49.0
  • boto3 ==1.20.5
  • django-storages ==1.9.1
src/mmw/requirements/test.txt pypi
  • coverage ==6.1.2 test
  • flake8 ==4.0.1 test
  • selenium ==4.0.0 test

Score: 9.602112152952344