Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

Collect Earth Online

Crowdsourced visual interpretation of on-demand satellite imagery for land use assessment.
https://github.com/openforis/collect-earth-online

Category: Natural Resources
Sub Category: Soil and Land

Keywords from Contributors

chirps climate climateserv emodis esi imerg ndvi servir servirglobal

Last synced: about 6 hours ago
JSON representation

Repository metadata

Collect Earth Online

README.org 

          * Collect Earth Online

Crowd-sourced visual interpretation of on-demand satellite imagery

** Installation Requirements

*** Required

- [[https://jdk.java.net][Java Development Kit (version 11, or 17)]]
- [[https://clojure.org/guides/install_clojure][Clojure CLI tools 1.10+]]
- [[https://www.postgresql.org/download][Postgresql (version 12)]]
- [[https://postgis.net/install/][PostGIS (version 3.2+)]]
- [[https://www.7-zip.org/][p7zip]]
- [[https://nodejs.org/en/][node 18]]
- [[https://www.python.org/downloads/][Python 3]]

** Application Configuration

On startup, the CEO application reads a file called ~config.edn~ in the top level directory of this repository, which contains various user-configurable system parameters. A sample file called ~config.example.edn~ is provided with this repository to get you started. To begin, simply copy it to ~config.edn~. Then edit it according to the following sections.

Please note that you must replace all values surrounded with angle brackets (e.g., "") with actual values for the application to work correctly. Most of the fields should be self-explanatory.

** PostgreSQL Database Setup

*PostgreSQL* needs to be installed on the machine that will be hosting this website. This installation task is system specific and is beyond the scope of this README, so please follow the instructions for your operating system and PostgreSQL version. However, please ensure that the database server's superuser account is named ~postgres~ and that you know its database connection password before proceeding.

*** PostgreSQL Server

To confirm that PostgreSQL is running as a server on your machine, you'll need to check its ~systmed~ or ~sysvinit~ process. To figure out which you're using on your system, run ~ps -p 1 -o comm=~. Once you know if you're using ~systemd~ or ~sysvinit~, you can run the following command:

#+begin_src sh
# If you're using systemd:
sudo systemctl status postgresql.service
# If you're using sysvinit:
sudo service postgresql status
#+end_src

If it's not running, you'll need to start it by running:
#+begin_src sh
# If you're using systemd:
sudo systemctl start postgresql.service
# If you're using sysvinit:
sudo service postgresql start
#+end_src

Finally, to ensure that your PostgreSQL server always restarts when your system reboots, you can run:
#+begin_src sh
# If you're using systemd:
sudo systemctl enable postgresql.service
# If you're using sysvinit:
sudo update-rc.d postgresql enable
#+end_src

*** CEO Database

Once the PostgreSQL database server is running on your machine, you should navigate to the top level directory (i.e., the directory containing this README) and run the database build command as follows:

#+begin_src sh
clojure -M:build-db build-all --dev-data
#+end_src

This will begin by creating a new database and role called ~ceo~ and then add the ~postgis~ and ~pgcrypto~ extensions to it. Next, the script will populate the database with the schemas, tables, and functions that are necessary for storing and processing ceo's data. Finally, it will load some default data into these tables that is necessary for the website to function properly. You can optionally load dev data with ~--dev-data~.  This will initialize the DB with 3 users, an imagery source, and a project.

If you wish to use a live copy of the CEO database instead of the dev data and you have a ~.dump~ file, run the following command:

#+begin_src sh
clojure -M:build-db restore -f /path/to/ceo/database/ceo-db---.dump
#+end_src

*** Performance Settings

If you want to improve the performance of your Postgresql server, one way is to visit the [[https://pgtune.leopard.in.ua/][pgtune]] website and input your system settings. This will generate a set of configuration options that you can add to the postgresql.conf file in your system-specific Postgresql data directory.

If you would like to keep these settings separate for your own reference, you can add them to a file called "performance.conf" that you should place in your Postgresql data directory. You can then import these settings into postgresql.conf by adding this line to the end of that file:

#+begin_example
include = 'performance.conf'
#+end_example

Note that the Postgresql server will need to be restarted for these changes to take effect.

** Environment Variable Setup

In order for your command line shell to find the programs installed above, you will need to update your environment variables. Under GNU/Linux, *BSD, or MacOS X, you can set these through your shell's .*rc or .*profile configuration files in your $HOME directory. In a Windows environment, you will need to set either the USER (for the current user only) or SYSTEM (for any user) environment variables under your system settings menu.

*** Step 1: Add JAVA_HOME

On a Unix machine using bash (e.g., GNU/Linux or MacOS X):

#+begin_src sh
export JAVA_HOME=
#+end_src

On a Unix machine using csh or tcsh (e.g., *BSD):

#+begin_src sh
setenv JAVA_HOME 
#+end_src

On a Windows machine:

#+begin_example
JAVA_HOME = C:\Program Files\Java\jdk-X.X.X
#+end_example

Replace *X.X.X* with the JDK version installed on your computer. In order to run CEO, your JDK version has to be either 11 or 17.

*** Step 2: Update PATH

On a Unix machine using bash (e.g., GNU/Linux or MacOS X):

#+begin_src sh
export PATH=$PATH:$JAVA_HOME/bin
#+end_src

On a Unix machine using csh or tcsh (e.g., *BSD):

#+begin_src sh
setenv PATH $PATH\:$JAVA_HOME/bin
#+end_src

On a Windows machine, add these entries to the PATH environment
variable:

#+begin_example
%JAVA_HOME%\bin
C:\Program Files\PostgreSQL\X\lib
C:\Program Files\PostgreSQL\X\bin
C:\Program Files\7-Zip
#+end_example

Replace *X* with the PostgreSQL version installed on your computer.

** Configuration

Copy ~config.default.edn~ to ~config.edn~ and update any settings that you wish. The base set of defaults should be sufficient to run CEO in a development setting.

** Python dependencies

Use ~pip~ to install dependencies.  The python / clj interop does not support virtual environments.

#+begin_src sh
pip install -r requirements.txt
pip install earthengine-api --upgrade
#+end_src

** Development Web Server

*** Google Earth Engine authentication

You will need a Google Earth Engine account to use the GeoDash. To apply for a GEE account, visit https://signup.earthengine.google.com/#!/. Once you receive an account, you can store local authentication information with:

#+begin_src sh
earthengine authenticate
#+end_src

This command will open a window in your browser containing a key that you should copy and paste back into your terminal. You do not need to update the ~:gee~ section of ~config.edn~ if you use this method. If the above method is giving you issues, try following [[the official step-by-step instructions provided by Google][https://developers.google.com/earth-engine/guides/python_install#expandable-2]].

*** Modifying hosts file

Add the following line to your hosts file. This local URL should be used for local development in place of ~localhost~ due to CORS settings on CEO's Mapbox account.

#+begin_src sh
127.0.0.1 local.collect.earth
#+end_src

*** Create accounts

To be able to crate accounts without having to send email for confirmation, set ~:auto-validate?~ of ~:mail~ to ~true~ in ~config.edn~.

There can only be one superuser account with ~user_uid~ of ~1~ ~adminstrator~ flag set to ~true~. Activating this flag for other users can cause their queries to sometimes not return the full result, as some of those queries assume only one root superuser to be existing.

*** Compile and run

To compile and run the web application, navigate to the top level project directory and run:

#+begin_src sh
npm install
npm run vite-dev
#+end_src

Now, in a separate terminal window (because webpack needs to be running in the background), run the following commands:

#+begin_src sh
npm run server-dev
#+end_src

The website will then be available at http://localhost:8080 unless a port is specified. These can also be configured using the ~:server~ section in your ~config.edn~ file. When using the ~clojure -M:server start~ command (which is what ~npm run server-dev~ is an alias for) an http port can be specified with -p and an https port can be specified with -P. In dev mode, server-side exceptions will be displayed in the browser and JavaScript source files will be reloaded whenever you refresh the page.

Going forward you will usually only need to run ~npm run webpack-dev~ and ~npm run server-dev~ (still in separate terminals) to get your dev environment set up.  Watch for updates to SQL files or ~package.json~ and run ~npm run build-db-functions~ or ~npm install~ respectively.

*** Checking for Reflection Warnings

From the top level project directory run:

#+begin_src sh
clojure -M:check-reflection
#+end_src

This will emit warnings when reflection is needed to resolve Java method calls or field accesses. To resolve any that appear, add [[https://clojure.org/reference/java_interop#typehints][type hints]] to your code. Resolving all reflection warnings can improve system performance.

** Production Web Server

*** Sessions

It is very important to change the default ~:session-key~ in ~config.edn~.  This key is used to encrypt user session data and should be unique to each deployment. The key must be exactly 16 characters long.

*** Email Server

To set up the email server for system emails, open the "config.edn" file in the root directory of the application. Edit the default EDN object containing server details to the file, replacing the values with your own.

*** Enabling HTTPS (optional)

To enable HTTPS from within the server, view the [[https://github.com/sig-gis/triangulum#triangulumhttps][Triangulum HTTPS]] page for further instructions on enabling HTTPS.

*** Google Earth Engine service account

For production it is recommended that you use a service account with a key file. You can obtain your key file by logging into your service account, navigating to the account menu, and clicking "Create key > JSON". Then, download that JSON key file and place it in the root directory of CEO. Set the email for your service account and key path in the ~:gee~ section of config.edn.

#+begin_src text
:gee {:ee-account  "example@gmail.com"
      :ee-key-path "ceo-gee-key.json"}
#+end_src

*** Launching the Web Server

To compile and run the web application, navigate to the top level project directory and run:

#+begin_src sh
npm install
npm run webpack-prod
clojure -M:build-db functions -d ceo
clojure -M:server start -m [dev|prod] [-p 8080] [-P 8443] [-r]
#+end_src

The website will then be available at http://localhost:8080 unless a port is specified. These can also be configured using the ~:server~ section in your ~config.edn~ file. An http port can be specified with -p and an https port can be specified with -P. In dev mode, server-side exceptions will be displayed in the browser and Clojure source files will be reloaded whenever you refresh the page. These features are disabled in prod mode. If -m is unspecified, it will default to prod mode.

*** Running the Web Server as a System Service

View the [[https://github.com/sig-gis/triangulum#triangulumsystemd][Triangulum Systemd]] page for further instructions on enabling the app as a system service.

*** Maintaining Daily Logs

By default the server will log to standard out. If you would like to have the system log to YYYY-DD-MM.log, use the "-o path" option to specify an output path. You can either specify a path relative to the top level directory of this repository or an absolute path on your filesystem. The logger will keep the 10 most recent logs.

*** Using the Announcement Banner

On each page load clojure will read the value of ~announcement.txt~. If text is found, the value will be inserted into a HTML element that displays as a red banner at the top of the page. To add a new announcement, edit ~announcement.txt~ and add a new message.  To remove the announcement, edit ~announcement.txt~ and remove all text.

** Contact

*Authors:*
- [[mailto:dsaah@sig-gis.com][David S. Saah (SIG)]]
- [[mailto:mspencer@sig-gis.com][Matt Spencer (SIG)]]
- [[mailto:gjohnson@sig-gis.com][Gary W. Johnson (SIG)]]
- [[mailto:billy.ashmall@nasa.gov][Billy Ashmall (NASA)]]
- [[mailto:roberto.fontanarosa@fao.org][Roberto Fontanarosa (FAO)]]
- [[mailto:obaldwinedwards@sig-gis.com][Oliver Baldwin Edwards (SIG)]]
- [[mailto:githika.tondapu@nasa.gov][Githika Tondapu (NASA)]]
- [[mailto:stefano.ricci@fao.org][Stefano Ricci (FAO)]]
- [[mailto:alfonso.sanchezpausdiaz@fao.org][Alfonso SanchezPausDiaz (FAO)]]
- [[mailto:bbhandari@sig-gis.com][Biplov Bhandari (SIG)]]

** License and Distribution

Copyright © 2016-2022 FAO.

Collect Earth Online is distributed by FAO under the terms of the MIT License. See LICENSE in this directory for more information.

Owner metadata

GitHub Events

Total
Last Year

Committers metadata

Last synced: 2 days ago

Total Commits: 4,690
Total Committers: 31
Avg Commits per committer: 151.29
Development Distribution Score (DDS): 0.676

Commits in past year: 212
Committers in past year: 4
Avg Commits per committer in past year: 53.0
Development Distribution Score (DDS) in past year: 0.434

Name Email Commits
Matt Spencer s****r@g****m 1519
Gary Johnson g****o@u****u 598
a-luz a****z@s****m 353
Gary Johnson l****c@g****m 331
NDC\gtondapu g****u@n****v 328
Billy Ashmall b****l@n****v 304
Roberto Fontanarosa r****a@g****m 279
Biplov Bhandari b****5@g****m 215
RJ Sheperd r****d@g****m 189
Oliver-BE o****s@g****m 110
Andrew Carlile a****e@s****m 108
Sif Biri s****i@s****m 72
Matt Spencer s****l@s****o 69
fefrain f****n 53
Jerome j****i@g****m 38
jonnvega j****a@g****m 30
John Dilger j****r@g****m 21
Thomas DeVera d****a@p****t 16
yzq.yang y****g@g****m 15
Roberto Fontanarosa r****a@g****m 8
fefrain f****n 5
kdyson k****n 5
tdev2308 t****v@c****t 5
Fefrain f****n@o****t 4
Alfonso Sanchez-Paus Diaz A****z@f****g 4
justineb17 7****7 3
Gary Johnson l****c@d****g 2
dependabot[bot] 4****] 2
David S Saah d****h@s****m 2
Andrew a****9@g****m 1
and 1 more...

Committer domains:

Issue and Pull Request metadata

Last synced: 2 days ago

Total issues: 22
Total pull requests: 244
Average time to close issues: 5 months
Average time to close pull requests: 16 days
Total issue authors: 12
Total pull request authors: 11
Average comments per issue: 1.23
Average comments per pull request: 0.18
Merged pull request: 200
Bot issues: 0
Bot pull requests: 0

Past year issues: 5
Past year pull requests: 49
Past year average time to close issues: 24 days
Past year average time to close pull requests: 8 days
Past year issue authors: 4
Past year pull request authors: 3
Past year average comments per issue: 1.4
Past year average comments per pull request: 0.0
Past year merged pull request: 32
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/openforis/collect-earth-online

Top Issue Authors

  • ricds (9)
  • rohansaw (2)
  • MatthieuStigler (2)
  • Olganep (1)
  • paullc (1)
  • fargotty (1)
  • Pedrord-gis (1)
  • MarshallLaw1581 (1)
  • ElieNgembo (1)
  • chitrez (1)
  • moralesluciacr (1)
  • twoldemariam (1)

Top Pull Request Authors

  • a-luz (128)
  • jollyblondgiant (54)
  • sirmspencer (17)
  • jonnvega (14)
  • sifbiri (13)
  • Oliver-BE (9)
  • jdilger (4)
  • lambdatronic (2)
  • ricds (1)
  • carloshormazabal (1)
  • kyle-woodward (1)

Top Issue Labels

Top Pull Request Labels

  • Bug Fix (1)

Dependencies

.github/workflows/linter.yml actions
  • actions/checkout v2 composite
  • actions/setup-node v2 composite
  • sibiraj-s/action-eslint v1.1.0 composite
package.json npm
  • @babel/core ^7.11.6 development
  • @babel/eslint-plugin ^7.14.5 development
  • @babel/plugin-proposal-class-properties ^7.10.4 development
  • @babel/preset-env ^7.11.5 development
  • @babel/preset-react ^7.10.4 development
  • babel-eslint ^10.1.0 development
  • babel-loader ^8.2.3 development
  • babel-plugin-lodash ^3.3.4 development
  • buffer ^6.0.3 development
  • css-loader ^4.3.0 development
  • eslint ^7.32.0 development
  • eslint-config-airbnb ^18.2.1 development
  • eslint-plugin-import ^2.25.4 development
  • eslint-plugin-jsx-a11y ^6.5.1 development
  • eslint-plugin-react ^7.29.2 development
  • eslint-plugin-react-hooks ^4.3.0 development
  • react-svg-loader ^3.0.3 development
  • style-loader ^1.2.1 development
  • tern ^0.23.0 development
  • webpack ^5.70.0 development
  • webpack-cli ^4.9.2 development
  • @babel/plugin-transform-runtime ^7.16.0
  • @ckeditor/ckeditor5-build-classic ^19.0.0
  • @ckeditor/ckeditor5-react ^2.1.0
  • airbnb-prop-types ^2.16.0
  • highcharts ^9.1.0
  • highcharts-react-official ^3.0.0
  • lodash ^4.17.20
  • ogr2ogr ^1.5.0
  • ol ^6.4.3
  • prop-types ^15.7.2
  • react ^16.13.1
  • react-dom ^16.13.1
  • react-grid-layout ^1.3.0
  • shpjs ^4.0.2
requirements.txt pypi
  • python-dateutil *
  • shapely *
  • shapely_geojson *
src/clj/collect_earth_online/utils/project.clj clojars

Score: 7.577121930876679