Marxan Cloud platform

Welcome to the Marxan Cloud platform. We aim to bring to the planet the finest
workflows for conservation planning.

Quick start

This repository is a monorepo which includes all the microservices of the Marxan
Cloud platform. Each microservice lives in a top-level folder.

Services are packaged as Docker images.

Microservices are set up to be run with or without Docker Compose for local
development - see the sections below for more details.

The recommended setup for new developers is to run all the backend services (api
and geoprocessing services, alongside their PostgreSQL and Redis databases) via
Docker Compose, and the frontend app natively.

In CI, testing, staging and production environments, microservices are
orchestrated via Kubernetes (see the relevant
documentation).

Most of the commands listed in this README and referenced elsewhere in the
repository are targeted at a GNU/Linux OS environment such as a recent Ubuntu,
Arch or Debian system, whether running natively or in a VM or under Windows
Subsystem for Linux 2 (WSL 2). They should also work identically on MacOS, while
they may need some adaptation to run on Windows systems.

Platform architecture

In a nutshell, the Marxan solution is composed by the following components:

• A frontend application accessible through the browser - the app
• A public, backend API - the api
• A geoprocessing-focused service used by the api - the geoprocessing api/application
• An HTML-to-PDF/HTML-to-PNG service - the webshot service

Besides these 4, there are other components that may be used in one-off
situations, like seeding source data (see /data), testing
(/e2e-product-testing) and others.

See ARCHITECTURE_infrastructure.md for
details.

Dependencies

• NodeJS v14.18 and v16.14
• Yarn v1
• PostgreSQL v14
• Postgis v3
• Redis v6
• A Sparkpost account

For development environments, a separate Sparkpost account than what is used for
staging/production should be used. Unless the transactional email components of
the platform are being actively worked on (email verification on signup, email
confirmation for password changes, email flow for resetting forgotten passwords,
etc.), there will be no need to set up email templates within the Sparkpost
account, and only a Sparkpost API key will be needed (see documentation on
environment variables for details on this).

Running Marxan using Docker

Before attempting to use the following steps, be sure to:

• Install Docker (19.03+):
• Install Docker Compose
• Create an .env at the root of the repository, defining all the required
  environment variables. In most cases, for variables other
  than secrets, the defaults in env.default may just work - your mileage may
  vary.

For development environments, a .env file can be generated with
default/generated values suitable to run a development instance, via the
following command:

make .env

This will only generate a file if no .env is present in the root of the
repository.

The PostgreSQL credentials set via environment variables are used to create a
database user when the PostgreSQL container is started for the first time.
PostgreSQL data is persisted via a Docker volume.

Running the Marxan Cloud platform

Run make start-api to start all the 4 services needed to run Marxan, as well
as the required database services, in containers via Docker Compose.

The docker build process may take a few minutes, depending on your hardware,
software and internet connection. Once completed, the applications will start,
and you should be able to access the Marxan site on localhost, on the port
specified as APP_SERVICE_PORT.

Debugging via Node inspector

To enable the Node inspector while running the MarxanCloud API services in
containers, use make debug-api instead. Example configuration
files for debugger setup in popular editors are provided in the
docs/developers/editors/ documentation folder.

When enabled, the Node inspector will start listening on the default port
9229/tcp both in the API and geoprocessing containers, and by default Docker
will forward this port to port 9230/tcp on the host for the API service, and
to port 9240/tcp for the geoprocessing service, where the inspector can be
reached by clients.

For security reasons (in case the host is, for example, a VM with a public IP
address and without a firewall in front, for whatever reason), the inspector
port will only be open on the loopback interface.

Running Marxan natively

Make sure you have installed and configured all the
dependencies locally. PostgreSQL (with PostGIS) and Redis need
to be up and running.

Running API and Geoprocessing services

When running the API and Geoprocessing services without relying on Docker
Compose for container orchestration, be sure to review and set the correct
environment variables before executing the application.
The env.default file and the docker-compose configuration files may give
you some example values that work for docker-based executions, and that may
be useful when implementing your native execution configuration.

The included Makefile has some useful build targets (commands) specifically
targeted at native execution (prefixed with native-) that you'll find helpful.
Refer to the Makefile inline documentation for more details.

If you'd like to run the application directly using Yarn, you can find a
package.json inside the /app folder with dependencies and commands for both
applications. After installing the nodejs dependencies, this is how you can
start either application:

// Run the API
yarn start

// Run the geoprocessing service
yarn start geoprocessing

Running the Frontend application

The Frontend application can be found in /app. Be sure to populate the
spp/.env file (note: this is an .env file distinct from the top-level .env
file which is used to configure backend microservices and data processing
pipelines) according to the app documentation, as well as
install the necessary nodejs packages.

To start the application, run:

yarn dev

The frontend app will then be available on http://localhost:3000 (or at the URL
shown when the app starts, if a different port has been configured).

Running the webshot service

The webshot service can be found in the /webshot folder. After installing
the necessary nodejs packages, you can start it by running:

yarn start:dev

Due to upstream packaging of the Chrome browser used by the Webshot service, it
may not be possible to run the webshot service in aarch64 environments (such
as MacOS on Apple silicon).

Setting up test seed data

make native-seed-api-with-test-data

Running tests

Running the whole test suite requires running 3 commands, each focused on a
specific type of test:

To run the unit tests for both the API and the Geoprocessing app:

yarn run test

To run the E2E tests for the API:

yarn run api:test:e2e

To run the E2E tests for the Geoprocessing app:

yarn run geoprocessing:test:e2e

Note that E2E tests may trigger cross-application requests, so:

• When running E2E tests for the API, you must have the Geoprocessing
  application running in the background.
• When running E2E tests for the Geoprocessing application, you must have the
  API running in the background.

Running tests require previously loading the test seed
data, and may modify data in the database - do not
run tests using a database whose data you don't want to lose.

Seed data

All fresh installations of Marxan (be it locally for development or in a cloud
provider for production) start off with empty databases, that need to be
populated with seed data before the Marxan platform is fully functional. The
seed data you'll want to import will depend on the goal of the installation you
are currently setting up.

Please make sure to wait for all of the backend services (api, geoprocessing and
webshot) to fully start, as database migrations will be run while the services
are started: attempting to import seed data before migrations have run fully
will result in errors.

There are three types of seed data available with the application:

• Geographic data: platform-wide spatial data for admin boundaries (GADM),
  protected areas (WDPA) and conservation features, such as the World
  Terrestrial Ecosystems database. These datasets should be available in every
  Marxan instance.
• User data: user accounts, intended only for development instances and for
  e2e/unit tests; these must not be imported in production-grade environments.
• Test data: intended only for environments where development or e2e/unit tests
  execution takes place, and must not be imported in production-grade
  environments.

Please review the following sections carefully to determine which best fits your
needs for each deployment

User data

User data is necessary for all types of Marxan installations, but different user
data import processes will best fit different use cases.

There are two ways to create user accounts:

Using the nodejs CLI

cd api
yarn run console create:user EMAIL_ADDRESS PASSWORD [-f, --firstname <first name>] [-l, --lastname <last name>] [-d, --displayname <display name>]

Using Make

// For Marxan running on Docker
make seed-api-init-data
// For Marxan running natively
make native-seed-api-init-data

The first option will allow you to create a custom user, and is targeted at
environments where user accounts are meaningful - for example, production. To
execute this on a cloud hosted version of Marxan, you should run the command
above on the VM instance/docker container running the api application.

In contrast, the second approach will batch-create several users with insecure
passwords and generic details, and it's only suited for development, testing
or otherwise ephemeral environments.

Geographic data

Importing the initial geographic data executes a long-running data ETL pipeline
that imports large amounts of data from publicly available datasets onto
Marxan's PostgreSQL server - using both api and geoprocessing databases.

First, either set up a new Marxan instance from scratch, or reset an existing
one to a clean-slate status (make clean-slate && make start-api) - this allows
to import spatial data into a clean database, avoiding that any user-uploaded
data may end up in seed data.

Once a clean Marxan instance is running, the easiest way to execute the spatial
data import process is using the following make task, which runs a dockerized
version of the tool:

make seed-geodb-data

Note this process can complete successfully and exit with code 0, but have
errors in the output logs. This is expected, and said log errors can be ignored.

The actual implementation can be found in the /data folder

This will populate the metadata DB and will trigger the geoprocessing ETL
pipelines to seed the geoprocessing DB with the full data that is needed for
production-grade instances of Marxan.

Please note that this full DB set up will require at least 16GB of RAM and 40GB
of disk space in order to carry out some of these tasks (GADM and WDPA data
import pipelines). Also, the number of CPU cores will impact the time needed to
seed a new instance with the complete GADM and WDPA datasets, which will be 1h+
on ideal hardware.

To execute this on a cloud hosted version of Marxan, you have a couple of
options:

• Run the import process locally on local running PostgreSQL servers, then
  export the resulting .sql locally and import it remotely.
• Run the import process locally, while having it connect directly to the remote
  api and geoprocessing databases using kubectl to set up port forwarding.
  You may need to modify the container's network
  mode
  to host for this to work.

While geographic data is technically necessary on all Marxan environments, there
is a faster alternative to import equivalent data on development/test
environments, which is discussed in the next section.

Test data

Test data includes both user data and (a small subset of) the geographical data
described above, as well as extra data necessary to run certain types of
automated tests. This data is meant for development/testing environments only,
and should not be imported in production environments.

// For Marxan running on Docker
make seed-dbs
// For Marxan running natively
make native-seed-api-init-data

These commands will:

• Import generic user data (equivalent to
  seed-api-init-data/native-seed-api-init-data described above)
• Import a precomputed subset of the geographical data
• Create sample/test Marxan resources, like organizations, scenarios, etc.

Maintenance

Resetting data to a clean slate status (docker only)

The main Makefile provides a way to reset db instances from scratch. This can
be useful to do regularly, to avoid keeping obsolete data in the local
development instance.

make clean-slate

Update seed data (GADM, WDPA) from newer upstream releases

The main Makefile provides a set of commands to create new db dumps from
upstream data sources, upload these dumps to an Azure storage bucket, and
populating both dbs from these dumps.

Populating clean dbs this way will be much quicker than triggering the full
ETL pipelines to import geographic data.

When uploading new dumps of seed data to an Azure storage container, or when
downloading pre-prepared data seeds from it, the following environment variables
must be defined in the root .env file:

DATA_SEEDS_AZURE_STORAGE_ACCOUNT_NAME=
DATA_SEEDS_AZURE_STORAGE_CONTAINER_NAME=

This will allow to run the az storage blob commands in the relevant Make
recipes with suitable authorization.

Users should have suitable access to the storage container configured.

For data uploads, they will need to be logged into an Azure account that is
allowed to write to the container:

• install the Azure CLI tool
  (az)
• get an Azure user set up, with suitable permissions to write to the relevant
  Azure storage account and container
• log in to this Azure account via the az CLI tool
  (https://learn.microsoft.com/en-us/cli/azure/authenticate-azure-cli)

For data downloads, the container itself needs to be created with "public blobs"
settings so that individual blobs can be fetched via non-authenticated HTTP
requests.

To run the geoprocessing ETL pipelines (such as when using the Seed data,
option 1 above) to dump data from a previously seeded instance and upload the
processed data to an Azure bucket:

make generate-content-dumps && make upload-dump-data

Other developers can then benefit from these pre-prepared data seeds when
populating new development instances after their initial setup, by running the
following command on a clean Marxan instance (that is, from empty databases, and
after letting all the migrations run for both api and geoprocessing
services):

make restore-dumps

Running the notebooks

This step is only needed when developing Python notebooks for Marxan.

Run make notebooks to start the jupyterlab service.

Development workflow

We use a lightweight git flow workflow. develop, main, feature/bug fix
branches, release branches (release/vX.Y.Z-etc).

Please use per component+task feature branches: <feature type>/<component>/NNNN-brief-description. For example:
feature/api/12345-helm-setup.

PRs should be rebased on develop.

As feature types:

• feature
• bugfix (regular bug fix)
• hotfix (urgent bug fixes fast-tracked to main)

Devops

Infrastructure

Infrastructure code and documentation can be found under /infrastructure

CI/CD

CI/CD is handled with
GitHub Actions. More details can be found
by reviewing the actual content of the .github/workflows folder but, in a nutshell,
GitHub Action will automatically run tests on code pushed as part of a Pull Request.

For code merged to key branches (currently main and develop), once tests run
successfully, Docker images are built and pushed to a
private Azure Container Registry.

The GitHub Actions workflows currently configured requires a few secrets
to be set on GitHub in order to work properly:

• AZURE_CLIENT_ID: Obtain from Terraform's azure_client_id output
• AZURE_TENANT_ID: Obtain from Terraform's azure_tenant_id output
• AZURE_SUBSCRIPTION_ID: Obtain from Terraform's azure_subscription_id output
• REGISTRY_LOGIN_SERVER: Obtain from Terraform's azurerm_container_registry_login_server output
• REGISTRY_USERNAME: Obtain from Terraform's azure_client_id output
• REGISTRY_PASSWORD: Obtain from Terraform's azuread_application_password output

Some of these values are obtained from Terraform output values, which will be documented
in more detail in the Infrastructure docs.

Bugs

Please use the Marxan Cloud issue
tracker to report bugs.

License

(C) Copyright 2020-2023 Vizzuality.

This program is free software: you can redistribute it and/or modify it under
the terms of the MIT License as included in this repository.

This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the MIT License for more details.

You should have received a copy of the MIT License along with this program. If
not, see https://spdx.org/licenses/MIT.html.