TSX

Provides reliable and robust measures of change in the relative abundance of Australia's threatened and near-threatened species at national, state and regional levels.
https://github.com/nesp-tsr3-1/tsx

Category: Biosphere
Sub Category: Terrestrial Wildlife

Keywords from Contributors

optimize archiving measur transforms observation animals generic projection compose conversion

Last synced: about 23 hours ago
JSON representation

Repository metadata

Australian Threatened Species Index

• Host: GitHub
• URL: https://github.com/nesp-tsr3-1/tsx
• Owner: nesp-tsr3-1
• License: mit
• Created: 2017-11-21T03:30:08.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2026-05-07T03:24:18.000Z (11 days ago)
• Last Synced: 2026-05-14T19:04:47.148Z (3 days ago)
• Language: Python
• Homepage: https://tsx.org.au
• Size: 53.8 MB
• Stars: 6
• Watchers: 1
• Forks: 1
• Open Issues: 3
• Releases: 0
• Metadata Files:
  • Readme: README.rst
  • License: LICENSE

README.rst

          ===================================================================
Welcome to the Australian Threatened Species Index (TSX) Repository
===================================================================

For information about the Australian Threatened Species Index project, see https://tsx.org.au

The code in this repository is responsible for data import and pre-processing of data, and visualsation of the generated trend graphs and associated data.


User Guide
==========

The easiest way to learn about running the TSX workflow is to check out the User Guide at https://tsx.org.au/user-guide/

Overview
========

The TSX consists of several components

- MySQL database:
    - For storage of raw data, which is either uploaded via the data interface, or imported via the command line scripts
    - For storage for the data interface (e.g. user accounts, custodian feedback data, etc.)
    - For processing of data using SQL queries and storing the results
    - database schema is found under `db/sql/create.sql`

- Python scripts:
    - For running the TSX workflow that transforms raw data into time series
    - For importing data such as taxonomic lists and species range polygons
    - For generating trend permutations to be displayed by the TSX visualiser
    - Found in the `tsx` directory

- LPI R package (https://github.com/Zoological-Society-of-London/rlpi)
    - Developed by Zoological Society of London
    - For producing trends from time series using the Living Planet Index method
    - We maintain a fork at https://github.com/nesp-tsr3-1/rlpi for bugfixes

- API backend:
    - Implemented in Python as a Flask app
    - Backend for TSX Data Interface and TSX Visualiser
    - Code lives under `tsx/api`

- Data interface front-end (https://tsx.org.au/data)
    - Implemented using the Vue Javascript framework
    - Provides an interface for users to upload, manage and analyse datasets
    - Requires login access (data custodians can self-register)
    - Code lives under `web/data`

- TSX Visualiser (https://tsx.org.au/tsx)
    - Implemented using the Vue Javascript framework
    - A single-page application that allows users to explore a set of trend permutations
    - A separate instance is deployed for each new version of the TSX
    - Code lives under `web/tsx`


Setup
=====

Prerequisites
-------------

- MySQL 8.x
- Python 3.12+
- Node 22+
- R 3.6+
- `uv `_

Clone this repository
---------------------

git clone https://github.com/nesp-tsr3-1/tsx.git


Native install vs Docker Compose
--------------------------------

There are two main options for getting a TSX development environment up and running:

1. Installing dependencies natively as per instructions below

2. Use the supplied Docker Compose configuation to automatically build containers with the necessary dependencies included. This is quicker to get up and running, but is arguably more complicated to use for development and requires some familiarity with Docker. (See `Docker Compose Instructions`_.)


Install native packages
-----------------------

Install development libraries (Ubuntu/Debian):

.. code:: bash

 sudo apt-get update
 sudo apt-get install -y --no-install-recommends libgdal-dev r-base r-base-dev git build-essential libharfbuzz-dev libfribidi-dev  libfontconfig1-dev libgit2-dev libssl-dev  default-mysql-client libbz2-dev curl

On macOS, we recommend using homebrew to install packages as needed while following the steps below.

Create MySQL database
---------------------

By default, the TSX software is configured to use a database called 'tsx' with a username of 'tsx' and a password of 'tsx'. This is configured in the `tsx.conf` file.

To create this database and user, connect to MySQL as root and run the following commands:

.. code:: sql

 CREATE USER 'tsx'@'%' IDENTIFIED BY 'tsx';
 CREATE DATABASE tsx;
 GRANT ALL PRIVILEGES ON tsx.* TO 'tsx'@'%';


Alternatively, you can avoid installing MySQL and instead run it via Docker (https://hub.docker.com/_/mysql/):

.. code:: bash

 docker run -d --name tsx-mysql -e MYSQL_DATABASE=tsx -e MYSQL_USER=tsx -e MYSQL_PASSWORD=tsx \
   -e MYSQL_ROOT_PASSWORD=root -p 3306:3306 mysql:8.3.0


Initialise MySQL database
--------------------------

Initialise the database using the provided scripts:

.. code:: bash

 mysql -u root -p tsx < db/sql/create.sql
 mysql -u root -p tsx < db/sql/init.sql
 mysql -u root -p tsx < sample-data/seed.sql


Copy example configuration
--------------------------

.. code:: bash

 cp tsx.conf.example tsx.conf

If necessary edit the ``[database]`` section in tsx.conf to match the database you just created.

Install R dependencies
------------------------

R dependencies are managed using `renv `_.

Install R dependencies by running:

.. code:: bash

  Rscript -e 'renv::restore()'

Install Python dependencies
----------------------------------

Python dependencies are managed using `uv `_.

Install uv: https://docs.astral.sh/uv/getting-started/installation/

Use uv to install Python and packages:

.. code:: bash

 uv sync


Install node and Javascript dependencies
----------------------------------------

Install Javascript dependencies:

.. code:: bash

 cd web
 npm install


Run API backend
---------------

This is required for the Data Interface and/or TSX Visualiser to function

.. code:: bash

  # Native environment
  uv run ./dev-server.sh

  # Docker compose (note this also runs the Data Interface front end)
  docker compose --profile webapp up


Run Data Interface
------------------

.. code:: bash

  cd web
  npm run dev-data


Run TSX Visualiser
------------------

.. code:: bash

  cd web
  npm run dev-tsx


Note: the TSX Visualiser will not function properly until time series and trend permutations have been generated.


Docker Compose Instructions
===========================

The following command will build necessary Docker containers and start a shell for running the workflow:

.. code:: bash

  docker compose run --build --rm workflow_cli

It will take a while to download the necessary packages and build the containers.

Important notes:

- When running via Docker Compose, ``tsx.conf`` must be configured with a database hostname of ``mysql``. (i.e. set ``host=mysql`` under ``[database]``)
- The project root directory is mounted to the default working directory (``/tsx/``) inside each container. In order to import your own files using the workflow, you will need to first put them within the project directory tree so that the container can see them.

To run the Data Interface and TSX Visualiser, use:

.. code:: bash

  docker compose --profile webapp up


Generate time series and permutations for sample data
=====================================================

A script is provided that will perform a full workflow run using the sample data and generate trend permutations for the TSX visualiser.

.. code:: bash

 uv run setup/test-workflow.sh

(Make sure you have completed `Initialise MySQL database`_ first.)

        

---

Owner metadata

• Name: NESP Threatened Species Recovery Hub (Project 3.1)
• Login: nesp-tsr3-1
• Email:
• Kind: organization
• Description:
• Website:
• Location:
• Twitter:
• Company:
• Icon url: https://avatars.githubusercontent.com/u/33853157?v=4
• Repositories: 1
• Last ynced at: 2023-03-08T18:46:30.031Z
• Profile URL: https://github.com/nesp-tsr3-1

---

GitHub Events

Total

• Pull request event: 1
• Push event: 77
• Create event: 1

Last Year

• Push event: 42

---

Committers metadata

Last synced: 3 days ago

Name	Email	Commits
James Watmuff	j****s@p****u	912
James Watmuff	j****s@J****l	144
James Watmuff	j****s@J****y	54
hoangnguyen177	h****7@g****m	23
Ubuntu	u****u@n****u	8
Ubuntu	u****u@n****u	2
dependabot[bot]	4****]	1

Committer domains:

• nesp-dev.coesra.org.au: 1
• nesp-dev1.coesra.org.au: 1
• jamess-mbp.home.gateway: 1
• planticle.com.au: 1

---

Issue and Pull Request metadata

Last synced: 13 days ago

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/nesp-tsr3-1/tsx

---

---

---

Dependencies

Score: 4.143134726391533