tree-detection-framework

This project has three main goals:

• Enable tree detection on realistic-scale, geospatial raster data with minimal boilerplate, using existing (external) tree detection/segmentation models
• Facilitate direct comparison of multiple algorithms
• Rely on modern libraries and software best practice for a robust, performant, and modular tool

This project does not, itself, provide tree detection/segmentation algorithms (with the exception of a geometric algorithm). Instead, it provides a standardized interface for performing training, inference, and evaluation using existing tree detection models and algorithms. The project currently supports the external computer vision models DeepForest, Dectree2, and SAM2, as well as a geometric canopy height model segmentor implemented within TDF. Support for other external models can be added by implementing a new Detector class.

We use the torchgeo package to perform data loading and standardization using standard geospatial input formats. This library allows us to generate chips on the fly of a given size, stride, and spatial resolution. Training and inference is done with modular detectors that can be based on existing models and algorithms. We have preliminary support for using PyTorch Lightning to minimize boilerplate around model training and prediction. Region-level nonmax-suppression (NMS) is done using the PolyGoneNMS library which is efficient for large images. Visualization and saving of the predictions is done using geopandas, a common library for geospatial data.

This project is under active development by the Open Forest Observatory. We welcome contributions and suggestions for improvement.

Tree detection models supported

TDF currently supports the following tree detection/segmentation algorithms.

DeepForest

• Github
• Uses RGB input data. Predicts tree crowns with rectangular bounding boxes.
• Provides a RetinaNet model trained on a large number of semi-supervised tree crown annotations and a smaller set of manual annotations.
• Trained using data from the US only but representing diverse regions. The model has been applied on data from outside the US successfully.

Detectree2

• Github
• Uses RGB input data. Predicts tree crowns with polygon boundaries.
• Provides a Mask R-CNN model trained on manually labeled tree crowns from four sites.
• Trained using data from tropical forests.

Segment Anything Model 2 (SAM2)

• Github
• Uses RGB input data. Predicts objects with polygon boundaries.
• Utilizes the Segment Anything Model (SAM 2.1 Hiera Large) checkpoint with tuned parameters for mask generation optimized for tree crown delineation.
• Does not rely on supervised training for tree-specific data but generalizes well due to SAM's zero-shot nature; however, non-tree objects are also detected and included in predictions.

Geometric Detector

• Implementation of the variable window filter algorithm of Popescu and Wynne
  (2004) for
  tree top detection, combined with the algorithm of Silva et al.
  (2016) for crown
  segmentation.
• Uses canopy height model (CHM) input data. Predicts tree crowns with polygon boundaries.
• This is a learning-free tree detection algorithm. It is the one algorithm that is implemented within TDF as opposed to relying on an existing external model/algorithm.

Software architecture

The tree-detection-framework is organized into modular components to facilitate extension including integration of additional detection models. The main components are:

1. preprocessing.py
   The create_dataloader() method accepts single/multiple orthomosaic inputs. Alternatively,
   create_image_datalaoder() accepts a folder containing raw drone imagery. The methods tile the
   input images based on user-specified parameters such as tile size, stride, and resolution and
   return a PyTorch-compatible dataloader for inference.
2. Detector Base Class
   All detectors in the framework (e.g., DeepForestDetector, Detectree2Detector) inherit from the
   Detector base class. The base class defines the core logic for generating predictions and
   geospatially referencing image tiles, while model-specific detectors translate the inputs to the
   format expected by the respective model. This design allows all detectors to plug into the same
   pipeline with minimal code changes.
3. RegionDetectionsSet and RegionDetections
   These classes standardize model outputs. A RegionDetectionsSet is a collection of RegionDetections, where each RegionDetections object represents the detections in a single image tile. This abstraction allows postprocessing components to operate uniformly across different detectors. These outputs can be saved out as .gpkg or .geojson files.
4. postprocessing.py
   Impelments a set of postprocessing functions for cleaning the detections by Non-Maximum Suppression(NMS), polygon hole suppression, tile boundary suppression, and removing out of bounds detections. Most of these methods operate on standardized output types (RegionDetections / RegionDetectionsSet).

Install

Some of the dependencies are managed by a tool called Poetry. I've found
easiest to install this using the "official installer" option as follows. Note that this should be run
in the base conda environment or with no environment active.

curl -sSL https://install.python-poetry.org | python3 -

Now create and activate a conda environment for the dependencies of this project.

conda create -n tree-detection-framework python=3.10 -y
conda activate tree-detection-framework

Now, from the root directory of the project, run the following command. Note that on Jetstream2, you
may need to run this in a graphical session and respond to a keyring popup menu.

poetry install

Finally, choose to either install the Detectron2 or SAM2 detection framework.

Detectron2:
The Detectron2 library is not compatible with poetry so must be installed directly with pip

# https://detectron2.readthedocs.io/en/latest/tutorials/install.html#build-detectron2-from-source
pip install git+https://github.com/facebookresearch/detectron2.git

Download the detectree2 checkpoint weights.

cd checkpoints
mkdir detectree2
cd detectree2
wget https://zenodo.org/records/10522461/files/230103_randresize_full.pth

SAM2:
Clone the SAM2 repository and install the necessary config files

git clone https://github.com/facebookresearch/sam2.git && cd sam2

pip install -e .

And download the associated checkpoints

cd checkpoints && \
./download_ckpts.sh && \
cd ..

And move into this repo

mv checkpoints ../tree-detection-framework 

Use

The module code is in the tree_detection_framework folder. Once installed using the poetry
command above, this code can be imported into scripts or notebooks under the name
tree_detection_framework the same as you would for any other library.

Examples

To begin with, you can access example geospatial data
here, which should be downloaded and placed in the data folder at the top level of this project. Our goal is to have high-quality,
up-to-date examples in the examples folder. We also have work-in-progress or one-off code in
sandbox, which still may provide some insight but is not guaranteed to be current or generalizable.
Finally, the tree_detection_framework/entrypoints folder has command line scripts that can be run
to complete tasks.

Evaluation and benchmark with NEON

Download the NEON dataset files and save the annotations and RGB folders under a new directory in the data folder.

wget -O annotations.zip "https://zenodo.org/records/5914554/files/annotations.zip?download=1"
unzip annotations.zip
wget -O evaluation.zip "https://zenodo.org/records/5914554/files/evaluation.zip?download=1"
unzip -j evaluation.zip "evaluation/RGB/*" -d RGB
rm annotations.zip
rm evaluation.zip

Follow the steps in tree-detection-framework/sandbox/evaluation/neon_benchmark.ipynb for detectors DeepForest & Detectree2, and tree-detection-framework/sandbox/evaluation/sam2_neon_benchmark.ipynb to use SAM2.

Evaluation and benchmark with Detectree2 datasets

1. Download the dataset. There are two ways to get the dataset:
   Download the site-specific .tif (for orthomosaic) and .gpkg (for ground truth polygons) files from https://zenodo.org/records/8136161. Then, follow steps in https://github.com/PatBall1/detectree2/blob/master/notebooks/colab/tilingJB.ipynb to do the the tiling.
   (OR)
   Download our pre-tiled dataset from https://ucdavis.box.com/s/thjmaane9d38opw1bhnyxrsrtt90j37m
2. Add the tiled dataset folder to the data folder in this repo.
3. For benchmark and evaluation see steps in tree-detection-framework/sandbox/evaluation/dtree2_benchmark.ipynb and tree-detection-framework/sandbox/evaluation/sam2_dtree2_benchmark.ipynb