Bee Tracker—an open‐source machine learning‐based video analysis software for the assessment of nesting and foraging performance of cavity‐nesting solitary bees

Abstract The foraging and nesting performance of bees can provide important information on bee health and is of interest for risk and impact assessment of environmental stressors. While radiofrequency identification (RFID) technology is an efficient tool increasingly used for the collection of behavioral data in social bee species such as honeybees, behavioral studies on solitary bees still largely depend on direct observations, which is very time‐consuming. Here, we present a novel automated methodological approach of individually and simultaneously tracking and analyzing foraging and nesting behavior of numerous cavity‐nesting solitary bees. The approach consists of monitoring nesting units by video recording and automated analysis of videos by machine learning‐based software. This Bee Tracker software consists of four trained deep learning networks to detect bees that enter or leave their nest and to recognize individual IDs on the bees’ thorax and the IDs of their nests according to their positions in the nesting unit. The software is able to identify each nest of each individual nesting bee, which permits to measure individual‐based measures of reproductive success. Moreover, the software quantifies the number of cavities a female enters until it finds its nest as a proxy of nest recognition, and it provides information on the number and duration of foraging trips. By training the software on 8 videos recording 24 nesting females per video, the software achieved a precision of 96% correct measurements of these parameters. The software could be adapted to various experimental setups by training it according to a set of videos. The presented method allows to efficiently collect large amounts of data on cavity‐nesting solitary bee species and represents a promising new tool for the monitoring and assessment of behavior and reproductive success under laboratory, semi‐field, and field conditions.


Overview
The Bee Tracker Software is able recognize positions of cavities in the nesting unit and bee IDs composed of a color-digit combination on the thorax of the bees (see image on the right above). The provided version can recognize the colors white, yellow and green and the digits 1 to 8. The software can further detect bees that enter or leave cavities (as in the image on the right below) and can therefore detect which bee has entered or left a cavity at what point in time. From these detected events further measurements can be extracted such as assigning marked females to the cavity they are nesting in. Additionally, flight durations and the number of cavities probed until a bee has found its own "correct" cavity are detected by the software.

Input videos
The input videos should contain a nesting unit with cavities as in the above image. The cavities will be numbered A1, A2,...,Ax for the first row, B1,B2,...,Bx for the second row and so on.

Important:
The camera should not be moved during the whole video! The cavities of the nesting unit must be in the same position during the whole video.
The input videos should be in MP4 format and the aspect ratio should be 16:9. For example 3860x2160 (4K).
The software is much more efficient if you provide more than one input video. For example, if you want to analyze three videos, do not start the software three times but choose all three videos as inputs and run the software for all three of them.

Results Folder
Choose an empty folder as the results folder. The software will create a new folder for each input video in the results folder and save all final and intermediate results to this folder. The result folders are labelled according to the file names of input videos. Input videos should therefore labelled with unique names.

Config File
Optionally, a config file can be provided as a third input. Inside the config file, the below parameters can be altered. If no config file is provided, the software will use the default values visible in the image below.

Outputs
The software will create a new subfolder within the results folder for each input video. Inside each such subfolder the following final outputs can be found: 1. all_events_unfiltered.csv: Inside this csv file, all detected events are listed in the following format: TIME, BEE_ID, MOVEMENT (enter or leave), CAVITY_ID. Each event has a timestamp and defines which bee has entered or left which cavity (see example output on the right). This list is completely unfiltered and may contain errors.

error_corrected_events.csv:
This csv file contains all events that remain after error deletion. The data is listed in the following format: TIME, BEE_ID, MOVEMENT (enter or leave), CAVITY ID, USED (event used to create measures in output files are listed as residential movements).

address_book.csv:
Inside this csv file, all detected "addresses" are listed in the following format: BEE_ID, CAVITY_ID, EVIDENCE. Each bee is assigned to one (or more) cavities. The evidence indicates the number of occurrences of a bee entering the cavity.

nest_recognition.csv:
Inside this csv file, the number of wrong cavities a female enters before finding its nest is listed in the following format: TIME, BEE_ID, CAVITIES.

flight_list.csv:
Inside this csv file, the flights are listed in the following format: TIME, BEE_ID, DURATION. Note that flights might be missing in this file due to the strict filtering of events in case of mispredictions by the deep learning models. (Read the "Extracting Measurements" section for more details).

visualization.mp4:
If you have activated the "Visualize Results" option, a video file called visualization.mp4 will be created with all detected events visualized.
In addition to the final outputs, some intermediate results are stored. Read the "Inner Workings" section for further information on these intermediate results.

Start / Pause / Resume
Once you have defined your inputs, you can start the computations by clicking on the "Start" button. You can pause the software anytime by clicking the "Pause" button. It can take a while (up to a few minutes) until the software is stopped. Please do not close the window or shut down your computer during this time. Otherwise it might not be possible to resume the computations. By clicking on "Start" again, the software can be resumed from the position it was paused.
Once the software has completed its computations it can be rerun with different parameters in the config file. This will not rerun all computations but only part of it. Therefore the computations will be much faster this time!

Detecting Bees
The first and most time consuming step of the software is to detect all bees in all frames of the video. The Faster RCNN Object Detection architecture with an image size of 1024x576 is used for this step. The software detects two types of classes: bee and bee flying.
The software will skip 12 frames (approximately 0.5 s) if no bees are detected. This is to speed up the computations in case of no bees being present in the video.

Detecting Cavities
The second step is to detect the positions of all cavities in the video. This is done with the Faster RCNN Object Detection architecture with an image size of 1024x576 as well. The cavities are only detected in a few frames. Therefore this step does not consume a lot of time.
For the detection of cavity positions the software requires cavities that are arranged in horizontal rows. The camera should not be moved during the whole video! The cavities of the nesting unit must be in the same position during the whole video.

Detecting Colors
The third step is to detect all colored IDs that are glued to the thorax of the bees. For this step, the images of the detected bees in step 1 are analyzed with the YOLOv3 Object Detection architecture. The YOLOv3 network will output the coordinates of the bounding box and the color of the ID as white, yellow or green (see image below).
In order to speed up this computation, the bee images of flights that do not end or start in a cavity are not analyzed.

Detecting Numbers
The fourth step is to recognize the numbers on the detected IDs from the previous step. A custom convolutional neural image classification network with 3 convolution layers and an input image size of 50x50 pixels is used. It has 9 possible output classes: the digits 1-8 and the indistinctive class. If the number is blurry, cut off or at a bad angle, the network will predict indistinctive. Due to its small size, this network can be executed extremely fast on a huge amount of detected number tags.

Event Detection
Having applied these four neural networks to the input video, the positions of all bees (along with the color-digit combination in the ID, if present) in each frame and the positions of all cavities are known to the software. The next step is to extract the events from this data. There are two possible events: entering or leaving a cavity.
The first step to reach this goal is to implement an object tracking algorithm. It tracks bees over multiple frames and assigns them an ID. A customized centroid tracking algorithm is used for this task. For a bee to be tracked over multiple frames, the centers of the bounding boxes within two consecutive frames are required to be within a certain max_distance (definable in the config file, default is 1.4 x average_hole_height). If this requirement is fulfilled the closest detections within two consecutive frames are assigned the same ID. However, if two detections of flying bees are close by (within 2 x max_distance), the bees are not tracked because the risk of ID swapping is high. Flying bees are only tracked if there are no other bees within 2 x max_distance).
After individual bees got assigned an ID they can be tracked over multiple frames. The color and digit of the ID can therefore be detected with very high accuracy as multiple frames can be used for identification.
Finally, the software can detect the events (entering and leaving a cavity). To do so, the starting and end points of all flights (tracked bee IDs) are analyzed. If the starting point of a flight is within the bounding box of a cavity and the detection is bee (and not bee flying), the cavity at that location is taken as the starting point of this flight. The detection of flight end points works analogous. There is a parameter in the config file which defines the maximal size of a bee detection to be considered as a starting or end point. Default is 1.2 x average_hole_area.

Extracting Measurements
To extract measurements a csv file with the detected events (all_events_unfiltered.csv) is used as input.
Every row contains the movement of a bee. The array (data) is organized as following: After the cavity-based error correction a second correction step is carried out which is based on bee IDs. For this purpose the data is ordered primarily by time stamp and secondarily by the bee ID. Unpaired events are again corrected by correct_missing_movementas as described above.
Based on this error-corrected data the measurements are extracted. First, nesting females are assigned to the cavities they are nesting in. A bee only gets assigned to a cavity, if it stays inside the cavity for min_nest_time after entering it and does not enter any other cavity within min_fly_time after leaving. The default values for both min_nest_time and min_fly_time are 40 s and can be changed in the config file. The implementation of this step is done in check_resident by iterating through the data.
check_resident always start with an Enter event and then calculates the two relevant times (time inside and outside cavities) which are compared to the set time thresholds. Considered events that fulfill the time criteria are saved in residential_movements and the further processing by take_measurement gets activated. Placeholder events are not considered for the calculations.
take_measurements calculates two values: drunkenness und collection time. The drunkenness is the number of cavities a bee probes until it finds the one it was assigned to (and is allegedly nesting in) while the collection time is calculated as the duration from leaving till returning to the nest.
Finally, three output csv files with the following measurements are created: address_book mit der bereinigten Liste der benisteten Löcher (und ihren Bewohnern) -address_book which contains the list of females and the cavities they are nesting in -nest_recognition which lists the number of cavites a female probed until it found ist nest -flight_book which liests flight durations

Troubleshooting
Not all cavities in the nesting unit are detected by the software: