An annotated Dataset and Benchmark for Detecting Floating Debris in Inland Waters

Abstract

Marine litter is a serious threat to marine ecosystems, and the timely removal of floating waste from inland waters is effective in preventing floating debris from entering the sea. An accurate object detection system is a prerequisite for efficiently clearing floaters. However, complex light conditions in the water, small size objects and other factors pose a huge challenge for floating object detection. In order to facilitate the solution of the floating object pollution problem and promote the application of AI technology in the water industry, we proposed the first floater dataset of waters collected from real water scenarios based on shore-based filming equipment, IWHR_AI_Lable_Floater_V1. The dataset consists of 3000 images containing accurate annotation information to support vision-based water surface floater detection tasks. We conducted a number of baseline experiments to evaluate the performance of mainstream object detection algorithms on this dataset. The results show that the detection accuracies of the models, including the state-of-the-art model YOLOv9, are all low, which also indicates that floating object detection is a challenging task.

Background & Summary

In recent years, the problem of marine litter, which includes plastics, metals, textiles, wood and a variety of organic and inorganic substances, has increasingly attracted widespread attention, with negative impacts on marine ecosystems, human health and economic development^1,2. As rivers serve as an important link between land and sea, a large number of them discharge waste, chemicals and pollutants into the sea, leading to an increase in marine litter. In addition, river debris tends to flow into the oceans as a result of currents and winds, making it one of the main sources of marine litter. Therefore, controlling and reducing the generation and discharge of river litter is of great significance in mitigating marine litter pollution and protecting the marine ecosystem. As a common water quality pollutant, river floating debris, including human discarded domestic rubbish, dead plants and animals, and industrial discharges, brings serious harm to the water environment and ecosystems, such as affecting water quality, endangering aquatic organisms, obstructing rivers, and endangering human health³. Timely detection of floating debris and the adoption of effective measures for its treatment and prevention are essential for improving water quality, protecting the aquatic ecosystem and safeguarding human health.

As shown in Fig. 1(a), the current river floating debris cleaning mainly relies on manpower, due to the floating debris of a wide variety of types, wide distribution, large number, and by the water flow, seasonal and other factors, the manual cleaning method of the task is heavy and inefficient, there is also a high cost. In addition, the watershed may be fast-flowing, complex terrain, to clean workers bring potential personal safety risks. As people’s awareness of the protection of the river environment gradually increases, the demand for river waste disposal is becoming more and more urgent. Traditional manual cleaning methods can clean up the rubbish, but the efficiency is low and there are potential safety hazards. In recent years, unmanned surface vehicles(USVs) have become popular because of their low maintenance costs, high mobility, long endurance, and the ability to operate in dangerous areas that are difficult for humans to reach^4–6. USVs equipped with floater capture devices can not only improve cleaning efficiency and reduce labour costs, but also reduce the impact of human factors on the environment. As shown in Fig. 1(b), an accurate floater detection system is indispensable for USVs to achieve efficient and reliable capture of floaters. With the development of computer vision, it has gradually become possible to support USVs to identify floating objects reliably, in real time and accurately.

Fig. 1.

Methods of cleaning up floaters. (a) Manual cleaning, (b) USV cleaning.

If machine vision is used to detect floating objects in the river, floating objects in the water such as plastic bottles and beverage cans are often too small in size, which makes it easy to lose features and impossible to locate and identify the object. The reason for this is that small objects have fewer pixels, fewer available features, and higher requirements for positioning accuracy, which undoubtedly increases the difficulty of accurate detection⁷. In addition, the complex environment of the waters also brings many challenges to machine vision-based detection methods, such as the light in the waters in the darker or strongly reflective conditions, the outline and color of the floater may not be easy to be accurately identified; The reflection of objects on the shore on the water surface inevitably adds to the difficulty of floating object detection. Examples of the above problems are illustrated in Fig. 2.

Fig. 2.

Challenges in floater detection. (a) Small objects, (b) Backlight, (c) Water reflections, (d) Reflection of objects on the shore.

Public datasets provide researchers and developers with a large amount of annotated data and experimental benchmarks, facilitating scientific research and technological innovation, and researchers can use these datasets for algorithm validation, model training, and performance evaluation, thus advancing the development of related fields. In order to promote the application of AI in the water industry, we released a dataset IWHR_AI_Lable_Floater_V1 (10.6084/m9.figshare.27376851.v1)⁸, which consists of a floater dataset from the viewpoints of video surveillance cameras and mobile devices. The dataset was collected in 2022 from the section of the Grand Canal from Beiguan Barrage to Tongji Road Bridge and the nearby Reduce Transportation Ditch in Tongzhou District, Beijing, and contains 3000 annotated images containing floating debris, with a total of 23692 annotated objects covering common household waste, including plastic bottles and foam boards, as well as floating debris such as water plants and algae. This dataset not only focuses on data authenticity and diversity, but also provides the basis for an integrated linkage technology between shore-based cameras and USVs, whereby the USVs automatically travel to clean up floating objects based on information detected by shore-based cameras. In addition to object detection, the dataset can also be used for floater tracking studies to obtain data on water flow and floater migration patterns, providing insight into the dynamics of floater distribution in inland waterways. This is an important reference for evaluating the pollution of waters and developing appropriate treatment measures.

Finally, we evaluate commonly used algorithms on our dataset. In summary, the contributions of this paper can be summarized as follows:

1. According to the survey, IWHR_AI_Lable_Floater_V1 is a dataset of floaters in real scenarios of inland waters under the viewpoints of video surveillance cameras and mobile devices, which provides a valuable resource for researchers and developers.

2. IWHR_AI_Lable_Floater_V1 contains a large number of images of small-sized floaters with diverse features and scenarios, which effectively contributes to the research of small object detection in waters.

3. We conducted baseline experiments based on some mainstream detectors.

Related work

Vision-based object detection

For water surface object detection, traditional water surface object detection algorithms rely on hand-designed feature extraction methods, such as image processing⁹ and Gaussian mixture model¹⁰. Although these methods can achieve certain results under specific conditions, they are often sensitive to environmental changes and lighting conditions, with low detection accuracy and robustness.

With the rise of deep learning, especially the application of Convolutional Neural Networks (CNNs), the performance of surface object detection has been dramatically improved^11–13. With trained deep learning models, salient features in images can be learned to improve the accuracy and robustness of object detection. For example, the one-stage object detectors YOLO (You Only Look Once) series^14–19, CenterNet²⁰, and RetinaNet²¹, which generate candidate regions directly from the input image and perform category prediction and bounding box regression, and two-stage classical detectors such as Faster R-CNN²², which are widely used in water surface object detection. Among them, YOLO truly realizes real-time object detection, and after continuous iteration, YOLOv9²³ and YOLOv10²⁴ were released in 2024, which well support the tasks of object detection, instance/semantic segmentation, and panorama segmentation.

In addition, the combination of image preprocessing^25–27, such as image enhancement and dehazing, and dehazing techniques can effectively eliminate the impact of hazy weather on the image quality, so that the detection algorithm can work stably under various weather conditions. For complex backgrounds such as reflections and ripples on the water surface, the use of image enhancement techniques can improve the image quality, highlight the object features, and improve the detection accuracy.

Related dataset

In the area of waste detection, scholars in the computer vision research community have spared no effort in actively promoting research and releasing a series of vision-based waste detection datasets. TrashNet²⁸ is an image dataset for waste classification. It contains images of different types of waste such as paper, glass, metal, plastic, etc. and can be used for training and evaluation of waste classification tasks. The Taco²⁹, released in 2020, is an instance segmentation dataset for litter identification in natural environments, containing photographs of litter taken in different environments, ranging from tropical beaches to the streets of London, and is continuously collecting and adding new annotations using its website (http://tacodataset.org/). Liu³⁰ et al. have created the Marine Image Dataset (MID), a dataset containing both large and small obstacles, which also provides ground-truth data for the horizon. The recently released FloW-Img³¹ is a dataset of floating objects on the surface of the water taken from the viewpoint of USVs, containing 2000 images and 5271 annotated objects. Based on these datasets, some vision-based float detection work has been carried out^3,32.

As shown in Fig. 3, although these datasets are useful for litter detection, they may not cover the full range of features of floating debris in inland waterways and their application scenarios are quite different from inland waterways. The images in our release IWHR_AI_Lable_Floater_V1 were captured by shore-based filming equipment, providing researchers and developers with a valuable data resource for real-world scenarios.

Fig. 3.

Comparison of the IWHR_AI_Lable_Floater_V1 dataset with other datasets.

Methods

This section describes the IWHR_AI_Lable_Floater_V1 dataset in detail.

Acquisition platform

We used shore-based devices such as video surveillance cameras and mobile phones to collect videos containing floating objects to collect data at real river and lake sites. These shore-based devices use CMOS sensors, and video surveillance cameras constructed on river/lake shores capture images with a resolution of 1920 × 1080, mobile phones capture images with a resolution of 1920 × 1080, 880 × 1920, and digital cameras capture images with a resolution of 3840 × 2160.

Data collection and annotation

In order to further enrich the diversity of the samples, videos and pictures with different angles, lighting conditions, and backgrounds were collected by adjusting the shooting angle, time, and environment during the collection of raw images, and one picture was extracted from the video every approximately 20 frames, and then, 3000 good-quality pictures were selected from all of them. Then, the well-known bounding box annotation software LabelImg was used to annotate the object and category information of the floaters in the images. It should be noted that the principle of annotation is that in addition to single targets such as bottles, plastic bags, etc., floating objects made up of water plants, algae or other litter accumulations also need to be annotated. The annotation team consists of experienced computer vision researchers to ensure the quality of the information annotated in the dataset. An example of classical annotation is shown in Fig. 4, from which it can be seen that the floating objects include common household wastes such as plastic bottles and foam boards, as well as accumulating floating objects such as water plants and algae.

Fig. 4.

Classically annotated images in the IWHR_AI_Lable_Floater_V1.

Dataset statistics

Currently, the IWHR_AI_Lable_Floater_V1 dataset contains 3000 images of floating objects on the surface of the water and the corresponding annotations, 672 of which were captured by a video surveillance camera, and the rest of which were captured by a mobile cell phone. The number of objects contained in each image is shown in Fig. 5, which contains a total of 23692 annotated floater objects.

Fig. 5.

Number of floaters included in each image.

If the statistics are based on the pixel ratio of the object mask relative to the image, it is found that as many as 94.5% of the object pixels account for less than 10%, as shown in Fig. 6.

Fig. 6.

The visualization of the IWHR_AI_Lable_Floater_V1. (a) Distribution of object center positions, (b) Distribution of object sizes.

Data Records

The dataset is available at Figshare⁸ (10.6084/m9.figshare.27376851.v1), which contains image files in JPG format and corresponding annotation files in XML format, and the folder structure is shown in Fig. 7.

Fig. 7.

Folder structure of the dataset.

Technical Validation

We have evaluated the performance of nine commonly used deep learning-based object detection algorithms based on the IWHR_AI_Lable_Floater_V1 dataset, including the two-stage object detection algorithms Faster R-CNN and CentreNet2, the single-stage object detectors SSD, RetinaNet, YOLOv5s, YOLOv6s, YOLOv7, YOLOv8s, and we also evaluate the newly released state-of-the-art model YOLOv9. The dataset was first partitioned into training, validation, and test sets in the ratio 8:1:1, and these models were trained with the help of a server equipped with NVIDIA RTX 3090 GPU, with 300 epochs for each training. In order to show the effect of different random seeds on the test results when splitting the dataset, we set three random seeds, 100, 101, and 102. During the training process, the training loss and validation mAP after each epoch are recorded, as shown in Figs. 8, 9, which show the trends of the training loss and validation mAP with the number of epochs for each detector, respectively. It can be seen that each detector converges to a lower loss level after the training is completed. Except for YOLOv5, YOLOv8, and YOLOv9, all the models contain 300 epochs, which is due to they use of early stopping to avoid overfitting during training while saving computational resources and time.

Fig. 8.

Training loss for different detectors. (a) Fast R-CNN, (b) RetinaNet, (c) SSD, (d) CenterNet2, (e) YOLOv5s, (f) YOLOv6s, (g) YOLOv7, (h) YOLOv8s, (i) YOLOv9.

Fig. 9.

Validation mAP for different detectors. (a) Fast R-CNN, (b) RetinaNet, (c) SSD, (d) CenterNet2, (e) YOLOv5s, (f) YOLOv6s, (g) YOLOv7, (h) YOLOv8s, (i) YOLOv9.

The trained detectors was evaluated on the test set at random seeds of 100, 101, and 102, respectively, and the test mean of the three seeds was used as the final evaluation result of this detector, as shown in Table 1.

Table 1.

Results of different detectors on the test set.

Model	Input Size (pixel)	mAPtest (%)	Speed RTX 3090 (ms/frame)	Params (M)
Faster R-CNN	600 × 600	54.6	110.0	514.8
RetinaNet	640 × 640	60.7	87.8	482.8
SSD	512 × 512	60.2	26.0	246.2
CenterNet2	640 × 640	60.1	111.1	127.1
YOLOv5s	640 × 640	65.9	2.0	14.4
YOLOv6s	640 × 640	63.6	2.7	40.6
YOLOv7	640 × 640	65.7	8.6	74.8
YOLOv8s	640 × 640	66.2	3.7	22.5
YOLOv9	640 × 640	67.7	5.8	252.4

The results on the test set are shown in Table 1. It can be seen that the mAP of different detectors on our dataset is at the level of around 65.0%. We believe that there are still some challenges for the detection of small objects on the water surface.

Author contributions

Conceptualization, G.Q. and M.Y.; methodology, G.Q.; software, G.Q.; validation, G.Q.; formal analysis, G.Q.; investigation, G.Q. and M.Y.; data curation, G.Q. and M.Y.; writing—original draft preparation, G.Q.; writing—review and editing, G.Q.; visualization, G.Q.; supervision, M.Y. and H.W.; project administration, M.Y. and H.W.; funding acquisition, M.Y. All authors have read and agreed to the published version of the manuscript.

Code availability

Python 3.9 was used for data processing and model training, and the reference code used for training and testing in this work can be found in the GitHub repository at the following links: https://github.com/jwyang/faster-rcnn.pytorch, https://github.com/yhenon/pytorch-retinanet, https://github.com/amdegroot/ssd.pytorch, https://github.com/xingyizhou/CenterNet2, https://github.com/ultralytics/yolov5, https://github.com/meituan/YOLOv6, https://github.com/WongKinYiu/yolov7, https://github.com/ultralytics/ultralytics, https://github.com/WongKinYiu/yolov9, https://github.com/open-mmlab/mmdetection. We also have hosted the algorithms and data processing code used into the FigShare⁸ repository with the 10.6084/m9.figshare.27376851.v1.

Competing interests

The authors declare no competing interests.