Egg mass classification considering the hatching process of Pomacea canaliculata

Abstract

Pomacea canaliculata feeds on seedlings that have been planted less than three weeks ago. This study aimed to construct an imaging system that can eliminate the egg masses of P. canaliculata before they hatch and multiply. An image classification method is proposed that can recognize the state of hatching of the egg masses. As hatching process proceeds, this state changes from “freshly laid” to “maturing” and then to “mature.” In the proposed method, first, egg image pixels are detected using a four-label semantic segmentation model that includes the background label. Next, the egg masses are classified by analyzing the distribution of the labeled pixels in the egg masses. We conducted an experiment in which we verified the effectiveness of the proposed method on images of egg masses from an agricultural canal and evaluated its classification accuracy. The F1-score of the proposed method was 1.00 when the weather was cloudy and 0.842 when it was sunny, demonstrating that the state of hatching could be identified accurately regardless of the brightness of the day. Using this classification method, only newly laid eggs can be dropped into water and eliminated, which is a step forward in the automation of this method for P. canaliculata control.

Introduction

Damage caused by the Channeled applesnail (Pomacea canaliculata) (Fig. 1a) has become increasingly serious in lotus root farms and rice paddies in recent years. P. canaliculata is designated as an invasive alien species and distributed throughout Japan and Southeast Asian countries¹. The problem is that they feed on young and soft lotus root paddy rice seedlings and up to about three weeks after planting².

Fig. 1.

Various P. canaliculata : (a) P. canaliculata , (b) Egg masses, (c) P. canaliculata and its egg masses, (d) Wall of an agricultural canal on which egg masses have been deposited.

Current measures to control P. canaliculata include shallow water management, which maintains the water depth at about 40 mm to suppress feeding behavior, and spraying chemicals to kill them³. However, because shallow water management does not actually exterminate P. canaliculata, when the water level rises during rainfall, feeding damage still occurs. Chemical sprays also have an environmental impact, and there are limits to the number of times they can be used. Therefore, the current P. canaliculata control measures are unable to completely exterminate them. It has become increasingly important to develop measures that eliminate not only the P. canaliculata but also their eggs.

P. canaliculata deposits multiple eggs at once, forming a single lumpy egg mass, as shown in Fig. 1b. An egg mass is a collection of approximately 200–300 eggs³. Therefore, removing just one egg mass should have a significant effect on reducing damage. As shown in Figs. 1c,d, the egg masses are deposited on the walls of agricultural canals and on the stems of plants above the water surface³.

Currently, the egg masses are removed manually, either dropped into the water with a stick or crushed. However, P. canaliculata has a high reproductive capacity, producing more than 3000 eggs per year⁴. If conditions are good, they will spawn once every three to four days. Good condition means that the density of P. canaliculata in the rice field is high. Therefore, extermination using this method is a heavy burden³. To address this issue, it is expected that automatic removal technology will be established to reduce the burden of egg mass removal work on farmers.

When exterminating egg masses, the methods of extermination differ depending on the stage of hatching. The duration of hatching is approximately two weeks when the average temperature is 25 °C. If dropped into the water, freshly laid eggs can be killed by suffocation, but eggs that are close to hatching will survive⁵. Therefore, not all eggs should be dropped into the water, and it is necessary to change the method of extermination depending on the state of hatching. To automate egg mass removal, image processing methods that can automatically recognize the hatching status of eggs are required.

Previous research in the agricultural field has led to several practical examples, including both extermination systems and image-processing systems. With respect to extermination systems, research has been conducted on robots that remove spotted lanternfly (Lycorma delicatula) eggs⁶ as well as systems that remove whitefly (Trialeurodes vaporariorum)⁷, Tessaratoma papillosa⁸, Pieris rapae⁹, and aphids (Aphidoidea)¹⁰.

With respect to image systems, some methods^11,12 use images and object-level detection to identify defective products and pests. Yanfei et al.¹¹ used a CNN-based model to classify the quality of apples. CNN is a convolutional neural network, which is one of the network architectures for deep learning. CNN-based model is an image classification model that uses CNN. To keep defective apples out of the market, the system identifies apples according to three quality levels (premium, middle, and poor), and has an overall accuracy of 95.33%. Additionally, Sarin et al.¹² conducted research on the automatic counting of brown planthopper populations, which cause damage to rice in paddy fields. For monitoring purposes, a support vector machine-based model was used to classify the developmental stages of brown planthoppers attached to rice (nymphal, brachypterous, or macropterous) with 83% accuracy.

Other methods^13,14 have been proposed that detect objects at pixel level instead of object level. Lucas et al.¹³ detected diseases and pests using images of coffee tree fields. This detection technique uses semantic segmentation, which allows detection in units of pixels. The models using U-Net and PSPNet architectures obtained mIoU results of 94.25% and 93.54%, respectively, and they were able to perform classification with high accuracy. mIoU is one of the evaluation indices for image segmentation, and indicates how closely the segmented pixels match the correct answer. mIoU is 100% when the pixels match perfectly. Here, U-Net is a model particularly for image segmentation and can capture the shape and texture of objects in an image with high accuracy. In U-Net, by connecting the encoder section that compresses image information and the decoder section that restores the compressed information, it is possible to improve the overall classification accuracy by simultaneously taking into account both the details and overall information of the image. Like U-Net, PSPNet is a model specialized for image segmentation, which analyzes an image multiple times at different scales and combines the results to improve the overall classification accuracy. Pawel et al.¹⁴ used a U-Net model to extract feed, chitin, and frass areas from images.

Furthermore, because weeds also cause harm to agricultural crops, some studies^15,16 have classified weeds and seedlings. Ma et al.¹⁵ used SegNet to classify three categories: rice seedlings, weeds, and background. So that chemical herbicides can be sprayed accurately, not only the category and location but also the outline and shape are determined with high accuracy. Milioto et al.¹⁶ used semantic segmentation to realize a system that can classify crops, weeds, and background in real time. Their system operates at approximately 20 Hz.

These studies demonstrate that semantic segmentation, which enables classification at pixel level, can perform tasks in agriculture such as pest detection and the classification of similar objects such as seedlings and weeds. Therefore, we believe that semantic segmentation has excellent potential as an effective image classification method for detecting egg masses and classifying hatching status. Semantic segmentation is a method that can accurately identify even the smallest objects in an image by performing pixel-by-pixel image analysis. The size of a P. canaliculata egg mass is about 30 mm in length and 15 mm in width, which is very small in the image. Therefore, semantic segmentation, which is a pixel-by-pixel image analysis method, works to detect tiny egg masses and identify the state of hatching.

We conducted preliminary research applying semantic segmentation and color information to the detection of egg masses and classification of the hatching process¹⁷. The eggs change color from dark pink to white during hatching¹⁸; therefore, we focused on this color change and clarified that it is possible to classify the hatching stage using the saturation channel of Hue Saturation Value (HSV) color space. However, this image processing system classifies the hatching stage using a predetermined saturation threshold. Fixed thresholds make it difficult to respond robustly to changes in illumination, and if recognition of the hatching state fails, an accurate method for eliminating egg masses cannot be developed.

Therefore, in this paper, we propose an image-processing method that recognizes the hatching stage of the egg masses of P. canaliculata and is robust to changes in the illumination of the external environment. The proposed method performs pixel-level detection using a four-label semantic segmentation model to identify egg pixels. Next, by analyzing the pixel distribution, we are able to classify egg masses.

The objective of this study is to clarify how to construct an imaging system that can eliminate the egg masses of P. canaliculata before they hatch and multiply. For that purpose, in experiments, we verify the effectiveness of the proposed method on images of egg masses from an agricultural canal and evaluate its classification accuracy. Furthermore, because agricultural canals are outdoor environments, we aim to be able to accurately identify the state of hatching regardless of the brightness of the day.

Methods

Hatching process of the egg masses

Before presenting the method for classifying egg masses, we first describe the hatching of egg masses. As an egg mass incubates, its status changes from freshly laid to maturing and then to mature. Example images of freshly laid, maturing and mature eggs are shown in Fig. 2. The period until hatching is approximately two weeks when the average temperature is 25 °C³. Freshly laid egg masses have a long time remaining before they hatch, maturing egg masses have some time remaining before they hatch, and mature egg masses are just about to hatch. Freshly laid is about two days after being laid. Because there are reports that about half of the freshly laid eggs will not hatch if they are submerged within two days after being laid⁵. However, there are no reports that quantitatively show the maturing and maturation periods, so it is not possible to quantitatively show all hatching conditions. Egg masses are calcareous and harden, generating calcium components as hatching progresses⁵. Since freshly laid have not yet solidified, they can be exterminated by suffocation if dropped into water. However, maturing and mature egg masses can survive even if dropped into water⁵. Therefore, not all egg masses should be dropped into the water. For practical use, it is necessary to be able to accurately discriminate freshly laid egg masses, which can be exterminated by dropping them into water, from maturing and mature egg masses, which cannot be exterminated this way. Therefore, in the proposed method, images are classified using a total of four labels: the three hatching states and the background.

Fig. 2.

Hatching states of egg masses: (a) freshly laid, (b) maturing, and (c) mature.

Flow of the proposed method

The flow of the proposed method is presented in Fig. 3. First, images of an agricultural canal wall are taken. Second, detection at pixel level (egg pixels) is performed using a semantic segmentation model with four labels: (1) freshly laid, (2) maturing, (3) mature, and (4) background. To classify egg masses and the background, each connected region of egg-mass pixels (all labels except the background label) is detected as a single egg mass.

Fig. 3.

Flow of the proposed method.

Next, we analyze the pixel distribution within each egg mass region. In this analysis, the number of pixels in each label and its variance are measured to classify the egg masses in units of clusters. Finally, we present the identification results of freshly laid egg masses, which can be exterminated by dropping into water, and maturing, and mature egg masses, which cannot be exterminated this way.

Whereas our previous study¹⁷ performed semantic segmentation using two labels (egg mass and background), the proposed method performs semantic segmentation using four labels. As a result, in the reference¹⁷, the hatching stage was identified using saturation in HSV color space based on the pixel information in regions labeled as egg-mass regions, which was not robust to changes in the brightness of the input image. By contrast, the proposed method performs a distribution analysis using pixel information based on four labels and does not use brightness information, resulting in egg mass identification that is robust to changes in brightness.

In our method, we used a GALLERIA XL 7c-R36 11800H PC, which was equipped with an Intel i7-11800H CPU and NVIDIA GeForce RTX 3060 Laptop GPU. All code was made using Python 3.10.11.

Four-label semantic segmentation model

This section describes the pixel-level detection method shown at the top of Fig. 3. Right side in Fig. 3 shows examples of the detection results. The original image is shown in (a), and (b) shows the output image after applying the four-label semantic segmentation. Note that multiple labels can be output in one egg mass. In this figure, the dark pink, light pink, and white regions indicate pixels labelled as freshly laid, maturing, and mature, respectively.

Next, we describe the four-label semantic segmentation model and training method. Left side in Fig. 3 shows examples of the input at this stage: the training images and their annotations by an expert are shown in (c) and (d), respectively. Since it is necessary to identify each egg mass, annotation is also performed on each mass for training.

A typical approach to semantic segmentation is an encoder-decoder structure. It consists of an encoder that extracts features from an image using a convolutional layer and a decoder that takes the extracted features and outputs a probability map. Details of the constructed model are given in Supplemental Table A.1.

We employed the U-Net model proposed by Ronneberger et al.¹⁹. ResNet-50 was used as the U-Net encoder. In the identification of crops and weeds conducted by Kamath et al.²⁰, the best results were obtained when the U-Net encoder was a ResNet-50, which is also useful for identifying hatching status in this study. We used the four labels, freshly laid, maturing, mature, and background. The encoder depth was set to 5. U-Net was implemented using a Python library segmentation model²¹.

We also performed data augmentation to improve the generalization performance of the model. A different mini-batch was generated for each epoch, and a random transformation was applied to the training images for each mini-batch. Supplemental Table A.2 lists details of the data augmentation. In this study, data augmentation was performed using the Albumentations library²², where p is a parameter representing probability, and its maximum value is one. In practice, some captured images may be blurred because the system will be used in an outdoor environment. Therefore, blurring and motion blur were also added to the data augmentation.

Supplemental Table A.3 shows the training settings. The input image size was 448 448. The initial learning rate was set to 1.0 , the batch size was set to 8, and the number of epochs was set to 20. If the learning rate is attenuated after learning progresses to a certain extent, the accuracy should improve²³. Therefore, when the number of epochs was 10, the learning rate was lowered to 1.0 . The data constituting each mini-batch was randomly shuffled at each epoch. The Dice loss was employed as the loss function, and Adam was used for optimization²⁴. mIoU was used as the evaluation index. Assuming that the overlap between the correct image and the prediction result for label c is , is a true positive, is a false positive, and is a false negative. In addition, there are four labels, and hence mIoU can be expressed by the following formulas:

1

2

At each epoch, test images were used to evaluate accuracy, and the model was saved when the highest score was obtained.

Finally, we detect the egg mass area. To distinguish between the background and egg mass regions at pixel level, connected egg-mass pixels (three labels excluding the background label) are detected as a single egg mass.

Distribution analysis of labeled pixels in egg masses

Egg mass identification using pixel distribution analysis, as shown in the lower part of Fig. 3, is described in this section. As mentioned in the previous section, in the four-label semantic segmentation model, multiple labels can occur in one egg mass (Fig. 4a). To eliminate egg masses, it is necessary to identify egg masses at cluster level rather than at pixel or egg level, and hence a process is required to determine one hatching state label from the region of an egg mass that contains multiple labels.

Fig. 4.

Classification of egg masses: (a) input images to the model, (b) ground truth, and (c) detection at pixel level.

Figure 4b shows the annotation result of the images in Fig. 4a labeled by an expert. In Fig. 4, the first three egg masses from the left are freshly laid (F1, F2, and F3), the middle three egg masses are maturing (M1, M2, and M3), and the two right egg masses are mature (NF1 and NF2).

From the pixel-by-pixel detection results in Fig. 4c, we can predict Fig. 4b. To do this, we first focus on the number of pixels of each label within the egg mass region. Let the number of pixels of the freshly laid label be , the number of pixels of the maturing label be , and the number of pixels of the mature label be . For example, F1 can be correctly identified as freshly laid if it satisfies the following formula:

3

Similarly, when M1, M2, and M3 satisfy (4), and when NF1 and NF2 satisfy (5), they can be correctly identified.

4

5

However, because in F2 and F3, the largest number is , they are incorrectly identified as maturing according to (4).

We compare in detail F1, F2, and F3, which should be classified as freshly laid egg masses, and M1, M2, M3, which should be classified as maturing egg masses. Egg masses that should be identified as freshly laid exceed a certain value of in the egg mass area. Furthermore, in F1, F2, and F3, the dark pink freshly laid labeled pixels are spread throughout the egg mass area. These results are observed not only in this example but also in other samples, and when these images are identified simply by comparing the number of pixels, there is a tendency to regard freshly laid egg masses as maturing ones.

Therefore, to correctly identify freshly laid egg masses, we decided to focus on the pixel distribution of each label in addition to the number of pixels in each label. The egg masses are identified based on the three steps set out below.

• Step 1. Classify each egg mass into three labels (freshly laid, maturing, or mature) using (3), (4), and (5).

• Step 2. For egg masses labeled as maturing, if the ratio is greater than or equal to threshold as expressed by (6), proceed to Step 3; other egg masses are treated as maturing.

  6

• Step 3. Calculate the standard deviation s of each pixel coordinate labeled freshly laid within the egg mass area as follows:

  7
  8

Here, , are the standard deviations in each direction of the x- and y-axes of the image. Let each pixel coordinate of the freshly laid label be ,, the average of each pixel coordinate be , and the length of the x-axis and y-axis of the egg mass be ,. Equation (7) performs normalization.

Then, from (8), when the standard deviation s of the pixel distribution of the freshly laid label is greater than or equal to threshold , the egg mass is regarded as freshly laid, otherwise it is regarded as maturing.

Preliminary experiment for threshold setting

To determine the threshold values and described in the previous section, we conducted a preliminary experiment. The input image data were egg mass images taken on three days under different weather conditions (2021/10/19, 2022/08/09, and 2022/11/24,) at an agricultural canal construction site. The captured image was cropped to a size of 448 448, and the resolution was 0.07 mm/pixel. The number of images created was 4119 images for training (519 images containing egg masses and 3600 images containing only the background) and 45 images for testing (45 images containing egg masses). Supplemental Table A.4. shows the number of egg masses for each label in the images that contain egg masses. Using these, we first trained the four-label semantic segmentation model. We next applied the trained four-label semantic segmentation model to the test images and output the results in pixels.

Then, we identified the egg masses using the pixel distribution analysis described in the previous section. For the 45 test egg masses, we appropriately adjusted the thresholds and of (6) in Step 2 and (8) in Step 3, and searched for parameter values that could correctly identify all egg masses, obtaining = 0.15 and = 0.20 as a result.

Supplemental Fig. A.1 shows some examples from the preliminary test. Figure A.1(a) shows the test images, Fig. A.1(b) shows the detection results in pixels, and Fig. A.1(c) shows the identification results based only on comparing the number of pixels of each label. In Fig. A.1, the first three egg masses from the left are freshly laid (F4, F5, and F6), the middle three egg masses are maturing (M4, M5, and M6), and the two right egg masses are mature (NF3 and NF4).

When only the number of pixels in each label are compared, F4, M4, M5, M6, NF3, and NF4 can be correctly identified. However, in F5 and F6, which are freshly laid, is the highest number in pixel-by-pixel identification, and hence they are incorrectly identified as maturing.

By contrast, by analyzing the pixel distribution using suitable parameters (= 0.15, = 0.20), the freshly laid egg masses in F5 and F6 can be correctly identified, as shown in Fig. A.1(d). Using these threshold values, we conduct an evaluation of the proposed method on new test data.

Experimental setup

Evaluation method

Because freshly laid egg masses are targeted for extermination, we treat freshly laid egg masses as positive and the maturing and mature egg masses as well as the background as negative. For example, Table 1 has True labels and Predicted labels. If TP is a number that is positive in the True label and positive in the Predict label, TN is the number of negatives in the True label and negatives in the Predicted label, FP is the number of positives in the Predicted label and negatives in the True label, and FN is the number of negatives in the Predicted label and positives in the True label, Precision, Recall, and F1-score can be expressed by the following formulas:

9

10

11

To demonstrate the effectiveness of the proposed method, we evaluated the label of the correct region relative to the label of the predicted region.

Table 1.

Classification results.

		Predicted label
		Freshly laid		Maturing		Mature		Background
		Cloudy	Sunny	Cloudy	Sunny	Cloudy	Sunny	Cloudy	Sunny
(a) Classification using the proposed method
True label	Freshly laid	5	8	0	0	0	3	0	0
	Maturing	0	0	7	8	2	8	0	0
	Mature	0	0	0	0	18	54	0	0
	Background	0	0	0	0	4	6	144	190
(b) Classification based on prior research(average saturation value)17
True label	Freshly laid	1	4	4	6	0	0	0	1
	Maturing	0	1	9	14	0	1	0	0
	Mature	0	0	1	23	16	28	0	0
	Background	0	0	1	3	0	1	144	191
(c) Classification using part of the proposed method(comparison of the number of pixels for each label)
True label	Freshly laid	3	4	2	4	0	3	0	0
	Maturing	0	0	7	8	2	8	0	0
	Mature	0	0	0	0	18	54	0	0
	Background	0	0	0	0	4	6	144	190

Comparison methods

In this experiment, we compared the proposed method with the following two methods. The first comparison method is our previous research¹⁷. The proposed method performs pixel-by-pixel detection using a four-label semantic segmentation model and identifies egg masses by analyzing the number and distribution of pixels for each label (Steps 1, 2, and 3 of previous section). In contrast, our previous method performs pixel-by-pixel detection using a two-label semantic segmentation and identifies each egg mass using the average saturation value. We performed this comparison to evaluate whether the proposed method can respond robustly to illumination changes in the external environment.

The second comparison method performs pixel-by-pixel detection using a four-label semantic segmentation model like that of the proposed method, and performs identification only by comparing the number of pixels of each label (Step 1 in previous section). We performed this comparison to verify whether pixel distribution analysis can more accurately identify freshly laid egg masses.

Image dataset

The egg masses of P. canaliculata are laid on the walls of agricultural waterways and on the stems of plants above the water surface. For this study, we captured egg masses in an agricultural canal (Supplemental Fig. A.2) at a lotus root farm in Tamura-cho, Tsuchiura City, Ibaraki Prefecture, Japan, which is suffering from damage from P. canaliculata. The camera used was a CANON EOS70D. The image size was 5472 3648. The width of the agricultural canal investigated was 1.0 m, and images of egg masses were captured while maintaining a distance of 0.5 m from the canal wall. Egg masses were captured randomly from the front, right, and left. The resolution of the camera is approximately 0.07 mm/pixel. When captured from the front, if the egg mass is 30 mm 30 mm, the image size is approximately 428 pixels 428 pixels.After performing pixel-by-pixel detection using four-label semantic segmentation, the connected egg mass pixels (three labels excluding the background label) were detected as single egg masses to distinguish between the background and the egg mass regions. Small clusters of less than 1000 pixels were removed as noise.

To test whether the proposed method can respond robustly to changes in illumination, we collected egg mass images on four days with different weather conditions (2021/10/19, 2022/08/09, 2022/11/24, 2023/08/08). One expert determined the hatching status and annotated the captured egg mass images. Supplemental Table A.5 presents the weather during image collection and the number of egg masses included in the collected images.

Images taken on 2021/10/19, 2022/08/09, and 2022/11/24 were used as training images, and images taken on 2023/08/08 were used as test images. Egg mass images taken in both cloudy and sunny weather were used for training. We evaluated the accuracy of actual use by testing with images from 2023/08/08 as data from a different day that have not been incorporated into the training data. In addition, the background should not be detected as an egg mass. Therefore, background images that do not contain egg masses were also randomly included in each of the training and test images.

Next, we created an image dataset for the evaluation experiments. To verify whether the hatching state of the egg mass could be correctly recognized, the images were cropped around the center of gravity of the egg mass. The created images were cropped to a size of 448 448 while maintaining the original resolution. The number of images created was 4164 training images (564 images containing egg masses and 3600 images containing only the background). There were 424 test images (88 images containing egg masses and 336 images containing only the background). Supplemental Table A.6 shows the number of egg masses for each label in the images containing egg masses. Using these images, we conducted an evaluation of the proposed method. Note that the preliminary experiments were conducted using only training images.

Results and discussion

Pixel-level detection results

First, we evaluated a four-label semantic segmentation model at pixel precision. Figure 5 shows the detection results in pixels. Figure 5a shows the input image, Fig. 5b shows the detection results of the previous method (two labels), and Fig. 5c shows the detection results of the proposed method (four labels).

Fig. 5.

Pixel-level detection results: (a) input images to model, (b) previous method using semantic segmentation with two labels, and (c) proposed method using semantic segmentation with four labels.

The Dice loss was used as the loss function used in image segmentation tasks to improve the prediction accuracy of the learning model. The lower the Dice Loss, the higher the accuracy of the learning model. The more stable the Dice Loss, the higher the stability of the learning model. Supplemental Fig. A.3 shows the change in Dice loss, where the training Dice loss is the loss for the training images, and the test Dice loss is the loss for the test images. Supplemental Fig. A.3 reveals that the Dice loss rapidly decreases after the first epoch, and a stable low loss is achieved.

It also does not cause overfitting or unstable behavior. The mIoU evaluation index indicates higher accuracy as it approaches 1.0. The mIoU of the proposed method on the 424 test images was 0.977. The IoU of the previous method (two labels) was 0.994 using prior research¹⁷, indicating that the identification accuracy was comparable that of the proposed method. However, even if annotation is performed on a mass-by-mass basis, the hatching states of the eggs in each egg mass are different, and hence it is difficult to match all the labels on a pixel-by-pixel basis, as shown in Fig. 5c. Therefore, a pixel analysis of each label is required to identify the egg mass states.

Classification results of the egg masses

In the egg mass classification evaluation, to confirm whether the proposed method can respond robustly to brightness, the test images were divided into images taken in sunny weather and those taken in cloudy weather, and the results were compared.

Table 1 presents the classification results for cloudy and sunny weather as confusion matrices.

Table 1(a) shows the identification results using the proposed method; Table 1(b) shows the classification results from the previous method; and Table 1(c) shows the identification results of a part of the proposed method (only comparing the number of pixels of each label). In addition, Table 2 presents the results of Precision, Recall, and F1-score defined by equation (9) through (11) when freshly laid egg masses were treated as positive results. From Table 1(a), FP and FN are 0 in cloudy condition. Therefore, Precision, Recall were 1.0 as shown in Table 2 of the proposed method. Table 1(b) reveals that the identification based on the previous method (average saturation) does not take into account the brightness of the external environment, resulting in undetected freshly laid egg masses. Furthermore, comparing Table 1(a) and (c), it can be confirmed that the proposed method using distribution analysis has a higher detection rate of freshly laid.

Table 2.

Classification result when treated freshly laid egg masses as positive results.

Weather	Comparison methods	Precision	Recall	F1-score
Cloudy	Classification using the proposed method	1.00	1.00	1.00
	Classification based on prior research17	1.00	0.200	0.333
	Classification using part of the proposed method	1.00	0.600	0.750
	(Comparison of the number of pixels for each label)
Sunny	Classification using the proposed method	1.00	0.723	0.842
	Classification based on prior research17	0.800	0.364	0.200
	Classification using part of the proposed method	1.00	0.364	0.533
	(Comparison of the number of pixels for each label)

By contrast, classification results using the proposed method resulted in no undetected freshly laid egg masses, which will reduce the burden of extermination work. Similarly, on images taken when the weather was sunny, false positives occurred in the previous method’s results in Table 1(b), which would result in the maturing egg masses accidentally being dropped into the water. By contrast, the proposed method does not cause false positives. However, in Table 1(a) where Freshly laid was mistakenly identified Mature under the Sunny condition. In this case, when an egg mass is present in the shadow of grass as shown in Supplemental Fig. A.4, it is detected as an egg mass, however, the identification was incorrect. As Supplemental Fig. A.4 shows, there is grass between the camera and the egg mass, so if we could take a picture from a different angle, it might be possible to recognize it. As we are considering implementing this research in a robotic system as the ultimate application, we believe that it may be possible to recognize it if the camera angle is changed.

Figure 6 shows an example of image recognition results for freshly laid egg masses. Figure 6a is the input image, Fig. 6b is the correct image, Fig. 6c is the detection result in pixels, Fig. 6d is the classification result obtained by the proposed method, Fig. 6e presents the identification results using prior research¹⁷, and Fig. 6f presents the results using part of the proposed method (comparison of the number pixels for each label).

Fig. 6.

Example of classification results for freshly laid egg masses (dark pink, light pink, white, and black regions indicate pixels labelled as freshly laid, maturing, mature, and background, respectively.): (a) input images, (b) ground truth, (c) pixel-level detection, (d) classification result using the proposed method, (e) classification based on prior research¹⁷ , and (f) classification using part of the proposed method.

In Fig. 6, the images of the freshly laid egg masses on the left were taken when it was cloudy, and those on the right were taken when it was sunny. The freshly laid egg masses in Fig. 6 were identified as maturing by the two comparison methods, but in the proposed method, they were correctly identified as freshly laid egg masses by Steps 2 and 3 of the pixel distribution analysis. As shown in Table 2, for the proposed method, the F1-score for cloudy days is 1.00, and the F1-score for sunny days is 0.842. This shows that the proposed method was able to identify objects with higher accuracy than other comparison methods, even when the illumination level changed. During experiment, the average processing time of the proposed method was 7.69 fps for one 448 448 image.

The limitations of this study are that the effectiveness of the proposed method was verified for unknown data, but the number of samples was small, and that in implementing the proposed method in the field, factors other than the influence of weather, such as different shooting angles and cases where the image is hidden by grass, must be taken into consideration. In the future, it will be necessary to take these factors into consideration and conduct additional validation of the proposed method using more images.

Conclusions

In this paper, we proposed an image classification method that can recognize the hatching state of egg masses to realize a system for exterminating the egg masses of P. canaliculata.The main contributions of this paper are the following:

• The proposed method uses a semantic segmentation model with four labels: eggs that still have a long time to incubate (freshly laid eggs), eggs that have incubated for some time (maturing eggs), eggs that are about to hatch (mature eggs), and the background. This model could detect the label of each pixel (egg pixels). We showed that this segmentation method allows the accurate classification of egg masses pixel and background pixel by pixel from pixel-level detection results.

• By analyzing the characteristics of the pixel distribution within the three types of egg mass areas, we could classify egg masses in a way that is robust to changes in the illumination of the external environment. This enables each labeled egg mass to be effectively exterminated.

• To demonstrate the practicality of the proposed method, we conducted an experiment in which we applied this method to classify egg masses attached to the wall of an agricultural canal adjacent to a lotus root field. Experimental results showed that the proposed method was able to identify the egg masses more robustly than other comparison methods to changes in the illumination of the external environment. We confirmed that the identification accuracy of the proposed method is not affected by weather.

In future, we plan to conduct additional validation of the proposed method using more images, taking into account factors such as different shooting angles and when the image is hidden by grass. Moreover, we plan to develop an egg-mass exterminating robot that incorporates this image processing system.

Author contributions

T.Yo. and T.Ya. conceived the experiments, T.Yo. and T.Ya. conducted the experiments, T.Yo analysed the results. All authors reviewed the manuscript.

Data availability

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-024-81010-z.