Abstract
In most of the countries, grapes are considered as a cash crop. Currently huge research is going on in development of automated grape harvesting systems. Speedy and reliable grape bunch detection is prime need for various deep learning based automated systems which deals with object detection and object segmentation tasks. But currently very few datasets are available on grape bunches in vineyard, because of which there is restriction to the research in this area. In comparison to the vineyard in outside countries, Indian vineyard structure is more complex, so it becomes hard to work in real-time. To overcome these problems and to make vineyard dataset for suitable for Indian vineyard scenarios, this paper proposed four different datasets on grape bunches in vineyard. For creating all datasets in GrapesNet, natural environmental conditions have been considered. GrapesNet includes total 11000+ images of grape bunches. Necessary data for weight prediction of grape cluster is also provided with dataset like height, width and real weight of cluster present in image. Proposed datasets can be used for prime tasks like grape bunch detection, grape bunch segmentation, and grape bunch weight estimation etc. of future generation automated vineyard harvesting technologies.
Keywords: Artificial intelligence, Grape bunch segmentation, Vineyard dataset, Deep learning, Grape bunch detection etc
Specifications Table
| Subject | Agriculture Engineering, Computer Vision and Pattern recognition, Data Science, Artificial Intelligence |
| Specific subject area | Grape cluster datasets for future generation automated vineyard harvesting system |
| Type of data | RGB & RGB-D Vineyard Image Datasets |
| How the data were acquired | Two image capturing devices are used to create dataset: 1. One-plus 7 Mobile phone with tripod 2. Intel Real-Sense D435I Depth Camera with raspberry pi 4 as a processing unit with proper necessary software installations. Each dataset is captured by following the standard protocols. |
| Data formats | .jpg, .png, .raw, .csv |
| Description of data collection | Three GrapesNet datasets are created at Indian vineyard in natural environment conditions to cover illuminance effect, camera position and various obstructions with two image capturing devices and remaining one dataset is created in to the indoor experimental environment with depth camera. |
| Data source location | Yard Owner: Mr. Jalindar Mane Venue: Village- Yelavi, Dist. Sangli, State-Maharashtra Latitude and longitude of location: 17.0408° N, 74.5126° E Temp: 25–34C°, Humidity: 73.0%, Rain: 4.72cm, Soil type: typically black soil |
| Data accessibility | Repository name: GrapesNet: Indian Grape Clusters RGB & RGB-D Image Datasets Data identification number (DOI):10.17632/mhzmzd5cwx.1 Direct URL to data: https://data.mendeley.com/datasets/mhzmzd5cwx/1 |
Value of the Data
-
•
In currently available grape bunch dataset only last rows of vines are considered so that it will not be confused with the background vines and bunches. Vines with limited grape bunches is taken into consideration, and pruning is also done to remove leaf occlusion on bunches. For making grape bunch detection more robust for real time applications and suitable for Indian vineyards, in proposed dataset all vine rows of vineyards and large number of bunches per vine are covered. GrapesNet dataset consists of grape bunch RGB and RGB-D dataset of Indian vineyard.
-
•
Proposed dataset consists of 4 sub-datasets which are created by the research point of view with total 11000+ versatile images which can be further increased with another different data augmentation processes.
-
•
90% of currently available grapes dataset are of red grapes, so proposed GrapesNet dataset is all on white grapes and with natural environmental conditions.
-
•
These datasets will be very much beneficial to the new researcher which are willing to work in future agriculture and technology based researches like grape bunch detection, grape bunch picking robot, grape quality assessment based AI system etc. because proposed dataset is created in successive pattern so that any beginner in this domain with prior knowledge about deep learning and python language can start working from scratch.
-
•
This datasets are created with single grape type (Sonaka) and work can be extended for various types of grapes.
1. Objective
-
•
To create RGB and RGB-D grape bunch datasets on Indian vineyard for deep learning applications like grape bunch detection and segmentation
-
•
To create RGB-D grape bunch dataset on Indian vineyard to predict weight of grape bunch for automated vineyard harvesting robotic systems
-
•
To create versatile RGB and RGB-D vineyard datasets which will results into automated deep learning systems compatible for real-time applications
2. Data Description
According to survey of global agriculture report, roughly 2 billion people (26.7% of the world population) derive their livelihoods from agriculture [1]. In all the fruit production business, grapes are considered as cash crop because it has huge demand in national and international markets. Currently lot of research is in progress to make grape production and harvesting process simpler. Grape picking robots and grape assessment AI based systems are part of enhancement in harvesting process of vineyard [2], [4], [5], [6], [7], [8], [9], [10], [11]. Prime objective while creating such advanced systems is accurately segment and detect the grape bunches on the vines. In reference [3], dataset is collected and labeling is done which is required to train deep learning models for grape bunch detection task. So to trigger this process further, GrapesNet dataset has been proposed which includes 4 different datasets of grape bunches of vineyard and are listed below:
-
1.
Dataset-1: RGB images with one grape bunch per image with black background and one grape bunch per image with natural background
-
2.
Dataset-2: RGB images with two or more grape bunches per image with natural background
-
3.
Dataset-3: RGB-D images of vineyard with real environmental conditions
-
4.
Dataset-4: RGB-D images with one grape bunch per image with experimental environment with coral background for grape weight prediction task Table 1.
Table 1.
Dataset with capturing device details.
| Dataset No. | Capturing Device | Format |
|---|---|---|
| 1 | One Plus 7 Mobile Camera | .jpg |
| 2 | One Plus 7 Mobile Camera | .jpg |
| 3 | Intel Real-sense D435I Camera | .png, .raw |
| 4 | Intel Real-sense D435I Camera | .png, .raw |
As mentioned above, the images have been captured in such a way that it will cover various illuminance effect, changes in camera positions and various obstructions which makes it versatile and reliable for real-time applications. Ground truth data like real height (cm), width (cm) and weight (kg) of each grape cluster used for dataset 4 creation which will be beneficial for grape cluster weight estimation task. Total 11000 + images have been captured in GrapesNet dataset. Device used to capture datasets are listed below:
GrapesNet dataset consists of 4 different types of datasets which are created with different set-ups and different environmental conditions. Here for some datasets artificial background as well as light is being considered and for remaining datasets, natural environmental conditions are considered with real background. Table 2 shown below gives the details idea about schedules for creating datasets, captured images and total images in each dataset.
Table 2.
Schedule of GrapesNet dataset creation.
| Dataset No. | Description | Date | Indian Time (24 Hours Clock) | Captured Images | Total Images |
|---|---|---|---|---|---|
| 1 | RGB images with one grape bunch per image with black background | 21/11/2019 | 12:00-15:30 | 250 | 615 |
| 22/11/2019 | 7:00-10:00 | 155 | |||
| RGB images with one grape bunch per image with natural background | 23/11/2019 | 16:00-18:30 | 210 | ||
| 2 | RGB images with two or more grape bunches per image with natural background | 2/04/2021 | 10:00 to 17:00 | 240 | 740 |
| 3/04/2021 | 10:00 to 17:00 | 240 | |||
| 4/04/2021 | 10:00 to 17:00 | 260 | |||
| 3 | RGB-D images of vineyard with real environmental conditions | 25/03/2021 | 10:00 to 17:00 | 55 | 424 |
| 26/03/2021 | 10:00 to 17:00 | 55 | |||
| 27/03/2021 | 10:00 to 17:00 | 55 | |||
| 28/03/2021 | 10:00 to 17:00 | 55 | |||
| 29/03/2021 | 10:00 to 17:00 | 55 | |||
| 30/03/2021 | 10:00 to 17:00 | 54 | |||
| 31/03/2021 | 10:00 to 17:00 | 50 | |||
| 1/04/2021 | 10:00 to 17:00 | 50 | |||
| 4 | RGB-D images with one grape bunch per image with experimental environment with coral background | 15/03/2022 | 10:00 to 16:00 | 84 | 350 |
| 16/03/2022 | 10:00 to 16:00 | 84 | |||
| 17/03/2022 | 10:00 to 16:00 | 98 | |||
| 18/03/2022 | 10:00 to 16:00 | 84 | |||
| Grand Total of Images | 2129 | ||||
Proposed dataset have considered only for Sonaka grape type but this strategy can be adapted further for any grape type. Detailed strategies of different datasets creations are described in brief section below.
2.1. Dataset 1
Dataset-1 is consists of 2 sub datasets: RGB images with one grape bunch per image with black background and one grape bunch per image with natural background. The samples of the dataset-1 are shown in Fig. 1.
Fig. 1.
Samples of dataset-1 (a) Single grape cluster with black background (b) Single grape cluster with natural background.
For achieving variation in illuminance, different time slots are considered while creating datasets which will insure improvement in system's accuracy. In 1st task, artificial background of black color have been considered to make grape bunch detection easier. And in 2nd complexity of detection task has been increase by removing the black background and taking real one.
2.2. Dataset 2
To make grape detection model more versatile and suitable for real-time situations, Dataset-2 is created with 2D RGB dataset of multiple grape bunches per image in vineyard at different light conditions which are captured by one plus 7 mobile camera mounted on a tripod. Samples of dataset-2 are given in Fig. 2.
Fig. 2.
Samples of dataset-2 with multiple grape cluster per image (a) Vines in background (b) Ground in background.
2.3. Dataset 3
Dataset-3 is created with D435I camera which provides RGB images and depth images along with its raw file. While capturing the dataset-3, to make model more robust and reliable different real-time scenarios have been taken into consideration like:
1. Occlusions 2. Capturing angle 3. Sunlight
In first case as seen in Fig. 3, images which contains normal grape bunches, occluded by leaves, occluded by another grape bunches and occluded by vine branches have been considered. (a) Normal (b) Occluded by leaves (c) Occluded by other bunches (d) Occluded by vine branches.
Fig. 3.
Diversity in dataset based on occlusion parameter.
In Fig. 4, images are taken in such a way that it will cover all the angles from the object like angle parallel to grape bunch, elevated angle and dropped angle.
Fig. 4.
Diversity in dataset based on camera angle parameter (a) Front angle (b) Elevated angle (c) Dropped angle.
In Fig. 5, various sunlight conditions have been considered depending on which color variations in grape bunches can be observed. While capturing images by taking sunlight conditions, scenarios like normal/front light, sheltered, sidelight and backlight have been taken into account. By considering all these above scenarios, the aim of versatile grape bunch dataset of vineyard creation has been fulfilled.
Fig. 5.
Diversity in dataset based on light angle parameter (a) Normal/Front light (b) Shelter (c) Side light (d) Backlight.
2.4. Dataset 4
Dataset-4 is created for the weight prediction tasks. To keep the task simple, Dataset-4 has been created with artificial environment of the fixed environmental conditions and constant artificial light. Also, fix background around grape bunch is considered which is made with card-board of coral color supported with chair and only one grape bunch at a time is hanged with the help of rope, stick and clip for capturing the each bunch's individual image. Fig. 6 shows the images of same grape bunches taken at different distances:
Fig. 6.
Images of same grape bunches from different distances.
For one grape bunch multiple images has been taken with different depths to get variety of depths. At the time of dataset creation, original height, width and weight of each grape bunch are also noted. This weight will be considered as ground truth while comparing with the predicted weight values.
3. Experimental Design, Materials and Methods
To create GrapesNet datasets for grape bunch segmentation and weight prediction task, 2 different cameras are used: One-plus 7 Mobile camera and Intel Real-sense D435I depth camera. Fig. 7 shows both the cameras used in this experiment:
Fig. 7.
Image capturing devices (a) One Plus 7 Mobile Camera (b) Intel Real-sense D435I Camera.
GrapesNet dataset created with one plus 7 mobile camera with 48-megapixel (f/1.7, 1.6-micron) camera and D435I camera provide .png images of RGB and depth with 1280 × 720 resolution. Along with that it also produces .raw files of same image with 424 × 240 resolution which contains bit depth values of each pixel in image. Intel® Real-Sense™ Depth Camera D435i have an inertial measurement unit (IMU). For best scanning results, an IMU provides an extra set of data which provide better dense reconstruction. It can capture depth data for distance range 0.3m to 3m. Detailed specification of the image capturing devices is mentioned in the Table 3 below:
Table 3.
Specification of image capturing devices.
| Camera Specification |
Device |
|
|---|---|---|
| One plus 7 Mobile Camera | Intel Real-sense D435I camera | |
| Brand | One-plus | Intel |
| Camera model | One plus 7 Mobile Camera | Intel Real-sense D435I Depth Camera |
| Camera output | RGB image | RGB image, Depth image, RAW image |
| Resolution | Color image - 4000 × 3000 | Color image - 1920 × 1080 Depth image - 1280 × 720 RAW image - 424 × 240 |
| Ideal range | - | 0.3 m to 3 m |
| Field of View (FOV) – (H × W) | 117° × 117° |
RGB sensor - 69° × 42° Depth sensor -87° × 58° |
| Focal Length | 35mm | 100mm |
| Flash Mode | No flash | No flash |
As shown in Fig. 8(a), both one plus 7 mobile camera and D435I camera have been mounted on tripod stand which is adjustable and with maximum height up to 4 feet.
Fig. 8.
Experimental Set-up.
As shown in Fig. 8(b), image capturing systems has been designed by assembling the multiple components like raspberry pi-4 board, keyboard, raspberry pi 7” display and one-plus power bank. Processing unit used for capturing device is raspberry pi-4 board. All the necessary software and drivers have been installed inside board which will be the prime need for capturing images and then board is connected to Raspberry pi 7” display with handy keyboard as an input device. To give power supply to image capturing system, one plus power bank is connected to it with cable.
All the 1st three dataset are created by directly taking images of grape bunches from random distances with real as well as black background. So no specific steps has been involved. But for creating dataset 4 specific steps has been followed which are given in flowchart given in Fig. 9. As shown in Fig. 9, single grape bunch is taken. Height, width and weight of same bunch are noted as a features for machine learning and deep learning models. Then depth which is nothing but distance between bunch and camera is varied from 30 to 120 cm with the scale of 15 cm. for each depth images will be captured and saved. When all depth images for distance equal to final distance in given range are captured, next grape bunch will be taken and same procedure will be followed all over again.
Fig. 9.
Flowchart of dataset 4.
In proposed dataset, in each dataset it contains RGB & RGB-D images which are large in size and utilizes large memory. So to avoid that image resizing pre-processing operation is performed on all datasets, which reduces the image sizes from 4000 × 3000 to 500 × 500 for one plus 7 camera images and 1280 × 720–500 × 500 for D435I camera images. When any dataset is considered for deep learning application, it is necessary that it contain huge number of images. Data augmentation is process to increase the size of dataset with some image processing operations like horizontal flip, left wrap, right wrap, bottom wrap and top wrap etc. To perform this pre-processing and augmentation operations python script has been used. As given in Table 4, highlights datasets and augmentations operations. Before augmentation number of images in GrapesNet dataset was 2129 which after augmentation becomes total 11000+ images.
Table 4.
Data augmentation details.
| Original Images |
After Augmentation |
||||
|---|---|---|---|---|---|
| Dataset No. | RGB | RGB-D | RGB | RGB-D | Augmentation Process |
| 1 | 405 | - | 2835 | - | Horizontal flip, vertical flip, bottom wrap, top wrap, left wrap, right wrap |
| 210 | - | 1470 | - | Horizontal flip, vertical flip, bottom wrap, top wrap, left wrap, right wrap | |
| 2 | 740 | - | 2960 | - | Horizontal flip, bottom wrap, top wrap |
| 3 | 424 | 424 | 1696 | 424 | Horizontal flip, bottom wrap, top wrap |
| 4 | 350 | 350 | 2100 | 350 | Horizontal flip, vertical flip, bottom wrap, top wrap, left wrap, right wrap |
Ethical Statement
Proposed data does no involved any human subjects, animal experiments or data collected from social media platforms.
CRediT authorship contribution statement
Dhanashree K. Barbole: Conceptualization, Methodology, Software, Writing – original draft, Investigation. Parul M. Jadhav: Supervision, Validation, Writing – review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
Research Fellowship from Department of Science and Technology (DST), Women Scientist (WOS) scheme-B, Kiran Division, Central Government, India. The equipment used for this research was funded under Department of Science and Technology (DST) Women Scientist (WOS) scheme-B of Kiran Division, Central Government, India.
Data Availability
References
- 1.D. Barbole, P. Jadhav, “GrapesNet: Indian grape bunchs RGB & RGB-D image datasets”, mendeley data, 2023. 10.17632/mhzmzd5cwx.1 [DOI] [PMC free article] [PubMed]
- 2.Global agriculture, “agriculture at crossroads: Findings and recommendations for future farming”, article, 2022
- 3.Ariza-Sentís M., Vélez S., Valente J. Dataset on UAV RGB videos acquired over a vineyard including bunch labels for object detection and tracking. J. Data Brief. 2023 doi: 10.1016/j.dib.2022.108848. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Meshram V., Patil K. FruitNet: Indian fruits image dataset with quality for machine learning applications. J. Data Brief. 2022 doi: 10.1016/j.dib.2021.107686. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Dang Thi P., Chung D., Van T. A fruits recognition system based on a modern deep learning technique. IOP Conf. Ser. J. Phys. Conf. Series. 2019 1327. [Google Scholar]
- 6.Santos T.T., Souza L.L.D., Santos A.A.D., Avila S. Grape detection, segmentation and tracking using deep neural networks and three-dimensional association. Comput. Vis. Pattern Recognit. 2020 [Google Scholar]
- 7.California Historical Society collection . via University of Southern California Digital Library (USC.DL); 2012. “Close-Up of a Grape Bunch on a Vine”, Dataset Published. [Google Scholar]
- 8.D.K. Barbole, D.P.M. Jadhav, “Comparative analysis of deep learning architectures for grape bunch instance segmentation”, IT in Industy, 2021
- 9.D.K. Barbole, D.P.M. Jadhav, “A review on fruit detection and segmentation techniques in agricultural field” springer book series advances in intelligent systems and computing, 2020
- 10.K. Anupriya & G.M. Sri, “Fruit freshness detection using machine learning”, Cognitive Informatics and Soft Computing pp 633–642, 2022
- 11.Barbole D.K., Jadhav D.P.M. Grape yield prediction using deep learning regression model. IEEE Explore. 2021 [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.