Evaluation of plum fruit maturity by image processing techniques

Abstract

Maturity is the key factor which determines the storage life and ripening quality of fruits. In order to provide marketing flexibility and to guarantee the acceptable eating quality to the buyer it is very critical to determine the right maturity stage. Maturity indices are also important for trade regulation, marketing strategy and for the efficient use of labor and resources. The proposed system is based on implementation of image processing techniques on the JPEG images of different maturity stages of the plum variety ‘Satluj Purple’ grown under sub-tropical conditions. The external quality features like color, texture and size were analyzed. Color feature was extracted by using mean RGB values. Entropy, Local Binary Pattern and Discrete Cosine transformation were used for extracting textural features. Correlation coefficients between images of various categories were recorded to determine the most dominant factor for classification. Multi-Attribute Decision Making theory was used for taking final decision. The developed system accurately determined the maturity level. Color was found to be the most dominant factor for classifying the plums according to maturity level. The error percentage was less than 2.4%, when the length and width computed from application were compared with the manual readings. When RGB indices of fruit images were correlated with chemical properties of fruits, strong association was found between fruit acidity and mean intensity of green color (R² = 0.9966). Significant variability in total soluble solids was also explained by variation in R/G ratio (R² = 0.8464).

Introduction

Precision agriculture is an emerging era of research particularly in developing countries like India. Punjab is a leading state for cultivation of sub-tropical plum in India. After harvesting fruits undergo various operations such as packaging and transportation. Possibility of fruits getting spoiled is likely to happen as this process takes long time. Due to lack of storage facilities in developing countries, huge quantity of fruits and vegetables are wasted. In India, 18% of the total amount of fruits and vegetables produced are wasted from the post-harvest stage until they reach consumers (FASAR 2014). As plums are extremely perishable, so it is necessary to classify the harvested crop on the basis of maturity level. Manual inspection method has high rate of errors because of fatigue and distraction. Moreover it is a time consuming and laborious job.

To attain the best eating quality, majority of the fruits are allowed to ripen on the plant. But for distant marketing of fruits, harvesting of climacteric fruits at pre-optimum stage is required. Maturity level of plum can be further correlated with the estimated-days-to-rot. The prediction of estimated-days-to-rot is more important than the maturity level in decision making on the account of shipment delay. The maturity level also aids in selecting storage methods and selecting processing operations for value addition The proposed system classifies the plums into four categories namely green mature, color break, full color development and over-ripe based on color, texture and size (Fig. 1).

Fig. 1.

Maturity stages a green mature, b color break, c full color development, d over-ripe

Various methods have been applied by the researchers to automate the process of maturity determination. Alfatni et al. (2008) developed an automated grading system for oil palm bunches using the RGB color model. The program results showed that a different category of fruit bunches can be differentiated on the basis RGB intensity. Lee et al. (2008) developed a robust color space conversion and color index distribution analysis technique for automated date maturity evaluation. Omid et al. (2010) divided the fruit image into a number of elementary elliptical frustums. Ohali (2011) proposed computer vision based date fruit grading system. From RGB images, the external date quality features like flabbiness, size, shape, intensity and defects were extracted.

Devi and Varadarajan (2013) decomposed the image into three separate planes namely red, green and blue. Improved Bacterial Foraging algorithm was used for calculating three different thresholds for three planes. Jadhav and Patil (2013) proposed a fruit quality management system to examine the color and size of the fruit. The canny method was used for edge detection and the color feature was evaluated on the basis of RGB values. Mohammadi et al. (2015) developed an automatic algorithm for classifying the fruits on the basis of color. Two classifiers based on linear (LDA) and quadratic discriminant analysis (QDA) were used to assess the applicability of vision system. The results showed that QDA classifier could be valuable in categorizing the fruits with overall accuracy rate of 90.24%. Pourdarbani et al. (2015) developed an online sorting system for date fruit. The system was comprised of a conveying unit, illumination and capturing unit, and sorting unit. Physical and mechanical features were extracted from the samples and the detection algorithm was designed. An index based on color features was defined to detect date samples.

Automated agricultural industry is expanding all over the world but in developing countries of Asia and Africa, automation of agricultural management tasks using image processing is still limited. In India, most of the tasks are performed manually which are costly as well as unreliable because human decision in identifying quality factors is inconsistent, subjective and slow (Patel et al. 2011). Huge post-harvest losses in handling of extremely perishable fruits like plums have necessitated the development of non-destructive maturity assessment technique.

Materials and methods

MATLAB R2010a framework was used for developing software. Image Processing Toolbox of MATLAB provides a set of comprehensive reference algorithms, standard functions and applications for image processing, analysis, visualization and algorithm development. Plum fruit samples were collected from the Fruit Research Farm of PAU Ludhiana during the period of April–May 2016. Samples of plum variety ‘Satluj Purple’ were randomly picked from about 40 different plants maintained under uniform cultural practices. Plum fruits were labeled for identification purpose. Manual readings for major axis length and minor axis length were recorded with digital Vernier Caliper (MITUTOYO Japan) for validation purpose. Total soluble solids (TSS) were determined with the help of handheld digital refractometer (Atago, Japan) and expressed in per cent. The readings were corrected with the help of temperature correction chart at 20⁰ C, whereas, fruit acidity was estimated by taking 2.0 ml of stained juice and was titrated against 0.1 N NaOH solution using phenolphthalein as an indicator and expressed in percentage (AOAC 2000). The images of plum variety ‘Satluj Purple’ were captured with black background using digital camera (Nikon Coolpix S3200, Resolution—4608 × 3456, and Format-JPEG). Camera was mounted vertically over the fruit samples at a distance of 15 cm. The black color in background makes it easier to extract the fruit edge characters for determining its properties because RGB value of black color is zero. Images were captured in natural light to avoid reflections and shadows. Twenty-five images of each category were used for calibration purpose and remaining ten images of each category were used for validation purpose. Steps involved in the development of plum fruit maturity evaluation algorithm are image preprocessing, segmentation and feature extraction.

Image preprocessing

The purpose of image preprocessing is to improve the quality of digital image by decreasing the noise level and by correcting the distorted or degraded data. The nearest neighboring pixels in an image were utilized for getting the new brightness value in the output image.

Segmentation

It is basically used for locating objects of interest in images. The success of image analysis depends on reliability of segmentation process. In the proposed study, uniform thresholding operator available in MATLAB Image Processing Toolbox was used for locating the plum fruit in an image. In uniform thresholding, pixels above the specified level were set to white while those below the specified level were set to black. In this manner, the pixels pertaining to the fruit were separated from the background pixels and the region of interest (ROI) was segmented. The output of this phase was a binary image which had 1’s in the plum fruit region and 0’s in the background region.

Feature extraction

Feature extraction is an area of research which calls for high degree of attention. As no single technique is suitable for all types of images, so various feature extraction techniques were applied. Several features like color, texture and size were extracted for determining the right maturity level of plums. Based on these features, the plums were classified into four categories namely green mature, color break, full color development and over-ripe.

Color

The averaged surface color is a good parameter for judging the quality of fruits. In the proposed study, RGB model was used in which three channels namely red, green and blue contribute to create a colored image. Firstly, the colored image of plum was converted into gray scale image and histogram was plotted. Histograms are invariant to the geometrical transformations because they do not relate spatial information with the pixels of a given color which makes this software independent of fruit orientation. Abscissa represents intensity and ordinate represents number of pixels. To define appropriate threshold values for categorization, correlation coefficients between images were calculated (Choong et al. 2006). Correlation coefficients (r) between the selected image and template images were calculated iteratively (Fig. 2) by using following equation:

$$\text{r}=\frac{{\sum }_{\mathrm{m}}{\sum }_{\mathrm{n}}\left({\text{A}}_{\mathrm{mn}}-\overline{\text{A}}\right)\left({\text{B}}_{\mathrm{mn}}-\overline{\text{B}}\right)}{\sqrt{\left({\sum }_{\mathrm{m}}{\sum }_{\mathrm{n}}{\left({\text{A}}_{\mathrm{mn}}-\overline{\text{A}}\right)}^2\right)\left({\left({\sum }_{\mathrm{m}}{\sum }_{\mathrm{n}}{\text{B}}_{\mathrm{mn}}-\overline{\text{B}}\right)}^2\right)}}$$ 1

where A, B are feature vectors or matrices and − 1 ≤ r ≤ + 1

Fig. 2.

Comparison of RGB Histograms

Further, it was used for calculating the strength of relationship by using Eq. 2.

$$\text{Strength}\text{of}\text{Relationship}={\text{r}}^2\ast 100$$ 2

The results were stored in an array. The biggest correlation coefficient was selected from the array and by this comparison the maturity level of the given plum was judged.

Texture

1. Local binary pattern (LBP)

Firstly, the input image was converted to gray scale. For every pixel in the gray scale image, a neighborhood was selected around the current pixel. For calculating the LBP value for a pixel in the gray scale image, the comparison of the central pixel value with all the eight neighboring pixel values was carried out. The order of traversal must be same for all the pixels i.e. it should be either clock-wise or anti-clockwise. If the current pixel intensity value was greater or equal to the neighboring pixel intensity value, then the corresponding bit in the binary array was set to 1 otherwise it was set to 0. In this way 8-bit binary code was calculated for all the pixels. The results of the comparisons were stored in 8-bit binary array. For calculating the LBP value, binary to decimal conversion was performed. LBP values of all the pixels were stored in the LBP mask which had the same dimensions as that of input image and then LBP mask was mapped to a histogram for the texture representation.

By comparing the LBP features of the uploaded image with the LBP features of the template images, the maturity level of the plum was judged.

• 2.Discrete cosine transformation (DCT)

To analyze the texture of a digital image, large storage space and a lot of computational time is required for calculating the matrix of features. To overcome these problems, the use of DCT for texture representation is advised. The neighboring pixels within an image are highly interrelated. An invertible transform was used for concentrating randomness into few de-correlated parameters. The DCT is an ideal approach for energy concentration and de-correlating for a large class of images. The RGB image was converted into gray scale image. The Block-based DCT transformation was used for spatial-localization. The DCT gives a very high energy compaction without degrading the image quality. For extracting textural features, DCT coefficients in compressed domain were used as the feature vectors.

• 3.Entropy

Image entropy is a key attribute which is used for describing the ‘business’ of an image. There is a built-in method in MATLAB to calculate the entropy value of any gray scale image to carry out a detailed analysis on images. This also helps in carrying out statistical analysis of the randomness. Firstly, the entropy filtering was applied to the uploaded image. Iteratively, the entropy filtering was applied to all the template images and the correlation coefficients were calculated. The resultant values were stored in an array. In order to filter out the irrelevant images from the results, a threshold value was chosen.

Size

Firstly, the plum fruit edge characters were extracted. For this purpose the background subtraction was performed and the binary image was obtained. Various size related properties like projected area, perimeter, major-axis length and minor-axis length were calculated in terms of number of pixels by using the inbuilt functions available in the MATLAB framework.

$$\text{size}\_\text{prop}=\text{regionprops}\left(\text{binary}\_\text{image},\text{'all'}\right)$$

The above function returns the values in the terms of pixels. To give out the results in standard units of measurement, calibration factor (pixels/cm) was required (Nandi et al. 2016). In the proposed study, distance from camera was kept fixed (15 cm) and the object of known dimensions was used as reference for calculating calibration factor.

Multi attribute decision making (MADM) theory

MADM theory refers to making decisions on the basis of multiple attributes. The MADM method specifies how the values of various attributes are to be processed to get the final result. Each attribute is assigned a weightage depending on its importance to arrive at a particular solution. The option which returns the highest value is selected. In this system the color and texture of the plum are important attributes for determining maturity level. Color and texture of the fruit are not conflicting attributes but in some rare cases the results of the two may differ. In order to resolve such a conflict MADM theory was applied. Color was given more weightage than texture because color is a better distinguishing factor than texture for classification. The sum of weights is always equal to 1 where n is the number of attributes taken into consideration (Eq. 3)

$$\sum _{i=1}^n{w}_i=1$$ 3

Results and discussion

Experiments were performed by using 140 images of plum variety ‘Satluj Purple’. Each image was saved in JPEG format. Various features were analyzed and effectiveness of each feature in overall decision making process was evaluated. Various images were tested to prove that the color is the most dominant factor for determining the maturity level. 25 images of each category were used for the calibration and 10 images of each category were used for testing purpose. Graphical User Interface was designed using GUIDE (Graphical User Interface Design Interface) Layout Editor (Fig. 3a).

Fig. 3.

a Graphical user interface for identifying maturity of plum fruit, b relationship between color indices and chemical properties

Size analysis

The size based classification of plums is done according to the length of major axis and minor axis. The error percentage between the manual and the calculated values was less than 2.4% (Table 1). Similarly, Prabha and Kumar (2013) assessed the maturity of banana fruit on the basis of size by using image processing techniques.

Table 1.

Comparison of manual and calculated plum fruit size values

Sample	Length (cm)				Width (cm)
	Manual	Calculated	Error	Error %	Manual	Calculated	Error	Error %
1	4.83	4.90	0.07	1.4	4.77	4.83	0.06	1.2
2	4.08	4.03	0.1	2.4	4.33	4.41	0.08	1.8
3	4.15	4.28	0.05	1.2	4.25	4.28	0.03	0.7
4	4.13	4.10	0.03	0.7	5.03	4.98	0.05	0.9
5	5.07	5.15	0.08	1.5	5.18	5.15	0.03	0.5
6	4.29	4.27	0.02	0.4	4.34	4.31	0.03	0.6
7	4.85	4.78	0.07	1.4	3.85	3.94	0.09	0.2
8	4.34	4.29	0.05	1.1	4.21	4.26	0.05	0.1
9	5.0	4.98	0.02	0.4	4.78	4.87	0.09	1.8
10	4.97	4.91	0.06	1.2	4.95	4.85	0.1	2.0

Color and texture analysis

Mean RGB intensity values of 40 samples were used for calculating range of RGB values for each category (Table 2). Results indicated that mean RGB values for various categories were not overlapping therefore mean RGB values were used for setting threshold values for categorization. The mean intensity value of green color decreased as the plum fruit ripened. No significant variation was observed in mean intensity value of blue color with respect to maturity level.

Table 2.

Range of RGB intensity values

Category	Red		Green		Blue
	Min	Max	Min	Max	Min	Max
Green mature	58.62	75.52	65.81	77.41	32.51	35.49
Color break	116.82	131.45	44.58	67.95	20.80	25.37
Full color development	92.02	107.07	22.23	32.34	14.82	30.04
Over-ripe	72.64	77.53	22.4	30.89	23.62	29.56

Pearson correlation coefficient indicates the degree of similarity between two images and was recorded to analyze the relationship on the basis of various parameters. Twelve samples of different categories were used for comparison purpose. Strength of Relationship between images of various categories was recorded on the basis of mean RGB values (Table 3a). It was observed that there was strong positive correlation between all the images of four categories. Intra-class coefficients were found to be significantly higher than Inter-class coefficients. So RGB indicies were used for setting threshold values for categorization. Color was found to be very dominating factor. These results are in agreement with the findings of Prabha and Kumar (2013) who reported the 99.1% accuracy in color intensity algorithm for assessment of the banana fruit maturity.

Table 3.

Strength of Relationship between different categories on the basis of color intensities (3a), texture (3b) and combination of color and texture (3c)

Strength of relationship		Green mature			Color break			Full color development			Over-ripe
		1	2	3	4	5	6	7	8	9	10	11	12
(a)
Green Mature	1	100	97.65	98.47	87.85	88.45	86.38	84.82	83.65	88.72	87.31	89.51	88.87
	2	97.65	100	97.56	87.31	88.08	86.08	83.85	82.01	89.45	89.00	90.40	90.16
	3	98.47	97.56	100	88.47	89.32	87.53	86.14	84.09	90.40	88.92	90.80	90.16
Color Break	4	87.85	87.31	88.47	100	97.52	96.94	92.56	93.35	91.81	91.83	93.05	92.62
	5	88.45	88.08	89.32	97.52	100	98.27	92.33	92.35	91.91	90.86	91.91	91.78
	6	86.38	86.08	87.53	96.94	98.27	100	89.98	89.66	89.36	88.23	89.32	88.81
Full color development	7	84.82	83.85	86.14	92.56	92.33	89.98	100	98.29	95.79	89.53	90.52	89.45
	8	83.65	82.01	84.09	93.35	92.35	89.66	98.29	100	91.97	88.02	89.06	88.36
	9	88.72	89.45	90.40	91.81	91.91	89.36	95.79	91.97	100	94.56	94.28	94.13
Over-ripe	10	87.31	89.00	88.92	91.83	90.86	88.23	89.53	88.02	94.56	100	98.55	99.12
	11	89.51	90.40	90.80	93.05	91.91	89.32	90.52	89.06	94.28	98.55	100	99.12
	12	88.87	90.16	90.16	92.62	91.78	88.81	89.45	88.36	94.13	99.12	99.12	100

		1	2	3	4	5	6	7	8	9	10	11	12
(b)
Green mature	1	100	6.62	7.95	6.66	4.86	5.99	3.46	5.14	5.22	5.23	5.31	3.86
	2	6.62	100	7.16	5.50	5.76	5.13	4.10	4.06	4.42	3.76	4.29	5.05
	3	7.95	7.16	100	5.99	4.84	4.95	3.77	4.87	4.91	3.43	4.59	3.55
Color break	4	6.66	5.50	5.99	100	5.59	5.75	3.34	4.05	3.81	3.81	4.30	3.83
	5	4.86	5.76	4.84	5.59	100	5.75	3.21	4.05	3.68	3.25	3.24	3.62
	6	5.99	5.11	4.96	5.75	5.75	100	4.01	3.66	3.40	3.37	4.31	3.14
Full color development	7	3.46	4.04	3.77	3.34	3.22	4.00	100	2.66	3.29	1.84	2.87	2.52
	8	5.14	4.05	4.88	4.06	4.06	3.66	2.66	100	3.66	2.80	3.74	2.83
	9	5.22	4.44	4.88	3.66	3.65	3.42	3.29	3.66	100	3.07	3.84	3.06
Over-ripe	10	5.23	3.76	3.43	3.81	3.25	3.37	1.82	2.85	3.08	100	4.76	3.51
	11	5.31	4.29	4.59	4.30	3.24	4.31	2.87	3.74	3.84	4.76	100	3.30
	12	3.86	5.05	3.55	3.83	3.62	3.14	2.52	2.83	3.06	3.51	3.30	100

		1	2	3	4	5	6	7	8	9	10	11	12
(c)
Green mature	1	100	79.44	80.36	71.61	71.73	70.30	68.54	67.94	72.02	70.89	72.67	71.86
	2	79.44	100	79.48	70.94	71.61	69.88	67.88	66.41	72.44	71.95	73.17	73.13
	3	80.36	79.48	100	71.97	72.42	71.01	69.66	68.24	73.29	71.82	73.55	72.83
Color break	4	71.61	70.94	71.97	100	79.13	78.70	74.71	75.49	74.18	74.22	75.30	74.86
	5	71.73	71.61	72.42	79.13	100	79.76	74.50	74.69	74.25	73.33	74.17	74.14
	6	70.30	69.88	71.01	78.70	79.76	100	72.78	72.46	72.17	71.25	72.31	71.67
Full color development	7	68.54	67.88	69.66	74.71	74.50	72.78	100	79.16	77.29	71.98	72.99	72.06
	8	67.94	66.41	68.24	75.49	74.69	72.46	79.16	100	79.30	70.98	71.99	71.25
	9	72.02	72.44	73.29	74.18	74.25	72.17	77.29	79.30	100	76.26	76.19	75.91
Over-ripe	10	70.89	71.95	71.82	74.22	73.33	71.25	71.98	70.98	76.26	100	79.79	79.99
	11	72.67	73.17	73.55	75.30	74.17	72.31	72.99	71.99	76.19	79.79	100	79.95
	12	71.86	73.13	72.83	74.86	74.14	71.67	72.06	71.25	75.91	79.99	79.95	100

For textural analysis, the values for Strength of Relationship between twelve samples of different categories were also recorded (Table 3b). There was no significant difference between inter-class and intra-class correlation coefficients for textural feature matrices. So classification of fruits solely on the basis of texture may give inaccurate results. The developed system was tested for two different test cases i.e. calibration set and validation set. When the developed system was tested for calibration set, it yielded 100% accuracy for both color and texture. 20% plum fruit samples from the validation set were misclassified when tested solely on the basis of texture.

For achieving the higher accuracy and for reducing the misclassification rate MADM theory was employed. MADM theory combined the color and textural features. Color was given more weightage than texture because misclassification was least when tested on the basis of color features. Strength of Relationship was also recorded for the combination of color and textural features (Table 3c). The combination of color and textural properties yielded accurate results.

Relationship between color indices and chemical properties

In plum fruits, development of purplish red color is considered as the maturity indices. As the intensity of purplish red color of plum fruits increased, the ripening attributes also improved. Total soluble solids mainly contribute towards the sweetness of fruits during ripening. Results showed that as the intensity of purplish red color increased, the TSS content of fruits also increased proportionately but at the overripe stage a decline in TSS content was observed that may be due to senescence of fruits.

But in case of fruit acidity the trend was reversed. As the fruit color development enhanced, the acid content of fruits decreased. During maturation, the fruit itself might utilize the acids that results in reduction of organic acids (Bhattarai and Gautam 2006).

The variation in the distribution of the colors in the images can be explained by variation in the chemical content in different parts (Teerachaichayut and Ho 2017). There are the relationships between RGB indices, TSS content and fruit acidity as shown in Fig. 3b. When the plum fruits ripened, acidity decreased from 2.28 to 0.6% and correspondingly the mean intensity of green color also decreased from 71.61 to 26.64. The following equation form was satisfied for predicting fruit acidity by mean intensity of green color:

$$\begin{array}{c}\hfill y=0.0011x^2-0.0764x+1.8758\\ \hfill {\text{R}}^2=0.9966\\ \hfill \end{array}$$ 4

The high value of the coefficient of determination (R² = 0.9966) indicated strong association between fruit acidity and mean intensity of green color.

As the ripening progressed, TSS content firstly increased from 10.1 to 13.88% and then declined to 13.76% at over-ripe stage. Similar trend was observed for R/G ratio which firstly increased from 0.936 to 3.857 and then decreased to 2.818 at over-ripe stage. Similarly, a positive correlation between TSS and color intensity of plum fruits was observed by Majeed and Jawandha (2016). The following equation form was satisfied for predicting TSS by R/G ratio:

$$\begin{array}{cc}\hfill \text{y}& =-0.111x^2+1.9549x+8.211\hfill \\ \hfill {\text{R}}^2& =0.8464\hfill \\ \hfill \end{array}$$ 5

High value of coefficient of determination (R² = 0.8464) indicated that significant variability in TSS content can be explained by variation in R/G ratio.

Thus, the system could potentially be applicable in predicting the right fruit maturity and its chemical properties to make the plum fruit production more profitable. The additional advantage of the developed software is that it can be easily extended for other fruits and vegetables with minor changes.

Conclusion

The developed system accurately determined the maturity level. Color was found to be the most dominant factor for classifying the plums according to maturity level. The error percentage was less than 2.4%, when the length and width computed from application were compared with the manual readings. When RGB indices of fruit images were correlated with chemical properties of fruits, strong association was found between fruit acidity and mean intensity of green color (R² = 0.9966). Significant variability in TSS content was also explained by variation in R/G ratio.