Determination of the olive maturity index of intact fruits using image analysis

Abstract

In this work, the maturity index of different samples of olives was objectively assessed by image analysis obtained through machine vision, in which algorithms of color-based segmentation and operators to detect edges were used. This method allows a fast, automatic and objective prediction of olive maturity index. This prediction value was compared to maturity index (MI), generally used by olive oil industry, based on the subjective visual determination of color of fruit skin and flesh. Machine vision was also applied to the automatic estimation of size and weight of olive fruits. The proposed system was tested to obtain a good performance in the classification of the fruit in batches. When applied to several olive samples, the maturity index predicted by machine vision was in close agreement with the maturity index of fruits visually estimated, values that are currently used as standards. The evaluation of weight of fruit also provided good results (R² = 0.91). These results obtained by image analysis can be used as a useful method for the classification of olives at the reception in olive mill, allowing a better quality control of the production process.

Introduction

The quality of virgin olive oil depends on both agronomic and technological factors. With regard to the agronomic aspects, there are several factors that can affect the quality of the olive oil, including olive variety, irrigation rate, pest attack, fruit diseases and harvest time (Uceda and Frias 1975). Harvest time is the first and most crucial decision to make in the production process of virgin olive oil. It is determined by several factors: fruit retention force, oil content in the olive fruit, chemical composition, sensory attributes of olive oil and industrial yield. Because these factors evolve as the fruit matures, it is necessary to use methods to monitor the maturity process. The most common method for evaluating maturity in olives is based on the visual determination of the maturity index (MI). This involves assessing changes in the color of the skin and flesh as the fruit matures and giving them a classification (from 0 to 7) based on the color (Mínguez-Mosquera and Gallardo-Guerrero 1995; Roca and Minguez-Mosquera 2001). It is a very time-consuming procedure and very subjective because it depends on the opinions and experience of the evaluator, as well as on environmental factors such as lighting, color appearance and condition of the fruit, all of which can be highly variable. In addition, evaluators often have to decide between colors that are difficult to distinguish. From a visual point of view and depending on the stage of ripeness, the color of olives can change from intense green to black (Ting and Rouseff 1986). This is because in the final stages most of chlorophylls are degraded and replaced by anthocyanins, which makes the fruits more vulnerable to external damage and infection (Gnanasekharan et al. 1992; Garcia et al. 1996a; Tovar et al., 2002).

Knowing the MI is useful for producers because it enables them to identify the optimal harvest time to improve the quantitative and qualitative characteristics of olive oil production. For example, for the Picual olive cultivar, an MI value of between 3 and 4 indicates a good balance between the yield and quality of olive oil, regardless of the extraction process (Baccouri et al., 2007). Other methods to determine the stage of maturity in olives include determining fruit firmness (Taylor et al. 1995; Lehman-Salada, 1996), measuring the fruit’s rate of respiration (Ranalli et al., 1997), using non-destructive methods such as a densimeter (Garcia et al. 1996b; Olmo et al. 2000; Garcia et al. 1998), measuring the transmission of acoustic waves through the fruit (Muramatsu et al. 1996) and using a flourescence method to evalute the ripening grade by anthocyanins.determination (Agati et al. 2005). Some authors have proposed a modification of some classical chemical parameters as indicators of the maturity process. Mickelbart & James (2003) proposed using the dry matter percentage in olive fruit and Cherubini et al. proposed using the sugar content as a technological MI. Recently, Near Infrared Spectroscopy (NIRS) has been also used to evaluate olive fruit maturity. Gracia & Leon (2011) studied the evolution of oil and moisture content in intact olives during the maturity process. Cayuela et al. (2009) carried out a similar study, but also included the prediction of acidity and oil content on the basis of fresh weight and dry matter. Both studies investigated predicting these quality parameters in olive fruit and virgin olive oil by directly measuring the fruit using NIRS. Bellincontro et al. (2012) applied NIR technology to the measurement in the field of the individual concentration of the main polyphenols (oleuropein, verbascoside, and 3, 4-DHPEA-EDA) and the total polyphenols in olives, proposing these parameters as indicators of fruit maturity. Morales-Sillero et al. (2011) studied the feasibility of NIR spectroscopy for the non-destructive characterization of table olive traits, including the MI. Salguero-Chaparro et al. (2012) studied the optimization of the acquisition parameters for that online analysis of intact olive fruits using vis-NIR spectroscopy, which is a first step in implementing NIRS technology for process control in olive oil production.

Another important method for evaluating the maturity stage of olive fruits is computer vision. Many applications have been developed in recent years using this method for classifying olive fruit according to various criteria. The maturity stage has been used to classify peaches during harvest by applying histograms in the red channel (R) and a combination of red and infrared images (red divided by the corresponding infrared image [R/IR]) (Cordero et al. 2006). Other studies have used a region-oriented segmentation algorithm to detect the most common peel defects in citrus fruits (Blasco et al. 2007). This detection focuses mainly on aspects of interest—the sound peel, the stem and the defects—and allows a successful segmentation to be made of smaller defects, such as scale [Kondo et al. 2000], quality control and defects in cherries (Rosenberger et al. 2004; Uthaisombut, 1996) and apples (Cheng et al. 2002; Kavdir and Guyer 2004; Mehl et al. 2004). The detection can also be based on the shape, size and maturity of the fruits, as shown for cucumbers (Abdullah et al. 2006). In this case, artificial classifiers such as linear discrimination analysis and multi-layer perceptron neural network were used, leading to a 100% success rate in placing cucumbers in three shape categories: well formed, slightly deformed and seriously deformed. Other studies have measured the volume of watermelon by using water displacement, ellipsoid approximation and image processing methods (Koc, 2007). Brosnan and Sun (2004) presented a comprehensive review of image processing techniques for various food products, including bakery products, meat, fish, vegetables and fruits. Earlier studies demonstrated the ability to sort fruits according to the color (Pla et al. 1999; Mendoza and Aguilera 2004; Oberti et al. 1999; Peri et al. 2003). Significant correlations between shape and color during aging in apples have been found by evaluating the diameter of the average per tree, using manual measurements at different development stages of apple fruits (Stajnko and Cmelik 2005). Some studies in the literature have used the correlation between images and degree of ripeness to characterize the ripening process and define appropriate threshold values for various cultivars in order to evaluate possible applications in field and postharvest practice (Bodria et al. 2002; Choong et al. 2006). Computer vision has also been applied to table olives to detect external damage (Diaz et al., 2000, 2004; Riquelme et al. 2008, Furferi et al. 2010) and has become a widely used technology in the production of table olives at industrial level.

Until now, all studies using vision systems enabled an overall and unique MI to be obtained for a set of olives. The new vision method proposed, based on the use of infrared and visible images, allows an MI to be obtained for each individual olive in a set, which provides more information about the olives entering the factory and can help in decision-making along the process line. To be accepted in the industry, therefore, the method for assessing the MI should be easy to use, inexpensive and reliable. In order to reduce the subjectivity of the current method for determining the MI of olives, this study aims to provide an automatic method that integrates a machine vision system for classifying olives. This is done by classifying them according to maturity class using an image segmentation algorithm based on color and edge detection. In order to improve the agreement of predicted values through computer vision, this study combines the information obtained by color analysis and the MI prediction for individual olive fruits. The novel methodology proposed also provides more quality information, such as olive size and average weight, and could be useful in improving the performance of commercial systems by allowing for the classification and removal of olives that do not meet the required standards and in improving quality control in the extraction process of olive oil.

Materials and methods

Samples

Olive samples of the Picual variety (2 kg) were collected from several olive mills in the province of Jaen, Spain in November 2011. Their colour ranged from dark green to completely black. They were used for the acquisition of images, and synthetic samples were created in order to ensure that the samples covered the full MI range.

Visual determination of the maturity index (MI)

On each sampling day, the MI was determined by a panel of experienced evaluators who assessed the skin and flesh color of the olives (Garcia et al. 1996a). This involved dividing the olives into eight groups according to the characteristics summarized in Table 1. The official method uses 100 olives and calculates an overall MI value for each sample or group of olives.

Table 1.

Maturity Index (MI) classification groups of olives based on skin and flesh color

Maturity index group	0	1	2	3	4	5	6	7
Description	Skin color deep green	Skin color yellow-green	Skin color with < half the fruit surface turning red, purple or black	Skin color with > half the fruit surface turning red, purple or black	Skin color all purple or black with all white or green flesh	Skin color all purple or black with < half the flesh turning purple	Color all purple or black with > half the flesh turning purple	Skin color all purple or black with all the flesh purple to the pit

The MI is calculated using the following equation:

$$\mathit{MI}={\displaystyle \sum i}*\mathit{ni}/100$$

where i is the group number and ni is the number of olives in the group.

This method involves manually separating the olives, cutting the pulp to examine it, counting them and identifying the group to which they belong. The method also involves calibrating other measurements of the fruit, such as weight, size and diameter.

Image system

Images were acquired using a JAI AD-080CL multi-spectral camera which combined a visible color channel (Bayer mosaic CCD) that can generate 24-bit RGB images and an NIR channel (monochrome CCD). The major advantage of this camera is its ability to capture both channels simultaneously along the same optical path.

The camera uses a standard Camera Link interface, whereby each channel can output images with 8- or 10-bit and 24-bits, and a resolution of 1024 × 768 active pixels per channel. The camera was installed in an enclosed cabin equipped with controlled lighting in order to achieve consistent images. The lighting was provided by a halogen lamp with some filters to produce a diffused light simulating the illuminant D65 (6,500 K), which is the standard as determined by the International Commission on Illumination, simulating the lighting conditions of a cloudy day at midday. Direct lighting on the olive fruits was avoided by using a reflecting surface placed between the fruits and the illuminant. The images were taken at a distance of about 45 cm, with a neutral white background.

Methodology

The algorithms used were developed with the image processing toolbox, version 7.4.0 of Matlab (The MathWorks, Inc., Natick, MA, USA), along with the software Image Pro-Plus version 6.0 (MediaCybernetics,Inc).

The images were taken in duplicate for both channels, IR (monochromatic) and visible (RGB image), where the pixels are specified by three values, one for each color component (red, green and blue).

Estimation of the maturity index (MI)

The technique used to estimate the MI was segmentation, which is based on identifying regions and edges using clusters of pixels selected according to various criteria (color, boundary, texture, etc.) (Medioni et al. 2000; Plataniotis and Venetsanopuolos 2000).

In order to develop a method for the automatic prediction of MI, the following steps were carried out:

Identification and separation of objects in the NIR image

Separating the objects involves using the NIR monochrome image in order to segregate each olive fruit. To identify objects in the NIR monochrome image, initially a contrast equalization is performed, followed by a spatial filter enhancement (‘Flatten’), which reduces the intensity variations in the background pixels. The spatial filtering operation was applied to the images to enhance or attenuate the spatial detail in order to improve the visual performance or facilitate further processing. Spatial filtering can be seen as a ‘local’ procedure in image processing, in that it changes the value of each pixel according to the values of the surrounding pixels. From a practical point of view, this steps focuses mainly on classifying the gray levels in the original NIR image according to the value in the neighboring pixels. This task was carried out using two algorithms: a) Border-4 Neighbor, which determines the edges of the olive taking account of the intensity values of pixels in the histogram and of the darker pixels as edges of objects; b) connected components algorithm, which is used to allocate a value to each olive. This edge-based algorithm enumerates each set of pixels that make up an olive (Chang and Saavedra 2001).

Creating a mask and applying it to the visible color image

From the data in the array of connected components, a mask can be created, to which various size filters can be added in order to measure complete objects or automatically remove those objects that are larger or smaller than the fixed size limit (Russ, 2006). A skeleton of the image is constructed on the basis of these objects. The skeleton is intended to represent the shape of an object with a relatively small number of pixels, and therefore all pixels of the skeleton are structurally necessary. The position, orientation and length of the skeleton lines correspond to those of the original image. The skeleton simplifies the task of extracting the features of an image.

The mask, which reduces the image to a skeleton and eliminates protrusions, is added the RGB image on which color-segmentation is performed for the separated objects.

Segmentation based on color and classification

Segmentation based on color can be carried out using supervised or unsupervised methods in order to obtain and quantify the predominant color in the olive fruits. This step requires transforming the RGB image using various functions for changing the RGB format in L*a*b color space. The CIE L*a*b* colour notation system was applied to determine the parameters L*, a* and b*, where L* is the lightness, a* is the color axis from green to red and b* is the color axis from blue to yellow.

With the supervised methods, a region containing the color of interest (color markers) is selected and then averaged. This classification is done by using the k-nearest neighbor (KNN) method (Richards, 1999), where each pixel is classified in the same class as color markers that have a similar intensity. The k-nearest neighbors are assigned a value of ‘a’ and ‘b’ for each marker, and it is then possible to classify each pixel in the image in order to calculate the Euclidean distance between pixels and color markers. Unsupervised methods for color segmentation techniques use clustering algorithms that essentially perform the same functions as classifiers methods, but without using the training data (Thomas, 1991; Floyd and Steinberg 1975). To compensate for the lack of training data, clustering methods iterate between image segmentation and characterize the properties of each class. In this way, the clustering methods are trained using the available data. The basic idea is to assume that image pixels are points in a three-dimensional space (RGB), so that the points of a area of similar color are grouped together. Clustering techniques allow a representative point for each group to be obtained, based on measures of similarity between these points. In this algorithm, point xi is assigned to group R whose centroid C is closer to it. Euclidean distance is usually used as a measure of similarity between the point and the centroid. The results of this classification are normally expressed in terms of the total percentage of pixels colored in each object. Using these results, fruits can be classified according to their stage of maturity (Table 2).

Table 2.

Olive classification based on the skin color

Class	Olive skin color
0	>50% Bright green
1	>50% Greenish-yellow
2	% greenish-yellow with black and/or reddish spots > % reddish-brown
3	% greenish-yellow with black and/or reddish spots < % reddish-brown
4	100% blackish-purple or black

Estimation of the size and weight

The study also sought to determine the size and weight of the fruit using image analysis. To do this, a calibration curve for different weights of the fruit and their respective areas in the image pixels was performed using a group of training samples. This was then validated with a set of external validation samples.

Results and conclusion

Estimation of the maturity index (MI)

All the available sets of olives were analyzed, and for each batch of olives the ripeness index was calculated following the procedure described in the previous section.

The procedure was performed first for 40 synthetic images with separate olives, and these algorithms were applied in order to obtain the MI for each fruit. The results were obtained as the percentage of each color in the object and the fruits were then grouped into the different classes. The effectiveness of the method was demonstrated with the Picual olive variety, in which the fruits vary greatly in color and size.

This test phase with sample images showed that the values given by the vision system were very similar to those obtained using the traditional method based on visual determination and used as the reference. Table 3 shows some results obtained from applying the proposed procedure to images containing olives with different MIs. Figure 1 shows an original image and the image after the analysis results were available, with each object class shown in a different pixel color.

Table 3.

Results of the maturation index (MI) estimated and visually calculated for some samples analyzed

Object	%	%	%	%	Estimated	Measured
	Bright green	Greenish-yellow	Reddish-brown	Black	MI	Visual MI
1	100	0	0	0	0	0
2	57	43	0	0	0	1
3	33.3	66.7	0	0	1	1
4	0	100	0	0	1	1
5	1	99	0	0	1	1
6	0.6	99.4	0	0	1	1
7	2.8	97.2	0	0	1	1
8	0.7	99.3	0	0	1	1
9	0	40.4	59.6	0	3	3
10	0	67.4	32.6	0	2	2
11	0	33.6	66.4	0	3	3
12	0	51.9	48.1	0	2	2
13	0	60.6	39.4	0	2	3
14	0	49.9	50.1	0	3	3
15	0	0	100	0	3	3
16	0	31.3	68.7	0	3	3
17	0	0	0	100	4	4
18	0	0	0	100	4	4
19	0	0	0	100	4	4
20	0	0	0	100	4	4

Fig. 1.

An original image and the image after the analysis results were available, with each object class in a different pixel color (black = % black; brown = %reddish-brown; green = % green)

Most of the errors (refers to the possible failures in the segmentation process) found in the pixel segmentation procedure were due to isolated or small clusters of pixels, located mainly at the boundaries of adjacent regions. These groups of pixels can be easily detected and discarded, as they do not form broad regions, but are fairly discontinuous and are divided into segments. They can therefore be discarded due to their small area, using the object’s center area to apply the color classification.

The results of the predicted MI for the fruit using the proposed vision method for images with individual olives were strongly related to the visual RI measured. The errors of prediction were almost negligible or zero for the other samples (results in Table 2). These values are acceptable given the subjectivity that characterizes MI evaluation using the traditional method.

This protocol was applied to 15 complex images with overlapping olives. About 100 olives in each image were analyzed and an average of 92% was correctly detected as separate objects and analyzed separately by the program. Figure 2 shows an example of the analyzed images.

Fig. 2.

a Image of sample in the RGB space, b image after segmentation and c image obtained after applying the complete procedure

The protocol therefore allows an effective analysis of samples and could substantially improve these results through the improvement and standardization of the lighting conditions for the samples.

The results of the analysis of these 15 images are shown in Fig. 3, where it is possible to compare the results obtained using both methods.

Fig. 3.

Histogram representation of the results of an analyzed image with relative errors values

These results confirm the high correlation between both methods, with the difference values provided by the vision system and the classical visual method both being very low and acceptable given the high subjectivity of the method used as the reference.

Estimation of olive size and weight

This part of the study sought to estimate the fruit weight by the area calculated in pixels in the image processing The real weights had been previously measured in 40 samples of olives and a positive correlation was found between these real weights and the area in pixels (correlation coefficient of 0.91) (Fig. 4).

Fig. 4.

Correlation between olive areas in pixel and real olive weight measured

These results show the statistical interdependence between the weight of fruit and the visual images observed in the sample as the measured area. The error values were higher than is desirable because in this case the type of ripeness of the samples was not taken into account. The weight of olives can depend on the stage of fruit maturity; the amount of water and oil also increases until it reaches its maximum in olives that are more ripe. The next step involved constructing a calibration line for each class of maturation, as shown in Fig. 5.

Fig. 5.

Correlation between olive areas in pixels and olive weight (in grams) for a maturity index (MI) of1, 2, 3 and 4, respectively

In order to validate the calibration data using external samples, the weight of 10 samples of olives was measured. The validation results are shown in Table 4.

Table 4.

Data validation samples and relative error values for both calibrations, global calibration (GC) and partial calibration (PC)

		Estimate Weight	Estimate Weight	Area	Relative Error	Relative Error
Object	Real Weight (grams)	Global Calibration (grams)	Partial Calibration (grams)	(pixel)	(Global Calibration)	(Partial Calibration)
1	3.16	3.4	3.1	5717	0.08	0.02
2	3.4	3.8	3.3	6054	0.12	0.03
3	2	1.8	1.9	4111	0.10	0.05
4	3.43	3.7	3.6	5977	0.08	0.05
5	4.21	4.7	4.5	6986	0.12	0.07
6	3.22	3.1	3.4	5406	0.04	0.06
7	4.3	4.5	4.5	6740	0.05	0.05
8	3.77	3.5	3.8	5786	0.07	0.01
9	4.36	4.4	4.5	6658	0.01	0.03
10	3.3	3	3.3	5274	0.09	0.00

As has been shown previously it is therefore important to check the influence of maturity on determining fruit size and weight. Therefore have compared the estimated weight of the olives using two methods, one without considering maturation status (global calibration) and another having account of the state of maturation olives belong to (partial calibration). This method can be a very useful tool for quickly and accurately assessing fruit maturity, with a relative error of not more than 0.07, ensuring a proper determination of size. From the visual and geometric characteristics of the fruit, an automatic vision system of fruit weight can be developed in order to improve the quality of information for farmers.

This method could be useful for selecting the date of collection, based on tracking changes in the color of the fruit on the tree until the selected value is achieved. Harvest time, therefore, should not depend on a subjective decision, but on a uniform measure, using a rapid and non-destructive method.

The proposed methodology has the potential to be used for classifying fruit according to quality during the extraction process of olive oil. The results of this study suggest that the implementation of the methodology and on-line image acquisition and analysis can occur at acceptable speeds; this will be investigated in future studies.

The method described provides an estimate of the MI value. This procedure requires a negligible amount of time; the only time required is for collecting the olive samples from the trees. This is important when the experts working in oil mills carry out real-time monitoring of the agronomical and technological parameters. In some cases, however, the purpose of oil mills is to increase the quality of the extracted olive oil by optimizing some technological parameters.

An objective of the online and non-destructive methods used in classification systems is to ensure that products meet a given set of quality characteristics and parameters. In particular, optical techniques offer potential in the assessment of surface characteristics and inner qualities or composition. The systems used to measure optical properties provide good information about the characteristics of each fruit, which is useful for the assessment of products.