Non-Destructive Estimation of Physicochemical Properties and Detection of Ripeness Level of Apples Using Machine Vision

ABSTRACT Nondestructive estimation of physicochemical properties, post-harvest physiology, and level of ripeness of fruits is essential to their automated harvesting, sorting, and handling. Recent research efforts have identified machine vision systems as a promising noninvasive nondestructive tool for exploring the relationship between physicochemical and appearance characteristics of fruits at various ripening levels. In this regard, the purpose of the current study is to provide an intelligent algorithm for estimating two physical properties including firmness, and soluble solid content (SSC), three chemical properties viz. starch, acidity, and titratable acidity (TA), as well as detection of the ripening level of apples (cultivar Red Delicious) using video processing and artificial intelligence. To this end, videos of apples in orchards at four levels of ripeness were recorded and 444 color and texture features were extracted from these samples. Five physicochemical properties including firmness, SSC, starch, acidity, and TA were measured. Using the hybrid artificial neural network-difference evolution (ANN-DE), six most effective features (one texture and five color features) were selected to estimate the physicochemical properties of apples. The physicochemical estimation was then further optimized using a hybrid multilayer perceptron artificial neural network-cultural algorithm (ANN-CA). The results showed that the coefficient of determinations (R2) related to the prediction models for the physicochemical properties were in excess of 0.92. Additionally, the ripeness level of apples was estimated based on physicochemical properties using a hybrid multilayer perceptron artificial neural network-harmonic search algorithm (ANN-HS) classifier. The developed machine vision system examined ripeness levels of 1356 apples in natural orchard environments and achieved a correct classification rate (CCR) of 97.86%.

For instance, Vis-NIR spectroscopy was used by several scholars for assessing apples' internal properties and/or ripening levels. Sharabiani et al. (2020) successfully estimated TA and taste index of gala apples. In another work, Sharabiani et al. (2021) reliably estimated gala apples' total soluble solids and BrimA. Pourdarbani et al. (2021) successfully identified Fuji apples' firmness, acidity, and starch content. Very recently, Pourdarbani et al. (2022) accurately predicted pH and TA of Fuji apples.
Moreover, color features and spectral data in Vis-NIR range were used by Pourdarbani et al. (2020c), Pourdarbani et al. (2020b) to assess different ripening stages of Fuji apples. The reported results showed that both spectral and color data have the potential to accurately determine the maturation levels of Fuji apples. In another work, color features and/or spectral data in Vis-NIR range were used to successfully predict the total chlorophyll content of Fuji Apple (Pourdarbani et al., 2020a). The results evidenced that the aforementioned chemical property could be estimated using an ordinary camera. The lower system cost is the primary motivation for implementing color imaging techniques instead of spectroscopy methods to evaluate fruits' ripening level and/or physicochemical properties.
In another relevant study, Cardenas-Perez et al. (2017) characterized the ripening level of Golden Delicious apples using a computer vision system. Three ripening levels of apples were characterized by CIE-lab color space, chroma, and hue angle. Strong correlations were observed between a ripening index (RPI) (derived from physicochemical parameters such as TA, SSC and firmness values) and the majority of color properties. Using multivariate discriminant analysis (MDA) over four color parameters, 100% correct classification rate (CCR) was achieved for predicting the ripening levels of Golden Delicious apples. The authors concluded that color information could be utilized in lieu of physicochemical analysis for ripening determination. Similar laboratory level studies have also been conducted for other fruits such as mango (Vélez-Rivera et al., 2014) and Medlar (Zandi et al., 2020).
One should note that applying color imagining algorithms is much simpler in the standard/ laboratory environment than in the field. In the latter case, light intensity continuously varies throughout the day, and weather conditions may also change from sunny to cloudy, affecting image quality (Sabzi et al., 2018a). Ultimately, the developed models should be applied in the real-world environment where remarkable lighting variations exist. One of the key challenges in model development for outdoor orchard environments is appropriate segmentation due to the aforementioned complexities. Indeed, previous studies outlined reliable segmentation algorithms for apple quality monitoring in orchard environments, which can be followed to overcome segmentation challenges (Sabzi et al., 2018a).
Furthermore, it should be noted that while specific internal properties (such as TA, SSC and firmness values) and external appearance of certain fruits (such as apple (Cardenas-Perez et al., 2017) and mango (Vélez-Rivera et al., 2014)) change drastically at different ripening levels, other fruits may not exhibit similar trends. In those cases, more involved techniques such as spectroscopy, biochemical assays, and electronic nose have been researched for quality evaluation (Beghi et al., 2017). In the case of apples, a substantial modification in physicochemical properties (such as TA, SSC and firmness values) and external appearance over maturation have been reported (Cardenas-Perez et al., 2017). However, previous work in this area has been limited to studies conducted in artificially controlled lab environments (Cardenas-Perez et al., 2017) and/or solely focused on evaluating ripening level (Sabzi et al., 2019a). There is a need to develop practically feasible techniques in a natural orchard setting where environmental conditions vary continuously. To this end, the present study was formulated to investigate the feasibility of using a computer vision system in orchard environments to (i) predict five physicochemical properties (i.e. firmness, SSC, starch, acidity, and TA) of Red Delicious apples with different maturity states; and (ii) to classify the apples into four levels of ripeness (i.e. unripe, semiripe, ripe, and overripe). Moreover, unlike the majority of previous imaging-based computer vision systems, a video processing algorithm was utilized in the present work. Implementing video processing provides more flexibility in data collection during fruits growth and development and facilitates applying the developed knowledge over large fields using state-of-the-art tools such as drones and/or robots. Furthermore, video-based imaging can provide beneficial information for efficient real-time resource management in the fields through variable-rate technologies. Figure 1 shows the various steps of the proposed machine vision algorithm to estimate the ripeness levels of Red Delicious apples in orchard environments, which will be discussed in detail next.

Video Acquisition and Segmentation
Videos of apples (cultivar Red Delicious) on trees were acquired under natural light in five orchards in Kermanshah-Iran (34°18ʹ48.87"N, 47°4ʹ6.92"E) using a digital camera (DFK 23GM021, 120 f/s CMOS, Imaging Source GmbH, Germany). The video files were converted into frames using a program code in MATLAB and Statistics Toolbox (The MathWorks, Inc., Natick, MA). A laptop (Intel Core i3, 2.13 GHz, 4GB RAM, Windows 10) equipped with MATLAB 2016b was used to process the video frames. Figure 2 illustrates each of the four ripening levels on the tree branch.
Segmentation of the acquired video frames was needed to extract useful information by eliminating the background, noise, and undesired components. Figure 3 shows four sample frames of Red Delicious apples in the garden. Herein, the background components that were required to be removed from the frames include trunks of trees, green and sick leaves, other flowers and plants, tree branches, and sky. Ground reference for each of the four ripeness levels was established by a panel of trained inspectors.
A successful segmentation algorithm for apples previously developed by Sabzi et al. (2018a) was adapted. To overcome the background complexity of the video frames and to remove background, noise, and undesired components, a combination of color thresholding method, texture method, and intensity conversion method was applied (Sabzi et al., 2018a(Sabzi et al., , 2019a(Sabzi et al., , 2019b. A total of 1500 apples were imaged for model development. The frames were initially converted into LUV color space, and a simple threshold ensured the removal of green leaves and other components with a pixel value greater than 95. Next, the image was converted to grayscale and the texture features of the local standard deviation were applied. Another part of the background was deleted upon further binarization and thresholding. The image was again converted to grayscale, and  the pixel intensity range was changed from 0-1 to 0-0.4. Since image pixel values were stored as 8-bit unsigned integers, these values were multiplied by 225. In the next step, image segmentation was done and pixels with values greater than 75 were considered as background and removed. In the final segmentation step, 92 different color thresholds in RGB color space were applied to further discriminate between the background and the desired object according to the algorithm developed by Sabzi et al. (2018a). The applied segmentation algorithm could eliminate the background, noise, and undesired components to extract the objects of interest. Thereafter, the performance of the segmentation algorithm (SA) was analyzed according to three criteria of sensitivity, specificity, and CCR, which are expressed in equations 1 to 3. Sensitivity measures the correct classification of the samples within a class; specificity is the ability to correctly identify other class samples (background objects) in the studied class; and CCR is defined as the percentage of correctly identified samples across all classes.
Sensitivity is calculated as such: Specificity is calculated as such: Correct Classification Rate is calculated as such: where TP (true positive) is the number of samples correctly classified in each class, TN (true negative) is the incidence of not identifying the samples when the class is not apple, FP (false positive) represents the incorrect identification of apple when the class is not apple, and FN (false negative) is the incorrect identification of apple when the class is apple (Wisaeng, 2013).

Extraction of Color and Texture Features
Previous studies have reported the successful implementation of various color spaces and/or texture features in combination with hybrid artificial neural networks for evaluating the ripening levels of apples (Sabzi et al., 2019a) and pH values of oranges (Sabzi et al., 2020a). Herein, we followed an analogous approach and extracted various color and texture features from each sample to identify physicochemical characteristics and maturity levels of apples. Sixteen different color spaces were included in color feature extraction viz. RGB, HSV, YIQ, YCbCr, CMY, Improved YCbCr, L* a* b*, JPEG-YCbCr, YDbDr, YPbPr, YUV, HSL, XYZ, LUV, LCH, and CAT02 LMS. The detailed definition of the aforementioned color spaces can be found elsewhere (Chaves-González et al., 2010;García-Mateos et al., 2015;Kahu et al., 2019;Wu et al., 2015). Two groups of color features were extracted from each color space, including features corresponding to mean and standard deviation (MSD), and features corresponding to vegetation indices (VI) (Sabzi et al., 2019a(Sabzi et al., , 2020a. Seven MSD features from each color space were obtained, including the mean and standard deviation of the three individual components and mean of all three components (Table 1). Thus, a total of 7 × 16 = 112 features were extracted. Additionally, 14 features were extracted from VI in each color space.    (Table 3). Features were extracted at 0°, 45°, 90°, and 135° for each sample resulting in 27 × 4 = 108 texture features in total. Overall, 336 (color)+108 (texture) = 444 features were extracted from each sample.

Experimental Measurement of Physicochemical Properties
Fifteen apples at each ripening level were randomly selected (15*4 = 60 samples in total) and experimentally analyzed to identify apples' reference physicochemical properties (firmness, SSC, starch, acidity, and TA) at each maturity level. The collected data were later used for developing nondestructive physicochemical and ripeness predictive models.
The apple firmness was measured using the Fruit and Vegetable Ripeness/Hardness Tester (HFH80 Series, OMEGA Engineering, Norwalk, CT) equipped with a cylindrical probe (11 mm diameter and 8 mm height). The device was calibrated as per the manufacturers' recommendation and the firmness values were measured at a uniform penetration speed of 5 mm/s. A portable refractometer (RF18, Extech Instruments, Waltham, MA) was used to measure SSC of the samples in terms of percentage of degrees brix. One-degree brix is defined as 1 g of sugar in 100 g of solution expressed as percentage by mass.
To measure the starch content, the precipitate was dissolved in a mixture of dimethyl sulfoxide/ hydrochloric acid (4:1) and centrifuged at 12,000 rpm for 15 minutes. Iodine chloride reagent was prepared by dissolving 0.6 g of potassium iodide in 100 mL of 0.05 N hydrochloric acid. 0.5 mL of  (Woebbecke et al., 1995) EXG ¼ 2 � G n À R n À B n Additional red (Meyer et al., 1998) EXR ¼ 1:4 � R n À G n Color index for extracted vegetation cover (Kataoka et al., 2003) CIVE ¼ 0:441 � R n À 0:811 � G n þ 0:385 � B n þ 18:78 Subtraction between additional green and additional red (Meyer and Neto, 2008) (Woebbecke et al., 1993) NDI ¼ G n À B n ð Þ= G n þ B n ð Þ Green index minus blue (Woebbecke et al., 1995) GB ¼ G n À B n ð Þ Red-blue contrast (Golzarian and Frick, 2011) RBI ¼ R n À B n ð Þ= R n þ B n ð Þ Green-red index (Golzarian and Frick, 2011) ERI ¼ R n À G n ð Þ � R n À B n ð Þ Additional green index (Golzarian and Frick, 2011) EGI ¼ G n À R n ð Þ � G n À B n ð Þ Additional blue index (Golzarian and Frick, 2011) EBI ¼ B n À G n ð Þ � B n À R n ð Þ Diagonal moment iodine (yellow reagent) and 0.5 mL of herbal extract were mixed thoroughly in a small plastic tube. After 15 minutes, the absorbance of the resulting solution was read by a spectrophotometer (Optizen 2120 UV plus, Company: Mecasys Co., Ltd., Korea) at 600 nm and the results were expressed in mg/g of fresh weight. Next, the standard starch solution was prepared with varying concentrations of starch (0 to 100 mg/L). Finally, data plotted between the absorbance of the standards and the starch concentration in g/mL showed a linear fit and starch concentration was determined accordingly. The acidity of the juice was measured using a pH meter (AZ-8689, Taiwan). To measure TA, 5 mL of fruit extract was diluted with 40 mL of distilled water. The pH of the mixture was adjusted to 8.2 with the addition of 5 N NaOH. TA was calculated using equation 4.
where V b = consumption volume (mL) N = normalized NaOH (0.5 mol/L) M = molecular weight (134 g/mol for malic acid) V s = fruit juice volume (mL) n = number of carboxylic acid groups. After measuring the five physicochemical properties, analysis of variance (ANOVA) was used to explore the statistical differences between properties at four different ripeness levels. Moreover, Tukey and Sheffes' methods were used as posthoc tests.

Selection of Optimal Features
The 444 color and texture features previously extracted, needed to be curtailed to remove redundant features, reduce computational time, and improve model performance. Statistical procedures such as Gamma test and Artificial Intelligence-based methods are typically used to select effective features (Wang and Paliwal, 2006). In this paper, the ANN-DE algorithm was chosen to determine the optimal features (Storn and Price, 1996). ANN-DE includes two main steps of initialization and evolution. With no initial information, a random population is used to optimize the problem, which is then recombined through mutation, and the selection process continues until optimization is finalized. The extracted features are considered as a vector. Different sized input vectors (based on a different number of extracted color and texture features) were sent to the multilayered perceptron artificial neural network (MPANN) to predict the properties of firmness, SSC, starch, acidity, and TA that were experimentally measured. The data (from 60 samples) was divided into three categories of training (70%), validation (15%), and testing (15%). Finally, the mean squared error (MSE) of each vector was recorded and the vector with the minimum MSE was chosen (Sabzi et al., 2020a). Table 4 shows the values of the parameters of the multilayer perceptron neural network used in this section.

Prediction Model for Physicochemical Properties
The optimum features chosen in the previous section were used as inputs for the ANN-CA algorithm to tune the parameters of MPANN and further optimize the prediction of the physicochemical properties. The cultural algorithm considers the cultural evolution and the impact of cultural and social space to determine the optimal parameters of the MPANN (Ali et al., 2016). This network has five adjustable parameters, namely the number of neurons, the number of layers, the transfer function, backpropagation network training function, and backpropagation weight/bias learning function. If the parameters have optimal values, the neural network will yield the highest performance. The cultural algorithm sends a vector form of the parameters to the MPANN and the modeling results are recorded in terms of the mean squared error. The data (from 60 samples) was divided into three groups of training (70%), validation (15%) and testing (15%). The vector with the minimum MSE was selected as optimal and its members were selected as optimal values of the adjustable parameters of the MPANN. In order to check the reliability of the physicochemical predictor, the model operation was evaluated using regression coefficient (R), and coefficient of determination (R 2 ) for 100 iterations. Each iteration involved physicochemical prediction under different combinations of training, validation, and testing sets (from the original data of 60 apples).

Classification Model for Ripeness Levels
In this phase, the output of the ANN-CA hybrid classifier that predicted the physicochemical properties was assigned as inputs to ANN-HS algorithm classifier. The harmonic search algorithm is a metaheuristic algorithm that works similarly to the cultural algorithm to optimize the identification of the ripeness level. The objective function predicted the values of the assigned variables (Lee and Geem, 2005) and classified the apples into four levels of unripe, semi-ripe, ripe, and overripe, based on their physicochemical properties (see (Sabzi et al., 2018b for more detail). After identifying the optimum classifier model (from 60 apples), the reliability of the recommended classifier was tested over 100 iterations. Each iteration involved ripeness levels classifications under different combinations of training, validation, and testing sets (from the original data of 60 apples).
In the last step, the performance of the developed machine vision system was examined in the orchard environments and the ripeness levels of 1356 apples were estimated. Figure 4 shows the result of applying SA on two different video frames. Even with a complex background, our SA was able to detect the target object (apples) while removing other objects from the background. The SA incorrectly categorized only 19 apples as background out of 1519 objects that were identified as apple samples (Table 5). It misclassified only 11 samples out of the 2380 background-related objects in the apple class. Therefore, SA achieved a CCR of 99.23%. Overall, the segmentation algorithm performed well in identifying apples as it reached a sensitivity, specificity, and CCR of over 98% (Table 6).

Statistical Analysis of Experimentally Measured Physicochemical Properties
The analysis of variance (ANOVA) showed significant differences among the four ripeness levels of apples for which the physicochemical properties were experimentally measured (Table 7). Moreover, posthoc tests showed significant differences at all ripeness levels for all physicochemical properties clearly establishing that our subset of 60 apples chosen for physicochemical properties determination was sufficient to represent each ripening level. Figure 5 displays the average values of the experimentally measured physicochemical properties at each ripeness level. A clear increase or decrease in mean values of starch, TA, and SSC was observed with the increase in ripeness. However, uneven trends were found for the acidity and firmness values.

Optimum Texture and Color Features
Application of ANN-DE algorithm led to the extraction of 6 optimal features among the available 444 features. They include one texture and five color features, namely the normalized first component of RGB color space, the first component of the CMY color space, the cluster shadow of 135 adjacency degree, the difference between the second additional component of the YCbCr color space and the first additional component of the YCbCr color space, the first additional component of the CAT02LMS color space, and the normalized second component of YCbCr color space.    SSC. Levels 1, 2, 3 and 4 refer to unripe, semi-ripe, ripe, and overripe levels, respectively. Table 8 lists the optimal values of the ANN parameters that were determined using the cultural algorithm. As previously discussed, the reliability of this ANN-CA classifier was tested over 100 different iterations for each of the physicochemical properties. Therefore, 100 regression coefficients and coefficients of determination were obtained for firmness, SSC, starch, acidity, and TA. Figure 6 displays the corresponding boxplots. The results indicate that high R (>0.96) and R 2 (>0.92) values were obtained over the entire 500 (5 × 100) iterations, confirming the capability of ANN-CA algorithm to estimate the physicochemical properties of apples. Excluding the very few existing outliers (Figure 6), R and R 2 values were above 0.98 and 0.96, respectively, confirming the reliable operation of the developed model in the long run. Table 9 lists the optimal values of the ANN parameters that were determined by the harmonic search algorithm. This hybrid ANN-HS classifier was run 100 times to classify apples into four ripeness levels. Figure 7 shows the boxplot diagram of the CCR corresponding to 100 repetitions. The results indicate the CCR values in excess of 91.8%, with the majority of values above 96%. Ultimately, the performance of the developed algorithms was evaluated in natural orchard environments. As shown in Figure 8, the developed machine vision system categorized apples into four ripeness levels (classes) of 1, 2, 3, and 4 corresponding to the unripe, semi-ripe, ripe, and overripe levels, respectively. The system achieved an overall CCR of 97.86% with the highest misclassification (2.92% error) for level 1 and the lowest misclassification (1.37%) for level 3 (Table 10). The high CCR and low misclassification errors validate the operation of the system in the orchard environments.

Classification Model for Ripeness Levels
The segmentation of each frame took 0.55 s whereas processing of other phases took 0.62 s. Considering the large number of steps (e.g., image acquisition, segmentation, background removal, and pattern classification) and their respective accuracies involved in such problems, direct comparison of the present results with other works is not possible. However, previous studies have also utilized nondestructive imaging methods to evaluate fruits' ripening and physicochemical properties. For instance, Cardenas-Perez et al. (2017) assessed the ripening levels of another apple variety (Golden Delicious) using a computer vision system at a laboratory scale. The authors identified RPI (using physicochemical parameters such as TA, SSC, and firmness) and categorized apples into three ripening level of unripe, ripe, and senescent, accordingly. The reported TA and SSC, and firmness had decremental, incremental and decremental patterns through the ripening process, respectively, which were in reasonable agreement with our observations. Strong correlations were obtained between RPI and several color parameters such as CIEL a*, b*, and h*(r = −0.949, −0.890, 0.965, respectively). The authors reported on 100% CCR for classifying the ripening level of apples using MDA (with four color parameters). In another study, Vélez-Rivera et al. (2014) implemented a computer vision system to identify the ripening levels of mango fruits. The relationship between mangos' physicochemical properties (such as SSC, TA and firmness) and color parameters were explored. Strong correlations were observed between physicochemical parameters and several color parameters during ripening.   Mangoes were classified into three levels of preclimacteric, climacteric, and senescence, according to their RPI index. The authors reported a classification rate of 90% using MDA (by implementing four color parameters). These studies confirm that the physicochemical characteristics of fruits influence their appearance through the ripening period, which is in line with the findings provided in our work. However, one should note that the applicability of the aforementioned works was only tested in a laboratory  Overall, the present work demonstrated the feasibility of evaluating apples' physicochemical properties (for apples with different maturity stages) and ripening levels in orchard environments using a video-based computer vision algorithm. Implementation of such a low-cost nondestructive approach will enable farmers to remotely practice efficient resource management in the field. Indeed, the operation of the developed models depends on the functionality of the segmentation algorithm. Therefore, the performance of the segmentation model should be evaluated over more orchards and for different apple varieties.
It should be noted that samples with similar ripening levels usually possess close physicochemical and color properties. On the other hand, samples at different ripening levels typically have different physicochemical and color properties. Hence, developing a predictive physicochemical model from color parameters among various ripening levels would be possible. Our results confirm such feasibility. However, calibration development for the exact prediction of physicochemical properties of apples within a single ripening level is more challenging and requires more extensive physicochemical data acquisition. This was beyond the scope of the present work and is a topic for future research.

Conclusion
A nondestructive approach was developed to estimate the physicochemical properties and ripeness levels of Red Delicious apples in orchard environments using artificial intelligence and a video processing algorithm. Videos of apples with four levels of ripeness were acquired. Due to the complexity of orchard environments, successful segmentation of the video frames required a combination of color threshold (LUV color space), texture, and intensity conversion methods. For a subset of orchards' apples, five physicochemical properties, viz. firmness, starch, acidity, SSC and TA, were experimentally measured at each ripening level.
Significant differences were found in the physicochemical properties at different ripening levels. Processing video frames led to identifying six optimum color and texture features (through the implementation of ANN-based algorithms) to reliably predict the measured physicochemical properties. The predicted physicochemical properties were then used to predict apples' ripening levels. The ultimate developed machine vision system was tested across five orchards to detect apples' ripening levels. A promising CCR of 97.86% was achieved, confirming the reliable operation of the developed models. However, further research is required to train the algorithm with different apple cultivars and fabricate a field-scale prototype. Such models can be integrated into harvesting robots and/or drones to facilitate efficient real-time resource management in the fields.

Data Availability Statement
The data that support the findings of this study are available from the corresponding author, upon reasonable request.

Disclosure Statement
No potential conflict of interest was reported by the authors.

Funding
This study was financially supported by Iran National Science Foundation (INSF) through the research project 96007466. The authors also acknowledge the financial support provided by the Canada Foundation for Innovation (CFI) and Natural Sciences and Engineering Council of Canada (NSERC).