Skip to main content
Heliyon logoLink to Heliyon
. 2021 Feb 2;7(2):e06162. doi: 10.1016/j.heliyon.2021.e06162

Study of consistency of expert evaluations of wine sensory characteristics by positional analysis

AA Khalafyan a, ZA Temerdashev a,, VA Akin'shina a, YuF Yakuba b
PMCID: PMC7859295  PMID: 33553783

Abstract

Individual characteristics inherent in the expert, as well as their physical and psycho-emotional state subject to the influence of random, uncontrollable factors, contribute to subjectivity in the sensory evaluation of wines. With great variability of opinions, the final results of sensory evaluation may become doubtful. The presence of a random component in the sensory evaluation justifies the use of statistical methods for analyzing the consistency of expert evaluations. Along with Spearman's correlation coefficients and Kendall matching, Cronbach alpha criterion was used to assess the consistency of expert opinions. The advantages of positional analysis have been discussed – Cronbach's alpha criterion is calculated not by the rank of expert points, but by the initial point scale considering its variability; it allows to evaluate the contribution of each expert to the consistency of expert evaluations, as well as the reliability of the total scale of points set for each wine sample. Based on the data analysis from sensory evaluation of the quality of dry red and white wines of Russian production, the results of the consistency study of expert evaluations as well as the reliability of the total score scale have been presented. What is more, analysis of the “loyalty” of experts in evaluating the quality of wines has been performed.

Keywords: Expert evaluation, Positional analysis, Wine tasting


Expert evaluation; Positional analysis; Wine tasting

1. Introduction

Despite the high level of development of tools for laboratory analysis of wine quality, the main way to comparatively evaluate their organoleptic properties is sensory evaluation conducted by specially trained professional experts. Statistical analysis methods have a wide range of applications in wine quality research. They are most often used in the sensory assessment of the quality of wines (Baker and Ross, 2014; Cortez et al., 2009; Etaio et al., 2008; Piclin et al., 2008; Rinaldi and Moio, 2018), differentiation of wines by grape growing regions (Khalafyan et al., 2019; Urvieta et al., 2018), analysis of the influence of the chemical composition of wines and grapes on their quality (Bindon et al., 2014; Kapusta et al., 2017; Pejina et al., 2016; Perestrelo et al., 2018; Vujovic et al., 2016), etc. When processing the results of sensory evaluation, various statistical methods are used – analysis of variance (ANOVA) (Etaio et al., 2008; Rinaldi and Moio, 2018; Taladrid et al., 2019), principal component analysis (PCA) (Piclin et al., 2008), discriminant analysis (Khalafyan et al., 2016), Cartesian coordinates mapping (Hopfer and Heymann, 2013), regression analysis (Baker and Ross, 2014; Cortez et al., 2009), statistical text analysis with Alceste (Rodrigues and Parr, 2019), etc.

Considering the publications describing sensory evaluation of wines (Jackson, 2015, 2017; Parker, 2003; Rodrigues and Parr, 2019; Spence, 2019; Spence and Wang, 2019; Taladrid et al., 2019; Wang and Spence, 2018), expert methods have certain disadvantages. The results of sensory evaluation of wines are influenced by the composition of experts, their qualification level and quantity as well as imbalance of wines. Individual characteristics inherent in each expert, along with their physical and psycho-emotional state subject to the influence of a large number of random and uncontrollable factors contribute to subjectivity in expert evaluations.

Processing expert evaluations includes checking the consistency (unanimity) of expert opinions and averaging expert opinions within an agreed group. Apparently, if expert opinions regarding the quality of wines are contradictory, the final result of sensory evaluation – wine ranking by the degree of preference by the experts, will be at least doubtful.

For an integrated assessment of the consistency of expert evaluations (Kuznetsov and Strijov, 2014), analysis of expert opinions on reaching consensus (Kozierkiewicz-Hetmanska, 2017) and conducting expert opinion procedures (Perez-Rodrıguez and Rojo-Alboreca, 2016), a matrix approach as well as theorems based on the relationship between expert opinions and their number have been proposed. Multicriteria decision-making tasks based on hierarchy analysis have been discussed to improve the consistency of experts (Khatwani and Kar, 2016) and establish the accuracy of expert evaluation of the object state by pairwise comparison with a quantitative assessment of preferences (Radaev, 2007). The consistency of expert evaluations is established by mapping on a Cartesian plane (Abramova and Telitsyna, 2013; Wilson et al., 2018; Risvik et al., 1994; Hopfer and Heymann, 2013).

Expert evaluations are empirical data in which a random component is present, therefore, to analyze the consistency of expert opinions, the statistical apparatus is widely used: correlation analysis, two-dimensional normal distribution, multiple factor analysis, Spearman's rank correlation coefficient, Kendall's concordance, etc (Cain and McLeay, 2016; Etaio et al., 2008; Khalaphyan et al., 2017; Le Dien and Pages, 2003; Radaev, 2007; Vaamonde et al., 2000; Wilson, 2017). The above statistical methods are not without drawbacks and do not fully allow to evaluate the contribution of each expert to their unanimity.

In this paper, some of the problems associated with establishing the consistency of expert evaluations of the wine quality have been considered. As an alternative statistical method for assessing the unanimity of experts, Cronbach's criterion has been proposed which is used in positional analysis to assess the reliability of the total scale of respondents, its advantages have been discussed as well. In contrast to Spearman's rank correlation coefficients and Kendall's correlation widely used for these purposes, Cronbach's alpha is calculated not according to the rank of expert points, but according to the initial point scale using the variability of expert evaluations. The criterion allows to determine the contribution of each expert to the overall consistency, to evaluate the reliability of the total scale of points set by each expert for wine samples.

2. Material and methods

2.1. Research objects

60 samples of natural dry red and white grape wines produced in 2010–2015 in the territory of main wineries of Krasnodar region (Russia) were analyzed: “Myskhako”, “Fanagoria Number Reserve”, “Kuban-Vino”, “Southern wine company (SWK)”, “Villa Victoria”, “Chateau Tamagne”, “Chateau le Grand Vostock”. All the wine samples were produced according to traditional technologies from European (Cabernet, Merlot, Aligote, Riesling, Saperavi, etc.) and hybrid grape varieties (Bianca, Viorica, Moldova, Pervenets Magaracha, etc.) and were kindly provided for research by their manufacturers. The wines were poured into dark green glass bottles with screw caps and stored until use at 10 °C. All wine samples were dry, alcohol content varied from 9 to 13 % (v/v) and pH values ranged from 3.61 to 3.79. Dissolved oxygen in wines was measured by the immersion of the probe before bottling in barrels, which was less than 1 mg/dm3.

Wines from European grape varieties obtained by traditional technologies without the use of sulfur dioxide were not considered, since this category significantly differs in taste from wines for which sulfiting was used.

2.2. Research methods

2.2.1. Sensory analysis

All experimental studies related to sensory analysis were carried out at the Federal Research Center of Horticulture, Viticulture, Wine-making (FSC HVW, Krasnodar, Russia). 11 specialists participated in the sensory evaluation procedure aged between 32 and 66 years (the average age was 50 years; 4 female and 7 male). The informed consent was obtained from the participants of the sensory test. All participants are considered experts in the field of wine, work in the wine industry and have professional experience in the field of sensory analysis. Questionnaire table for experts is presented in Table 1.

Table 1.

Questionnaire table for experts.

Parameter Value (0–100)
Typicality
Transparency
Color
Aroma
Taste

The wine sample (50 cm3) was poured into each glass and covered with a Petri dish with diameter of 5.7 cm 30 min before sensory evaluation. The tests were carried out in a well-lit tasting room with controlled temperature conditions. All samples were fed at 16–22°С at tables with white napkins. Experts were prohibited to communicate during the sensory assessment procedure. The wines were served in transparent tulip-shaped glasses with a volume of 220 dm3. After evaluating each sample, participants were asked to wait at least 30 s, clear their taste with water and crackers. The intervals between tasting of each sample were 2 min. During each interval, experts rinsed their mouths with water. Experts evaluated each sample in triplicate during the working week.

The sensory assessment results of wine quality were expressed on a scale from 50 to 100 according to the well-known rating system (Parker, 2003). For a consolidated assessment of the organoleptic characteristics of wines, the average values of sensory evaluations were used according to the results of tasting by a group of experts.

Table 2 shows the results of sensory evaluation for the first 10 wine samples. A table with the results of sensory evaluation of all 60 wine samples is given in the electronic supplement to the article. The top row shows the experts, their gender, age, while the first column shows the sample number.

Table 2.

Results of sensory evaluation of tested wine samples.

Sample Number Expert 1, f, 66 Expert 2, m,32 Expert 3, m, 59 Expert 4, f, 57 Expert 5, m, 42 Expert 6, f, 38 Expert 7, m, 58 Expert 8, m, 55 Expert 9, f, 59 Expert 10. m, 43 Expert 11, m, 41 Sum
1 77 80 80 80 81 82 77 85 85 83 81 891
2 83 83 79 79 79 63 82 79 83 78 78 866
3 89 79 81 83 82 76 83 79 86 77 85 900
4 90 85 82 78 82 78 85 76 85 84 78 903
5 90 87 79 85 85 82 84 83 85 85 85 930
6 90 84 80 84 83 76 77 86 83 81 84 908
7 85 87 82 85 84 78 83 86 84 86 87 927
8 90 86 80 83 84 84 83 87 82 82 79 920
9 95 92 85 84 86 80 88 88 84 84 80 946
10 88 86 82 79 81 64 85 80 83 79 86 893

2.2.2. Data analysis

All calculations were implemented using the STATISTICA software (v. 10) (Hill and Lewicki, 2007). The pairwise consistency of experts was determined using Spearman's rank correlation coefficient, the “individual” consistency was established by the multiple correlation coefficient, group consistency – by means of Kendall's concordance coefficient and Cronbach's alpha criterion (Reliability and Item Analysis). However, the listed statistical criteria for the consistency of expert evaluations – Spearman's rank correlation coefficients, Kendall's and Kronbach's alpha correlations do not have generally accepted ranges of variation for their interpretation in the nominal scale, therefore, we focused on the degree of their proximity to 0 and 1. The closer the value of the criteria is to 0, the lower the consistency is; following this trend, if the value is closer to 1 – the consistency is higher. Scatter plots for experts and wine samples were built using the Multidimensional Scaling module.

3. Results and discussion

3.1. A study of the consistency of expert evaluations through Spearman's rank correlation coefficients and Kendall's concordance

Traditionally, for a statistical assessment of the overall consistency of 2 experts, Spearman's rank correlation coefficient is used. When the tasting results coincide for 2 arbitrary experts, they turn out to be “completely” consistent and the correlation coefficient of the points they set is equal to 1. The complete inconsistency of the two experts means that their evaluations are random independent variables and, in accordance with the provisions of probability theory, the correlation coefficient will be 0. Therefore, the correlation coefficient can also serve as an assessment of the “pairwise” consistency of experts (Table 3). In the first 11 columns of Table 3, Spearman's rank correlation coefficients are shown (statistically significant coefficients are shown in bold). The highest r value of 0.530 was observed between experts 2 and 5; the consistency of evaluations between them was higher than between other pairs. Less consistency of evaluations was between experts 4 and 8 (r = 0.522); 3 and 4 (r = 0.489), etc.

Table 3.

Correlation coefficients between the experts.


Expert
Expert 2 Expert 3 Expert 4 Expert 5 Expert 6 Expert 7 Expert 8 Expert 9 Expert 10 Expert 11 R
Expert 1 0.289 0.235 0.354 0.302 -0.02 0.198 0.192 0.240 0.089 0.254 0.650
Expert 2 0.467 0.314 0.530 0.304 0.258 0.411 0,097 0.400 0.393 0.689
Expert 3 0.489 0.406 0.411 0.399 0.387 0.294 0.470 0.276 0.842
Expert 4 0.487 0.310 0.213 0.522 0.121 0.164 0.295 0.783
Expert 5 0.486 0.261 0.461 0.134 0.198 0.157 0.784
Expert 6 0.162 0.465 -0.080 0.324 0.066 0.680
Expert 7 0.147 -0.117 0.321 0.001 0.574
Expert 8 0.071 0.474 0.177 0.803
Expert 9 0.129 0.224 0.741
Expert 10 0.202 0.689
Expert 11 0.713

For a more objective evaluation of the “individual” consistency of experts, we used the multiple correlation coefficient R which is a generalization of pairwise correlation coefficient in cases where the number of random variables is more than 2. R values are given in the last column of Table 2. Obviously, when multiple correlation coefficient increased, the individual consistency of expert evaluations also increased. The highest consistency with the evaluations of other experts was observed for expert 3 (R = 0.842), while the lowest was for expert 7 (R = 0.574). The predominantly close to 1 R values indicate a supposedly high overall consistency of expert evaluations which is of undoubted practical interest for researchers and practitioners. In the case where there are more than 2 experts, Kendall's concordance coefficient is used for the statistical consistency evaluation, the calculated value of which was 0.335 showing the low consistency of expert evaluations and discrepancy with the results obtained by correlation analysis.

3.2. Positional analysis of the consistency of expert evaluations

As an alternative to Kendall's concordance coefficient, positional analysis should be considered which was developed to build reliable profiles (scales) in psychology. Cronbach's alpha (α) statistics is used as the criterion for the reliability of the questionnaire (Eq. (1)):

α=(n/(n-1))·[1- i=1n(si2)/ssum2] (1)

where si2 is sample variance of the i-th statement (position); ssum2 is sample variance of the total scale (sum of respondent ratings for all questionnaire positions); n – number of positions. The closer α is to 1, the higher the reliability of the total scale is observed; similarly, the closer α is to 0, the lower the reliability is. In relation to the results of sensory evaluation of wine quality given in Table 1, respondents are wine samples, positions are experts, the total scale is the sum of points set by experts for each sample (Sum column). In fact, the reliability of the total scale of expert evaluations also means the general consistency of expert opinions. If the expert evaluations coincide (the highest consistency), then the scores in the columns of the table are the same, consequently, by the variance property ssum2=n2·s2, where s2 = si2 (i = 1,…, n). Substituting the obtained values in (1), the following Eq. (2) is obtained:

α=(n/(n-1))·[1-i=1n(si2)/ssum2]=(n/(n-1))·[1-n·s2/(n2·s2)]=(n/(n-1))·((n-1)/n)=1. (2)

If there is no correlation between expert evaluations (the least consistency), then they are independent random variables. Therefore, using the dispersion property, i.e., the variance of the sum of independent random variables is equal to the sum of the variances, the equation ssum2=i=1n(si2) is obtained (Eq. (3)), then

α=(n/(n-1))·[1-i=1n(si2)/ssum2]=(n/(n-1))·[1-i=1n(si2)/i=1n(si2)]=(n/(n-1))·[1-1] = 0 (3)

In some cases, in the presence of negative covariance relationships between expert evaluations, ssum2 may be less than i=1n(si2), then α will be a negative value, which also means expert opinions inconsistency.

The final results of experimental data positional analysis are presented in Table 4. Cronbach's alpha criterion of 0.843 indicated in the information part of the table characterizes the consistency of expert evaluations as high and does not contradict the results of correlation analysis. From the above data it follows that Kendall's concordance coefficient does not fully take into account the consistency of expert evaluations in comparison to positional analysis. We believe that the reason is in the undoubted advantage of positional analysis. The calculation of Kendall's concordance coefficient is associated with the transition from a “strong” presumably numerical point scale to a “weak” rank scale accompanied by a significant loss of information regarding the variability of expert evaluations. So, if an expert set 85, 77 and 95 points for three wine samples, then when moving to the rank scale they would correspond to ranks 2, 3 and 1, respectively, leveling the degree of difference between samples in 8, 18 and 10 points, i.e. their variability sets only the order – more or less. When calculating Cronbach's alpha criterion, all calculations are carried out in the original, more differentiated point scale, considering the variability of expert evaluations. Therefore, the conclusions drawn on the basis of Cronbach's alpha criterion regarding the consistency of expert evaluations are more informative and mathematically justified.

Table 4.

The results of positional analysis of unanimity among experts.


Variable
Summary for scale: Mean = 905,900; Standard Deviation = 34,139; Valid N:60. Cronbach alpha: 0,843, standardized alpha: 0,872.
Mean if deleted Variable if deleted Standard Deviation if deleted Item-Total
Correlation
Alpha if deleted Consistency rating
Expert 1 820.983 952.216 30.858 0.513 0.831 8
Expert 2 821.017 978.983 31.289 0.587 0.826 5
Expert 3 824.733 982.696 31.348 0.806 0.819 2
Expert 4 824.300 976.310 31.246 0.686 0.822 3
Expert 5 822.750 1027.188 32.050 0.680 0.829 7
Expert 6 824.650 934.928 30.577 0.391 0.849 11
Expert 7 822.867 1074.582 32.781 0.255 0.847 10
Expert 8 824.767 700.812 26.473 0.759 0.816 1
Expert 9 824.150 1008.494 31.757 0.458 0.835 9
Expert 10 824.950 961.347 31.006 0.601 0.824 4
Expert 11 823.833 984.706 31.380 0.576 0.827 6

Experts 6 and 7 reduce the overall consistency.

An important advantage of positional analysis is also that the indicators given in the columns of Table 4 allow to observe changes in the main statistics of the total scale with the sequential removal of the sensory evaluation of each expert. Obviously, if α value in the column Alpha if deleted exceeds 0,843 in the information part of the table, then the expert reduces the overall consistency of expert evaluations; otherwise, the expert makes a positive contribution to the overall consistency by increasing it. As follows from the table, only experts 6 and 7 reduce the overall consistency, the remaining 9 experts increase it. The last column of Table 4 shows the rating of experts in descending order of their contribution to the overall consistency of expert evaluations which does not contradict the results of correlation analysis. Naturally, the removal of experts who reduce the consistency should lead to an increase in the overall consistency of expert evaluations. So, if we remove experts 6 and 7 from the consideration, α will increase and assume a value equal to 0.857.

3.3. Graphical analysis of the consistency of expert evaluations

Analysis of a matrix file of pairwise correlation coefficients by the Multidimensional Scaling method, created by positional analysis using the original data table, allows to graphically represent experts as points on a plane according to their contribution to the overall consistency. With the decrease in the geometric distance between experts, the consistency of expert evaluations increases. The smaller the geometric distance between experts is observed, the higher the consistency of their expert evaluations is. It can be seen from the scattering diagram (Figure 1) that experts 3 and 8 who have the greatest contribution to the overall consistency of expert evaluations are located at the closest distance to each other. At an insignificant distance from them are experts 4, 10, 2, 11, 5, etc., following them in the ranking. Experts who make a positive contribution to the consistency of experts are located in relative proximity, forming a group of unanimous experts indicated by blue rhombi in the diagram. At the same time, experts 6 and 7 are located at the periphery of the chart at some distance from the other experts and from each other – they are indicated by red triangles.

Figure 1.

Figure 1

Expert scattering diagram constructed by multidimensional scaling.

3.4. Analysis of the loyalty of experts using Cronbach's alpha criterion

A comparative analysis of the average points given by experts for all 60 wine samples shows that some of the experts are less demanding (loyal) and assess the wines with a higher average score than the more demanding experts with a lower average score. The level of expert requirements was expressed as a loyalty scale and defined as the aggregate of average scores given by them in the sensory evaluation of wines. For the credibility of the interpretation of the obtained loyalty scale, its reliability was assessed by Cronbach's alpha criterion. For this purpose, the original data file presented in columns 2–12 of Table 1 was analyzed by means of the STATISTICA software. Cronbach's alpha criterion was found to be 0.869, thus showing high reliability of the loyalty scale. Loyalty “rating”, i.e., number of experts after their ranking in descending order according to the loyalty scale, is presented in the column Rating for all wines in Table 5. The most loyal was expert 1 with 84.917 points. The least loyal was expert 10 with 80.95 points. Student's t-test for independent variables showed the statistical significance of the difference between the indicated mean values (p = 0.00002 < 0.05), which shows the practical feasibility of analyzing the loyalty of experts. Males and females turned out to be of the same loyalty on average, since their average point amounts were approximately the same: – 82.379 и 82.340, respectively.

Table 5.

Expert loyalty rating for sensory evaluation of wines.


Expert number
Dry wines
Gender expert Expert loyalty scale Rating for all wines Expert loyalty scale for white wines Rating for white wines Expert loyalty scale for red wines Rating for red wines
1 female 84.917 1 87.200 1 83.775 3
2 male 84.883 2 84.850 2 84.900 1
3 male 81.167 9 80.900 9 81.300 9
4 female 81.600 7 81.350 7 81.725 7
5 male 83.150 3 82.200 6 83.625 4
6 female 81.250 8 75.600 11 84.075 2
7 male 83.033 4 82.700 4 83.200 5
8 male 81.133 10 80.750 10 81.325 8
9 female 81.750 6 84.200 3 80.525 11
10 male 80.950 11 81.200 8 80.825 10
11 male 82.067 5 82.700 5 81.750 6

The use of positional analysis made it possible to assess the contribution of all 60 wine samples to the reliability of the loyalty scale. Cronbach's alpha values after a successive removal of wine samples from positional analysis allowed to identify samples reducing/increasing the reliability of the scale. Cronbach's alpha values after a successive removal of wine samples from positional analysis made it possible to isolate samples that reduce/increase the reliability of the loyalty scale. The reliability of the loyalty scale is reduced by samples 1, 12, 22, 25, 29, 32, 33, 34, 38, 42, 46, 47, 48, 49, 52, 53, 54, 55, 56, 60, the rest increase it. From the scattering diagram (Figure 2), it can be seen that wine samples that reduce the reliability of the scale (indicated by red triangles) are located on the periphery of the graph — in the right and lower parts of the scattering diagram. Samples that increase the reliability of the scale (indicated by blue rhombs) are localized mainly in the left and central parts of the scattering diagram.

Figure 2.

Figure 2

Scattering diagram of wine samples constructed by multidimensional scaling.

Analysis of expert loyalty separately for white wines (Rating for white wines column) showed that expert 1 retained the position of a loyal expert with 87.2 points, the least loyal was expert 6 with 75.6 points. The position of a disloyal expert was also retained by expert 10 with 81.2 points. An analysis of expert loyalty for red wines (Rating for red wines column) showed that experts 2 and 1 with 84.9 and 83.775 points, respectively, were the most loyal. Experts 9 and 10 with 80.525 and 80.825 points, respectively, were the least loyal. The average scores set by experts for white and red wines were 82.146 and 82.455, respectively. A slight difference in average values suggests that expert loyalty is resistant to wine types.

4. Conclusions

Cronbach's alpha criterion for positional analysis allowed a deeper look at the issues of the consistency and variability of expert evaluations. The advantages of Cronbach's alpha criterion over the traditionally used statistical criteria – Spearman's rank correlation coefficients and Kendall's concordance are that the criterion is calculated not according to the rank of expert points, but according to the initial point scale. The criterion allows to evaluate the contribution of each expert to the overall consistency highlighting those experts who increase or decrease it. Using the criterion, the reliability of the total score scale set by each expert can be evaluated and their loyalty can be analyzed.

In relation to the sensory quality evaluation data of 60 natural dry red and white wines of Russian origin, the pairwise and individual consistency of experts was investigated using pairwise and multiple correlation coefficients. The analysis of the general consistency of expert evaluations using Cronbach's alpha criterion and Kendall's concordance coefficient has been carried out. Cronbach's alpha criterion characterizes the overall consistency of expert evaluations as high and does not contradict the results of correlation analysis. Kendall's concordance coefficient does not fully take into account the consistency of expert evaluations compared to positional analysis, as its calculation is associated with a transition from a “strong” numerical point scale to a “weak” rank scale accompanied by a significant loss of information regarding the variability of expert evaluations. The positional analysis allowed to identify experts increasing/decreasing the overall consistency. The reliability of the loyalty scale has been shown – the scale of the average points set by each expert. The absence of the influence of expert gender and wine type on the loyalty of experts has been established.

Declarations

Author contribution statement

A. A. Khalafyan, V. A. Akin'shina: Analyzed and interpreted the data.

Z. A. Temerdashev: Conceived and designed the experiments; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Yu. F. Yakuba: Performed the experiments.

Funding statement

This work was supported by the Russian Foundation for Basic Research (18-03-00059).

Data availability statement

Data included in article/supplementary material/referenced in article.

Declaration of interests statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.

Acknowledgements

Experiments were carried out with the use of scientific equipment of the Ecological and Analytical Center of the Kuban State University.

References

  1. Abramova N., Telitsyna T. An approach to analysis of expert estimation validity in cognitive mapping. IFAC Proceed. Vol. 2013;46(9):893–898. [Google Scholar]
  2. Baker A.K., Ross C.F. Sensory evaluation of impact of wine matrix on red wine finish: a preliminary study. J. Sensory Stud. 2014;29(2):139–148. [Google Scholar]
  3. Bindon K., Holt H., Williamson P.O., Varela C., Herderich M., Francis I.L. Relationships between harvest time and wine composition in Vitis vinifera L. cv. Cabernet Sauvignon 2. Wine sensory properties and consumer preference. Food Chem. 2014;154:90–101. doi: 10.1016/j.foodchem.2013.12.099. [DOI] [PubMed] [Google Scholar]
  4. Cain M., McLeay S. Statistical auditing of non-transparent expert assessments. Sankhya B. 2016;78(2):362–385. [Google Scholar]
  5. Cortez P., Cerdeira A., Almeida F., Matos T., Reis J. Modeling wine preferences by data mining from physicochemical properties. Decis. Support Syst. 2009;47:547–553. [Google Scholar]
  6. Etaio I., Pérez Elortondo F.J., Albisu M., Gaston M., Ojeda M., Schlich P. Development of a quantitative sensory method for the description of young red wines from Rioja Alavesa. J. Sensory Stud. 2008;23(5):631–655. [Google Scholar]
  7. Hill T., Lewicki P. StatSoft; Tulsa, OK: 2007. Statistics Methods and Applications. [Google Scholar]
  8. Hopfer H., Heymann H. A summary of projective mapping observations - the effect of replicates and shape, and individual performance measurements. Food Qual. Prefer. 2013;28(1):164–181. [Google Scholar]
  9. Jackson R.S. Wines: wine tasting. In: Caballero B., Finglas P., Toldra F., editors. Encyclopedia of Food and Health. Academic Press; 2015. pp. 577–584. [Google Scholar]
  10. Jackson R.S. Oral sensations (taste and mouth-feel) In: Jackson R.S., editor. Wine Tasting: A Professional Handbook. 3 rd Edition. Academic Press; 2017. pp. 103–136. [Google Scholar]
  11. Kapusta I., Cebulak T., Oszmiański J. The anthocyanins profile of red grape cultivars growing in south-east Poland (Subcarpathia region) J. Food Meas. Char. 2017;11(4):1863–1873. 2017. [Google Scholar]
  12. Khalafyan A.A., Temerdashev Z.A., Yakuba Yu.F., Guguchkina T.I. Computer analysis of the sensory qualities of red wines as a method to optimize their blend formulation. Heliyon. 2019;5(5) doi: 10.1016/j.heliyon.2019.e01602. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Khalafyan A.A., Yakuba Yu.F., Temerdashev Z.A. Application of ranging analysis to the quality assessment of wines on a nominal scale. J. Anal. Chem. 2016;71:205–214. [Google Scholar]
  14. Khalaphyan A.A., Temerdashev Z.А., Guguchkina T.I., Yakuba Yu.F. Computer analysis of expert evaluation of the organoleptic quality indexes of wines. Analitika i kontrol’ [Analytics and Control] 2017;21(2):161–172. [Google Scholar]
  15. Khatwani G., Kar A.K. Improving the Cosine Consistency Index for the analytic hierarchy process for solving multi-criteria decision making problems. Appl. Comput. Inform. 2016;34:118–129. [Google Scholar]
  16. Kozierkiewicz-Hetmanska A. The analysis of expert opinions’ consensus quality. Inf. Fusion. 2017;34:80–86. [Google Scholar]
  17. Kuznetsov M.P., Strijov V.V. Methods of expert estimations concordance for integral quality estimation. Expert Syst. Appl. 2014;41(4 PART 2):1988–1996. [Google Scholar]
  18. Le Dien S., Pages J. Hierarchical Multiple Factor Analysis: application to the comparison of sensory profiles. Food Qual. Prefer. 2003;14(5-6):397–403. [Google Scholar]
  19. Parker R.M. 4th. ed. Simon & Schuster; London: 2003. Bordeaux: a Consumer's Guide to the World's Finest Wines; p. 1244. [Google Scholar]
  20. Pejina B., Stanimirovic B., Vujovic D., Djordjevic J.P., Velickovic M., Tesevic V. The natural product content of the selected Cabernet Franc wine samples originating from Serbia: a case study of phenolics. Nat. Prod. Res. 2016;30(15):1762–1765. doi: 10.1080/14786419.2015.1135148. [DOI] [PubMed] [Google Scholar]
  21. Perestrelo R., Silva C., Silva P., Câmara J.S. Rapid spectrophotometric methods as a tool to assess the total phenolics and antioxidant potential over grape ripening: a case study of Madeira grapes. J. Food Meas. Char. 2018;12(3):1754–1762. [Google Scholar]
  22. Perez-Rodrıguez F., Rojo-Alboreca A. The triangle assessment method: a new procedure for eliciting expert judgement. Expert Syst. Appl. 2016;72:139–150. [Google Scholar]
  23. Piclin N., Pintore M., Lanza C.M., Scacco A., Guccione S., Giurato L., Chrétien J.R. Sensory analysis of red wines: discrimination by adaptive fuzzy partition. J. Sensory Stud. 2008;23(4):558–569. [Google Scholar]
  24. Radaev N. Accuracy in expert evaluation of object states by pairwise comparison with quantitative preference estimation. Meas. Tech. 2007;50(9):908–915. [Google Scholar]
  25. Rinaldi A., Moio L. Effect of enological tannin addition on astringency subqualities and phenolic content of red wines. J. Sensory Stud. 2018 [Google Scholar]
  26. Risvik E., McEwan J.A., Colwill J.S., Rogers R., Lyon D.H. Projective mapping: a tool for sensory analysis and consumer research. Food Qual. Prefer. 1994;5(4):263–269. [Google Scholar]
  27. Rodrigues H., Parr W.V. Contribution of cross-cultural studies to understanding wine appreciation: a review. Food Res. Int. 2019;115:251–258. doi: 10.1016/j.foodres.2018.09.008. [DOI] [PubMed] [Google Scholar]
  28. Spence C. Perceptual learning in the chemical senses: a review. Food Res. Int. 2019;123:746–761. doi: 10.1016/j.foodres.2019.06.005. [DOI] [PubMed] [Google Scholar]
  29. Spence C., Wang Q.J. Wine expertise: perceptual learning in the chemical senses. Curr. Opin. Food Sci. 2019;27:49–56. [Google Scholar]
  30. Taladrid D., Lorente L., Bartolomé B., Moreno-Arribas M.V., Laguna L. An integrative salivary approach regarding palate cleansers in wine tasting. J. Texture Stud. 2019;50(1):75–82. doi: 10.1111/jtxs.12361. [DOI] [PubMed] [Google Scholar]
  31. Urvieta R., Buscema F., Bottini R., Coste B., Fontana A. Phenolic and sensory profiles discriminate geographical indications for Malbec wines from different regions of Mendoza, Argentina. Food Chem. 2018;265:120–127. doi: 10.1016/j.foodchem.2018.05.083. [DOI] [PubMed] [Google Scholar]
  32. Vaamonde A., Sanchez P., Vilarino F. Discrepancies and consistencies in the subjective ratings of wine-tasting committees. J. Food Qual. 2000;23(4):363–372. [Google Scholar]
  33. Vujovic D., Pejin B., Djordjevic J.P., Velickovic M., Tesevic V. Phenolic natural products of the wines obtained from three new Merlot clone candidates. Nat. Prod. Res. 2016;30(8):987–990. doi: 10.1080/14786419.2015.1079191. [DOI] [PubMed] [Google Scholar]
  34. Wang Q.J., Spence C. Wine complexity: an empirical investigation. Food Qual. Prefer. 2018;68:238–244. [Google Scholar]
  35. Wilson K.J. An investigation of dependence in expert judgement studies with multiple experts. Int. J. Forecast. 2017;33(1):325–336. [Google Scholar]
  36. Wilson С., Brand J., du Toit W., Buica A. Polarized projective mapping as a rapid sensory analysis method applied to South African Chenin Blanc wines. LWT - Food Sci. Technol. (Lebensmittel-Wissenschaft -Technol.) 2018;92:140–146. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Data included in article/supplementary material/referenced in article.


Articles from Heliyon are provided here courtesy of Elsevier

RESOURCES