Skip to main content
NCBI home page
Food Science and Biotechnology logoLink to Food Science and Biotechnology
. 2023 Aug 22;33(5):1177–1187. doi: 10.1007/s10068-023-01413-y

Comparison of check-all-that-apply and intensity rating in orange juice and yogurt studies

Jihye An 1, Jeehyun Lee 1,✉
PMCID: PMC10908659  PMID: 38440678

Abstract

This study compared check-all-that-apply (CATA) and rating methods using simple flavor foods and determined the discrimination ability of two consumer-based methods. Orange juice (simple flavor and liquid) and yogurt (simple flavor and semi-solid) samples were used. Six samples with different flavors and textures were evaluated for each food group. One hundred twenty consumers participated in each food session. CATA and rating were performed in two visits at weekly intervals. Consumers in each session distinguished the sample characteristics, and similar results were obtained using the CATA and rating methods. Although the number of characteristics with a significant difference in the rating method was relatively higher than that of CATA, some attributes with low frequency and intensity values may not have a significant effect on sample discrimination. Therefore, the types of questionnaire should be selected considering the test objectives and how similar the samples were.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10068-023-01413-y.

Keywords: CATA, Discrimination ability, Consumer test, Sensory characterization, Intensity, Threshold to check

Introduction

In the food industry, sensory evaluation plays an important role in product development, product reformation, sensory characteristics of products that affect consumer acceptability, and marketing (Lawless and Heymann, 2010). Sensory evaluation methods based on untrained consumers have recently been introduced. Check-all-that-apply (CATA) is a typical consumer-based method (Ares et al., 2014). CATA asks consumers to select all attribute terms that apply to each sample. This method can be easily and quickly evaluated by untrained consumers and provides valid and reproducible results for the sensory characterization of a wide range of products (Ares and Jaeger, 2023). However, binary responses cannot measure the strength of the sensory attributes, which can prevent detailed descriptions and identification of products with similar sensory profiles (Antúnez et al., 2017; Ares et al., 2014). To compensate for these problems, other methods for evaluating the intensity of the characteristic terms were developed: CATA with intensity (Reinbach et al., 2014) and rate-all-that-apply (RATA) (Ares et al., 2014). RATA is a modified form of CATA that selects the intensity of the sensory characteristics (Ares et al., 2014). Although rating methods have also been used in consumer tests of intensity (Yang and Lee, 2021), findings regarding rating methods have rarely been reported.

Many authors continue to question the reliability of the results of experiments conducted by consumers, compared to those of trained panels (Ares et al., 2011; Ares and Varela, 2017). Previous studies have shown that the results of the CATA, RATA, and other methods are comparable to those generated by trained panels (Ares et al., 2018; Bruzzone et al., 2012; Bruzzone et al., 2015; Danner et al., 2018; Jaeger et al., 2013; Oppermann et al., 2017; Pineau et al., 2022). Additionally, these studies suggest that consumer-based methods can be useful alternatives to descriptive analysis for understanding the perception of product sensory attributes. Based on these results, participants should be assessed with a focus on consumers, rather than trained panels. And it is necessary to check whether the consumer-based questionnaire is suitable for use and can give reliability to the results. Reinbach et al. (2014) compared three consumer profiling methods (CATA, CATA with intensity, and napping) and they found that the three methods could discriminate between samples and high similarities to each other. Vidal et al. (2018) also compared RATA and CATA. CATA and RATA performed similarly, but RATA tended to have more distinction of characteristic terms than CATA. When analyzing RATA data, all attributes that consumers did not check were considered as a score of 0 (Meyners et al., 2016). However, one of limitations of RATA is that sometimes not checked terms may be perceived as having some intensity (Jaeger et al., 2020a, b; Yang and Lee, 2021). Consumers’ wine evaluation study (Yang and Lee, 2021) provided 108 attributes and there were twelve and nine attributes with significant differences using CATA and rating, respectively, for user groups and this may be due to the large number of attributes to choose from or rate. For non-user groups, unfamiliarity to wine characteristics may have resulted in only four significant attributes when utilizing CATA, but 10 significant attributes when utilizing rating, which is similar to user group. Recently, CATA and intensity rating was compared using images for sensory and non-sensory terms and the citation frequency of CATA tended to have sigmoidal relationship with intensity, which means CATA term selection increased slowly at low- and high-end of intensity scale (Jaeger et al., 2023). Therefore, rating methods that select intensity for all attributes may have better statistical power (Yang and Lee, 2021). Jaeger et al. (2023) also suggested further rating to be used for CATA with the similar intensity samples at low- or high-intensity. Although many studies have compared various methodologies in consumer tests, few have used a rating method to compare CATA using the same attribute terms.

Some previous studies defined complexity as a number of sensory attributes, aromas, ingredients or flavors (Bitnes et al., 2009; Lévy et al., 2006; Palczak et al., 2019). The complexity not only affects food preference and acceptability, but can also contribute to positive or negative responses (Lévy et al., 2006; Pierguidi et al., 2019). Yang and Lee (2021) conducted a consumer test comparing CATA and rating methods using wine samples with 108 attribute terms (complex flavors). Additionally, complexity is defined as the opposite of simplicity (Palczak et al., 2019). Therefore, it is necessary to compare CATA and rating using simple flavor foods to confirm the effect of samples’ simplicity between the two methods.

The objectives of this study were to compare the CATA and rating methods for the same attributes using simple flavor foods and to determine the discrimination ability of the two methods.

Materials and methods

Samples and preparation

This study included the evaluation of two food categories, a liquid beverage and a semi-solid liquid, considering their different flavors and textures. Orange juice and yogurt were selected to represent these textural characteristics. Six commercially available products with different flavors and textures were selected as samples for each food type (Table 1). Simple flavor foods were determined according to the number of sensory characteristics (Palczak et al., 2019). Orange juice and yogurt with relatively few sensory characteristics were classified as simple-flavor foods. References listed between 20 and 37 attributes (Kim et al., 2013; Lotong et al., 2003; Pérez-Cacho et al., 2008, for orange juice; Brown and Chambers, 2015; Coggins et al., 2008; Desai et al., 2013, for yogurt) and terms used in this study are listed in Table 2 (25 attributes for the orange juice and 29 attributes for the yogurt evaluation).

Table 1.

Sample information

Sample Product name Company Ingredients Amount Other information
Orange juice Sugar content/100mL (%)
 T’aom T’aom orange Binggrae Co., Ltd. (Namyangju, Korea) Purified water, concentrated orange juice, orange pulp, natural orange flavor 1600 mL 10.0
 Sunup Sunup orange 100% Maeil Dairies Co., Ltd. (Seoul, Korea) Purified water, concentrated orange juice, synthetic flavor, Vitamin C 1000 mL 9.5
 I’m real I’m real orange Pulmuone Co., Ltd. (Seoul, Korea) Squeezed orange juice, orange pulp 700 mL 10.0
 Delmonte Del Monte orange 100 Lotte Chilsung Beverage Co., Ltd. (Seoul, Korea) Purified water, concentrated orange juice, synthetic flavor, Vitamin C 1500 mL 12.0
 Noah Noah’s valencia orange juice Noah’s Creative Juices (Caulfield, Australia) Valencia orange juice, Vitamin C 260 mL 8.5
 Vitamont Vitamont pur jus orange douce Vitamont Co. (Monflanquin, France) Pure orange juice 250 mL 8.6
Yogurt Fat content/100 g (%)
 Bulgaris Bulgaris unsweetened plain yogurt Namyang Dairy Products Co., Ltd. (Seoul, Korea) Milk(Grade 1 A), purified water, milk cream, nonfat dry milk (NFDM) mixture, modified starch, milk base KSA104, milk protein powder, lactase, lactic acid bacteria 435 g 8.2
 Maeil Maeil Bio plain yogurt Maeil Dairies Co., Ltd. (Seoul, Korea) Milk, NFDM mixture, milk protein powder, fish gelatin, emulsifier, lactic acid bacteria (S.thermophilus, L.bulgaricus, LGG) 450 g 7.0
 Binggrae Yoplait plain Binggrae Co., Ltd. (Namyangju, Korea) Milk, plain blend (oligosaccharide, sugar, modified starch, amidpectin, synthetic flavor, natural aromatics), dry milk mixture, purified water, dextrin, lactic acid bacteria 85 g 5.9
 Foodis Foodis Greek yogurt plain Ildong Foodis Inc. (Seoul, Korea) Milk(Grade 1 A), raw sugar, lactic acid bacteria 80 g 13.6
 Danone Greek yogurt plain Danone Pulmuone Co., Ltd. (Seoul, Korea) Milk, purified water, cheese plain syrup, dry milk mixture, condensed milk protein powder, crystalized fructose, fructooligosaccharide, fish gelatin, whey protein powder, modified starch, maltodextrin, lactic acid bacteria 90 g 4.4
 Fage2% FAGE Total 2% Greek yogurt FAGE USA Dairy Industry, Inc. (NY, USA) Grade A pasteurized skimmed milk and cream, live active yogurt cultures (L.Bulgaricus, S.Thermophilus, L.Acidophilus, Bifidusm L. Casei) 150 g 2.7
Open in a new tab

Table 2.

Sensory attributes of simple-flavor samples (orange juice and yogurt)

Sample CATA and rating attribute terms
Orange juice (25) Flavor (21)
 Citrus/non-orange Overripe/near fermented Fruity/non citrus
 Metallic Peely Floral
 Raw/fresh Sweet Sweet aromatics
 Sour Sour aromatics Bitter
 Bitter aromatics Orange Candy-like
 Artificial sweetener Cooked Cardboard*a
 Musty Green* Plastic
Mouthfeel (4)
 Tongue burn/numbing Astringent Toothetch*
 Pungent
Yogurt (29) Flavor (23)
 Processed Grain-like Oil-like
 Sharp/bite Sweet Animalic
 Astringent Lemon Buttery
 Oxidized Sour Lactic
 Bitter Butyric* Dairy fat
 Whey Overall dairy Cooked
 Cardboard Salty Acetaldehyde
 Moldy Plastic
Texture (6)
 Firmness Smoothness Thickness
 Degree of dissolving Toothetch Chalky mouthcoating
Open in a new tab

aTerms with asterisks (*) indicate “no significant difference” when evaluated using CATA when Cochran’s Q test was performed (p < 0.05)

Orange juice

Six orange juice samples with different characteristics in terms of ingredients, packaging, and distribution (shelf stable or refrigerated) were selected for this study. Table 1 shows the information of the orange juice samples. T’aom (T’aom orange, Binggrae Co., Ltd., Namyangju, Korea), Sunup (Sunup orange 100%, Maeil Dairies Co., Ltd., Seoul, Korea), I’m real (I’m real orange, Pulmuone Co., Ltd., Seoul, Korea), Delmonte (Del Monte orange 100, Lotte Chilsung Beverage Co., Ltd., Seoul, Korea), Noah (Noah’s valencia orange juice, Noah’s Creative Juices, Caulfield, Australia), Vitamont (Vitamont pur jus orange douce, Vitamont Co., Monflanquin, France) were used for orange juice samples. All samples were stored in a refrigerator (ETE5107TA-RKR; Electrolux, Seoul, Korea) and removed immediately before preparation. Orange juice was mixed well before being poured into a cup. The serving temperature was 5–7 °C. Each sample was provided in 100 mL in a 200 mL transparent plastic cup and labeled with three-digit random numbers.

Yogurt

Six products with different sweetness and texture (viscosity) were selected from commercially available and easily accessable plain yogurts in Korea (Table 1). Bulgaris (Bulgaris unsweetened plain yogurt, Namyang Dairy Products Co., Ltd., Seoul, Korea), Maeil (Maeil Bio plain yogurt, Maeil Dairies Co., Ltd., Seoul, Korea), Binggrae (Yoplait plain, Binggrae Co., Ltd., Namyangju, Korea), Foodis (Foodis Greek yogurt plain, Ildong Foodis Inc., Seoul, Korea), Danone (Greek yogurt plain, Danone Pulmuone Co., Ltd., Seoul, Korea), Fage2% (Fage total 2%, FAGE USA Dairy Industry Inc., NY, USA) were used for yogurt samples. All samples were stored in a refrigerator (ETE5107TA-RKR, Electrolux, Seoul, Korea) at 4 °C, and the serving temperature was 5–8 °C. Yogurt was served in a transparent sauce cup (60 mL) by scooping about 60 g without mixing to preserve the original yogurt texture and provided with a plastic spoon. The consumers were provided with a new transparent plastic spoon for each sample. All samples were labeled with three-digit random code.

Participants

Each experiment was conducted with 120 consumers per the food category (240 consumers in total). The participants were aged between 19 and 65 years. As this study included food intake, a screening stage was performed when recruiting participants. Individuals who had food allergies (especially orange juice, yogurt, and milk), those who were sensitive to caffeine (anxiety, arrhythmias, etc.), pregnant women, and those who had difficulty in ingesting and swallowing were excluded from participation at the time of recruitment.

Questionnaires

Selection of sensory attribute terms

The sensory attribute terms were selected using literature. Flavor-, aroma-, mouthfeel- and texture-related terms were considered (Table 2).

Orange juice

The questionnaire used in the orange juice session was determined by referring to several previous studies on orange juice (Kim et al., 2013; Lotong et al., 2003). Lotong et al. (2003) first developed sensory characteristics (lexicons) of commercial orange juice. The attributes used in a preliminary descriptive analysis of orange juice in Korea (Kim et al., 2013) were also considered. In this study, 21 flavor and four mouthfeel attributes were selected based on two lexicons.

Yogurt

The sensory characteristics used in the yogurt consumer test were described in a plain yogurt study by Brown and Chambers (2015). Additionally, terms developed for milk yogurt (Coggins et al., 2008) and Greek yogurt (Desai et al., 2013) were checked and used in the questionnaire. In this study, the sensory terms consisted of 23 flavor attributes and six texture attributes.

Types of questionnaire

Two types of questionnaire were used in this study (CATA and rating). CATA was used to check all sensory attribute terms recognized by consumers while consuming the provided sample. Rating questionnaire allows participants to select the intensity of the characteristic felt in each sample. A 0 to 5 category scale was used to perform the rating where 0 means “Not at all”, 1 means “Very weak”, and 5 means “Very strong” (Yang and Lee, 2021). The RATA method used a 1–3 or a 1–5 point scale (Ares et al., 2014). In the interpretation of RATA data, all missing checks were considered as a score of 0 (Meyners et al., 2016), which turned 5-point scale into 6-point scale. When interpret the score of the lowest point “1” in RATA data, it can be difficult to interpret intuitively whether the intensity of an attribute was ‘absent’ or ‘weak’ (Yang and Lee, 2021). Therefore, in the rating method used in this study, a 0–5 point (6-point) scale was used to allow consumers to select the intensity of all characteristics (Yang and Lee, 2021). The CATA and rating questionnaires used the same terms for each sample. At the end of each sensory evaluation, consumers were additionally asked whether the response to the questionnaire (CATA or rating) was easy or boring, and a 5-point scale was used (“1”: not at all, “2”: Not really, “3”: Neutral, “4”: Somewhat “5”: Very much) (Jaeger et al. 2015). Each test method was conducted separately at 1 week interval.

Consumer test

Consumer tests were conducted in an individual booth at a sensory laboratory, which the participants visited twice a week apart. In the first session, half of the participants (60) evaluated samples using the CATA questionnaire, and the other half (60) used rating to prevent errors that may appear because of the questionnaire order. In the second session, the consumers conducted the evaluation using the opposite method from the first session. Six samples were evaluated for each session, and each sample was given at 7 min intervals. A sequence of six samples was balanced according to a 6 × 6 Williams Latin square design (Williams, 1949). Bottled water (500 mL Samdasoo, Kwang Dong Pharmaceutical Co., Seoul, Korea) and crackers (− 30% di sale rispetto alla media dei crackers più venduti, Nuova, Industria Biscotti Crich S.p.a Zenson di Piave, Italy) were provided to rinse the mouth between samples. Consumers were encouraged to take a sip of water and a bite of crackers between the sample evaluations. After evaluating six samples, consumers were asked to provide demographic information, including sex, age, job, frequency of food intake (orange juice, yogurt, respectively, of each session), boredom, and ease of questionnaires.

Ethics statement

This study was reviewed and approved by the Institutional Review Board at Pusan National University (PNU IRB/2018_21_HR). All participants voluntarily signed an informed consent form before the experiment.

Data analysis

Cochran’s Q test was performed to confirm the frequency of selection of CATA terms and significant differences between the six samples in each food category. When significant difference existed, the critical difference (Sheskin) procedure was followed as post hoc pairwise comparisons analysis. Rating data were assessed using analysis of variance (ANOVA) to determine significant differences between samples for each attribute (p < 0.05) and Fisher’s Least Significant Difference post hoc test (LSD) was used for significant attributes. A correspondence analysis (CA) was conducted to visualize the correlations between the samples and attributes using non-parametric data. The intensity data of the sensory attributes were analyzed using principal component analysis (PCA) for comparison with CA. The RV coefficient was calculated to confirm similarity (closeness) between the two methods. The closer the RV coefficient is to 1, the higher the similarity (association) between the two matrices. Linear regression was performed to determine how the mean value of intensity varies with selection frequency (%) and to verify whether a linear model makes sense. ANOVA, CA, and PCA were performed using SAS version 9.4 (SAS Institute Inc., Cary, NC, USA). Cochran’s Q test, RV coefficient, and linear regression were conducted using XLSTAT software, version 2022.2. (Addinsoft Inc., Paris, France).

Results and discussion

Comparison between CATA and rating

Orange juice

The two biplots in Fig. 1 visualize the correlations between the variables in the two consumer-based orange juice sessions. The CA dimensions explained 83.84% (dimension 1 = 52.38%, dimension 2 = 31.46%), and the PCA explained 87.85% of principal components 1 and 2, accounting for 59.18 and 28.67%, respectively. The two methods showed similar results for the relationship between the samples and sensory characteristics. Terms, such as artificial sweetener, candy-like, and cooked, were related to Sunup and Delmonte samples, whereas the peely and raw/fresh attributes were closely related to T’aom and I’m real. Among the six samples, T’aom and I’m real were samples with the pulp among the orange juice samples (Table 1). Flavor characteristics, such as raw/fresh, and peely, could have been affected by the texture of the pulp. Attributes such as musty, metallic, plastic, and numbing, were located close to those of Vitamont and Noah. The RV coefficient for sample configuration was analyzed to confirm the similarity between the two consumer-based methods, and was 0.982 (p = 0.003) for orange juice. However, RV coefficient for attribute configuration was 0.675 (p < 0.0001). Therefore, consumers who consumed simple-flavor foods (orange juice) showed similar patterns of sample classification and were able to discern differences between samples in both the CATA and rating tasks but differed in attribute evaluations.

Fig. 1.

Fig. 1

Open in a new tab

Correspondence analysis and principal component analysis from the orange juice study. a Correspondence analysis biplot using check-all-that-apply results. b Principal component analysis biplot using intensity rating. Filled rhombuses (♦) indicate samples; empty circles (○) indicate attributes. A aroma

Yogurt

The CA and PCA biplots of the six yogurt samples are shown in Fig. 2. CA dimensions 1 and 2 accounted for 57.90 and 33.39% (a total of 91.29%), respectively. PC 1 explained 57.77% of the data variability, and PC 2 explained 35.85% of the data variability (a total of 93.62%). In both biplots, sensory characteristics and samples were distributed in similar locations. The RV coefficient for sample configuration was 0.991 (p = 0.005), but only 0.773 (p < 0.001) for attribute configuration. The Bulgaris and Maeil samples were characterized as more dissolving, sharp/bite, and smooth. Attributes such as acetaldehyde (green apple flavor), sweet, cooked, and processed were positioned close to Binggrae and Dannon in both biplots. Foodis was located near thickness and firmness. Bulgaris and Maeil products have texture characteristics (firmness and smoothness) opposite to those of Foodis, so they were located opposite to each other in the two-dimensional CA and PCA maps. Fage 2% was associated with attributes such as astringent, moldy, cardboard, bitter, and toothetch.

Fig. 2.

Fig. 2

Open in a new tab

Correspondence analysis and principal component analysis from the yogurt study. a Correspondence analysis biplot using check-all-that-apply results. b Principal component analysis biplot using intensity rating. Filled rhombuses (♦) indicate samples; empty circles (○) indicate attributes

The CA and PCA biplots were compared to determine whether the responses were affected by the two evaluation methods when consumers evaluated samples. As shown in Figs. 1a, b and 2a, b, each sample and its characteristics were similarly located in the two-dimensional maps (CA and PCA). In addition, because the RV coefficient was higher than 0.9 in all samples, the correlation between the two methods was very high. Similar to this study, other various studies have confirmed the correlation between CATA and various intensity methods (Ares et al., 2015; Ares et al., 2018; Bruzzone et al., 2015; Reinbach et al., 2014; Vidal et al., 2018). Based on these results, when a consumer test was conducted with simple-flavor foods, the similarity was high in the two scales, and consumers were able to identify samples and attributes using both evaluation methods.

In this study, we confirmed which type of questionnaire could provide consumers with a better discernment of simple flavor samples. CATA data were analysed using Cochran’s Q test (non-parametric statistical analysis), and ANOVA was conducted for rating data to confirm attributes with significant differences between samples (p < 0.05). Table 2 shows the sensory attribute terms used for the CATA and rating. Attributes with an asterisk (*) indicate no significant differences in CATA. In the CATA questionnaire, 88% of orange juice characteristics showed significant difference and 96.6% did of yogurt characteristics (Supplementary Tables 1 and 3). In contrast, all characteristics showed significant differences in the rating method, for both orange juice and yogurt (Supplementary Tables 2 and 4). When using the rating method, the number of characteristics with significant differences was relatively high, suggesting that the rating method is more effective than CATA for sample discrimination in general. In a previous study evaluating wines as having complex flavors (Yang and Lee, 2021), there were more characteristics with significant differences in rating than in CATA when analyzed with total consumer data. Among 108 terms for wine, terms with significant differences were 14 (13.0%) in CATA and 18 (16.7%) in rating (Yang and Lee, 2021). However, in this case of wine CATA evaluation, there were lower ratio of characteristics with significant differences between samples probably because the number of sensory terms in questionnaires was higher than that of orange juice or yogurt. Furthermore, evaluating complex foods with consumers were influenced by their consumption frequency as previously mentioned in the introduction.

In the orange juice session, cardboard, green, and toothetch characteristics showed significant differences in rating, but no differences in CATA (Supplementary Tables 1 and 2). Among the attributes of orange juice, the intensity of cardboard characteristics had a maximum value of 0.92 (Noah) and a minimum value of 0.26 (Sunup) on a 0–5-point intensity scale, with a difference in intensity of 0.66 and a least significant difference (LSD) value of 0.1995 (Supplementary Table 2). In CATA frequency, cardboard was selected low, ranging from 4.2 to 11.7%. This may imply that ‘cardboard’ character did not exceed many consumers’ individual threshold to check (Jaeger et al., 2020a, b). For the green characteristics, the maximum intensity was 1.68 (Noah), the minimum value was 0.96 (Sunup), and the difference in intensity was 0.72 (LSD: 0.2767). Although not significant, CATA frequency showed similar tendency as Noah having the most frequently checked as 27.5% and Sunup being checked the least as 15.8%. Toothetch may have been present in all samples having subtle differences. For rating, the maximum intensity value of toothetch was 1.26 (Vitamont), the minimum value was 0.78 (Sunup), and the difference in intensity was 0.48 (LSD: 0.2153). However, in CATA, frequency ranged from 12.5 to 15.8%. Although the difference in intensity seemed small, the rating method showed significant differences between the samples for most characteristics. For artificial sweetener characteristic, CATA showed bigger difference between samples as the frequency ranged from 20.8 to 70.0% whereas rating ranged from 1.56 to 2.76. This observation may also be explained by ‘person-, attribute-, and category-specific thresholds’ (Jaeger et al., 2020a, b) for artificial sweetener in orange juice. Interestingly, none of orange juice contained artificial sweeteners, but Sunup and Delmonte included synthetic flavor and vitamin C, and Noah had vitamin C listed in the ingredients list (Table 1). Consumers may have used the artificial sweetener to dump foreign perception to orange juice flavor.

In the yogurt study, butyric acid showed significant differences in rating, but not in CATA (Supplementary Tables 3 and 4). For the intensity of the butyric attribute, the maximum value was 1.25 (Fage 2%) and the minimum value was 0.75 (Binggrae). Although the difference in intensity was as small as 0.5, a significant difference was observed between the samples in rating (LSD: 0.2552). Characteristics, such as cardboard, green, and toothetch in orange juice, and butyric acid in yogurt, with subtle differences in intensity indicate that it is difficult to distinguish between samples using CATA. These results are in line with the reasons for developing intensity methods to overcome the potential limitations of binary data (Ares et al., 2014). Ares et al. (2015) noted that the use of CATA questions may not be recommended when dealing with samples with subtle differences. Jaeger et al. (2023) also suggested that although rating could result statistical differences among samples, it may not be important if characteristics are not prominent in terms of intensity, indicated by low absolute intensity value in rating as well as low frequency in CATA. It just may not be that important if the intensity was below individual threshold to check. Because of this limitation, Choi and Lee (2019) suggested asking consumers to rate intensity of a few important attributes while concurrently utilizing CATA for overall perception.

To determine whether the type of questionnaire had a psychological effect on consumers when performing sensory evaluations, consumers additionally responded whether responding each method was easy or boring (Table 3). Most consumers answered similarly that both methods were neither difficult, nor boring. However, in the question asking whether CATA is easy to use in yogurt session, “Not really” was 3.3%, which was much lower than answers to the same question for rating of 17.5%. It may be relatively difficult for some consumers to check the intensity of all characteristics rather than to determine the presence or absence of each attribute. These results are in line with findings of Jaeger et al. (2015).

Table 3.

Frequency of ease and boredom responses to CATA and rating

Participants (n = 240)
Orange juice (n = 120) Yogurt (n = 120)
Frequency (%) Frequency (%)
Ease of CATA response
   Not at all 0 0.0 0 0.0
   Not really 3 2.5 4 3.3
   Neutral 24 20.0 36 30.0
   Somewhat 65 54.2 66 55.0
   Very much 28 23.3 14 11.7
Boredom of CATA response
   Not at all 24 20.0 24 20.0
   Not really 56 46.7 54 45.0
   Neutral 27 22.5 27 22.5
   Somewhat 11 9.2 13 10.8
   Very much 2 1.7 2 1.7
Ease of rating response
   Not at all 0 0.0 0 0.0
   Not really 10 8.3 21 17.5
   Neutral 20 16.7 36 30.0
   Somewhat 63 52.5 44 36.7
   Very much 27 22.5 18 15.0
Boredom of rating response
   Not at all 24 20.0 36 30.0
   Not really 66 55.0 52 43.3
   Neutral 23 19.2 29 24.2
   Somewhat 6 5.0 3 2.5
   Very much 1 0.8 0 0.0
Open in a new tab

Relationship between CATA frequency and intensity

Figure 3 shows the results of the visualization of the correlation between the frequency of selection in the CATA and the mean value of intensity in the rating for each sample. R2 (coefficient of determination) indicates the extent to which the independent variable explains the dependent variable in a regression model (Nagelkerke, 1991). The value of the coefficient of determination (R2) ranges from 0 to 1, and scores closer to 1 indicate a higher correlation between the dependent and independent variables. In this study, R2 had high explanatory power for orange juice (0.847) and yogurt (0.834). The relationship between CATA and the rating was linear for all samples. In other words, consumers chose the characteristic term selected in CATA with a high intensity in proportion to the term selected.

Fig. 3.

Fig. 3

Open in a new tab

Linear regression of intensity means from rating by selection frequency from CATA (0–100%) of two food categories. a Orange juice session. b Yogurt session. Using a 6-point intensity scale (0–5 points) with 1-point increment. Value converted to percentage of selection frequency of CATA data. The confidence interval is represented by a gray line (Obs 95%) and a dotted line (Mean 95%)

In this study, a linear relationship was found between the selection frequency of CATA and rating intensity, regardless of the sample type (Fig. 3). Additionally, line graphs of average ratio between CATA citation per intensity rated were drawn for each and all attribute (Jaeger et al., 2023) and overall line graph showed the linear relationship (data not shown). Similar results were found for the selection frequency and intensity of attributes in previous studies (Bruzzone et al., 2012; Choi and Lee, 2019; Jaeger et al., 2020a, b; Vidal et al., 2018; Yang and Lee, 2021). Regardless of the method, consumers were able to consistently evaluate the flavor and texture characteristics of each sample, and a correlation was found between the CATA and rating methods.

Therefore, both food evaluation methods had similar consumer responses and effects. However, different evaluation methods must be selected depending on the objectives of study and similarity of the test sample. If objective of study were to classify samples based on flavor and/or texture similarity for screening purposes, CATA would be a good candidate. If samples are similar in flavor and/or texture intensity and includes several weak but important characteristics, rating would be more appropriate methods. Based on recommendation of Jaeger et al. (2023), characteristics with strong intensity should be also rated.

This study had some limitations. The meaning of specific sensory attributes that each consumer considers can be interpreted differently (Ares and Varela, 2017). Because CATA contains many sensory terms in the questionnaire, consumers may experience difficulty in knowing the meaning of some terms or may not be able to locate terms appropriate for their perception. Similar to previous study by Bianchi et al. (2021), conducting qualitative research to confirm the difference between the definition of attribute term intended by researcher and what consumers thought. In addition, future consumer research is needed to determine how consumers utilize terms when they do not know the exact meaning when characterizing products using given terminology. Adapted flash profiling or free sorting (Delarue, 2023) may be an alternative method for overcoming these limitations.

To conclude, this study was conducted to confirm the relationship between CATA and rating with the same terms and to determine whether sample discrimination is affected by the type of consumer-based method. Samples were grouped according to flavor and textural characteristics and were in similar positions in biplots created using data from CATA and rating (CA and PCA, respectively). Although it cannot be concluded that any evaluation method is better than the others, the type of method is considered to affect sample discrimination. Consumer-based methods should be selected according to the purpose of the study. CATA would be recommended to classify or screen samples based on flavor and/or flavor similarity and differences, however, rating might be a better choice if samples were similar in flavor and/or texture intensity and included several weak intensity characteristics that were important.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 36.1 kb) (36.1KB, docx)

Acknowledgements

This study was supported by the National Research Foundation of Korea grant funded by the Korean government (Ministry of Science, ICT, & Future Planning) (No. 2017R1C1B2006191).

Declarations

Conflict of interest

The authors declare no conflict of interest.

Footnotes

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. Antúnez L, Vidal L, de Saldamando L, Giménez A, Ares G. Comparison of consumer-based methodologies for sensory characterization: Case study with four sample sets of powdered drinks. Food Quality and Preference. 2017;56:149–163. doi: 10.1016/j.foodqual.2016.09.013. [DOI] [Google Scholar]
  2. Ares G, Jaeger SR. Check-all-that-apply (CATA) questions with consumers in practice: Experimental considerations and impact on outcome. In: Julien D, Lawlor JB, editors. Rapid sensory profiling techniques. 2. Cambridge: Woodhead Publishing; 2023. pp. 257–280. [Google Scholar]
  3. Ares G, Varela P. Trained vs. consumer panels for analytical testing: Fueling a long lasting debate in the field. Food Quality and Preference. 2017;61:79–86. doi: 10.1016/j.foodqual.2016.10.006. [DOI] [Google Scholar]
  4. Ares G, Bruzzone F, Gimenez ANA. Is a consumer panel able to reliably evaluate the texture of dairy desserts using unstructured intensity scales? Evaluation of global and individual performance. Journal of Sensory Studies. 2011;26:363–370. doi: 10.1111/j.1745-459X.2011.00352.x. [DOI] [Google Scholar]
  5. Ares G, Bruzzone F, Vidal L, Cadena RS, Giménez A, Pineau B, Hunter DC, Paisley AG, Jaeger SR. Evaluation of a rating-based variant of check-all-that-apply questions: Rate-all-that-apply (RATA) Food Quality and Preference. 2014;36:87–95. doi: 10.1016/j.foodqual.2014.03.006. [DOI] [Google Scholar]
  6. Ares G, Antúnez L, Bruzzone F, Vidal L, Giménez A, Pineau B, Beresford MK, Jin D, Paisley AG, Chheang SL, Roigard CM, Jaeger SR. Comparison of sensory product profiles generated by trained assessors and consumers using CATA questions: Four case studies with complex and/or similar samples. Food Quality and Preference. 2015;45:75–86. doi: 10.1016/j.foodqual.2015.05.007. [DOI] [Google Scholar]
  7. Ares G, Picallo A, Coste B, Antúnez L, Vidal L, Giménez A, Jaeger SR. A comparison of RATA questions with descriptive analysis: Insights from three studies with complex/similar products. Journal of Sensory Studies. 2018;33:e12458. doi: 10.1111/joss.12458. [DOI] [Google Scholar]
  8. Bianchi I, Branchini E, Torquati S, Fermani A, Capitani E, Barnaba V, Savardi U, Burro R. Non experts’ understanding of terms frequently used by experts to describe the sensory properties of wine: An investigation based on opposites. Food Quality and Preference. 2021;92:104215. doi: 10.1016/j.foodqual.2021.104215. [DOI] [Google Scholar]
  9. Bitnes J, Ueland Ø, Møller PER, Martens M. Reliability of sensory assessors: Issues of complexity. Journal of Sensory Studies. 2009;24(1):25–40. doi: 10.1111/j.1745-459X.2008.00193.x. [DOI] [Google Scholar]
  10. Brown MD, Chambers DH. Sensory characteristics and comparison of commercial plain yogurts and 2 new production sample options. Journal of Food Science. 2015;80(12):S2957-S2969. doi: 10.1111/1750-3841.13128. [DOI] [PubMed] [Google Scholar]
  11. Bruzzone F, Ares G, Giménez ANA. Consumers’ texture perception on milk desserts. II–Comparison with trained assessors’ data. Journal of Texture Studies. 2012;43:214–226. doi: 10.1111/j.1745-4603.2011.00332.x. [DOI] [Google Scholar]
  12. Bruzzone F, Vidal L, Antúnez L, Giménez A, Deliza R, Ares G. Comparison of intensity scales and CATA questions in new product development: Sensory characterisation and directions for product reformulation of milk desserts. Food Quality and Preference. 2015;44:183–193. doi: 10.1016/j.foodqual.2015.04.017. [DOI] [Google Scholar]
  13. Choi Y, Lee J. The effect of extrinsic cues on consumer perception: A study using milk tea products. Food Quality and Preference. 2019;71:343–353. doi: 10.1016/j.foodqual.2018.08.004. [DOI] [Google Scholar]
  14. Coggins PC, Schilling MW, Kumari S, Gerrard PD. Development of a sensory lexicon for conventional milk yogurt in the United States. Journal of Sensory Studies. 2008;23:671–687. doi: 10.1111/j.1745-459X.2008.00179.x. [DOI] [Google Scholar]
  15. Danner L, Crump AM, Croker A, Gambetta JM, Johnson TE, Bastian SE. Comparison of rate-all-that-apply and descriptive analysis for the sensory profiling of wine. American Journal of Enology and Viticulture. 2018;69(1):12–21. doi: 10.5344/ajev.2017.17052. [DOI] [Google Scholar]
  16. Delarue J. Flash profile, its evolution and uses in sensory and consumer science. In: Delarue J, Lawlor JB, editors. Rapid sensory profiling techniques. 2. Cambridge: Woodhead Publishing; 2023. pp. 141–172. [Google Scholar]
  17. Desai NT, Shepard L, Drake MA. Sensory properties and drivers of liking for Greek yogurts. Journal of Dairy Science. 2013;96(12):7454–7466. doi: 10.3168/jds.2013-6973. [DOI] [PubMed] [Google Scholar]
  18. Jaeger SR, Chheang SL, Yin J, Bava CM, Gimenez A, Vidal L, Ares G. Check-all-that-apply (CATA) responses elicited by consumers: Within-assessor reproducibility and stability of sensory product characterizations. Food Quality and Preference. 2013;30(1):56–67. doi: 10.1016/j.foodqual.2013.04.009. [DOI] [Google Scholar]
  19. Jaeger SR, Beresford MK, Paisley AG, Antúnez L, Vidal L, Cadena RS, Cadena RS, Giménez A, Ares G. Check-all-that-apply (CATA) questions for sensory product characterization by consumers: Investigations into the number of terms used in CATA questions. Food Quality and Preference. 2015;42:154–164. doi: 10.1016/j.foodqual.2015.02.003. [DOI] [Google Scholar]
  20. Jaeger SR, Beresford MK, Lo KR, Hunter DC, Chheang SL, Ares G. What does it mean to check-all-that-apply? Four case studies with beverages. Food Quality and Preference. 2020;80:103794. doi: 10.1016/j.foodqual.2019.103794. [DOI] [Google Scholar]
  21. Jaeger SR, Chheang SL, Jin D, Roigard CM, Ares G. Check-all-that-apply (CATA) questions: Sensory term citation frequency reflects rated term intensity and applicability. Food Quality and Preference. 2020;86:103986. doi: 10.1016/j.foodqual.2020.103986. [DOI] [Google Scholar]
  22. Jaeger SR, Chheang SL, Jin D, Ryan GS, Ares G. How do CATA questions work? Relationship between likelihood of selecting a term and perceived attribute intensity. Journal of Sensory Studies. 2023 doi: 10.1111/joss.12833. [DOI] [Google Scholar]
  23. Kim MK, Lee YJ, Kwak HS, Kang MW. Identification of sensory attributes that drive consumer liking of commercial orange juice products in Korea. Journal of Food Science. 2013;78:S1451-S1458. doi: 10.1111/1750-3841.12227. [DOI] [PubMed] [Google Scholar]
  24. Lawless HT, Heymann H. Sensory evaluation of food: Principles and practices. New York: Springer; 2010. [Google Scholar]
  25. Lévy CM, MacRae A, Köster EP. Perceived stimulus complexity and food preference development. Acta Psychologica. 2006;123:394–413. doi: 10.1016/j.actpsy.2006.06.006. [DOI] [PubMed] [Google Scholar]
  26. Lotong V, Chambers IV E, Chambers DH. Categorization of commercial orange juices based on flavor characteristics. Journal of Food Science. 2003;68:722–725. doi: 10.1111/j.1365-2621.2003.tb05739.x. [DOI] [Google Scholar]
  27. Meyners M, Jaeger SR, Ares G. On the analysis of rate-all-that-apply (RATA) data. Food Quality and Preference. 2016;49:1–10. doi: 10.1016/j.foodqual.2015.11.003. [DOI] [Google Scholar]
  28. Nagelkerke NJ. A note on a general definition of the coefficient of determination. Biometrika. 1991;78:691–692. doi: 10.1093/biomet/78.3.691. [DOI] [Google Scholar]
  29. Oppermann AKL, De Graaf C, Scholten E, Stieger M, Piqueras-Fiszman B. Comparison of rate-all-that-apply (RATA) and descriptive sensory analysis (DA) of model double emulsions with subtle perceptual differences. Food Quality and Preference. 2017;56:55–68. doi: 10.1016/j.foodqual.2016.09.010. [DOI] [Google Scholar]
  30. Palczak J, Blumenthal D, Rogeaux M, Delarue J. Sensory complexity and its influence on hedonic responses: A systematic review of applications in food and beverages. Food Quality and Preference. 2019;71:66–75. doi: 10.1016/j.foodqual.2018.06.002. [DOI] [Google Scholar]
  31. Pérez-Cacho PR, Galan-Soldevilla H, Mahattanatawee K, Elston A, Rouseff RL. Sensory lexicon for fresh squeezed and processed orange juices. Food Science and Tchnology International. 2008;14:131–141. doi: 10.1177/1082013208094723. [DOI] [Google Scholar]
  32. Pierguidi L, Spinelli S, Dinnella C, Prescott J, Monteleone E. Individual differences in perceived complexity are associated with different affective responses to alcoholic cocktails. Food Quality and Preference. 2019;76:47–59. doi: 10.1016/j.foodqual.2019.03.010. [DOI] [Google Scholar]
  33. Pineau N, Girardi A, Gregorutti CL, Fillion L, Labbe D. Comparison of RATA, CATA, sorting and napping® as rapid alternatives to sensory profiling in a food industry environment. Food Research International. 2022;158:111467. doi: 10.1016/j.foodres.2022.111467. [DOI] [PubMed] [Google Scholar]
  34. Reinbach HC, Giacalone D, Ribeiro LM, Bredie WL, Frøst MB. Comparison of three sensory profiling methods based on consumer perception: CATA, CATA with intensity and Napping®. Food Quality and Preference. 2014;32:160–166. doi: 10.1016/j.foodqual.2013.02.004. [DOI] [Google Scholar]
  35. Vidal L, Ares G, Hedderley DI, Meyners M, Jaeger SR. Comparison of rate-all-that-apply (RATA) and check-all-that-apply (CATA) questions across seven consumer studies. Food Quality and Preference. 2018;67:49–58. doi: 10.1016/j.foodqual.2016.12.013. [DOI] [Google Scholar]
  36. Williams EJ. Experimental designs balanced for the estimation of residual effects of treatments. Australian Journal of Chemistry. 1949;2:149–168. doi: 10.1071/CH9490149. [DOI] [Google Scholar]
  37. Yang J, Lee J. Consumer perception of red wine by the degree of familiarity using consumer-based methodology. Foods. 2021;10:749. doi: 10.3390/foods10040749. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary material 1 (DOCX 36.1 kb) (36.1KB, docx)

Articles from Food Science and Biotechnology are provided here courtesy of Springer

RESOURCES