Validation of the Arabic version of the ORTO-R among a sample of Lebanese young adults

Abstract

Background

Within the literature, there is a variety of different measurement methods for orthorexic behaviours. The ORTO-15 is the one that attracted most research attention. Many scholars criticized the ORTO-15 for its unstable factor structure and over-estimation of the prevalence of orthorexia nervosa. For this purpose, Rogoza and Donini (Eat Weight Disord 26:887–895, 2020) re-assessed the original data and created a new tool, ORTO-R. The development of the ORTO-R theoretically solved many ambiguities associated with its parent measure. However, to date, no study, including the original one, tested the validity of the ORTO-R, leaving its utility somewhat speculative.

Methods

We gathered data from 363 Lebanese individuals, who answered the ORTO-R questions and a set of measures used to determine the validity of the scale (eating attitudes, perfectionism, body dissatisfaction, self-esteem, anxiety, and depression). Within this study, we analysed the internal consistency of the scale and different aspects of its validity (factorial, convergent, and divergent).

Results

Results supported all expectations; we successfully confirmed a one-factor measurement model of the ORTO-R, which appeared to be internally consistent. The ORTO-R score correlated positively to other orthorexic behaviours as well as to disordered eating attitudes, perfectionism, anxiety, and depression. It was also negatively related to self-esteem, but was unrelated to body dissatisfaction.

Conclusion

The ORTO-R may be deemed as a valid instrument for the assessment of orthorexic behaviours.

Level of evidence

V: Opinions of authorities, based on descriptive studies, narrative reviews, clinical experience, or reports of expert committees.

Introduction

Orthorexia nervosa (ON) is broadly defined as an obsession with eating healthy and proper food [1, 2]. Within the literature, there is a broad agreement that it regards not only fixation on healthy nutrition, but also behaviours more typical for eating disorders (e.g., excessive desire to consume specific food associated with strong feelings of fear and guilt) and obsessive–compulsive disorders (e.g., rigidness in following the diet), which ultimately leads to physical, emotional, and psychosocial impairments [3–7]. Although the concept of ON seems to be well defined, the issue of its measurement seems to be still unresolved.

The most popular measure of orthorexic behaviours is the ORTO-15 [1, 8]. It comprises 15 items designed to capture different aspects of orthorexic behaviours. It has been analysed and validated in multiple countries, more than any other measure for orthorexic behaviours [9–17]. The common point between these studies is that they raised criticism regarding the factor structure of the scale, which appeared to be randomly varying across them. Few items, however, were repeatedly replicating within all these studies, suggesting that there might be at least some similarities [18]. Furthermore, ORTO-15 has been frequently criticized due the over-estimation of ON prevalence [6], which repeatedly exceeded 50% [19–22], whereas the estimated prevalence within general population is less than 1% [23].

In response to this criticism, Rogoza and Donini re-analysed the original data [24], proposing the ORTO-R, a new scale that comprises several advantages over its parent measure. First, ORTO-15 was subject to analyses in the sense of the classical test theory. On the basis of confirmatory factor analysis, the one-dimensional structure comprising six items was advocated to be the best measurement model for the revised scale (here and after referred to as ORTO-R). These selected items were further revised to increase their relevance in terms of the characteristic of ON. Also, the response scale was changed from four- to five-point Likert-type scale, with higher scores indicating higher orthorexic behaviours (as opposed to ORTO-15) [1]. Moreover, the order of the retained items was mixed to reduce potential effects of method bias. Finally, the ORTO-R has been advocated not to be a diagnostic tool used for the assessment of prevalence of ON, but rather as a continuous assessment of orthorexic behaviours [24]. While the results presented by Rogoza and Donini are promising [24], they were limited by the fact that they were restricted to a dataset without any possibility to (a) test the proposed modifications and (b) assess the validity of the proposed measure.

University represents the first period in life where young adults start making their own food choices, as mirrored by high rates of overweight/obesity among these individuals [25]. Recent findings suggest a severe increase in disordered eating among university students since the beginning of the COVID-19 pandemic [26]. The current study aims to fill this gap within the literature on the measurement of orthorexic behaviours. First, we aim to provide a first-to-date test of the measurement of the ORTO-R. Although the content of items in ORTO-R was modified [23], the factor validity of this modified measure has never been actually assessed. Furthermore, existing studies, including the one conducted in the Lebanese population, used items in their original, unmodified meaning [27]; thus, they are re-assessment of the best working items of the ORTO-15 rather than an adaptation and validation of the ORTO-R items. Second, we aim to assess its validity in regards to disordered eating attitudes, perfectionism, and other relevant variables such as self-esteem, depression, anxiety, and body dissatisfaction.

Current study

In the current study, we assess different aspects of validity of the ORTO-R, namely factorial, convergent, and divergent validity as well as internal consistency. Although Rogoza and Donini [24] were unable to unambiguously confirm whether the ORTO-R should assume one- or two-dimensional structure, we expect it to capture one factor (H1). This is due to the fact that the factors in the original paper [24] were organized around items with similar wording, which were moreover presented in a sequential order within the ORTO-15. Thus, these results seem to support a one-factor model. As a pragmatic reason, estimation of a factor with only three indicators seems to be of limited use. Finally, we expect that the resulting one-factor structure would be internally consistent (H2), as expressed by the removal of neither item would not decrease it and that the item-total correlation would be at least of 0.50.

We hypothesize that ORTO-R scores would have a good convergent validity. For this purpose, we administered a different measure of orthorexic behaviours (the Teruel Orthorexia Scale). We expect a positive correlation of ORTO-R to it given they assess the same construct (H3). Given the phenotypical characteristic of ON, we expect that ORTO-R scores would be positively related to disordered eating attitudes and perfectionism (H4 and H5). As ON seems not to evaluate elements of body dissatisfaction or body image disturbances [5], we do not expect a significant correlation between those two variables (H6). It is worth noting, however, that the relation between body dissatisfaction and orthorexic behaviours is somewhat more complex as some studies reported a positive relation between them [28, 29]. This might be due to the fact that body satisfaction depends on compliance with self-defined “healthy” eating behaviour, thus, might be a function of it [6]. Finally, according to the results of previous studies, we expect ORTO-R scores to be positively related to anxiety and depression, but negatively to self-esteem (H7) [4, 5, 30, 31].

Methods

Participants and procedure

This cross-sectional study was carried out between July and September 2021. A total of 363 university students aged between 18 to 37 years (M = 22.65; SD = 3.48; 61.7% females; BMI range = 14.84–44.98; M = 23.62; SD = 4.13) were recruited through convenience sampling in several universities in Lebanon’s governorates. Participants received an online link to the survey, which included a small introduction about the purpose of the current study, ensuring the anonymity of participants and voluntariness of consent to research, followed by the study questionnaire. All participants responded willingly to the survey. There were no fees for participating in the study. All university students over the age of 18 years were eligible to participate. Excluded were those who refused to complete the survey. We have only collected complete responses and response to every question was mandatory. As a result, no missing data were recorded. There were no gender differences in orthorexic behaviours as assessed by ORTO-R (M_female = 8.63; SD = 5.18; M_male = 8.20; SD = 4.93; t₍₃₆₁₎ = − 0.77; p = 441), nor it was related to BMI (r = 0.06; p = 0.251).

Measures

ORTO-R

The ORTO-R [24], the revised version of ORTO-15 [8], consists of six items [24]. Those items were scored on a five-point Likert scale (never, rarely, sometimes, often, and always). The adaptation procedure to create an Arabic version of the scale was carried out in six steps: (1) items were translated from English to Arabic, (2) the translated items were consulted regarding linguistic and cultural suitability of questionnaire, (3) items were independently back-translated to English, (4) the back-translated items were consulted with the author of the scale, (5) the comments and suggestions from the author of the scale were introduced, and (6) the revised items were again back-translated and approved by the author of the scale. Higher scores highlight more ON tendencies (α = 0.78).

Teruel Orthorexia Scale (TOS)

The Teruel Orthorexia Scale (TOS) [32] is composed of 17 items, covered the two differentiable, yet linked, aspects of orthorexia: Orthorexia Nervosa (OrNe) and Healthy Orthorexia (HeOr). Responses are evaluated on a 4-point Likert scale ranging from 0 (completely disagree) to 3 (completely agree) [32]. Higher scores are reflective of more ON tendencies. The Arabic version of TOS has been validated in Lebanese adolescents [33] and adults [34] (α = 0.91 for the OrNe subscale and α = 0.88 for the HeOr subscale).

Eating Attitude Test (EAT)

The EAT, validated in Lebanon [35], is used to assess disordered food attitude. The questionnaire comprises 26 questions each with six response options, varying from infrequently/almost never/never (0) to always (3). The total score is calculated by summing all questions answers. Higher scores indicate more inappropriate attitudes (α = 0.94).

Big three perfectionism scale-short form

This scale is composed of 16 items and yields three subscales: rigid perfectionism (4 items), self-critical perfectionism (6 items), and narcissistic perfectionism (6 items) [36] (α’s = 0.87, 0.88, and 0.81 for each subscale, respectively).

Body Dissatisfaction subscale of the Eating Disorder Inventory-second version (BD-EDI-2)

It consists of nine items, measured on a four-point Likert scale, ranging from 0 (sometimes, rarely, never) to 3 (always). Five questions were reversed while doing the score calculation. Higher scores are indicative of greater body dissatisfaction [37] (α = 0.60).

Lebanese Anxiety Scale-10 items (LAS-10)

The LAS-10 scale [38, 39] includes elements about anxiety symptoms experienced during the past week. Higher scores indicate higher anxiety (α = 0.89).

The Patient Health Questionnaire-9 (PHQ-9)

Validated in Lebanon [40], it is a self-administered scale to screen for depressive symptoms [41], with items rated from “0” (not at all) to “3” (nearly every day), in the 14 days prior to evaluation (α = 0.90).

Rosenberg Self‐Esteem Scale

The Rosenberg Self-Esteem Scale (RSES) is a 10-item scale that reflects self-worth, where higher scores reflect a better self-esteem [42]. The translated Arabic version was used in previous papers [43, 44] (α = 0.99).

Statistical analyses

Each question was mandatory in the survey; as a result, no missing data were recorded. The Mplus v7.2 and SPSS v.23 were used for statistical analyses. The assessment of factor validity was carried out in two steps. To test this hypothesis, we first executed the parallel analysis [45] and minimum average partial [46] tests of dimensionality. Afterwards, we computed a one-factor confirmatory factor analysis (CFA), which was estimated using maximum-likelihood estimation with robust standard errors.1 To evaluate model fit of the CFA, we have relied on recommendations provided by Schermelleh-Engel et al. [47] [Comparative Fit Index (CFI) ≥ 0.95 and Root-Mean-Square Error of Approximation (RMSEA) ≤ 0.08 indicate a good fit of the model]. Internal consistency was assessed using the Cronbach’s α coefficient. In addition, we also report item-total correlation coefficients as well as the estimate of α if the item would be deleted from the scale. Finally, to assess convergent and divergent validity, we have computed a series of Pearson’s correlations. To assess the difference in strength of correlations, we also reported the values of the Z test; significant values inform that two compared correlation coefficients are different in terms of strength. The raw data, statistical script, and supplementary materials are available at the OSF project site: https://osf.io/wv4tj/.

Results

Factor validity: hypothesis 1

The results of the parallel analysis are presented in Fig. 1. The eigenvalue of simulated data was higher than the actual data starting from the third factor, suggesting that the two-factor model represents the data best. However, the eigenvalue of the second factor was apparently low (i.e., 0.5), which speaks against selection of such a model. Results of the minimum average partial test were in line with such interpretation as the minimum was achieved with one factor. Thus, both tests of dimensionality seem to be in favour of a one-factor model. The results of the confirmatory factor analysis also supported that the one-factor model fits the data well (χ²₍₇₎ = 23.53; p = 0.001; CFI = 0.959; RMSEA = 0.081; SRMR = 0.044). The standardized factor loadings are given in Fig. 2. Thus, the first hypothesis was confirmed in full.

Fig. 1.

Scree plot presenting actual and simulated Eigenvalues in parallel analysis

Fig. 2.

Standardized factor loadings of the ORTO-R

Internal consistency: hypothesis 2

The one-dimensional structure appeared to be not only valid in terms of factorial validity, but also internally consistent as the estimate of which reached the value of α = 0.78. The item-total correlations and α when item deleted are presented in Table 1. Each item was reasonably correlated with the total score. Only one of the items (i.e., item 6) was correlated less than 0.50, but was still within a reasonable range. In terms of importance, each item contributed to the total estimate of internal consistency as the removal of neither item did improve this estimate. Thus, the second hypothesis was also confirmed in full.

Table 1.

Item-total correlations and estimates of internal consistency if item deleted

Item	Item-total correlation	α When item deleted
1	0.57	0.74
2	0.54	0.74
3	0.50	0.75
4	0.56	0.74
5	0.52	0.75
6	0.47	0.76

Convergent and divergent: hypotheses 3, 4, 5, 6, and 7

In Table 2, we jointly present results (i.e., Pearson’s correlation coefficients) regarding hypotheses 3–7. The ORTO-R was moderately and positively correlated to the orthorexic behaviours as measured by the TOS but also with healthy orthorexia. Interestingly and providing further information on the ORTO-R validity, when both subscales of the TOS were entered into a regression model as predictors of the ORTO-R, the orthorexia nervosa subscale showed to be a stronger predictor (β = 0.42; p < 0.001) of ORTO-R compared to healthy orthorexia (β = 0.14; p = 0.023), supporting the third hypothesis. With respect to the disordered eating attitudes and perfectionism, ORTO-R score was positively associated with both, providing support for H4 and H5. As expected for H6, ORTO-R scores remained unrelated to body dissatisfaction. Finally, ORTO-R scores have had a theoretically predicted pattern of relations to criterion variables, supporting H7. Furthermore, through the means of the Z test, we assessed the differences in the strength of correlations between ORTO-R and both TOS subscales (orthorexia nervosa and healthy orthorexia) and all study variables. Results revealed that ORTO-R presented a more theoretically accurate pattern of relations as it was more strongly related to rigid perfectionism, self-critical perfectionism, and depression. All relations between ORTO-R and healthy orthorexia TOS scale, except for body dissatisfaction, were statistically different with healthy orthorexia presenting a more adaptive pattern of relations. In sum, ORTO-R seems to be characterized by good convergent and divergent validity, supporting all our expectations.

Table 2.

Convergent and divergent validity of the ORTO-R

Scale	ORTO-R	TOS-ON	TOS-HE	ORTO-R vs TOS-OrNe	ORTO-R vs TOS-HeOr
				Z	Z
Orthorexia nervosa (TOS-OrNe)	0.51*	–	0.66*	–	3.54***
Healthy orthorexia (TOS-HeOr)	0.41*	0.66*	–	− 6.09***	–
Eating attitudes (EAT-26)	0.40*	0.46*	0.27*	− 1.31	2.46**
Rigid perfectionism (BTPS)	0.21*	0.10	0.11	2.14*	1.95*
Self-critical perfectionism (BTPS)	0.32*	0.21*	0.05	2.21*	4.86***
Narcissistic perfectionism (BTPS)	0.24*	0.26*	0.13	− 0.40	1.97*
Body dissatisfaction (EDI)	− 0.08	− 0.03	0.01	− 0.96	1.57
Anxiety (LAS)	0.34*	0.27*	0.00	1.43	6.12***
Depression (PHQ-9)	0.41*	0.32*	0.06	1.89*	6.43***
Self-esteem (RSES)	− 0.21*	− 0.15*	0.03	− 1.17	4.24***

TOS Teruel Orthorexia Scale, OrNe orthorexia nervosa, HeOr healthy orthorexia, EAT-26 Eating Attitudes Test, BTPS Big Three Perfectionism Scale, EDI Eating Disorders Inventory, LAS Lebanese Anxiety Scale, PHQ-9 Patient Health Questionnaire, RSES Rosenberg Self-Esteem Scale

Bonferroni correction applied. Marked as significant with *when p ≤  0.005. For assessment of the difference in strength of correlations (i.e., Z tests), significance was marked using following thresholds: *p < 0.05; **p < 0.01; ***p < 0.001

Discussion

Our first hypothesis (H1) tackled ORTO-R factor validity. While we proceeded through a dimensionality assessment to test whether we retain a one-factor or two-factor model of this version of the ORTO-R, the two-factor solution turned to be a suboptimal choice for these three following reasons; first, empirically, the eigenvalues of the second factor were below 1, suggesting that such factor would not explain any variance beyond itself [48]. Thus, its empirical value would be of limited utility. Second, theoretically, the extraction of two factors seems to be of limited relevance as the content of all items regards highly associated features of orthorexic behaviours, with little theoretical space to provide a clear distinction between them. Third, pragmatically, the validity of such an ultra-brief scale could be potentially compromised [49]. Such partial disagreement between the test results could be explained as well in terms of the order in which the items were presented. Although their order in the ORTO-R version was randomized [24], they were presented in the current study as they appeared in the ORTO-15 due to an administration error [1], thus, producing a source of measurement bias [24, 50]. This issue was addressed during the assessment of the measurement model of the ORTO-R and the assessment of its internal consistency. A one-factor model fits well the data, and each item was positively, and to a similar extent, related to the total ORTO-R score. Thus, this could be a possible pull out from the ambiguous results found by Rogoza and Donini in their original paper, about the one-factor model of the ORTO-R [24]. As for our second hypothesis (H2), the one-factor model showed a satisfying internal consistency. Hence, the ORTO-R might be deemed as a brief, one-dimensional measure for orthorexic behaviours.

Our subsequent three hypotheses addressed the convergent and divergent validity of the ORTO-R. Measurement of this convergent validity was done through assessment of how well the ORTO-R could be related to an already established measurement of orthorexic behaviours (i.e., the TOS), as well as to other indicator variables, such as disordered eating attitudes and perfectionism. Furthermore, we contrasted the correlations coefficients of ORTO-R and both TOS subscales regarding other validity variables such as depression, anxiety, self-esteem, and body dissatisfaction. Results of the current study confirmed a positive correlation between ORTO-R and the two TOS subscales. As previously reported [44], ORTO-R and both TOS subscales were positively correlated to disordered eating attitudes. There was, however, a difference in how they correlated to perfectionism, especially to its rigid facet. While the ORTO-R showed a positive correlation with all perfectionism facets, the TOS-OrNe subscale was unrelated to rigid perfectionism. It is worth noting that the TOS healthy orthorexia subscale revealed a much more adaptive correlational pattern; in other terms, it showed a weaker correlation to depression, anxiety, disordered eating attitudes, and perfectionism, but a stronger one to self-esteem compared to ORTO-R. This supports the claim that ORTO-R and TOS-OrNe subscale assesses qualitatively different aspects. Furthermore, neither of the scales assessing orthorexic behaviours were related to body dissatisfaction. Despite that some studies report a positive relation between those two variables [28, 29], that association is dependent on the compliance with the self-defined “healthy” eating behaviour, which might differ due to cultural reasons [6]. This finding therefore requires further and more in-depth investigation. Thus, ORTO-R provided not only a valid pattern of relations to the variables of interest, but also was more theoretically compelling than TOS-OrNe given the lack of its relation to perfectionism. Our results should be interpreted with caution, since they are based on one study only; future studies are needed to confirm our findings. Altogether, the results of the current study support that ORTO-R may be an appropriate tool to assess orthorexic behaviours, within a theoretically meaningful manner.

Limitations

The ORTO-R is a revision of the ORTO-15 [13]; thus, it shares some of its limitations. ORTO-R [24], as well as the TOS [32], being novel instruments; limited literature may be found about their psychometric properties. Our study design being cross-sectional, causality cannot be inferred. The orthorexic behaviours were assessed using scoring tools and not through a clinical patient–specialist interview. Although self-reported questionnaires could be the most commonly used instruments, the use of questionnaire can lead to information bias taking the risk of misunderstanding the questions.

Conclusion

The goal of the current study was to provide evidence of validity of the ORTO-R, which was accomplished through the examination of its factor, convergent, and divergent validity as well as assessing its internal consistency. All formulated hypotheses were supported by our data. In a head-to-head comparison, ORTO-R appeared to be as good as the TOS-OrNe subscale in general, and outperformed it in terms of rigid perfectionism, providing a more accurate pattern of relations. The results of the current study provided evidence that ORTO-R is a valid tool for dimensional assessment of orthorexic behaviours. Thus, the current study further contributes to the initial findings of Rogoza and Donini [24] as a first-to-date validation of the different aspects of validity of the revised scale.

The current paper is a psychometrical report on the evaluation of the validity of the ORTO-R. This is an essential step, which demonstrates that this scale measures indeed what it was originally intended to. Results of the current study demonstrate that with the introduction of the ORTO-R, all the criticism of the ORTO-15 and clinical populations.

What is already known on this subject?

ORTO-R is a promising refined tool for assessing ON behaviours, based on the previous most used instrument to screen for ON, the ORTO-15 [24], but it still lacks more validations and testing, for a better future use.

What does this study add?

In a sample of Lebanese students, the Arabic version of the ORTO-R seems to be a valid tool for orthorexic behaviours assessment, which adds to the field of science the needed psychometric corrections for an ON screening previous scale, and offers a promising new scale to be used.

Author contributions

SO, RR, and SH designed the study; RR drafted the manuscript; RR carried out the analysis and interpreted the results; MM, SG, LD, and SH involved in the data collection and assisted in drafting and reviewing the manuscript; all authors reviewed the final manuscript and gave their consent.

Funding

The work of Radosław Rogoza was supported by the Foundation for Polish Science (FNP).

Availability of data and materials

All data generated or analysed during this study are not publicly available to maintain the privacy of the individuals’ identities. The dataset supporting the conclusions is available upon request to the corresponding author.

Declarations

Conflict of interest

The authors have nothing to disclose.

Ethical approval

The Psychiatric Hospital of the Cross Ethics and Research Committee approved this study protocol (HPC-007-2021).

Consent to participate

Submitting the form online was considered equivalent to obtaining a written informed consent.

Consent for publication

Not applicable.