Commentary on “Gender differences in orthorexic eating behaviors: A systematic review and meta-analytic integration”

Meta-analytic findings are limited by the quality of the original studies. In the case of orthorexia nervosa (ON), an already-controversial proposed eating disorder manifestation characterized by excessive preoccupation with, and adherence to, rigid healthy eating rules, the field’s longstanding reliance on a psychometrically flawed measure limits the interpretability of statistical summaries of ON’s prevalence, characteristics, and potential mechanisms. The ORTO-15, as noted by the current authors, is by far the most widely used measure of ON symptoms or traits (Donini, Marsili, Graziani, Imbriale, & Cannella, 2005). 62% of the studies in the current meta-analysis were based on this measure. Others authors have extensively documented the ways in which the ORTO-15 lacks internal consistency or a replicable factor structure, and drastically over-estimates the “prevalence” of ON (see Missbach, Dunn, & Konig for a critical review). Perhaps most notable is the ORTO-15’s lack of face validity. Many items fail entirely to measure the construct of interest (e.g., “Do you feel guilty when transgressing?” and “At present, are you alone when having meals?”), while others appear to measure general healthy eating intentions, but not healthy eating obsessions or rigid adherence to health rules (e.g., “Are you willing to spend more money to have healthier food?” “Do you think that on the market there is also unhealthy food?”).

This makes the results of ORTO-15 studies difficult to interpret at best and invalid and unworthy of interpretation at worst. Many authors who use the ORTO-15 or subsets of its items readily acknowledge its limitations. Authors cited in the current meta-analysis have described it as “a mediocre tool for assessing orthorectic tendencies” (Missbach, Hinterbunchinger, Dreiseitl, Zellhofer, Kurz, & Konig, 2015), and stated that “there is no evidence of the validity and reliability of the ORTO-15” while reporting questionable, unacceptable, and negative internal consistency estimates for the three factors of their proposed 12-item version (0.63, 0.47, and −0.51, respectively; e.g., Cortina, 1993) while (Alvarenga, Martins, Sato, Vargas, Phillippi, & Scagliusi, 2012). Despite the qualifications by their own authors, results from both of these studies are included in the current meta-analysis.

The quantification of gender differences in ON is an important entry into the debate about whether ON is truly distinct from anorexia nervosa (AN), which is also characterized by rigid, restrictive, and impairing eating behavior that, on the surface, may appear to follow “healthy” rules. There are well-established gender differences in the prevalence of AN and in levels of AN symptoms, with eating restrictions associated with weight/shape concerns being more common and more pronounced in women and girls than in men and boys. The current meta-analysis, even if results from the ORTO-15 are under-weighted or excluded, suggests that there are minimal or no gender differences in levels of ON symptomatology. This offers initial evidence that ON is distinct from AN. However, it could still be the case that ON is a variant of AN, with healthy eating perhaps being a more palatable explanation for men to give themselves and others for restrictive eating (e.g., Depa, Barrada, & Roncero, 2019). Before ON can be considered a distinct clinical entity, there is a need for more research aimed at mapping its boundaries with normative healthy eating and AN. The ORTO-15 is the wrong tool for this job.

The problems with the ORTO-15 are well-known, and several groups have responded by producing new ON symptom measures. Rogoza (2019) conducted a meta-analysis of psychometric studies of the ORTO-15 items, and concluded that the six most face-valid items also show stable loadings onto a single factor. To date, however, no empirical studies using the ORTO-6 have been published. The Dusseldorf Orthorexia Scale (DOS; Barthles, Meyer, & Pietrowsky, 2015), the Eating Habits Questionnaire (EHQ; Gleaves, Graham, & Ambwani, 2013), and the Teruel Orthorexia Scale (TOS; Barrada & Roncero, 2018) were all developed using literature reviews and expert consensus to generate item pools, and exploratory and confirmatory factor analysis for item retention. All three demonstrate face validity, internal consistency, and replicable factor structures. The EHQ and the TOS have separate scales capturing normative healthy eating (Healthy Behavior on the EHQ and Healthy Orthorexia on the TOS), and interference due to rigid healthy eating (Problems on the EHQ and Orthorexia Nervosa on the TOS), allowing these measures to be used to distinguish pathological from normative healthy eating. Because they are more recent, the literature based on these measures is relatively small; 18% of the studies in the current meta-analysis used the DOS, and 9% used the EHQ; none used the TOS.

The ORTO-15’s availability in numerous European languages, as well as in Arabic and Turkish, is likely to be a factor in its continued use. The DOS is currently validated in English, German, and Chinese, with a Spanish translation in press; the EHQ and TOS are only validated in English and Spanish respectively (Barthles et al., 2015; Chard, Hilzendegen, Barthles, & Stroebele-Benschop, 2019; He et al., 2019). The ON research community should prioritize validating these measures in a wide range of languages to facilitate the phasing out of the ORTO-15 perhaps temporarily replacing it with the ORTO-6 until these translations become available. While the DOS, EHQ, and TOS are all superior to the ORTO-15, none fully captures all of the proposed features of ON, or potential diagnostic rule-out criteria like intentional caloric restriction for weight loss, or distorted perception of weight and shape. There is room in the ON field for more theoretically and empirically derived measurement instruments. Before we can arrive at a gold standard measure of ON, there is a need for direct validity comparisons among the measures now available, and future entrants into the field.

In addition to using more valid measures, ON researchers should take other steps to improve the methodological quality of this literature. Most significantly, authors should statistically control for normative healthy eating (using scales from the EHQ or TOS) and AN symptoms. Until this approach is widespread, and the ORTO-15 abandoned, there can be no clarity on what ON is and is not.