Comparing and Contrasting Consensus versus Empirical Domains

Abstract

Background

Since the publication of the CFS case definition [1], there have been a number of other criteria proposed including the Canadian Consensus Criteria [2] and the Myalgic Encephalomyelitis: International Consensus Criteria. [3]

Purpose

The current study compared these domains that were developed through consensus methods to one obtained through more empirical approaches using factor analysis.

Methods

Using data mining, we compared and contrasted fundamental features of consensus-based criteria versus empirical latent factors. In general, these approaches found the domain of Fatigue/Post-exertional malaise as best differentiating patients from controls.

Results

Findings indicated that the Fukuda et al. criteria had the worst sensitivity and specificity.

Conclusions

These outcomes might help both theorists and researchers better determine which fundamental domains to be used for the case definition.

Introduction

In the mid 1990s, Fukuda et al. [1] developed a case definition for chronic fatigue syndrome (CFS), which used polythetic criteria: a set of symptoms in which all do not need to be present to make a diagnosis. Because the Fukuda et al. [1] criteria only require four symptoms out of a possible eight, critical CFS symptoms such as post-exertional malaise and memory/concentration problems were not required for a patient to receive a diagnosis. This could have increased the heterogeneity of the population and complicated identification of comparable samples. In order to deal with this problem, two other case definitions have emerged including the Canadian Consensus Criteria [2] and the Myalgic Encephalomyelitis-International Consensus Criteria. [3] The Canadian ME/CFS consensus clinical criteria defined seven specific symptom requirements [2], whereas the ME-ICC criteria further increased the number of symptoms required to eight. [3] The justification for each of these domains was consensus-based rather than empirical.

Using a more empirical approach with the Receiver Operating Characteristic (ROC) curve analysis, Jason, Evans, Brown et al. [4] found the post-exertional factor of the ME/CFS Fatigue Types Questionnaire [5] had 90% sensitivity and a specificity of .93. This domain and its 5 symptoms have been included in the DePaul Symptom Questionnaire (DSQ). [6] Other investigators have used statistical techniques to determine the critical domains experienced by patients with this illness. [7,8,9,10,11] From these studies, the domains of neurocognitive impairments and post-exertional malaise are more common than others including pain, autonomic, immune and neuroendocrine factors.

Several investigators have compared and contrasted case definitions to determine which symptoms best differentiate patients from controls. [12] For example, Jason, Kot et al. [13] used several methods (i.e., continuous scores of symptoms, theoretically and empirically derived cut off scores of symptoms) to identify core symptoms best differentiating patients from controls. In addition, 100 sets of analyses of data mining with decision trees were conducted, with equal numbers of participants in patient and control groups. Outcomes from these analyses suggest that individuals identified using fewer, but empirically selected symptoms (i.e., fatigue or extreme tiredness, physically drained/sick after mild activity, difficulty finding the right word to say or expressing thoughts, and unrefreshing sleep) can help guide the creation of more reliable case definitions.

Another recent study sought to discover latent factors among patients and controls, in order to contribute to understanding critical domains. Using patient samples collected in the US, Great Britain, and Norway, as well as controls, exploratory factor analysis was used to establish the underlying factor structure using a standardized self-report questionnaire. Jason, Sunnquist et al. [14] found a 4-factor solution including: post-exertional malaise, cognitive dysfunction, sleep difficulties and a combined factor involving neuroendocrine, autonomic and immune dysfunction. These types of more empiric methods could help investigators better understand the fundamental domains within this illness.

The present study examined advanced statistical methods to determine which domains of symptoms best differentiate patients from controls. Three consensus-based domains were used including the Fukuda et al. [1] criteria, the Carruthers et al. [2] clinical case definition of ME/CFS, and the ME-ICC case definition [3], in addition to more empirically derived factors. Our investigation used data mining analyses to compare these methods for differentiating patients from controls. We hypothesized that empiric methods that identify latent factors as well as domains that identified core features of this illness, using the ME/CFS and ME-ICC case definitions, would be more effective in differentiating patients from controls than the more general Fukuda et al. [1] criteria.

Method

Research Participants and procedure

DePaul sample

An international convenience sample of adults self-identifying as having CFS, ME/CFS, or ME was recruited. To be eligible, an individual needed to be at least 18, capable of reading and writing English, and have a self-reported current diagnosis of ME, CFS, or ME/CFS. Following approval by DePaul University's Institutional Review Board, participants were recruited from a variety of sources including postings on internet forums, support group visits, re-contacting of individuals who have participated in the DePaul Research Team's studies in the past and have indicated interest in future studies, and contacting of individuals who have emailed the team's address in the past with interest in future studies.

Participants were given three options for completion of the surveys: an electronic survey, a hard-copy survey, or a verbal survey over the telephone. All participants were given the opportunity to complete these surveys at home or in person at DePaul. Participants were not given a timeline for survey completion, as this illness can be unpredictable and result in a rapid decline of functioning on any given day. The first 100 individuals who completed the survey received a $5.00 gift card to Amazon.com for their participation.

Of the original 217 individuals who completed the DSQ, 189 participants were included in the present study. Twenty-eight participants were excluded due to active medical conditions, active psychological conditions, and/or the endorsement of lifelong fatigue, all of which preclude a diagnosis of CFS based upon the Fukuda et al. [1] case definition.

Demographically, the sample of 189 participants was 83.5% female and 16.5% male. 97.9% of the sample identified as Caucasian, 0.5% as Asian, and the remaining 1.6% identified as ‘Other.’ 55.3% of the sample stated that they were currently on disability, with only 12.8% of the sample working part or full-time. With regards to educational level, 39.9% of the sample held a professional degree, 35.6% held a standard college degree, 17.6% attended college for at least one year, and 6.9% completed high school or had a GED. The mean age was 51.6 (SD 11.2).

A sample of DePaul University undergraduate students was recruited via the Introductory Psychology Research Participation Pool. Students were required to be at least 18 years old to participate. Following approval from DePaul University's Institutional Review Board, participants provided informed consent and completed the study measures in person at DePaul. They received credit in their Introductory Psychology courses for participation in a research study.

Of the 96 control participants, 70.8% were female, and the remainder male. The majority of the sample (60.0%) identified as Caucasian, 13.7% as Asian or Pacific Islander, 9.5% as Black or African American, 1.1% as American Indian or Alaskan Native, and 15.8% as another race. Almost all (88.5%) of participants selected Student when reporting their work status, but 1.0% indicated Unemployed, and 10.4% stated that they were working part time. Regarding the highest level of education achieved, 4.2% held a standard college degree; 51.0% had completed at least one year of college; and 44.8% had a high school degree. The mean age of the sample was 20.6 (SD = 2.6).

SolveCFS BioBank Sample

A separate sample of individuals was collected by the Solve ME/CFS Initiative. This patient data originated from the SolveCFS BioBank, a resource with clinical information and blood samples on a sample of individuals diagnosed by a licensed physician specializing in CFS, ME/CFS, and ME. The sample used in the present study included only those over 18. The participants studied here were recruited by the Solve ME/CFS Initiative through physician referral. All participants who met eligibility criteria completed a written informed consent process. Participants completed the study measures electronically or by hard copy. SolveCFS BioBank data was de-identified and shared with the DePaul research team following submission and peer review of a research protocol to the Solve ME/CFS Initiative. Of the 239 patients who participated, 237 were included in the current study; two participants were excluded due to missing data.

The SolveCFS BioBank patient sample was 99.1% Caucasian, and 0.9% selected ‘Other’ for their race. With regards to gender, 73.0% of the sample was female. Only 10.5% of the sample was working full- or part-time, with 65.4% on disability. Regarding education level, 24.7% of the sample held a graduate or professional degree; 43.0% had completed college; 20.9% had completed some college; and 11.5% had a high school degree or GED. The average age of the sample was 49.7 (SD = 12.9).

Control participants were also recruited through the Solve ME/CFS Initiative and completed the same written informed consent process as the patient sample. Control participants were required to be in generally good physical and mental health and could not have a substance use disorder or any disorder that could cause immunosuppression. Furthermore, controls could not have any medical condition or mental health disorder that caused fatigue.

Of the 83 control participants, 80.7% were female, and 19.3% were male. Regarding race, 98.8% of the sample was Caucasian, and 1.2% was Black or African American. Most (66.3%) of the sample was working; 13.3% was retired; and the remainder was not working for other reasons. This sample was also highly educated, with 22.0% holding a graduate or professional degree and 39.0% a standard college degree; 25.6% had completed at least one year of college, and 13.4% had a high school degree. The mean age of the sample was 49.7 (SD = 13.6).

Newcastle Sample

Participants in the Newcastle sample had been referred for a medical assessment at the Newcastle-upon-Tyne Royal Victoria Infirmary clinic due to a suspected diagnosis of CFS. An experienced physician performed a comprehensive medical history and examination, and individuals who met eligibility criteria completed a written informed consent process. A total of one hundred participants completed study measures by hard copy, but three were excluded due to incomplete data.

The Newcastle sample was 99.0% Caucasian and 1.0% multiracial, and 82.5% of participants were female. Of this sample, 37.5% of participants were working either part- or full-time, and 30.2% were on disability. With regard to education level, 20.9% had a graduate or professional degree; 29.7% had a college degree; 24.2% had completed at least one year of college; 14.3% had a high school degree; and 11.0% had not completed high school. The average age of the sample was 45.6 (SD = 14.0).

Norway Sample 1

Individuals with CFS were invited to participate in a randomized controlled trial of a CFS self-management program. Participants were recruited from four mid-sized towns in southern Norway, two suburbs of Oslo, and some surrounding communities. Recruitment sources included: healthcare professionals, the waiting list for a patient education program, and CFS patient organizations. Information about the study was disseminated through brochures and personal communication. In addition, study announcements for participants were placed on the Oslo University Hospital website.

Participants were required to be older than 18 years of age and diagnosed with CFS by a physician or medical specialist. In addition, participants could not be pregnant and needed to be physically able to attend the self-management program. Those who were interested in participation were given additional information by telephone. Participants completed a consent form that provided permission to request confirmation of their CFS diagnosis from their physician or medical specialist. The study gained approval from the Regional Committee for Medical Research Ethics (Health Region North) and the Privacy Ombudsman for Research at Oslo University Hospital. Of the 176 participants, 175 were included in this study; one participant was excluded due to missing data.

This sample was 86.8% female and 13.2% male. Almost all participants were Caucasian (99.4%); one participant selected ‘Other’ when asked about race. Only 9.7% of participants were working, while 84.0% were on disability. Regarding education, 9.9% of participants had a graduate or professional degree, 40.1% a standard college degree, 41.9% a high school degree, and the remainder had not completed high school. The mean age of the sample was 43.4 years (SD = 11.7).

Norway Sample 2

Participants were recruited from an inpatient medical ward for severely ill patients as well as from the outpatient clinic at a multidisciplinary CFS/ME Center. To be eligible for inclusion, participants needed to be between 18 and 65 years old and capable of reading and writing Norwegian. Individuals with a suspected diagnosis of CFS were referred for evaluation and completed the study measures. All participants took part in a comprehensive medical history interview and a detailed medical examination conducted by experienced consultant physicians and a psychologist. The examinations were conducted to rule out exclusionary medical and psychiatric conditions. Participants completed a written informed consent, and the study measures were completed by hard copy. The project gained approval from the Privacy Ombudsman for research at Oslo University Hospital. Of the 64 total participants, 63 were included in this study; one was excluded due to missing data.

This sample was 82.5% female and 17.5% male. The majority of the sample identified as Caucasian, but 1.6% identified as Asian, and 3.3% as ‘Other.’ Most participants (76.2%) were on disability, while 19.0% were working. With regard to education, 11.1% held a graduate or professional degree; 25.4% held a standard college degree; 46.0% had a high school degree; and 17.5% had not completed high school. The mean age of the sample was 34.9 years (SD = 11.6).

Case Definitions

CFS Case Definition

A case of chronic fatigue syndrome is defined by Fukuda et al. [1] as the presence of the following criteria: (1) clinically evaluated, unexplained, persistent or relapsing chronic fatigue that is of new or definite onset (has not been lifelong); is not the result of ongoing exertion; is not substantially alleviated by rest and results in substantial reduction in previous levels of occupational, educational, social, or personal activities, and (2) the concurrent occurrence of four or more core symptoms, all of which must have persisted or recurred during six or more consecutive months of illness and must not have predated the fatigue. [1] Because frequency and severity criteria were not specified for the symptoms required in this case definition, participants needed to report frequency and severity scores of at least 1 for a symptom to be counted toward the four required symptoms.

The Canadian clinical ME/CFS consensus case definition [2] specifies that post-exertional malaise must occur with a loss of physical or mental stamina, rapid muscle or cognitive fatigability, usually taking 24 hours or longer to recover. In addition, two or more neurological/cognitive manifestations must be present (e.g., confusion, impairment of concentration and short-term memory). Unrefreshing sleep or poor sleep quantity or rhythm disturbance must be reported, as well as a significant degree of arthralgia and/or myalgia. Finally, one symptom from two of the following categories must be present: autonomic manifestations (e.g., neurally mediated hypotension, light headedness), neuroendocrine manifestations (e.g., recurrent feelings of feverishness, cold extremities), and immune manifestations (e.g., recurrent sore throats).

ME-ICC Case Definition

The International Consensus Criteria for ME [3] state that symptom severity must result in a 50% or greater reduction of a patient's premorbid activity level for a diagnosis. Additionally, symptoms from four major groupings are required. First, to meet criteria, a person must experience Post-Exertional Neuroimmune Exhaustion. Within the Neurological Impairment symptom grouping, a patient must have at least one symptom from three of the following four symptom categories (1) neurocognitive impairments (e.g., difficulty processing information, short-term memory loss), (2) pain, (3) sleep disturbance, and (4) neurosensory, perceptual and motor disturbances (e.g. inability to focus vision, sensitivity to light, feeling unsteady on feet). The third category is Immune, Gastro-intestinal and Genitourinary Impairments, and individuals must have at least one symptom from three of the following five symptom categories: (1) flu-like symptoms, (2) susceptibility to viral infections with prolonged recovery periods (3) gastrointestinal tract symptoms (e.g., nausea, abdominal pain), (4) genitourinary symptoms (e.g., urinary urgency), and (5) sensitivities to food, medications, odors, or chemicals. The final category is Energy Production/Transportation Impairments, and at least one symptom from one of the following four symptom categories must be present: (1) cardiovascular (e.g. orthostatic intolerance), (2) respiratory (e.g. labored breathing), (3) loss of thermostatic stability (e.g. subnormal body temperature), and (4) intolerance of extremes of temperature. Participants needed to report frequency and severity scores of at least 2 for a symptom to meet criteria.

Empirical Criteria

Using patient samples collected in the US, Great Britain, and Norway, as well as controls, exploratory factor analysis (EFA) was used to establish the underlying factor structure using a standardized self-report questionnaire, the DePaul Symptom Questionnaire (DSQ).[6] With the sample of patients and controls, and with patients alone, Jason, Sunnquist et al. [14] found a 4-factor solution using EFA, including : post-exertional malaise, cognitive dysfunction, sleep difficulties and a combined factor involving neuroendocrine, autonomic and immune dysfunction. These factors were used in the analyses below.

Measures

The DePaul Symptom Questionnaire

All participants completed the DePaul Symptom Questionnaire (DSQ),[6] a self-report measure of symptomatology, demographics, and medical, occupational, and social history. Participants were asked to rate each of the symptom's frequency and severity over the past six months on a 5-point Likert scale. Symptom frequency was rated: 0=none of the time, 1=a little of the time, 2=about half the time, 3=most of the time, and 4=all of the time. Likewise, severity was rated: 0=symptom not present, 1=mild, 2=moderate, 3=severe, and 4=very severe. The DSQ has evidenced good test-retest reliability among both patient and control groups [15] with factors evidencing good internal consistency.[8] The DSQ is available at REDCap's shared library: https://redcap.is.depaul.edu/surveys/?s=tRxytSPVVw

Statistics

Data mining with decision trees was used to further analyze domain data, and all variables are placed into the analyses, rather than one domain or a limited group of domains. Data mining techniques, such as classification using decision trees, provide statistical analyses that identify which questionnaire items best predict class membership and are useful for indicating which domains should be required in the diagnostic process to ensure the most accurate classifications. In the current study, decision trees were used to determine which domains were most effective at accurately classifying participants as patient or control.

Decision trees consist of a series of successive binary choices (branch points) that will ideally result in an accurate classification of participants. At each branch point of the tree, all of the domain variables are examined to determine which symptom has the most effect on the entropy of the classifications. Here, entropy indicates the certainty of the diagnosis. The domain selected at each branch point is the one that best predicts classifications at that point in the tree, and is used to split all of the cases into two groups. This process is repeated, and more symptoms are chosen, until the resulting series of branch points produces groupings of correctly classified participants.

SPSS Statistics software was used to build our decision tree models. To build the models, a Classification and Regression Tree (CART) algorithm was applied to a training set consisting of 66% of the cases, stratified to reflect the distribution of patient and control groups. The value of the model was measured by evaluating its classification performance when applied to cases reserved for testing (34% of the data), allowing this technique the ability to be generalized to new data.

Decision trees were used to determine which domains were most effective at correctly classifying participants as patients or control. Because data mining is best conducted with similar numbers of participants in groups, we took a random sample of 80 patients along with the 80 Controls. We created 100 sets of analyses.

Methods for replacing missing values

In examining the frequency and severity ratings of the 54 DSQ symptoms, participants missing responses to 10% or more items were removed. Of the remaining participants, missing values were replaced using the following method: For the cases that had a score of 0 for either frequency or severity of a symptom and were missing the other field, the missing value was set to 0; the rationale was that a symptom should occur “none of the time” (frequency=0) if the symptom is “not present” (severity=0). Otherwise, if a subject was missing data in only one of the two fields (frequency or severity) for a symptom, then the missing field was replaced with the mode of the cases that had the same score for the non-missing field. When both fields were missing for a symptom, the values were replaced with the overall medians in those fields for that symptom. After missing values were replaced, frequency and severity scores were multiplied by 25 to create 100-point scales. The 100-point frequency and severity scores for each symptom were averaged to create one composite score per symptom.

Results

Demographic Characteristics

As reported in Jason, Sunnquist et al. [14], significant differences existed between patients and controls. The control sample was significantly younger than the patient sample and also more diverse. While most of the control sample held a high school or college degree, a larger proportion of the patient sample had not completed high school or had a graduate or professional degree, and a larger proportion of patients were on disability, while more control participants were students.

Data Mining

Using the Fukuda et al. [1] criteria, Table 1 indicates that 3 symptoms (lymph node pain, joint pain and PEM) were best able to discriminate patients from controls, but with only a sensitivity of 77.8%, specificity of 75.5%, and accuracy of 76.2%. In contrast, Table 2 shows that the ME-ICC criteria [3] achieved a higher sensitivity (92.7%), specificity (80.9%), and accuracy (86.7%) with only two domains (Post-Exertional Neuroimmune Exhaustion and Energy Production/Transportation). Table 3 shows that the ME/CFS Canadian criteria [2] also did better than the Fukuda et al. criteria, using just two domains (PEM, and at least two Symptoms from Autonomic, Neuroendocrine, Immune), achieving 97.1% sensitivity, 76.8% specificity, and 93.9% accuracy. Finally, as noted in Table 4, using the latent variables from the empiric criteria, only one factor (PEM) was need to reach a sensitivity of 90.8%, specificity of 92.5% and accuracy of 91.6%, and this was the only data mining where all percentages were over 90%.

Table 1.

New Symptom Domains Analysis on Fukuda criteria, BioBank, DePaul, Newcastle, Norway n = 977

	Number of Times the Domain Was Used
Selected Symptom Domain	As Node 1	As Node 2	As Node 3	As Node 4	Total Use
Lymph Node	82	11	1	0	94
Joint Pain	15	33	20	1	69
PEM	0	32	29	1	62
Muscle Pain	3	16	3	0	22
Sore Throat	0	5	10	3	18
Unrefreshing Sleep	0	1	1	1	3
Headaches	0	0	1	0	1
Memory / Concentration	0	0	0	0	0
Sensitivity	77.8%
Specificity	75.5%
Accuracy	76.2%

Table 2.

New Symptom Domains Analysis on ME-ICC criteria BioBank, DePaul, Newcastle, Norway n = 977

	Number of Times the Domain Was Used
Selected Symptom Domain	As Node 1	As Node 2	As Node 3	As Node 4	Total Use
Post-Exertional Neuroimmune Exhaustion	100	0	0	0	100
Energy Production/Transportation	0	90	10	0	100
Immune, Gastro-intestinal and Genitourinary	0	3	15	7	25
Neurological Impairment	0	6	4	0	10
Sensitivity	92.7%
Specificity	80.9%
Accuracy	86.7%

Table 3.

New Symptom Domains Analysis on Canadian ME/CFS criteria, CFS BioBank, DePaul, Newcastle, Norway n = 977

Selected Symptom Domain	Number of Times the Domain Was Used					Total Use
	As Node 1	As Node 2	As Node 3	As Node 4	As Node 5
PEM	100	0	0	0	0	100
At least two present: Autonomic, Neuroendocrine, Immune	0	79	1	2	0	82
Sleep Dysfunction	0	1	12	5	0	18
Neurocognitive	0	3	6	0	0	9
Pain	0	0	0	3	0	3
Sensitivity	97.1 %
Specificity	76.8 %
Accuracy	93.3 %

Table 4.

Decision Tree Results on Factor Groups With Empirical Thresholds, BioBank, DePaul, Newcastle, Norway n = 977

	Number of Times the Factor Was Used
Selected Factor	As Node 1	As Node 2	As Node 3	Total Use
Factor 3 – PEM	99	0	0	99
Factor 4- Sleep	0	3	1	4
Factor 1- Autonomic, Neuroendocrine, Immune	0	2	0	2
Factor 2 – Cognitive	1	1	0	2
Sensitivity	90.8 %
Specificity	92.5 %
Accuracy	91.6 %

Discussion

The study's major finding was that the ME/CFS Canadian criteria [2], the ME-ICC criteria [3] as well as the empirical criteria [14] achieved higher accuracy in discriminating patients from controls than the Fukuda et al. [1] criteria. It is likely that the findings are due to the greater specification of core symptoms in the non-Fukuda et al. criteria. Also, of significance was that the PEM was the third predictor in the Fukuda et al. solution, whereas it came out as the first predictor in the other solutions. However, the fact that PEM came out in all analyses supports the importance of this domain in the case definition. Finally, the empiric criteria was the only one where sensitivity, specificity and accuracy were all over 90%.

For the Canadian ME/CFS case definition [2] only one symptom within 5 core symptom domain needs to be present for a patient to meet criteria for that domain, but for the neurocognitive domain, two symptoms are required. Thus 7 symptoms are required. However, 8 domains are required for the ME-ICC criteria within 4 domains [3]. It is unclear what statistical methods were used to designate domains or the number of symptoms within the domains. In addition, there are many possible symptoms within each domain, and some of these symptoms are highly prevalent and others are not among patients. An advantage in using domains rather than individual items to different patients from controls is that a person might not endorse a particular symptom in a domain, but still have impairment in that area, and thus there are more ways and items to meet that criteria with questions being asked in different ways.

The current study found that the empirical criteria only needed one domain to discriminate patients from controls, whereas the other case definitions needed more domains. Therefore it is possible to identify fewer domains than the ME/CFS Canadian criteria [2] and the ME-ICC criteria. [3]Also, only the empiric criteria achieved sensitivity, specificity and accuracy above 90%. In addition, it does seem that the criteria in these three different approaches tends to do better than the Fukuda et al. criteria. It should be noted, however, that while both the Canadian ME/CFS [2] and ME-ICC criteria [3] specify cardinal symptoms such as post-exertional malaise, by requiring seven or eight symptoms, these criteria could inadvertently increase the rate of psychiatric co-morbidity within the group of individuals who meet criteria [15], as was noted with the original Holmes et al. [16] criteria.

There are a number of limitations in this article. The Fukuda et al. [1] criteria involved symptoms whereas the other criteria involve domains, so that the comparison is really between symptom-based versus domain-based methods. In addition, the data are from self-report sources, so there needs to be efforts to replicate these findings with either behavioral or physiological measures. Although the sample was large, case ascertainment methods did vary, and having a large data set collected using similar methods would strengthen the findings. Finally, the controls sample was different from patients on sociodemographic characteristics, and having more comparable samples in terms of age would provide better data for comparisons.

Refining the case definition and bringing the varying gatekeepers (scientists, clinicians, patients, government) into this process is an important need that is still unmet. Such a development could result in the identification of more homogenous patient samples, which could possibly assist in the pursuit of biomarkers for ME and CFS. Once critical domains or symptoms are identified for an illness, it is also critical to use uniform methods and assessment tools to capture those symptoms. In this article, we reviewed critical domains within the ME/CFS literature, and suggested ways to develop a more empirical rather than consensus based case definition.