Evaluating the Reliability and Construct Validity of the Eckardt Symptom Score as a Measure of Achalasia Severity

Abstract

Introduction

Achalasia is a disease of mechanical esophageal dysfunction characterized by dysphagia, chest pain, regurgitation, and malnutrition. The Eckardt Symptom Score (ESS) is the gold standard self-report assessment tool. Current guidelines outline a three-step approach to patient reported outcomes measure design. Developed prior to these policies, the ESS has not undergone rigorous testing of its reliability and validity.

Methods

Adult achalasia patients retrospectively identified via a patient registry were grouped based on treatment history. Patients were grouped PREPOST (completed ESS, GERDQ, brief esophageal dysphagia questionnaire, NIH PROMIS Global Health, high resolution manometry, timed barium esophagram prior to treatment and after) and POST (completed measures only after treatment). Clinical characteristics, treatment type and date were obtained via medical record. Standardized psychometric analyses for reliability and construct validity were performed.

Results

107 patients identified; 83 POST and 24 PREPOST. The ESS has fair internal consistency and split-half reliability with a single factor structure. Dysphagia accounts for half the variance in ESS, while chest pain and weight loss account for 10% each. Pre-post-surgical assessment demonstrates improvements in ESS, except for weight loss. Effect sizes range from 0.24 to 2.53, with greatest change in regurgitation. Validity of the ESS is supported by modest correlations with GERDQ, HRQOL, and physiological data.

Conclusions

The ESS demonstrates fair reliability and validity, with a single factor structure mostly explained by dysphagia. Based on psychometric findings, weight loss and chest pain items may be decreasing ESS reliability and validity. Further assessment of the ESS under FDA guidelines is warranted.

Introduction

Achalasia is a chronic condition of the esophagus that can present with dysphagia, regurgitation, chest pain and malnutrition^{1, 2}. Mechanistically, achalasia results from improper relaxation of the lower esophageal sphincter and absent peristalsis without obstructive pathology. Currently, upper endoscopy, timed barium esophagram (TBE), and esophageal high-resolution manometry (HRM) are used to establish a diagnosis of achalasia. and follow treatment outcomes^2–5. While these tests are also used to monitor treatment response, a self-reported measure of symptom severity is often the primary outcome in clinical studies and is often the principal factor that drives clinical management decisions^{2, 6–8}.

Although other achalasia-symptom scoring tools are used, including the Achalasia Severity Score, Vantrappen Dysphagia Score, and Watson Dysphagia Score^7–10, the Eckardt Symptom Score (ESS) is a simple and more commonly used measure to grade symptom severity for achalasia patients in both clinical and research settings^{6, 11–13}. Nevertheless, a formal validation study to determine if the ESS is an optimal means to gauge achalasia symptoms has not been performed. In 2009, the US Food and Drug Administration (FDA) guidelines for scale development outlined three steps to produce a measure that is adequately validated for use in clinical trials: initial patient interviews/focus groups to generate scale items, administering the scale to a large and representative sample of patients, and reviewing the scale items via structured cognitive interviews with an additional small cohort of target patients¹⁴. As the ESS was developed in 1992, prior to these new FDA guidelines, it has not undergone rigorous validation and reliability testing.

The purpose of this study is to systematically evaluate the factor structure, reliability, and construct validity of the ESS for the grading of achalasia symptoms. Specifically, we aim to 1) Evaluate the factor structure and inter-item correlations of the ESS items via standard statistical practices, 2) Determine the reliability of the ESS via traditional measures of internal consistency and split-half reliability, and 3) Determine the content and construct (convergent) validity of the ESS as a measure of achalasia symptoms via comparisons with other, validated achalasia symptom questionnaires and physiological data obtained via HRM and TBE.

Methods

This study utilizes a cross-sectional, correlational, non-randomized design. A convenience sample obtained from a university-based outpatient gastroenterology practice is included.

Patient selection

Patients were identified for retrospective evaluation using a query of the Esophageal Center at Northwestern (ECN) Motility Laboratory Registry, which includes English-speaking patients aged 18–85 that were evaluated at the ECN for esophageal symptoms with HRM. The registry, from September 2011 to September 2016, was queried for patients with achalasia with the diagnosis established by pre-treatment HRM meeting Chicago Classification version 3.0 criteria for achalasia (subtypes I, II, or III). For patients in which pre-treatment manometry was not available for review (such as for patient referred to the ECN following treatment at a different facility), the diagnosis of achalasia was determined based on historic report of previous manometry results or subsequent treatment as achalasia. Patients were dichotomized into PREPOST and POST groups for evaluation. The PREPOST group includes patients with complete evaluations encompassing the ESS, the Gastroesophageal reflux disease questionnaire (GERDQ), the brief esophageal dysphagia questionnaire (BEDQ), and the NIH PROMIS Global Health Scale (measure of health-related quality of life (HRQOL)), as well as HRM and TBE studies completed both prior to treatment with pneumatic dilation, laparoscopic Heller’s myotomy, or per-oral endoscopic myotomy (POEM) and after. The POST group includes patients referred to the ECN after receiving treatment for achalasia who completed the ESS, GERDQ, BEDQ, HRQOL, HRM, and TBE. Patient charts were reviewed to obtain additional clinical characteristics and confirm treatment type and date. The study protocol was approved by the Northwestern University Institutional Review Board.

Symptom assessment tools

Participants completed the following paper-based questionnaires at the time of HRM.

Eckardt Symptom Score (ESS)¹³

The ESS is a 4 item self-report scale measuring weight loss in kilograms, chest pain, regurgitation, and dysphagia. Each item is graded on a score of 0 to 3, with a maximum score of 12. Scores greater than or equal to 3 are considered suggestive of active achalasia. The ESS is widely used in both clinical and research settings as a gold standard for measuring achalasia symptom severity.

Gastroesophageal Reflux Disease Questionnaire (GERDQ)¹⁵

The GERDQ is a widely used, 6-item self-report measure of reflux symptom severity. Questions evaluate four positive predictors for GERD (heartburn, regurgitation, sleep disruption due to symptoms, and increases in medication use to control GERD), as well as two negative predictors (epigastric pain and nausea). Across studies, the GERDQ demonstrates good reliability and validity.

Brief Esophageal Dysphagia Questionnaire (BEDQ)¹⁶

The BEDQ is a 10-item, recently validated self-report measure of esophageal dysphagia that also assesses for food impactions. The frequency and difficulty with swallowing solid foods, soft foods, and liquids are rated on a 5-point Likert scale for 8 items over the past 30 days. An additional 2 items measure how many instances of food impaction lasting more than 30 minutes or requiring an emergency department visit occurred in the past year.

NIH PROMIS Global Health Scale¹⁷

The PROMIS Global Health short form is a 10-item instrument representing degradations in multiple domains of health-related quality of life: overall physical health, mental health, social health, pain, fatigue, and overall perceived quality of life. The scale yields an overall total score as well as a Global Physical Health (GPH) score and a Global Mental Health (GMH) score. Higher scores denote better HRQOL.

High Resolution Manometry (HRM)

Manometry studies were completed using a 4.2-mm outer diameter solid-state assembly with 36 circumferential pressure sensors at 1-cm intervals (Medtronic Inc, Shoreview, MN). After a minimum 6-hour fast, the HRM assembly was placed transnasally and positioned to record from the hypopharynx to the stomach with approximately three intragastric pressure sensors. The HRM protocol included a 5-minute baseline recording and ten 5-ml swallows in a supine position at 20–30 second intervals. Manometry studies were analyzed using ManoView version 3.0 analysis software as described via the Chicago Classification^{3, 18, 19}. The basal esophagogastric junction pressure (EGJP) was measured at end-expiration during the baseline recording. The integrated relaxation pressure (IRP) was the mean pressure of four contiguous or non-contiguous seconds of maximal lower esophageal sphincter relaxation during the 10-second deglutitive period as referenced to gastric pressure; the median IRP of 10 supine swallows was used for each patient. Esophageal motility diagnoses were in accordance with the Chicago Classification v3.0, using a median IRP of > 15 mmHg as the upper-limit of normal¹⁹. Although the Chicago Classification was designed and intended for patients without previous surgery, we also utilized the classification scheme during the follow-up (post-treatment) evaluation to objectively describe the motility patterns.

Timed Barium Esophagram (TBE)

TBEs were performed in the upright position with x-ray images of the esophagus obtained at one, two, and five minutes after ingestion of 200-ml of low-density (45% weight to volume) barium sulfate. The height of the barium column was measured vertically from the EGJ.

Statistical Analyses

All data were imported into SPSS v. 24 for Macintosh (Chicago, IL) for analyses. Preliminary descriptive statistics evaluated the dataset for normality to determine need for non-parametric tests. Continuous variables are presented as Mean (SD) while categorical variables are shown as percentage (N). Patients are dichotomized as those with pre- and post-surgical intervention assessments (PREPOST) or post-surgical assessment only (POST). PREPOST and POST participant data are entered into the reliability, validity, and principle component factor analysis (PCFA) analyses. Cronbach alpha evaluated the internal consistency of the ESS and split-half reliability was determined via the Guttman statistic (standard acceptable cutoffs = 0.70). PCFA with varimax rotation determined the subscale structure of the ESS with a minimum Eigen value of 1.0 to determine relevant factors. Minimal acceptable sample size standards for PCFA of 20 responses for each scale item (20 x 4 = 80) was determined a priori²⁰. Inter-item correlations are calculated between ESS items to evaluate multicollinearity (standard acceptable cutoff of r > 0.70). Convergent and content validity is evaluated via a series of Pearson’s correlations with the GERDQ, BEDQ, HRQOL, and physiological data. For PREPOST participants, paired samples t-Tests and effect sizes (Cohen’s d standard cutoffs: small ≤ 0.2, medium 0.2 – 0.7, large ≥ 0.8) evaluated significant changes in ESS, GERDQ, BEDQ, HRQOL, and physiological data before and after achalasia treatment. Statistical significance is set to p < .05 for all analyses; we did not correct for multiple correlations.

Results

A total of 107 patients with achalasia were identified via the initial registry query; of these, 83 had only complete post-treatment evaluations (POST) and 24 had complete pre-and post-treatment evaluations (PREPOST). Demographic and clinical characteristics of the study sample by group are outlined in Table 1.

Table 1.

Demographic and Clinical Characteristics of Study Sample by Group

	POST (N=83)	PREPOST (N=24)
Gender
Male	44.6 (37)	66.7% (16)
Female	55.4 (46)	33.3% (8)
Age	51.3 (14.9)	51.5 (18.6)
Most Recent Treatment
PD	14.5% (12)	12.5% (3)
LHM	47.0% (39)	25.0% (6)
POEM	38.6% (32)	62.5% (15)
Achalasia Type (Pre-Treatment)
I	-	20.8% (5)
II	-	62.5% (15)
III	-	16.7% (4)
Post-Treatment Motility HRM Pattern
Achalasia I	12.0% (10)	8.2% (2)
Achalasia II	10.8% (9)	0% (0)
Achalasia III	9.6% (8)	12.5% (3)
Absent contractility	37.3% (31)	29.2% (7)
EGJ outflow obstruction	10.8% (9)	4.2% (1)
Distal esophageal spasm	6.0% (5)	4.2% (1)
Ineffective esophageal motility	10.8% (9)	41.7% (10)
Normal	2.4% (2)	0% (0)
ESS	3.57 (2.60)	2.17 (1.76)
GERDQ	7.92 (2.73)	6.96 (2.14)
BEDQ	11.42 (11.10)	3.54 (6.71)
HRQOL	13.05 (8.52)	10.63 (6.39)
TBE 1 min	8.18 (7.33)	5.90 (5.76)
TBE 5 min	5.38 (5.73)	4.43 (5.25)
Median IRP	14.68 (7.58)	12.21 (7.10)
EGJ Pressure	11.92 (7.27)	9.08 (6.95)

Psychometric Properties of the Eckardt Symptom Score

The ESS demonstrated fair internal consistency (Cronbach α = 0.67) and split-half reliability (Guttman statistic = 0.66). From the PCFA, dysphagia accounted for 53% of the variance in symptom score, followed by regurgitation (24%), chest pain (12%) and weight loss (11%). Inter-item correlations range from small and non-significant to modest, indicating no multicollinearity present in the ESS, but also suggesting that weight loss may be unrelated to the other ESS items (Table 2). Convergent and content validity of the ESS is supported by modest and significant correlations with the GERDQ, HRQOL, and physiological data with the exception of EGJP (Table 3). Individual ESS item correlations with questionnaire and physiological data are of mixed size and significance, with weight loss the most consistently showing weak or no relationship with other study measures.

Table 2.

Inter-Item Correlations for Eckardt Symptom Score Items

Item	1	2	3	4
1 Dysphagia	-	0.53**	0.49**	0.22*
2 Regurgitation		-	0.55**	0.16
3 Chest Pain			-	0.08
4 Weight Loss				-

^*P < .05

^**P < .01

Table 3.

Item Level Pearson’s Correlations for ESS, GERDQ, BEDQ, HRQOL, and Physiological Data

	Total	Dysphagia	Regurgitation	Chest Pain	Weight Loss
GERDQ	0.43**	0.28**	0.46**	0.45**	0.08
Regurgitation		0.38**	0.73**	0.49**	0.22*
Epigastric Pain		0.22*	0.27**	0.26**	0.16
Heartburn		0.31**	0.40**	0.50**	0.10
Nausea		0.20*	0.35**	0.31**	0.14
Sleep Disturbance		0.24*	0.42**	0.42**	0.05
Medication Increase		0.23*	0.22*	0.36**	0.09
BEDQ	0.78**	0.72**	0.82**	0.53**	0.24*
Dysphagia		0.73**	0.70**	0.55**	0.23*
Chest Pain		0.63**	0.66**	0.45**	0.27**
Impactions		0.44**	0.47**	0.26**	0.12
ER Visits		0.10	0.15	0.17	−0.06
HRQOL	−0.54**	−0.42**	−0.45**	−0.42**	−0.22*
Physical		−0.37**	−0.23**	−0.19	−0.22*
Mental		−0.38**	−0.24*	−0.19	−0.23*
TBE 1 min	0.28**	0.28**	0.23**	0.15	0.13
TBE 5 min	0.41**	0.40**	0.31**	0.19	0.23*
Median IRP	0.24*	0.28**	0.25**	0.13	−0.01
EGJ Pressure	0.15	0.13	0.16	0.14	−0.03

^*P < .05

^**P < .01

Mean Change and Effect Sizes for Eckardt Symptom Score Following Intervention

Follow-up evaluations for the PREPOST group were obtained at a median (interquartile range) 12 (8–14) months after treatment. Evaluation of the ESS between pre- and post-surgical assessment demonstrates significant improvements across the total score and each item, with the exception of weight loss (Table 4). Effect sizes are moderate to large, with the greatest clinically significant change in regurgitation score, followed by dysphagia, chest pain then weight loss. Changes in physiological, GERDQ, BEDQ, and HRQOL variables are of similar size and range.

Table 4.

Paired Samples t-Test and Effect Sizes for ESS, Physiological Variables, GERDQ, BEDQ, and HRQOL for Pre- and Post-Surgical Assessment

N = 24	Baseline Mean (SD)	Post-Surgery Mean (SD)	t (23)	P	Cohen’s d
Eckardt Total	6.63 (2.83)	2.17 (1.76)	6.71	.000	1.89
Dysphagia	2.38 (0.77)	0.88 (0.90)	6.43	.000	1.79
Regurgitation	1.92 (0.93)	0.21 (0.42)	8.38	.000	2.37
Chest Pain	1.13 (0.95)	0.50 (0.51)	3.72	.001	0.83
Weight Loss	1.21 (1.18)	0.58 (1.02)	1.81	.083	0.57
TBE Column Height (cm)
1 minute	16.43 (8.05)	5.87 (5.76)	5.26	.000	1.51
5 minutes	11.93 (8.62)	4.43 (5.25)	3.61	.001	1.05
IRP	34.23 (10.97)	12.21 (7.10)	8.24	.000	2.38
EGJ Pressure	31.3 (10.3)	9.1 (6.9)	9.63	.000	2.53
GERDQ	9.00 (3.26)	6.96 (2.14)	2.96	.007	0.74
Regurgitation	2.13 (1.12)	0.21 (0.42)	7.98	.000	2.27
Epigastric Pain	1.17 (1.20)	0.33 (0.70)	3.39	.003	0.86
Heartburn	0.96 (1.12)	0.50 (0.83)	2.11	.046	0.47
Nausea	1.29 (1.23)	0.25 (0.53)	3.92	.001	1.10
Sleep Disturbance	1.38 (1.21)	0.29 (0.55)	4.25	.000	1.16
Medication Increase	1.00 (1.29)	0.54 (0.93)	1.47	.156	0.41
BEDQ	28.17 (14.87)	3.54 (6.71)	7.70	.000	2.13
Dysphagia	15.88 (8.57)	1.63 (3.59)	7.80	.000	2.17
Chest Pain	8.79 (4.97)	1.08 (2.06)	7.36	.000	2.03
Impactions	3.21 (2.3)	0.83 (1.76)	4.31	.000	1.16
ER Visits	0.29 (0.46)	0.00 (0.00)	3.08	.005	0.89
HRQOL	16.61 (9.70)	10.94 (7.00)	3.44	.003	0.67
Physical	4.56 (3.96)	3.67 (3.55)	1.04	.315	0.24
Mental	3.39 (2.81)	2.72 (2.76)	1.22	.277	0.24

Discussion

While the Eckardt Symptom Score is a widely used, gold standard measure of achalasia symptom severity, this is the first study to systematically evaluate its reliability and validity in a well-defined patient population. Overall, the ESS demonstrates fair reliability, with measures of internal consistency and split-half reliability falling just under the standard acceptable cutoff score of 0.70. This may simply be a statistical artifact of the short nature of the ESS (4 items)²¹. However, based on inter-item correlations, we see moderate and significant relationships between each ESS item with the exception of weight loss, suggesting the weight loss question may be decreasing the ESS reliability. Weight loss also demonstrates the smallest correlations across other study measures and the smallest effect size in the PREPOST intervention cohort compared to other ESS items.

Weight loss is not as commonly encountered in achalasia as other symptoms captured by the ESS, so the smaller range of reported scores (and less potential for improvement after surgery) may, in turn, reduce the ESS’s reliability. Weight loss is the only objectively measured variable on the ESS, so less correlation with subjective responses is somewhat predictable. The ESS also does not specify intentional versus unintentional weight changes, which may confuse some patients when responding. Dietary modifications²², whether self-directed or under the guidance of the patient’s treatment team, are not captured by the ESS and were not collected as part of this study. Future studies should incorporate dietary changes to help understand not only how the weight loss item may impact the ESS, but also how diet may influence the other ESS symptom items.

Additionally, there is potential for variability in the ESS based on assessment points. In the present study, the ESS specifies weight loss from symptom onset, if done at baseline, or since surgery if measured at follow-up. However, no timeframe is included in ESS description of the measure. Thus, assessment times could vary substantially between patients. Lastly, obese patients with achalasia may manage to eat in spite of their esophageal dysfunction or patients may opt for high calorie, easy to eat foods such as ice cream which would translate to weight maintenance or even gain²³.

As expected for a short measure, the ESS consists of one uniform scale of which over 50% of the variance is explained by the dysphagia item. Both weight loss and chest pain account for a significantly smaller amount of the variance (around 10% each) in the ESS, suggesting further the weight loss item may be problematic, but also suggesting the chest pain item may require review. This finding is likely related to the potential multifactorial nature of esophageal chest pain, which may include obstruction of bolus flow and subsequent esophageal stasis, as well as spastic esophageal contractions. While the outflow obstruction is the primary target of achalasia interventions, which may also improve esophageal stasis, esophageal spasm may still persist and generate symptoms after treatment, particularly if an extended myotomy is not performed. Additionally, patients may be susceptible to gastroesophageal reflux following otherwise successful treatment that can manifest as chest pain and may not be differentiated from other mechanisms of chest pain in achalasia. Interestingly, the ESS chest pain item has only small, yet significant, correlations with the GERDQ epigastric and BEDQ chest pain items; large correlations exist with the ESS dysphagia and regurgitation items and the corresponding items on the GERDQ and BEDQ. Regurgitation (and chest pain) related to esophageal outflow obstruction and stasis can be difficult to distinguish from those related to reflux.

Construct validity for the ESS is supported by modest correlations with the GERDQ, BEDQ, HRQOL, and physiological data. At item level analyses, the chest pain and weight loss items demonstrate several small and non-significant correlations, most notably with all physiological data with the exception of weight loss and TBE at the 5-minute interval. Based on these data, 50% of the items on the ESS are not related to standard physiological assessment of achalasia severity.

Overall, the ESS is a fair measure of achalasia symptom severity with its strengths being in the assessment of dysphagia and regurgitation, items that consistently performed well when assessed for reliability and validity. The apparent weakness of the ESS lies in the chest pain and weight loss items. This is logical based on the fact that chest pain is heterogeneous in achalasia and may not correlate with improved bolus clearance. Although weight loss is probably a good indicator of poor outcome in achalasia, it may be affected by multiple factors and is likely not scaled properly on the ESS to reflect the positive outcome that is associated with weight gain.

There are limitations to the present study that should be considered when interpreting its results. While achalasia is a rare disease, the present study sample is smaller than desired for psychometric evaluations, especially PCFA and internal consistency. However, the short nature of the ESS likely offsets some of these statistical issues. Test-retest reliability was not measured in the present study, so we are unable to comment on the ESS’s temporal stability.

To improve evaluation of achalasia symptom severity, refinement of the ESS may be necessary. Based on FDA guidelines, the ESS should undergo further evaluation via cognitive interviews with individual achalasia patients to understand their interpretation and answers to each item, a process undertaken for more recently developed measures such as the Northwestern Esophageal Quality of Life Scale (NEQOL)²⁴ and the Eosinophilic Esophagitis Quality of Life Scale for Adults (EoE-QOL-A)²⁵. Additional focus groups may be needed to more comprehensively capture the clinical presentation of chest pain and weight loss symptoms in achalasia patients, thereby strengthening the construct validity of the ESS.

Key Points.

• The Eckardt Symptom Score (ESS) is the gold standard assessment for achalasia and was developed prior to current guidelines for patient reported outcome measure development.

• The ESS demonstrates fair internal consistency and split-half reliability. Validity is supported by modest relationships with measures of esophageal symptoms and physiological data. Weight loss and chest pain appear to have weak psychometric properties.

• As the ESS is widely used, item refinement may be warranted to ensure the most accurate assessment of patient outcomes.