Refining measurement of swallowing safety in the Dynamic Safety Grade of Swallowing Toxicity (DIGEST) criteria: validation of DIGEST version 2

Abstract

Background:

DIGEST is a validated method to grade the severity of pharyngeal swallowing impairment as a toxicity of cancer based on degree and patterns of penetration/aspiration and pharyngeal residue over a standardly acquired radiographic modified barium swallow (MBS) study. Since implementation in 2016, areas for refinement of grading mild safety impairment were identified by clinical and research users. The objective was to assess performance and validity of refined DIGEST_safety grading criteria (per DIGEST_v2).

Methods:

Refined safety criteria were developed and vetted with clinical and research users. DIGEST_v2 includes two changes to safety criteria. All MBS with blinded DIGEST_v1 grading were sampled from a registry database (1,331 patients underwent MBS over the period 12/2005 to 07/2019). New criteria were applied to derive DIGEST_safety grade version 2. Measures of criterion validity, including MDADI composite score, MBSImP^™© Pharyngeal Total, MBSImP^™© hyolaryngeal components (items 8–11), and PSS-HN diet, were correlated with DIGEST_safety and overall DIGEST grade from v1 and v2, and compared pairwise between re-assigned grades.

Results:

With application of v2 Safety criteria, 112/1,331 (8.4%) and 79/1,331 (5.9%) of exams changed in DIGEST_safety and overall grades, respectively. Safety and overall DIGEST grades v1 and v2 significantly correlated with criterion measures including MBSImP pharyngeal total, laryngeal MBSImP parameters of interest, MDADI, and PSS-HN (p<0.0001); correlations maintained similar magnitude between v1 and v2. 46 exams up-graded (re-assigned from Safety grade 1 per version 1 to grade 2 per version 2) performed more similarly to other safety grade 2 (v1), and likewise for 66 exams down-graded (re-assigned from Safety grade 1 per version 1 to grade 0 per version 2).

Conclusions:

Refined criteria defining mild safety impairment using the DIGEST methodology changed grades in small numbers of examinations. DIGEST_v2 criteria maintained criterion validity, demonstrated ordinality, and improved performance of the method in these rare scenarios.

Lay summary

DIGEST is a method developed and validated by the investigators in 2016 to grade the severity of pharyngeal swallowing dysfunction (dysphagia) using a decision-tree or flowsheet to guide clinician’s review of a standard radiographic modified barium swallow study. This work reports the validity of updated (version 2) DIGEST criteria that incorporate two modifications to the decision-tree.

Precis

Version 2 of DIGEST (DIGEST_v2) incorporated two modifications to the decision-tree used by clinicians to grade mild (grade 1) safety impairment based on patterns of laryngeal penetration/aspiration on a radiographic modified barium swallow study. Key findings of this work include: 1) criterion validity of DIGEST_v2 relative to references measures of clinician-graded pharyngeal swallowing impairment and patient-reported dysphagia, and 2) ordinality of DIGEST_v2.

INTRODUCTION

DIGEST^™ (Dynamic Imaging Grade of Swallowing Toxicity) is an analysis method developed to grade the severity of pharyngeal dysphagia based on results of videofluoroscopy commonly referred to as the modified barium swallow study (MBS). DIGEST uses a basic flowsheet and rubric as a decision tree to summarize the patterns of penetration/aspiration and pharyngeal residue observed on the MBS as markers of swallowing safety and efficiency. This process provides a reproducible way to achieve a summary grading of dysphagia from the modified barium swallow observations into an overall severity grade of impairment on a 5-point scale - in accordance with National Cancer Institute’s (NCI) CTCAE toxicity framework¹.

The initial development and validation publication in a sample of 100 head and neck cancer patients pre- and post-treatment found DIGEST was reliable (intra- and inter-rater weighted k 0.82–0.84 and 0.67–0.81, respectively) and demonstrated construct validity relative to reference measures of pharyngeal pathophysiology per MBSImP^™© (r = 0.77, p<0.001), perceived dysphagia per MDADI (r = −0.41, p<0.001), and diet per PSS-HN normalcy of diet subscale (r = −0.49, p<0.001)¹. DIGEST subsequently proved sensitive to longitudinal change after head and neck radiotherapy², head and neck surgery³, and swallowing therapy⁴, and was found to associate with RT dose to swallowing regions of interest² as detailed in Supplementary Table 1.

The analysis steps raters use to derive the DIGEST grade from the MBS are detailed in Figure 1. Visuoperceptual measures of swallowing safety (per penetration-aspiration scale scores)⁵ and efficiency (per estimated percentage of pharyngeal residue) are first assigned for each bolus in a standard protocol. Worst performance on penetration-aspiration scale and pharyngeal residue is classified in nominal, ordered bins, then a decision tree is used to assign DIGEST_safety and DIGEST_efficiency grades ranging from 0 (no impairment) to 4 (life threatening/profound impairment) that align to CTCAE framework. Finally, a 2×2 rubric derives the summary or overall grade of pharyngeal dysphagia (DIGEST) based on the interaction of DIGEST_safety and DIGEST_efficiency.

Figure 1.

Version 1 DIGEST criteria (Hutcheson KA, et al, Cancer, 2017) to derive grade of dysphagia (per MBS)

DIGEST is a simple, but robust tool with evidence of uptake in clinical and research practice^6–11. Since its implementation began in 2016, clinical and research speech-language pathology users of DIGEST identified areas for refinement specifically related to the grading of mild safety impairment. The objective of this work was, therefore, to assess the performance of refined DIGEST safety grading criteria in DIGEST version 2 (DIGEST_v2) and structural validity of DIGEST_v2. We hypothesized that DIGEST_v2 would maintain significant positive correlations with reference measures of clinician-graded pharyngeal dysphagia severity (per MBSImP^™©) and significant negative correlations with swallowing-related QOL (per the MD Anderson Dysphagia Inventory, MDADI) and diet (per the Performance Status Scale of Head and Neck Cancer, PSS-HN). We further hypothesized that DIGESTv2 would demonstrate ordinality in reference to clinician-graded dysphagia severity (per MBSImP).

METHODS

DIGEST version 2 refinements

Refined safety criteria were developed based on iterative feedback from real-world implementation of DIGEST in clinical and research settings. The impetus for refining the decision-tree for safety grading was the observation by real-world users that two patterns of penetration/aspiration events graded as mild in the original DIGEST flowchart frequently did not match the clinician’s perceived severity of dysphagia or clinical management of dysphagia in those scenarios. Revised DIGEST_v2 criteria were vetted and refined with users from multiple institutions. Two primary changes were made to the decision flowsheet for safety criteria in DIGEST_v2, as summarized in Figure 2.

• First, frequency modifiers were added for examinations in which the maximum penetration-aspiration scale (PAS) score is 3–4. In DIGEST_v2, a single PAS 3–4 bolus trial is now DIGEST_safety grade 0 (formerly Safety grade 1 in DIGEST_v1).

• Second, an additional decision node was added for examinations in which the maximum PAS is 7–8 occurring on a single bolus trial. In DIGEST_v2, a single bolus trial scored PAS 7–8 in an examination with additional bolus trials scored PAS 5–6 is now DIGEST_safety grade 2 (formerly DIGEST_safety grade 1 in DIGEST_v1).

Figure 2. Safety grading criteria per DIGEST v1 and v2.

Refinements in v2 depicted in red.

Other minor clarifications were made to language in the decision tree without changing structure or criteria. No structural changes were made to the DIGEST_efficiency flow chart or the DIGEST_safety by DIGEST_efficiency rubric.

MBS acquisition details

MBS studies were acquired as routine, clinical procedures according to a standard protocol. The standard bolus protocol included 2 trials each of 5-mL, 10-mL, and self-administered sips from a cup of thin liquid barium (Varibar; Bracco Diagnostics, Inc), barium pudding (Varibar; Bracco Diagnostics, Inc), and a cracker coated in barium pudding. MBS images were recorded at 30 frames per second with audio synchronization (TIMS® Medical, Foresight Imaging, Chelmsford, MA).

Sample selection

The sample was drawn from the Communication, Hearing, and Swallowing Outcomes in Oncology Patients registry database (PA11–0849) at MD Anderson Cancer Center, Houston, Texas, USA. MBS examinations with blinded laboratory DIGEST grading conducted in patients with diagnosed or suspected head and neck cancer (pre- or post-treatment) over the period of December 2005 to July 2019 were consecutively sampled deriving n=2,420 MBS in 1,331 patients. DIGEST_v1 was previously graded on all MBS studies by blinded, trained laboratory speech pathology raters who previously met reliability standards (>80% exact agreement). New criteria were applied to derive DIGEST_v2 Safety grade then overall DIGEST_v2 grade for analysis.

DIGEST_v2 validation

Reference measures of criterion validity included MBSImP^™© Pharyngeal Total, MDADI composite scores, and PSS-HN Normalcy of Diet subscale, as used in the original DIGEST validation. MBSImP pharyngeal total was the primary criterion reference measures representing a summary of pharyngeal phase swallowing physiology based on summation of 10 components rated on ordinal scale with higher scores reflecting greater impairment (possible range: 0 to 29 pharyngeal total; 0 to 4 component scores)¹². MDADI is a widely used 20-item patient reported outcome instrument that measures swallowing-related quality of life with higher scores reflecting better QOL (possible range: 20 to 100). The 19-item MDADI composite score (weighted average of 19-items excluding global item) was analyzed¹³. PSS-HN Normalcy of Diet subscale represents range of oral intake on an 11-point scale where 0 reflects no oral intake and 100 reflects regular oral diet¹⁴.

We additionally examined individual hyolaryngeal physiologic components because of their proposed role in mechanism of swallowing safety. These included laryngeal elevation (range 0–3), anterior hyoid excursion (range 0–2), epiglottic movement (range 0–2), and laryngeal vestibule closure (range 0–2), all graded on a 3 to 4-point ordinal scale in which higher scores represent greater impairment.

Statistical analysis

Distributions of DIGEST safety, efficiency and overall grades were descriptively summarized and visually inspected. 2×2 tables were constructed to visualize the concordance between versions; discordance represents the number and proportion of MBS that changed grade (DIGEST_safety and/or overall DIGEST grade) with application of v2 criteria. Spearman’s correlation coefficients with 95% CI were computed to assess strength of criterion validity for DIGEST (v1 and v2) in reference to other swallowing-related outcome measures including MBSImP pharyngeal total, MDADI composite, and PSS-HN, as well as between MBSImP hyolaryngeal components and DIGEST safety grade (v1 and v2). Variance in correlation coefficients between v1 and v2 was inspected. Associations between DIGEST with MBSImP and MDADI measures were assessed using the Kruskal-Wallis test with the Dwass-Steel-Critchlow-Fligner procedure for multiple pairwise comparisons. The chi-square test with the Bonferroni correction was used for all other criterion measures due to fewer categories risking many ties in rank-based tests. Pairwise comparisons between grades were evaluated for evidence of ordinality. For the primary analysis, the sample was restricted to a single MBS per patient (last MBS in the study period) because the variability between ratings made on different patients is expected to exceed the variability between ratings on a single patient, thus multiple MBS per patient may inflate degrees of freedom in statistical testing. A sensitivity analysis was conducted that included all MBS in the dataset (n=2,420 MBS in 1,331 patients).

RESULTS

Clinical history and demographics of the sample are summarized in Table 1.

Table 1.

Patient characteristics (n=1,331)

Characteristic	Value
Sex, No. (%)
Male	1133 (85.1)
Female	198 (14.9)
Age at first MBS, mean, y	60.2
Tumor site, No. (%)
OPC	1072 (80.5)
Oral cavity	8 (0.6)
Larynx	70 (5.3)
Hypopharynx	21 (1.6)
Nasopharynx	16 (1.2)
CUP/NOS	125 (9.4)
Other	19 (1.4)
Chemotherapy, No. (%)	907 (68.1)
Surgery, No. (%)	232 (17.4)
Radiation, No. (%)	1115 (83.8)
No treatment, No. (%)	12 (0.9)
Multimodality, No. (%)	933 (70.1)

Abbreviations: CUP, cancer unknown primary; MBS, modified barium swallow; NOS, not otherwise specified; OPC, oropharynx cancer

Distribution of DIGEST grades

With the application of v2 DIGEST-Safety criteria, 112/1,331 (8.4%) of MBS examinations changed in DIGEST_safety grade, while 79/1,331 (5.9%) of exams changed in overall DIGEST grade as depicted in Table 2. Forty five of 1,331 (3.4%) of examinations were down-graded, and 34/1,331 (2.6%) were up-graded.

Table 2.

Cross-tabulation between DIGEST grade distributions using version 1 and version 2 criteria (n=1,331)

SAFETY1 (version 1)	SAFETY2(version 2)
	0	1	2	3	4	Total
0	719	0	0	0	0	719
1	66	191	46	0	0	303
2	0	0	157	0	0	157
3	0	0	0	142	0	142
4	0	0	0	0	10	10
Total	785	191	203	142	10	1331
DIGEST1 (version 1)	DIGEST2 (version 2)
	0	1	2	3	4	Total
0	496	0	0	0	0	496
1	36	349	14	0	0	399
2	0	9	190	20	0	219
3	0	0	0	196	0	196
4	0	0	0	0	21	21
Total	532	358	204	216	21	1331

Abbreviations: DIGEST, Dynamic Imaging Grade of Swallowing Toxicity; MBS, modified barium swallow

Criterion validity

As expected, overall DIGEST grade and DIGEST_Safety grade (v1 and v2) significantly correlated in the expected direction and magnitude with all tested criterion measures, including all MBSImP parameters of interest, MDADI composite score, PSS-HN Normalcy of Diet subscale score, and hyolaryngeal physiologic MBSImP item component scores (p<0.0001) with similar magnitude of correlations between v1 and v2 as shown in Table 3. Figure 3 depicts stable performance of DIGEST_v1 and DIGEST_v2 for both overall DIGEST grade and DIGEST_safety grade in reference to MBSImP pharyngeal total as the primary criterion measure of interest.

Table 3.

Spearman’s rho and 95% Confidence Intervals (CI) to Assess Criterion Validity of Versions 1 and 2 DIGEST_Safety and Overall DIGEST Grades with Reference Measures of Swallowing Function (n=1,331)

	n	SAFETY version 2				SAFETY version 1				DIGEST version 2				DIGEST version 1
		r	95% CI			r	95% CI			r	95% CI			r	95% CI
Sum of MBSImP items	1323	0.57	0.53	0.61		0.56	0.52	0.60		0.61	0.57	0.65		0.62	0.58	0.65
MDADI composite score at time of MBS	1175	−0.44	−0.49	−0.40		−0.45	−0.50	−0.40		−0.49	−0.53	−0.44		−0.49	−0.54	−0.45
Normalcy of diet (PSS-HN)	1323	−0.31	−0.36	−0.26		−0.29	−0.34	−0.24		−0.37	−0.42	−0.32		−0.37	−0.42	−0.32
Laryngeal elevation	1323	0.46	0.42	0.51		0.46	0.42	0.50		0.45	0.40	0.50		0.45	0.41	0.50
Anterior hyoid excursion	1323	0.25	0.20	0.30		0.24	0.19	0.29		0.26	0.21	0.31		0.27	0.21	0.32
Epiglottic movement	1323	0.51	0.46	0.56		0.49	0.44	0.53		0.50	0.46	0.55		0.50	0.45	0.54
Laryngeal vestibule closure	1323	0.58	0.54	0.62		0.58	0.54	0.62		0.55	0.51	0.59		0.55	0.51	0.59

Abbreviations: DIGEST, Dynamic Imaging Grade of Swallowing Toxicity; MBS, modified barium swallow; MBSImP, Modified Barium Swallow Impairment Profile; MDADI, MD Anderson Dysphagia Inventory; PSS-HN, Performance Status Scale for Head and Neck Cancer Patients. p <.0001 for all correlations; testing H₀: r = 0. Analysis includes 1 MBS per patient.

Figure 3. Measures of criterion validity by DIGEST grades (v1 vs v2) (n=1,331 MBS).

DIGEST, Dynamic Imaging Grade of Swallowing Toxicity. Analysis includes one MBS per patient.

Ordinality

Results support the ordinality of DIGEST_v2 grades. Specifically, pairwise comparisons between DIGEST_v2 grades and criterion measures (MBSImP pharyngeal total, MDADI, and PSS-HN Diet) revealed statistically significant differences between grades after adjustment for multiple comparison (0 v 1, 1 v 2, 2 v 3, 3 v 4) for criterion measures, with a few exceptions in PRO and diet measures: DIGEST_v2 grades 1 versus 2 do not statistically significantly differ on PSS-HN diet; DIGEST_v2 grades 2, 3, and 4 do not statistically significantly differ on MDADI scores.

Sensitivity analysis

An analysis including all MBS in the study period yielded similar results for the primary analysis of criterion validity as demonstrated in Supplementary Table 2.

DISCUSSION

Swallowing remains a high priority outcome in head and neck cancer. The DIGEST method was developed to offer a pragmatic, reproducible clinician-rated measure of pharyngeal dysphagia severity from the MBS or videofluoroscopic swallow study. DIGEST was developed as an outcome measure for research and clinical practice. The present work represents a step forward for DIGEST, refining DIGEST_safety criteria to more accurately represent mild safety impairment using the DIGEST flowchart methodology. The two changes made to the DIGEST_safety criteria flowchart in DIGEST_v2 changed grades in only a small number of examinations (8% and 6% for DIGEST_safety and overall DIGEST grades, respectively). Continued use of the DIGEST method, therefore, should require minimal change for users to move from version 1 to version 2 grading. DIGEST_v2 criteria maintained criterion validity with improved performance of the method in the rare scenarios in which version 2 changed DIGEST grades, and for the first time, we have confirmed ordinality of the DIGEST grade.

A recent systematic review of 45 studies rated the psychometric quality of visuoperceptual measures from instrumental swallowing studies according to the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) checklist¹⁵. Among 9 studies reporting measures of reliability and validity for videofluoroscopy, DIGEST reportedly had the highest rated evidence overall, yet the authors acknowledged the need for further psychometric analysis specifically pertaining to internal consistency, measurement error, and structural validity. The present work further establishes structural validity of DIGEST demonstrating hypothesized correlations in the expected direction and magnitude. Furthermore, the present analysis confirms, for the first time, the ordinality of DIGEST, with pairwise comparisons finding that each step in DIGEST grade represents significantly worse pharyngeal swallowing (per MBSImP pharyngeal total). The ordinality of DIGEST was not previously reported in the original validation and thus represents an important knowledge gap filled by this analysis.

DIGEST is based on the pattern of airway entry (penetration/aspiration) and pharyngeal residue over a standard set of bolus trials administered in the MBS exam. The bolus protocol used in the present analysis for validation of DIGEST_v2 matched that of the original DIGEST validation study. The validity of DIGEST depends on a bolus protocol that allows detection or discrimination of patterns in the DIGEST flowsheet/decision-tree, with skipped trials limited to the extremes - truly normal swallows or severe or profound dysphagia. While outside the scope of the current investigation, validity of the DIGEST method under alternate bolus protocols represents an area for future work. The investigators posit that a minimum standard bolus protocol necessary to give high confidence of valid DIGEST results should include 5 to 6 uncompensated thin liquid bolus trials (IDDSI 0), and at least a single trial each of pudding thick (IDDSI 4) and dry solid (i.e., cracker or cookie, IDDSI 7). This range of consistencies aligns to industry standards for MBS bolus protocols¹² and matches standardized contrast agents designed for MBS (Varibar; Bracco Diagnostics). The number of thin liquid trials is recommended to allow discrimination between single, intermittent, and chronic episodes of penetration/aspiration, and supported by data suggesting 4 trials necessary to reach saturation on new airway entry¹⁶. As with the original DIGEST, deviation from the protocol is expected in extreme cases of dysphagia and the method maintains validity in this dataset despite skipped bolus trials or early application of strategies/compensations in these cases.

It is plausible that, although relatively minor, the new DIGEST_safety decision nodes and clarifications to language on the DIGEST_v2 form could impact reliability of the measure. Given the nature of the existing dataset used for this analysis, reliability was not tested in this study as version 1 to version 2 conversion was done by applying new syntax to apply the version 2 flowsheet logic to bolus level penetration-aspiration and residue ratings in an existing database. This represents a limitation to address in future work. Methods to enhance and maintain reliability of the measure are of paramount priority to the investigators. The authors are currently undertaking systematic investigation of reliability with emphasis on understanding training conditions and needs supporting reliable adoption of the DIGEST method, dissemination and implementation of a training curriculum, and opportunities for semi-automated methods to enhance training and reliability.

Another potential limitation of this work is the overwhelming proportion of non-surgically treated oropharyngeal cancer patients in the sample. In an NCCN designated comprehensive cancer center, the authors have implemented DIGEST clinically in all MBS studies conducted at the institution since the development of the instrument in 2016 (i.e., >10,000 patients referred for outpatient or inpatient MBS). However, the diversity of the analysis sample for this study was more limited due to sampling from ongoing research studies focused more directly on the population of patients treated with organ preservation strategies (i.e., minimally invasive surgeries and definitive radiotherapy or chemoradiotherapy) in which the majority of cases present with HPV-associated oropharyngeal cancer. While this sampling reflects practice patterns in the investigators’ institution, this raises an important opportunity for external validation among more diverse head and neck cancer populations and more broadly in oncology and other populations with dysphagia. The clinical observation of the investigators is that the validity of DIGEST appears sound in more diverse populations of head and neck cancer patients and oncology patients referred for MBS. This statement is based on the anecdotally perceived ‘match’ of DIGEST to the clinician’s subjective interpretation and clinical decision making, however, requires further research to psychometrically examine the validity in more diverse samples.

Real-world user feedback was the impetus for refinements made to the decision-tree in version 2 of the instrument. The changes were motivated by a perceived mis-match between the grade 1 ‘mild’ safety impairment assigned using the original DIGEST flowsheet and the clinician’s perceived dysphagia severity reflected in differences in clinical management decisions among patients with these two patterns of penetration-aspiration events. These patterns of airway invasion are relatively rare (observed in 8% of exams in this sample) and the adjustment to the safety grade flowchart in these clinical scenarios is supported by the validation statistics presented in this analysis. The perceived severity of swallowing safety impairment is presumed to be related to risk of complications (i.e., pneumonia) and thus should relate to clinical management decisions. The relationship between DIGEST safety grade, health complications, and clinical management decisions is an opportunity for future investigations.

Dissemination and implementation of DIGEST as a clinical and research tool is under way. Investigators have published use of DIGEST in the clinical environment and research laboratories^{1–3,17–19}, primarily in HNC populations. DIGEST has also been published as an outcome measure in patients with neurodegenerative disease^11,20. DIGEST_v2 updates have been included in translations and external validations that are under way, as well as the adaptation of DIGEST for flexible endoscopic evaluation of swallowing (DIGEST-FEES)¹⁸; the version 2 adaptations were already included in the DIGEST-FEES adaptation and the ALS validation.

Conclusion

DIGEST offers a simple, singular grading of swallowing toxicity ideal for use in large scale clinical trials in head and neck oncology as an imaging-based companion to the widely used clinical CTCAE grading. DIGEST_v2 demonstrated criterion validity and ordinality in this analysis with refinements targeting more accurate grading of mild safety impairment. Using the International Classification of Functioning Disability and Health (ICF) framework, DIGEST measures body function impairment domains but complementary measures are necessary to capture changes in activity, participation, and environmental impacts of swallowing function. For this reason, the investigators continue to emphasize the need for coupling the DIGEST results with complementary measures of pathophysiology (e.g., MBSImP, spatiotemporal measures)¹², function (e.g., diet scales such as PSS-HN, FOIS, or IDDSI-FDS)^14,21,22, and patient-reported outcomes (e.g., MDADI, EAT-10, SSQ)^23–25.

Supplementary Material

1

Supplementary Figure 1, Table 1, and Table 2.