How Important Is Randomization of Swallows During Kinematic Analyses of Swallow Function?

Abstract

Purpose

In dysphagia research involving kinematic analyses on individual swallow parameters, randomization is used to ensure judges are not influenced by judgments made for other parameters within the same swallow or by judgments made for other swallows from the same participant. Yet, the necessity of randomizing swallows to avoid bias during kinematic analyses is largely assumed and untested. This study investigated whether randomization of the order of swallows presented to judges impacts analyses of temporal kinematic events from videofluoroscopic swallow studies.

Method

One hundred twenty-seven swallows were analyzed from 18 healthy adults who underwent standardized videofluoroscopic swallow studies. Swallows were first analyzed by two trained raters sequentially, analyzing all kinematic events within each swallow, and then a second time in random order, measuring one kinematic event at a time. Intrarater reliability measurements were calculated between random and sequential swallow judgments for all kinematic events using intraclass correlation coefficient and percent exact agreement within a three-frame tolerance.

Results

Intraclass correlation coefficients (1.00) and percent exact agreement (89%) were excellent for all kinematic events between analyses methods, indicating there were no significant differences in measurements performed in random or sequential order.

Conclusions

This study provides preliminary evidence that randomization may be unnecessary during temporal swallow kinematic data analyses for research, which may lead to more efficient analyses and dissemination of findings, and alignment of findings with clinical interpretations. Replication of this design with swallows from people with dysphagia would strengthen the generalizability of the results.

Within the clinical realm, randomization is an important element for both study design and data analyses. For example, researchers may randomize the order of procedures to eliminate an order effect, or they may randomize the order of measurements performed on collected data to reduce the likelihood of measurement bias. The double-blinded, randomized controlled trial is considered to be the gold standard for research experiments (Broglio, 2018; Ferreira & Patino, 2016; Fonarow, 2016; Naci, 2017). Randomization is an important clinical trial design element that is frequently used to assign participants to treatment conditions to determine the effect of interventions. When implemented correctly, randomization can be useful for reducing selection bias, ensuring comparability across groups, and avoiding confounding variables in order to more definitively determine treatment effects during a clinical trial (Broglio, 2018; Ferreira & Patino, 2016; Fonarow, 2016; Naci, 2017). While there are numerous benefits to double-blinded randomized controlled trials, they often have stringent enrollment criteria (limiting the participants who are eligible to enroll), require more rigorous research routines and infrastructure to achieve high internal validity, and are not always feasible depending on the research question being investigated. These clinical trial requirements result in limited external validity/generalizability of results to the clinical setting.

Similar to clinical trial design, which involves randomization of participants to treatment conditions to minimize selection bias, randomization of swallows and/or measurements is standard practice in swallowing research for data analyses to avoid judgment bias. When trained raters complete kinematic analyses of temporal swallowing events, swallows are analyzed in random order to ensure that judgments from one swallow or measurement within a swallow do not influence judgments of other swallows or measurements from the same participant. For example, in well-designed and validated methods of swallow kinematic analyses and clinical ratings, such as the Modified Barium Swallowing Impairment Profile (MBSImP) and the Analysis of Swallowing Physiology: Events, Kinematics, and Timing, swallow randomization was used during analyses to ensure objective measurements (Martin-Harris et al., 2008; Steele et al., 2019). Other research studies that have conducted temporal kinematic swallow measures report swallow randomization as part of their data analyses protocol, as well (Herzberg et al., 2018; Mendell & Logemann, 2007). Similarly, within the Computational Deglutition Lab at the University of Pittsburgh, measurement randomization is part of the protocol that is used for all raters conducting kinematic analyses.

Although randomization is standard practice within the Computational Deglutition Lab, we recently discovered that a subset of our data was inadvertently analyzed in sequential order instead of random order. Instead of discarding this data set, this incidental error inspired us to conduct a systematic, preliminary research investigation to determine the importance of using randomization for swallow kinematic analyses. While swallow randomization is a research procedure that is frequently used when swallow kinematic measures are being performed, its necessity remains unknown. Similar to clinical trial designs that use random assignment of participants to groups, swallow randomization, a common practice in dysphagia research, is likewise believed to result in a reduced threat of systematic measurement error by judges performing several measurements arising from individual events (e.g., a swallow), though the literature provides little substantiation of this presumption (Akobeng, 2008; Dekkers et al., 2010). When clinicians analyze videofluoroscopy images, they consider the entire videofluoroscopic exam sequentially rather than measuring all necessary parameters one at a time among a data set. This is believed to introduce measurement bias due to judgments of other events within a swallow being known by the judge. Therefore, this study investigated whether randomization of swallows impacts analyses of temporal kinematic swallow events from videofluoroscopic swallow studies (VFSSs) to determine whether randomly ordering swallows and measures during analyses (measure-by-measure analyses) is advantageous compared to sequential (swallow-by-swallow) analyses. We investigated the question: Does randomization of swallows impact analyses of temporal kinematic events from VFSS images?

Method

We analyzed 127 swallows from 18 healthy community-dwelling adults (21–74 years of age) who underwent standardized VFSSs after they provided written informed consent. The data used for analyses were collected at the University of Pittsburgh Medical Center Presbyterian Hospital as a part of an ongoing research study examining swallowing physiology in healthy community-dwelling adults across the life span. All participants underwent standardized VFSSs to minimize radiation exposure (average fluoroscopy time of 0.66 min) and were imaged in the lateral plane. The standardized VFSS protocol consisted of 10 thin liquid boluses that were presented in a randomized order (five at 3 ml via spoon, five unmeasured self-selected volume cup sips). The instruction given to participants by the researcher for presentations by spoon was “Hold the liquid in your mouth and wait until I tell you to swallow it,” and the instruction for presentations by cup was “Take a comfortable sip of liquid and swallow it whenever you're ready.” Participants enrolled in the study had no history of swallowing difficulties, neurological disorder, or surgery to the head or neck region according to participant report. All female participants reported not being pregnant.

VFSS images used for data analyses were obtained using a standard fluoroscopy system set at a pulse rate of 30 pulses per second (PPS; Precision 500D System, GE Healthcare, LLC). The raw videos were captured at a higher sampling rate (73 frames per second) through a frame grabber module that did not compress or preprocess the images (AccuStream Express HD, Foresight Imaging). The video sampling rate was higher than the pulse rate to align with Shannon's sampling theorem for the signal processing work that we do in our lab (unrelated to this study; Oppenheim & Schafer, 2014) and because we are simultaneously collecting data from electronic sensors that require a higher sampling rate. Since kinematic analyses of swallow events were performed prior to down-sampling of the data to 30 frames per second, trained raters blinded to each other's measurements selected the first of duplicate frames to ensure consistent and accurate measurements of temporal swallowing measures.

Following data collection, trained raters in our lab began and completed kinematic analyses on 127 swallows before it was discovered that they were analyzing data in sequential rather than randomized order (as is the protocol within our research laboratory). To ensure that temporal kinematic measurements were not impacted by analyzing them in sequential rather than random order, the raters recoded all kinematic measurements from all swallows in random order. The bolus characteristics of the swallows included in data analyses for this study are summarized in Table 1.

Table 1.

Bolus characteristics for all healthy community-dweller swallows used for data analyses.

Bolus viscosity and utensil	No. of swallows	Percentage of swallows
Thin by spoon	68	53.5
Thin by cup	59	46.5

Note. Thin by spoon swallows were 3 ml, and thin by cup swallows ranged from 2 to 40 ml.

For the purposes of this study, sequential order for data analyses was defined as analyzing the swallows in the same order that they were collected during data collection, and random order was defined as analyzing the swallows in random order using a random number generator. For the swallows analyzed in sequential swallow-by-swallow order, all temporal swallow kinematic measures (see Table 2) within each individual swallow were performed before performing the same measurements on the next swallow. For the randomly ordered swallows analyzed on a measure-by-measure basis, judges coded one kinematic event across all swallows in the data set at a time (e.g., upper esophageal sphincter [UES] opening and closing for all videos), moving on to the next measure for all randomly ordered swallows. The judges first performed all sequential swallow-by-swallow measurements. Following this, trained judges remeasured the same events from the randomly reordered swallows one kinematic event at a time (measure by measure). Intraclass correlation coefficients (ICCs) and percent exact agreement within a three-frame (0.1-s) tolerance was used to calculate intrarater reliability between sequential and random swallowing judgments for all kinematic events (Lof & Robbins, 1990).

Table 2.

Definitions of temporal swallow kinematic events.

Swallow kinematic event	Definition
Bolus crosses mandible	The first frame in which the organized bolus head first reaches or crosses the plane of the ramus of the mandible and is associated with oral propulsion
Onset of hyoid movement	The first movement of hyoid leading to maximal hyolaryngeal excursion
Maximal hyoid displacement	The first frame in which the hyoid is at its maximally displaced position (superior and anterior) during the pharyngeal phase
Offset of hyoid movement	The first frame in which the hyoid is clear and in a stable position for at least two frames after descent at the end of the swallow (the bolus will typically have passed through the UES)
Laryngeal vestibular closure	The first frame in which no air or barium contrast is seen in the collapsed laryngeal vestibule
Laryngeal vestibular re-opening	The first frame in which the laryngeal vestibule reopens
UES opening	The first frame in which separation of the posterior and anterior walls of the UES has begun
UES closure	The first frame in which no column of air or barium contrast is seen separating the posterior and anterior walls of the UES

Note. UES = upper esophageal sphincter.

Prior to performing data analyses, all judges were trained and tested in all temporal kinematic swallow measures summarized in Table 2 (i.e., bolus passes mandible, onset of hyoid movement, maximal hyoid displacement, offset of hyoid movement, laryngeal vestibular closure, laryngeal vestibular reopening, UES opening, UES reclosing). To avoid drift across measurements, trained judges completed ongoing intrarater reliability during coding by randomly selecting one swallow to reanalyze following analysis of every set of 10 swallows. Ongoing interrater reliability during coding was measured and maintained by having another blinded rater randomly reanalyze 10% of every 100 swallows.

Results

ICCs for interrater reliability in the training data set was 1.00. There were no significant differences between measurements performed sequentially (swallow by swallow, all measures per swallow) and those performed on a measure-by-measure basis from randomly reordered swallows. Interrater reliability was excellent for ICCs (1.00) and percent exact agreement (89%) between sequential and randomized data analyses methods (see Table 3).

Table 3.

Intrarater reliability for sequential and randomized coding of all swallow kinematic events.

Swallow kinematic event	ICC	PEA
Bolus crosses mandible	1.00	98%
Onset of hyoid movement	1.00	97%
Maximal hyoid displacement	1.00	89%
Offset of hyoid movement	1.00	77%
Laryngeal vestibular closure	1.00	66%
Laryngeal vestibular re-opening	1.00	97%
UES opening	1.00	99%
UES closure	1.00	89%

Note. ICC = intraclass correlation coefficient; PEA = percent exact agreement; UES = upper esophageal sphincter.

Discussion

This study investigated the effects of randomization when trained judges perform temporal measurements of swallow kinematic events from videofluoroscopy images. We found that there was no significant difference between measurements of temporal swallow kinematic events when they were analyzed by trained judges in sequential order, all kinematic events within one swallow, or random order, one kinematic event at a time. The results from this study contribute to a growing body of literature regarding best practices and standardization for conducting videofluoroscopic examinations and for analyzing swallow kinematic events following the examination (Burns et al., 2015; Martin-Harris et al., 2008; Martin-Harris & Jones, 2008; Peladeau-Pigeon & Steele, 2013; Steele et al., 2019; Thompson et al., 2018). In order for VFSSs to be most useful within clinical and research settings, it is imperative to identify the aspects of conducting and analyzing VFSSs that are most important for ensuring accurate, timely, and nonbiased judgments. For example, it is recommended that VFSSs are completed at a rate of 30 PPS, because reduced pulse rates (15 PPS) have been shown to impact judgments of swallowing impairment and treatment recommendations (Bonilha et al., 2013; Cohen, 2009). While some studies have shown that both pulse and frame rates (30 PPS or 30 frames per second vs. lower pulse and/or frame rates) during VFSSs impact the accuracy of image analyses, there is limited research evidence that exists regarding best practices for the conduct of VFSS image analyses (e.g., frames per second, screen calibration, ambient lighting, randomization). A recent study examined the impact of recording rate, screen calibration, and ambient lighting on penetration–aspiration scale scores and MBSImP component scores (Mayer et al., 2017). In line with our study's findings, results revealed no significant differences in penetration–aspiration scale or MBSImP scores across conditions (recording rate, screen calibration, and ambient lighting). While this study and our research study were preliminary in nature, they provide initial evidence that some analyses conditions (e.g., randomization) may be unnecessary for conducting temporal measurements of swallow function.

While the majority of research laboratories analyze VFSS images on a swallow-by-swallow (one kinematic event at a time) basis, analyzing swallows in sequential order (all kinematic events within one swallow) may be beneficial for both researchers and clinicians. Sequential kinematic analyses of swallowing may increase the efficiency of data analyses and improve the external validity of research studies by mirroring how VFSS image analyses are conducted in ordinary clinical settings, which would increase the generalizability of research findings. For example, many clinicians use the MBSImP (a clinical ordinal scale that rates 17 physiological components of swallowing) to rate VFSS images. Rather than having clinicians rate 17 components for each swallow during a VFSS, these individual component scores are combined across all swallows to obtain total oral impairment and pharyngeal impairment scores. Therefore, analyzing swallows sequentially within research settings may more closely replicate swallow analyses in clinical settings. Increased external validity and generalizability of research findings may be especially important as the health care industry pushes clinicians toward greater reliance on evidence-based practice patterns and researchers to pursue research work that produces real-world evidence to answer relevant research questions that have immediate generalizability to patients (Maissenhaelter et al., 2018).

Limitations

While the results from this study are promising, there are several limitations. We used a relatively small subset of our large database for this preliminary study. Future studies should examine the importance of randomization for swallow kinematic analyses with a larger data set. In addition to this, our data set consisted of only healthy community-dwelling adults across the life span that underwent videofluoroscopy in a standardized manner. This limitation likely skewed/compressed the range of judgments within each measure toward the higher (i.e., less impaired) end of the normal-impaired continuum. Hence, replication of our methods using patient VFSS data is warranted to confirm their validity in the clinical setting and to further assess the necessity of the practice of performing measurements using only strictly randomly reordered exemplars.

Conclusion

This study found no significant difference between swallow kinematic measurements of temporal measures (hyoid onset, UES opening, etc.) when VFSS swallows from healthy people were rated by trained judges in sequential order, all kinematic events within one swallow, or in random order, one kinematic event at a time. The results from this study provide preliminary evidence that it may be unnecessary to randomize swallows for data analyses of swallow kinematic events, which may result in more efficient data analyses and result in improved external validity by using methods that are more reflective of clinical practice.

Funding Statement

Research reported in this publication was supported by Eunice Kennedy Shriver National Institute of Child Health & Human Development Awards R01HD092239-1 (awarded to Ervin Sejdic) and 2R01HD074819-04 (awarded to James L. Coyle and Ervin Sejdic).