Reliability of an Automated High-Resolution Manometry Analysis Program Across Expert Users, Novice Users, and Speech-Language Pathologists

Abstract

Purpose

The purpose of this study was to investigate inter- and intra-rater reliability among expert users, novice users, and speech-language pathologists with a semi-automated high-resolution manometry analysis program. We hypothesized that all users would have high intra-rater reliability and high inter-rater reliability.

Method

Three expert users, 15 novice users, and 5 speech-language pathologists participated in this study. Following a 20-minute training session, users analyzed 30 high-resolution manometry plots using an automated analysis program. Output parameters included two- and three-dimensional pressure integrals over 5 anatomical regions of interest. Intraclass correlations were used to examine inter- and intra-rater reliability. Analysis of variance was also performed to determine any differences in mean output parameter values.

Results

Within-group inter-rater reliability ranged from 0.54-0.99 and inter-group reliability ranged from 0.92-0.99. Intra-rater reliability ranged from 0.67-1.00 across all groups. There were no significant differences of output parameters between groups.

Conclusion

The high reliability observed after a short training session demonstrate that individuals with little to no prior knowledge of swallowing physiology can perform at a similar level as those with expertise. Given the quickness and ease of training in the use of this program, it has the potential for research and clinical utility.

Introduction

The introduction of high-resolution manometry (HRM) to evaluate pharyngeal swallowing has provided dysphagia researchers and clinicians with unique physiological information not otherwise obtainable by other clinically accepted methods, such as videofluoroscopy, fiberoptic endoscopic evaluation of swallowing (FEES), ultrasound, or electromyography. In its brief research history, pharyngeal HRM has been shown to distinguish between normal and disordered swallowing (Geng, Hoffman, Jones, McCulloch, & Jiang, 2013; Mielens, Hoffman, Ciucci, McCulloch, & Jiang, 2012), quantify changes in pressure due to bolus size (Hoffman, Ciucci, Mielens, Jiang, & McCulloch, 2010) or maneuvers (Hoffman et al., 2012; McCulloch, Hoffman, & Ciucci, 2010; Takasaki, Umeki, Hara, Kumagami, & Takahashi, 2011; Takasaki, Umeki, Kumagami, & Takahashi, 2010; Umeki et al., 2009), and serve as an outcome measure following surgical management of dysphagia (Takasaki, Umeki, Enatsu, Kumagami, & Takahashi, 2010). The American Speech-Language-Hearing Association recognizes pharyngeal manometry as an appropriate instrumental assessment of swallowing ((ASHA), 2000, 2004) and has identified it as an emerging area of clinical practice for speech-language pathologists ((ASHA), 2008). Thus, HRM is gaining popularity in the comprehensive swallowing assessments performed by speech-language-pathologists across the United States. Commercially available manometry hardware and software systems, however, are currently marketed for esophageal use and provide limited built-in features that support pharyngeal manometry analysis.

Automated HRM analysis tools have been developed outside of the manufacturer-supplied software that require limited user input to determine regions of interest (Mielens, Hoffman, Ciucci, Jiang, & McCulloch, 2011; Omari et al., 2011). These tools are shown to be efficient in generating comprehensive datasets that are sensitive to changes in pharyngeal pressure patterns associated with disease states (Geng et al., 2013; Mielens et al., 2012) and can detect risk for penetration and aspiration (Omari et al., 2011) and post-swallow residue (Omari, Dejaeger, Tack, Vanbeckevoort, & Rommel, 2012). The reliability of these tools has not yet been systematically investigated with respect to the likeliest intended users: the research assistant and the speech-language pathologist.

In this study, we test inter- and intra-rater reliability of a newly developed pharyngeal HRM analysis tool among expert users, novice users, and speech-language pathologists. We hypothesized that, following a brief training period, both speech-language pathologists with experience analyzing HRM studies and undergraduate research assistants with no previous exposure to HRM output would have high intra-rater reliability and high inter-rater reliability comparable to the reliability for expert users.

Methods

Participants

This study was approved by the University of Wisconsin Health Sciences Institutional Review Board. Program users were separated into three groups: 1) expert users; 2) novice users; and 3) speech-language pathologists. Expert users consisted of two doctoral and one post-doctoral researchers who designed the HRM analysis program, its input parameters, and the output variables (CAJ, MRH, ZG). The experts have an average of 2.6 ± 1.1 years of experience analyzing HRM output, and one is a licensed and certified speech-language pathologist (CAJ). Fifteen undergraduate research assistants (10 females and 5 males; mean age of 21.3±1.3 years) with no experience analyzing HRM data comprised the novice user group. The last group included five licensed and certified speech-language pathologists who specialize in dysphagia evaluation and treatment, have completed a competency program in performing and analyzing pharyngeal HRM studies, and combined have performed over 150 clinical HRM evaluations. The speech-language pathologist group had an average of 11.3 ± 5.6 years of practice with patients with dysphagia and 1.3 ± 0.2 years of experience performing and analyzing clinical HRM studies.

Analysis Program

Data were extracted using a customized MATLAB program (The MathWorks, Inc., Natick, MA), slightly modified from that described in Geng et al. (Geng et al., 2013). Pattern segmentation was based on the selection of regions, rather than selection of individual sensors within a region. Automated calculations of the parameters were then made based on the data in the user-defined, three-dimensional region.

After loading the data file, two-dimensional plots and three-dimensional contours were presented (Figure 1). Five regions of interest were identified for each swallow, requiring four clicks from the user. The five regions of interest are the velopharynx, mesopharynx, pre-opening UES pressure, UES during sphincteric relaxation, and post-closure UES pressure. The user clicks on the regions corresponding to the velopharynx, inferior border of UES high-pressure zone prior to swallow, pre-opening UES pressure drop, and post-closure UES pressure rise. The mesopharynx and UES during sphincteric relaxation regions are identified automatically by the program based on previous region selections by the user. Local pressure extrema in the regions of interest are then identified by the program. A box encompassing the region of interest based on predetermined temporal and spatial constraints is generated and can be modified by the user if so desired.

Figure 1.

Display of user interface portraying a normal swallow. Horizontal red line in spatiotemporal plot represents user input for inferior border of resting UES prior to the swallow. A = Velopharynx; B = Mesopharynx; C = Pre-opening UES zone; D = UES opening; E = Post-closure UES zone.

Regions of interest were defined manometrically using a method similar to that described by McCulloch et al. (McCulloch et al., 2010). The velopharynx is the region of swallow-related pressure change just proximal to the area of continuous nasal cavity quiescence and extending 3 cm distally. The user selects a point midway between the superior and inferior boundaries of velopharyngeal pressure. The nearest sensor, one sensor above, and one sensor below are defined as sensors corresponding to the velopharynx. If a swallow only displays pressure from two sensors in the velopharynx, the box will contain three sensors with one having a pressure reading close to zero, corresponding to the region just above the velopharynx. Provided the region has been selected correctly, this has no impact on parameters calculated from this region. The box corresponding to this region has a duration of two seconds; duration is meant only to ensure all data are recorded and it does not adversely affect temporal or manometric measurements. The mesopharynx is the region of swallow-related pressure change between the velopharynx and UES. It is defined in the program as consisting of all sensors between the velopharynx and anterior boundary of post-closure UES pressure. Duration of pressure events in the mesopharynx is automatically detected by the program. A test window with a 2-second duration is selected first, followed by a background window, also with a 2-second duration, that determines the mean and standard deviation of the background noise. The mesopharynx is defined temporally as the continuous area with pressure higher than the background, with starting and ending times as the first and last points that are above baseline. The inferior border of the UES high-pressure zone is identified by the user as the most inferior sensor that displays pressure at rest before the swallow. This sensor is then used as the inferior border for UES parameter calculations. Pre-opening UES pressure is identified by the user as the most inferior sensor in the UES high-pressure zone at the point where the pressure starts to decline to a nadir. This region has a 1.5-second duration and is defined spatially from the most superior sensor in the UES high-pressure zone to the inferior border of the UES high-pressure zone. Post-closure UES pressure is identified by the user as the most superior sensor that displays pressure rising from a nadir, corresponding to UES closure. Pressure rise from nadir in this region is temporally distinct from that in the mesopharynx. This region has a 1-second duration and is defined spatially from the most superior sensor as the UES starts to close to the inferior border of the UES high-pressure zone. The final region of interest is the UES during sphincteric relaxation, and is defined temporally as the region between the pre-opening and post-closure UES pressure peaks, with spatial constraints of the superior-most sensor that displays pressure corresponding to UES closure and the most inferior sensor that displays pressure at rest before the swallow. This region is identified automatically by the program.

For the purposes of this study, two- and three-dimensional pressure integrals were calculated for all five regions as in Geng et al. (Geng et al., 2013). Two-dimensional pressure integrals were defined as the area under the curve of the single sensor in the region with the highest recorded pressure, and three-dimensional integrals were defined as the total pressure generated in the entire area spanning all sensors.

Training and Testing

The first and second authors participated in training and testing the novice users and speech-language pathologists. Novice users were given a brief introduction to swallowing anatomy and physiology. Novice users and speech-language pathologists were led through a training program to familiarize themselves with the features of the program and region selection. During training, the trainer gave feedback as necessary on accurate or inaccurate selection of regions of interest. The training session lasted between 15-20 minutes, and no data from it were used in the analysis. After the user completed the training, testing began. Users were not limited in time to complete the testing portion of the experiment.

Swallows Included in Analysis

Thirty-four swallows were randomly selected from a database of HRM studies completed in research protocols approved under University of Wisconsin-Madison Institutional Review Board purview. For training, 8 swallows were obtained from normal swallowers and 1 was obtained from subjects with dysphagia (9 swallows in total). For testing, 15 swallows were obtained from normal subjects, 10 swallows were obtained from subjects with dysphagia, and 5 swallows were randomly chosen from the pool of 25 to be repeated to test intra-rater reliability (30 swallows in total). Swallows from the training session were not repeated in the testing session. Demographics of the subjects from which the data were extracted can be found in Table 1. The HRM data included swallows of different bolus consistencies and volumes. The data from each swallow were pooled into one text file, which the analysis program used to create the displays, as seen in Figure 1.

Table 1.

Descriptions of subjects from whom swallows used in analysis were obtained. Age is reported as mean±standard deviation.

Paradigm	Health Status	Number Included	Age (years)	Dysphagia Etiology
Training	Normal	8	28.0±6.5	–
	Dysphagia	1	81	Parkinson's Disease
Testing	Normal	15	33.7± 8.2	–
	Dysphagia	10	69.5±10.5	Cervical web, cricopharyngeal dysfunction (3), vocal fold paralysis, post-radiation dysphagia (2), pharyngeal pouches, unknown (2)

Statistical Analysis

Intrarater reliability was calculated for each rater using intraclass correlation coefficients (ICC) based on the five repeated swallows within the testing set. Interrater reliability within each group of users was evaluated using intraclass correlation analysis as well. Agreement across groups was determined by calculating the ICC across group means (i.e., input data for each group were the mean values across all raters within that group) to avoid potential bias due to unequal group sizes. Analysis of variance (ANOVA) was performed to determine if there were any differences in mean measurements as calculated by the three groups, and a Kruskal-Wallis rank sum test was used to evaluate ICC differences between ratings from normal swallowers and subjects with dysphagia.

Results

Novice users analyzed all 30 swallows in an average of 30.2±11.3 minutes. Speech-language pathologists analyzed all swallows in 31.8±3.83 minutes.

Average intrarater reliability was comparable across the three groups, though highest in the expert group (table 2). Average intrarater reliability across parameters was 0.94±0.08 for expert raters, 0.90±0.13 for novice raters, and 0.90±0.09 for speech-language pathologists. As there were three rater groups and ten parameters, 30 intrarater ICC values were determined; of these, 22 were at least 0.9 or higher.

Table 2.

Intraclass correlation coefficients demonstrating intrarater reliability across rater groups. Values are presented as mean±standard deviation. SLP = speech-language pathologist; 3D = three-dimensional integral; 2D = two-dimensional integral; VP = velopharynx; MP = mesopharynx; UES = upper esophageal sphincter.

Parameter	Expert	Novice	SLP
3D VP (mmHg*s)	0.99±0.02	0.70±0.35	0.86±0.32
3D MP (mmHg*s)	0.75±0.26	0.77±0.29	0.85±0.15
3D Pre UES (mmHg*s)	0.92±0.05	0.99±0.01	1.00±0.00
3D Post UES (mmHg*s)	0.90±0.13	0.94±0.10	0.93±0.13
3D UES (mmHg*s)	0.99±0.01	0.97±0.07	0.97±0.04
2D VP (mmHg*s)	1.00±0.00	0.99±0.02	0.91±0.20
2D MP (mmHg*s)	0.97±0.04	0.67±0.39	0.70±0.34
2D Pre UES (mmHg*s)	0.97±0.01	0.98±0.03	0.87±0.17
2D Post UES (mmHg*s)	0.95±0.05	0.96±0.09	0.97±0.05
2D UES (mmHg*s)	1.00±0.00	0.99±0.02	0.98±0.02

Interrater reliability within each group was also comparable across the three groups (table 3). Average interrater reliability ICC values across parameters were 0.89±0.11 for expert raters, 0.84±0.15 for novice raters, and 0.86±0.13 for speech-language pathologists. As with intrarater reliability analysis, 30 interrater ICC values were determined; of these, 15 were at least 0.9 and 24 were at least 0.8. Interrater reliability was lowest for measurements made for the mesopharynx, though still greater than 0.7 for the three-dimensional integral and greater than 0.48 for the two-dimensional integral.

Table 3.

Intraclass correlation coefficients demonstrating interrater reliability for each group of raters. SLP = speech-language pathologist; 3D = three-dimensional integral; 2D = two-dimensional integral; VP = velopharynx; MP = mesopharynx; UES = upper esophageal sphincter.

Parameter	Expert (n=3)	Novice (n=15)	SLP (n=5)
3D VP (mmHg*s)	0.9348	0.8669	0.8338
3D MP (mmHg*s)	0.704	0.7073	0.7519
3D Pre UES (mmHg*s)	0.9887	0.9706	0.9571
3D Post UES (mmHg*s)	0.8985	0.891	0.8827
3D UES (mmHg*s)	0.9361	0.8547	0.9268
2D VP (mmHg*s)	0.9851	0.9614	0.9424
2D MP (mmHg*s)	0.6551	0.4883	0.5467
2D Pre UES (mmHg*s)	0.9328	0.9385	0.8947
2D Post UES (mmHg*s)	0.9167	0.879	0.9293
2D UES (mmHg*s)	0.942	0.8768	0.9208

There were no differences in mean parameter values determined by the three groups (table 4). Inter-group reliability was high, with ICC values for all ten parameters exceeding 0.92 and eight of the values exceeding 0.98. There were no differences in ICC values of ratings from normal swallowers and subjects with dysphagia (χ²(2, N = 30) = 0.75, p = 0.69).

Table 4.

Mean values for each parameter across groups as well as inter-group reliability assessment using intraclass correlation analysis and analysis of variance (ANOVA). Intraclass correlation coefficients (ICC) demonstrate interrater reliability while F statistics and corresponding p-values represent results of ANOVA evaluating if there were differences in each parameter across groups.

Parameter	Expert	Novice	SLP	ICC	F	p-value
3D VP (mmHg*s)	118±58	116±54	102±55	0.9841	0.2532	0.778
3D MP (mmHg*s)	93±42	106±51	119±60	0.9537	0.6651	0.5219
3D Pre UES (mmHg*s)	221±106	229±106	218±101	0.994	0.0317	0.9688
3D Post UES (mmHg*s)	257±60	263±81	258±84	0.9953	0.0151	0.985
3D UES (mmHg*s)	114±92	127±108	109±99	0.9816	0.0853	0.9185
2D VP (mmHg*s)	53±34	53±32	52±34	0.9969	0.0084	0.9916
2D MP (mmHg*s)	29±11	33±12	35±15	0.9244	0.7711	0.4718
2D Pre UES (mmHg*s)	73±42	77±40	73±41	0.9912	0.0345	0.9662
2D Post UES (mmHg*s)	64±33	68±35	64±34	0.9936	0.0459	0.9552
2D UES (mmHg*s)	42±32	45±34	42±34	0.9901	0.0317	0.9688

Discussion

This is the first study evaluating intra- and inter-rater reliability of intended users (speech-language pathologists and research assistants) against expert users of a novel program for the analysis of high-resolution manometry studies. Reliability was high within users, across users within a group, and across groups of users. The findings demonstrate that after a brief training session, individuals with little to no prior knowledge of swallowing physiology can perform at a similar level as those with expertise.

The high level of reliability is likely due to the numeric, objective nature of the HRM data as well as the program used for analysis. The program was designed to minimize variability of user input; users guided the program to look at certain sensors, and the program then created boxes surrounding the region of interest based on mathematical criteria. The reliability for user-defined regions calculated in this study compares to that found in a study of an impedance, HRM, and videofluoroscopic analysis program (Omari et al., 2011). A commonly cited disadvantage in instrumental swallowing studies is the variability in reliability (Kelly, Drinnan, & Leslie, 2007; Stoeckli, Huisman, & Martin–Harris, 2003) and the amount of training needed to perform various clinical judgments. Use of the Modified Barium Swallowing Impairment Profile (MBSImP) (Martin-Harris et al., 2008) requires 20-25 hours of training and 80% agreement with standard scores to become a registered MBSImP clinician. Our program required only a 15-20 minute training session. This not only speaks to the usability of our method of analysis specifically, but also the intuitive nature of HRM generally. The basic underlying physical principles of swallow-related pressure change and user-friendly two- and three-dimensional displays are easy for any student or clinician with some background in science and data analysis to understand. No other currently widely used swallowing evaluations instrumental swallowing evaluations have demonstrated the ease or quickness in training for novice users with limited knowledge of swallowing physiology.

Although reliability was similar among all groups, inter-rater reliability in the expert group was slightly higher than the speech-language pathologists or novice users. This was expected, given the expert group's intimate knowledge of program design. Speech-language pathologists could be considered to be intermediate users, likely due to their competency in swallowing evaluation and previous experience interpreting HRM spatiotemporal plots. While the speech-language pathologists were proficient in clinical use of HRM, they had never used the program prior to the study. Importantly, the similarities in judgments among groups speaks to the ease of learning and using the program; a novice rater with 20 minutes of experience can perform at a similar level as a speech-language pathologist with 20 years of experience in dysphagia evaluation or even a developer of the program. A thorough understanding of swallowing physiology may be beneficial, particularly for severely disordered swallows; however, it is not a prerequisite for accurate use of the program.

Since all calculations were automated based on user-identified regions, the variability in the data are due to the identification of these regions. The region of interest that showed the lowest reliability was the mesopharynx. This region is not selected by the user but is instead determined automatically according to its relationship to user-defined regions inferior and superior to it. Variability in the interrater reliability, especially with the 2-dimensional integral, could thus be due to inconsistencies in sensor selection for the velopharynx or UES. If a user included a sensor picking up UES pressure in the mesopharynx, it could potentially be chosen by the software to calculate the 2-dimensional integral. This is a potential source of the variability in this parameter. Thus, a clearer definition of the superior boundary of the post-closure UES zone may be needed. Further, this high variability is suggestive that the 2-dimensional integrals may not be as robust against user error as 3-dimensional integrals, which are calculated over an entire region of interest. As user-defined regions had higher reliability, we will also explore whether allowing the user to define this region further improves reliability.

Our knowledge of and the evidence base for pharyngeal HRM are growing rapidly. This tool can bring a new dimension to the clinician's armamentarium, offering objective, functional, and quantitative data. More complex calculations than what can be performed using commercially available manometry software systems, such as integral-based measurements, can increase the clinical utility of these data and guide in clinical decision-making (Geng et al., 2013; Hoffman et al., 2013; Mielens et al., 2012). The integral-based analysis calculated by the automated program in this study provides a complete picture of swallowing physiology that factors in pressure generation at all levels and time points during the swallow.

Conclusion

A novel, automated program for the analysis of high-resolution manometry studies has high intrarater reliability as well as high interrater reliability within and across expert users, novice users, and speech-language pathologists. Complex parameters calculated by this program have the potential to augment clinicians' judgment with little additional invested time. Given the quickness and ease of training in the use of this program, it has the potential for research and clinical utility.