Scoring the Penetration–Aspiration Scale (PAS) in Two Conditions: A Reliability Study

Abstract

A widely applied metric for identifying airway invasion events is the Penetration–Aspiration Scale (PAS). PAS scores are often included as primary outcome measures in clinical interventional studies, applied to characterize airway protection in a particular disease, used to establish a normal referent for control group comparisons without dysphagia, and as determinants or predictors of clinical outcomes. Despite the widespread use of the PAS, there is variability in scoring condition. One common method used in research studies includes rater scores applied to each single swallow that occurred during a modified barium swallow study (MBSS) of the same patient. A second common method includes raters scoring single swallow segments that have been spliced from full MBSS from different patients. These single swallow segments are then randomly distributed and the rater is blinded to all swallows that occurred during that patient MBSS. The potential effects of different scoring conditions on rater reliability and score accuracy have not been studied and may have high relevance for the conclusion drawn from the result. The primary aim of this investigation is to determine the impact of two scoring conditions on rater reliability and score accuracy: 1. Contextual, unblinded scoring condition and 2. Randomized, blinded condition. Results of the present study show that no statistically significant differences in PAS rater reliability and score accuracy were found between the two scoring conditions. If findings from this pilot study are reproduced in larger sample sizes, the time and intensity involved in splicing and randomizing MBSS for scoring may not be necessary.

Introduction

Dysphagia, or disordered swallowing, impacts 1 in 25 adults every year in the United States [1, 2] and leads to serious health complications including malnutrition, dehydration, and aspiration pneumonia [3–5]. Penetration and Aspiration, characterized as varying degrees of airway invasion of ingested material, are major sequalae of a swallowing impairment that increase the likelihood of aspiration pneumonia [4, 6–8]. Penetration is operationally defined as invasion of ingested material or pathogens into the laryngeal vestibule (i.e., airway) to a level above the vocal folds, and aspiration, a more severe outcome, refers to airway invasion of material below the vocal folds into the trachea. As such, detection of aspiration and penetration and most importantly, the physiologic cause of airway invasion events are critical outcomes of a modified barium swallow study (MBSS), the most commonly applied diagnostic method that uses videofluoroscopic imaging of deglutition. MBSS permits visualization of critical structural movements of swallowing relative to flow of standardized liquid, semisolid, and solid barium consistencies, and allows the detection of swallowing impairment and airway invasion in real time [9–12].

A widely applied metric for identifying and rating the severity of penetration and aspiration events during MBSS is the Penetration–Aspiration Scale (PAS) [13]. The PAS is a rank ordered scale that classifies the depth of penetrated or aspirated material and the patient’s response to airway invasion.

PAS scores are often included as a primary outcome measure in clinical interventional studies [14–18], applied to characterize airway protection in a particular disease [19–24], and used to establish a normal referent for control group comparisons without dysphagia [25–28]. Despite the widespread use of the PAS, there is variability in scoring condition. One common method used in research studies includes rater scores applied to each single swallow that occurred during the course of a MBSS of the same patient. In this condition, the rater is not constrained by factors such as order of scoring and has visual access to all the swallows produced during the MBSS. A second common method includes raters scoring single swallow segments that have been spliced from full MBS studies from different patients. These single swallow segments are then randomly distributed and the rater is blinded to all other swallows that occurred during that patient MBSS. The potential effects of different scoring conditions on rater reliability and accuracy of PAS scores have not been studied and may have high relevance for the conclusion drawn from the result.

The Penetration–Aspiration Scale (PAS)

The PAS was initially developed and tested by Rosenbek and colleagues in 1996 to describe aspiration and penetration events [13]. It is an 8-point ordinal scale, with 1 representing the least and 8 representing the highest or most severe score. PAS scores are multidimensional, i.e., include several observations within each score: (1) depth of airway invasion (material above, contacting, or below the level of vocal folds; (2) whether or not there is material remaining after the swallow (ejected, not ejected); and (3) the patient’s response to material present in the airway (effort to clear the material) (Table 1).

Table 1.

8-point Penetration-Aspiration Scale (PAS) [13]

Score	Description of events
1	Material does not enter airway
2	Material enters the airway, remains above the vocal folds, and is ejected from the airway
3	Material enters the airway, remains above the vocal folds, and is not ejected from the airway
4	Material enters the airway, contacts the vocal folds, and is ejected from the airway
5	Material enters the airway, contacts the vocal folds, and is not ejected from the airway
6	Material enters the airway, passes below the vocal folds, and is ejected into the larynx or out of the airway
7	Material enters the airway, passes below the vocal folds, and is not ejected from the trachea despite effort
8	Material enters the airway, passes below the vocal folds, and no effort is made to eject

Variability in implementation of the PAS is not limited to scoring conditions. The clinical perception and statistical analyses of PAS scores also vary considerably between clinicians and research studies [29, 30]. A recent systematic review [31] thoroughly examined the discrepancies in statistical approaches and reporting practices when applying the PAS. The review [31] divided studies into three categories based on whether PAS scores were treated as ordinal, interval, or categorical variables in their statistical analysis. In addition, the review highlighted inconsistencies in reporting measures of reliability, power analyses and study protocols from which the PAS scores were derived (e.g., consistencies, bolus size).

Scoring Conditions

A scoring condition is operationally defined as the visual context during which a rater applies a PAS score. Two of the most commonly used PAS scoring conditions in clinical research laboratories include (1) Contextual, unblinded scoring characterized by rating swallow tasks that occurred during the course of the entire MBSS of the same patient, without any blinding of scores that precede or follow the targeted swallow task [25, 28, 32], and (2) Randomized, blinded scoring characterized by scoring randomized single swallow tasks with no information regarding swallowing performance that occurred prior to and/or following the targeted swallow task [13, 24, 27, 33].

The primary hypothesis underlying this pilot investigation is that PAS scoring condition influences rater reliability and accuracy of scores. In a Contextual scoring condition, we predict that rater reliability and accuracy will be higher given the circumstantial information provided from the patient’s overall swallowing performance when compared to the Randomized scoring condition in which the rater is blinded to surrounding swallowing performance. To test our hypothesis, this current investigation aimed to determine the impact of the two commonly used PAS scoring conditions on rater reliability and score accuracy: 1. Contextual, unblinded scoring condition and 2. Randomized, blinded condition.

Methods

Participants

Seven graduate students participated as student raters. In addition, four certified speech-language pathologists participated as clinician raters. The Institutional Review Board determined that the proposed study was exempt from human subject review (IRB ID: STU00207458). Inclusion criteria for participation included completion of (1) a graduate course in swallowing physiology and pathophysiology, (2) the Modified Barium Swallow Impairment Profile (MBSImP^™) online course [34, 35], and (3) a PAS scoring training video. Student raters had some observational experiences with patients with dysphagia but had no experience in scoring the PAS prior to participation in the study. Clinician raters had an average of 8.5 years (range = 5–13 years) of experience in scoring the PAS. In addition to student and clinician raters, two certified speech-language pathologists with an average of 20 years of experience in scoring the PAS participated as “gold standard” raters.

Material

The sample included 15 MBSS digital recordings that were retrospectively extracted from a larger database of a heterogenous patient population referred for a routine swallowing evaluation (e.g., head and neck cancer, stroke, pulmonary, and general ENT). The MBSS were selected based on their representation of the full range of PAS scores (Table 2).

Table 2.

Frequency of PAS^* scores across 15 MBSS^* digital records

PAS score	1	2	3	4	5	6	7	8	Total
Frequency	45	16	22	1	12	14	8	8	126

^*Penetration-Aspiration Scale

^*Modified barium swallow study

All MBSS were conducted in a standard fluoroscopy suite at one location using the MBSImP protocol [35]. Continuous fluoroscopy was employed, and digital recordings were made with a resolution of 60 fields (30 frames) per second. The TIMS 2000 SP DICOM System (TIMS Medical, Chelmsford, Massachusetts, USA) was used for signal acquisition, digital storage, and retrieval of the swallowing data. Patients were imaged in an upright, seated, or standing position. The MBSImP protocol includes 12 swallow tasks via presentations of varying bolus consistencies, bolus volumes, and presentation methods. Ten swallow tasks are captured in a lateral view, and 2 in an anterior–posterior (AP) view. However, PAS scores are only applied to swallow tasks captured in the lateral portion of a MBSS.

Patients were positioned in a lateral plane, and regions of visualization included the oral cavity, pharyngeal cavity, larynx, and cervical esophagus. The fluoroscopic angle was 70° and the visualization field included the lips anteriorly, nasal cavity superiorly, cervical spinal column posteriorly, and the entire pharyngoesophageal segment inferiorly [9, 11, 36, 37].

Patients were administered standardized, commercial preparations of barium contrast agents (VARIBAR® barium sulfate 40% weight/volume; Bracco Diagnostics, Inc., Monroe Township, NJ) that included thin (< 15 cps) barium (two trials of 5 ml via teaspoon, one cup sip [20 mL], and sequential swallows from cup [40 mL]), nectar (150–450 cps) barium (one trial of 5 mL via teaspoon, one cup sip [20 mL], and sequential swallows from cup [40 mL]), thin-honey (800–1800 cps) barium (one trial of 5 ml via teaspoon), pudding (4500–7000 cps) barium (one trial of 5 mL via teaspoon), a solid (one-half portion of a Lorna Doone shortbread cookie coated with 3 mL pudding barium) in the lateral view.

The total radiation exposure averaged 3 min. The average duration for the patient’s participation during the MBS procedure, including set-up and positioning, was 15 min [35, 38].

Each MBSS followed the full MBSImP protocol whenever possible depending on the swallowing severity of the patient. That is, the clinicians who conducted the swallowing studies refrained from evaluating certain consistencies and/or bolus sizes for safety reasons. Therefore, the full protocol was not completed for all MBSS included in the analysis but allowed for the full range of PAS scores to be included in the dataset.

MBSS Samples for Contextual Scoring Condition

Fifteen digitized records from the lateral portion of an entire MBSS using the MBSImP protocol were de-identified and converted from a Digital Imaging and Communications in Medicine (DICOM) format into an Audio Video Interleave (AVI) format using TIMS DICOM Review Software (TDRS) (Foresight Imaging, Chelmsford, MA). Raters used Quick-Time Player 7 to view and score each MBSS in its entirety.

MBSS Samples for Randomized Scoring Condition

The same 15 digitized records from the lateral portion of the entire MBSS were trimmed into single swallow tasks (n = 126). The reduction in single swallow tasks from 150 (10 lateral view tasks/MBSS) to 126 was due to incomplete MBSImP protocol administration. Patients were unable to tolerate the full protocol because of swallowing severity concerns. Each swallow task was assigned a unique code to blind raters from viewing swallow tasks that proceeded or followed the targeted swallow task for scoring. Raters in the Randomized condition were informed of the bolus consistency type and volume for each scored swallow.

Procedure

Pre-Scoring Phase

Each rater preceded their scoring by reading two classic articles [13, 28] describing the PAS and viewing a 30-min training video that included illustrations of relevant anatomical landmarks, operational definitions, scoring guidelines, and video examples of each PAS score (Fig. 1). Raters had unlimited access to the training video throughout their participation.

Fig. 1.

Modified schematic representation of the PAS [13]

Scoring Phase

Raters accessed and scored the MBSS records for both the Contextual and Randomized scoring conditions through a secured server using laptop computers (n = 126 swallow tasks/scoring condition). Quick Time Player 7 was used to permit forward and backward, frame-by-frame capability view which raters were encouraged to utilize while scoring the PAS. The duration and order of scoring swallowing tasks within each scoring condition were not controlled. Raters had unlimited access to the MBSS records within the period designated for each scoring condition (2 weeks).

Inter-rater Reliability

A cross-over design was applied to assess score agreement between raters, i.e., inter-rater reliability. Student and clinician raters (N = 11) were divided into two groups to counter-balance scoring conditions and control for practice and order effects. The first group (N = 6) scored in the Contextual scoring condition first, and the second group (N = 5) scored in the Randomized scoring condition first, and then the groups switched. A two-week rest period of no scoring was inter-posed between scoring conditions to minimize recall bias.

Intra-rater Reliability

To assess score agreement within each rater, a 4-week period of no scoring was allotted before raters (N = 10) rescored 25% of total swallow tasks from each scoring condition. One student rater did not complete intra-rater portion of the study.

PAS Score Accuracy

For accuracy, a measure of score precision and a reflection of a true observation, PAS scores were compared to the “gold standard” scores in the Contextual and Randomized scoring conditions. The two “gold standard” clinician raters consensus scored all MBSS records in both scoring conditions.

Statistical Analysis

Intraclass Correlation Coefficients (ICCs) were used to determine (1) inter-rater reliability, (2) intra-rater reliability, and (3) PAS score accuracy. ICC is a reliability index calculated through analysis of variance to detect the effect caused by raters, observations (e.g., PAS scores), and random error [39, 40]. There are various versions of ICCs depending on the study design and research question/s. In the current investigation, we applied a single-rating, absolute agreement, 2-way random effect ICC to determine inter-rater reliability and PAS score accuracy. For intra-rater reliability, a single-rating, consistency, 2-way mixed effect ICC was applied [39–42]. ICC estimates and their 95% Confidence Intervals (CIs) were calculated using R software version 3.5.2 (2018–12-20).

Missing Data

Fourteen swallowing tasks were omitted (i.e., listwise deletion) from student raters data analysis (n = 224). Two student raters did not complete scoring all swallow tasks due to reported scoring uncertainty related to difficulty visualizing the videofluoroscopic image. Based on “gold standard” scores, the 14 missed swallow tasks included a range of PAS scores. In the Contextual scoring condition, 3 (PAS = 1), 1 (PAS = 2), 1 (PAS = 3), 4 (PAS = 6), and 1 (PAS = 8) scores were missing. In the Randomized scoring condition, 1 (PAS = 4) and 3 (PAS = 7) scores were missing.

Clinician raters scored all swallowing tasks in both scoring conditions with no missing data (n = 252).

Results

Student Raters (N = 7)

PAS scores applied in the Randomized scoring condition showed a higher inter-rater reliability (ICC = 0.67, CI 0.60–0.74) compared to scores applied using the Contextual scoring condition (ICC = 0.55, CI 0.45–0.65) (Table 3). Similarly, the mean ICC for intra-rater reliability was higher when scores were completed in the Randomized scoring condition (ICC = 0.76) compared to the Contextual scoring condition (ICC = 0.73) (Table 4). ICC estimates were also higher for PAS score accuracy, when raters used the Randomized scoring condition (ICC = 0.74) compared to the Contextual scoring condition (ICC = 0.69) (Table 5. These ICC differences in rater reliability and PAS score accuracy between the two scoring conditions were not statistically significant based on 95% Confidence Intervals.

Table 3.

ICC^* estimates and 95% Confidence Intervals (CIs) for inter-rater reliability

Scoring condition	Swallowing tasks	Student raters N = 7
Contextual	112	ICC = 0.55 (CI 0.45–0.65)
Randomized	112	ICC = 0.67 CI 0.60–0.74)

^*Interclass correlation coefficient

Table 4.

ICC^* estimates and 95% Confidence Intervals (CIs) for intra-rater reliability

Scoring condition	Student raters						Mean
	1	2	3	4	5	6
Contextual	0.60 (0.34–0.77)	0.79 (0.62–0.89)	0.92 (0.85–0.96)	0.58 (0.32–0.76)	0.71 (0.50–0.84)	0.81 (0.66–0.90)	0.73
Randomized	0.77 (0.57–0.88)	0.68 (0.43–0.84)	0.85 (0.71–0.92)	0.62 (0.35–0.80)	0.86 (0.73–0.93)	0.79 (0.61–0.90)	0.76

^*Interclass correlation coefficient

Table 5.

ICC^* estimates and 95% Confidence Intervals (CIs) for PAS score accuracy

Scoring condition	Student raters							Mean
	1	2	3	4	5	6	7
Contextual	0.72 (0.62–0.80)	0.60 (0.47–0.71)	0.81 (0.67–0.89)	0.76 (0.68–0.83)	0.61 (0.35–0.76)	0.58 (0.07–0.83)	0.74 (0.64–0.81)	0.69
Randomized	0.78 (0.69–0.84)	0.73 (0.64–0.81)	0.80 (0.53–0.90)	0.81 (0.74–0.87)	0.77 (0.67–0.84)	0.63 (0.37–0.78)	0.70 (0.60–0.79)	0.74

^*Interclass Correlation Coefficient

Clinician Raters (N = 4)

Clinician raters showed an equivalent inter-rater reliability in scores applied using the Contextual and Randomized scoring conditions (ICC = 0.70, CI 0.57–0.80, 0.58–0.79, respectively) (Table 6). For intra-rater reliability, the mean ICC was higher when scores were completed in the Randomized scoring condition (ICC = 0.86) compared to the Contextual scoring condition (ICC = 0.80) (Table 7). For PAS score accuracy, ICC estimates were slightly higher when clinician raters used the Contextual scoring condition (ICC = 0.78) compared to the Randomized scoring condition (ICC = 0.76) (Table 8). ICC differences in rater reliability and PAS score accuracy between the two scoring conditions were not statistically significant based on 95% Confidence Intervals.

Table 6.

ICC^* estimates and 95% Confidence Intervals (CIs) for inter-rater reliability

Scoring condition	Swallowing tasks	Clinician raters N = 4
Contextual	126	ICC = 0.70 (CI= 0.57–0.80)
Randomized	126	ICC = 0.70 (CI= 0.58–0.79)

^*Interclass correlation coefficient

Table 7.

ICC^* estimates and 95% Confidence Intervals (CIs) for intra-rater reliability

Scoring condition	Clinician raters				Mean
	1	2	3	4
Contextual	0.93 (0.84–0.98)	0.77 (0.48–0.91)	0.63 (0.25–0.84)	0.88 (0.71–0.95)	0.80
Randomized	0.89 (0.75–0.96)	0.91 (0.78–0.96)	0.80 (0.56–0.92)	0.83 (0.61–0.93)	0.86

^*Interclass correlation coefficient

Table 8.

ICC^* estimates and 95% confidence intervals (CIs) for PAS score accuracy

Scoring condition	Clinician raters				Mean
	1	2	3	4
Contextual	0.79 (0.71–0.85)	0.81 (0.69–0.88)	0.82 (0.76–0.87)	0.73 (0.33–0.87)	0.78
Randomized	0.80 (0.71–0.86)	0.80 (0.65–0.88)	0.78 (0.69–0.84)	0.67 (0.32–0.82)	0.76

^*Interclass correlation coefficient

Discussion

The current study aimed to examine the potential effects of different PAS scoring conditions applied in research investigations (Contextual unblinded vs. Randomized blinded) on rater reliability and score accuracy. We hypothesized that rater access to swallowing performance that occurs prior to and after the targeted swallow would improve rater’s performance, i.e., reliability and accuracy. The lack of significant difference in either rater reliability or accuracy between conditions, however, was not supportive of our clinical hypothesis. In a recent study, the influence of randomization on the analysis of temporal kinematic swallow events was examined to determine whether there was an advantage of random analysis compared to sequential analysis of temporal kinematic measures [43]. In their investigation however, they hypothesized that randomization was more advantageous due to the fact that it reduces measurement or judgement biases that may potentially occur from sequential analysis. Similar to our results, no significant differences were found in measurements performed in random or sequential order.

In this pilot investigation, student raters had no previous experience in scoring the PAS and were provided with equal resources prior to scoring the studies. Controlling experience and training in our student raters was intended to reduce potential biases on measuring the effect of different scoring conditions. Yet, a limited number of studies have shown a relationship between improvement in PAS rater reliability with increased clinical experience [32, 44, 45]. As such, the addition of clinician raters to our pilot investigation aimed to test whether clinical experience in scoring PAS has an effect on rater reliability and/or score accuracy when applying different scoring conditions. However, the lack of significant differences between the two scoring conditions holds even for experienced clinicians; therefore, the time and intensity involved in trimming and randomizing MBSS may not be necessary for either reliable and/or accurate PAS scoring in clinical research investigations.

In clinical practice, unlike research studies, speech-language pathologists (SLPs) typically apply the PAS while viewing the full MBSS for each patient, and thus score the PAS using a similar form of the Contextual scoring condition. Despite the fact that the PAS was designed to rate single swallows and not overall performance, SLPs do not necessarily score every single swallow task within a complete MBSS, but rather review the patient’s overall swallowing performance and assign a maximum (i.e., most severe) PAS score for each bolus consistency (e.g., thin, nectar, honey) and/or swallow task (e.g., cup sip, sequential swallow). PAS max scores, along with the physiologic cause(s) of the aspiration and/or penetration are considered when developing appropriate treatment plans. Findings from a recent study support this practice and show that when reported with MBSImP Impairment composite scores, PAS max scores contribute to the probability of accurate oral and pharyngeal physiologic swallowing impairment classification (mild, moderate, severe) [46].

Throughout the consensus scoring process completed by our “gold standard” raters, several ambiguities were identified within the tool’s scoring guidelines such as the time point in the swallow when the score should be applied, should aspiration of residue from a previous swallow task be scored within the current swallow, and several others. As such, our research laboratory developed specific guidelines with operational definitions for scoring (Appendix 1 in Table 9). Given the lack of PAS standardization and operational definitions, clinicians are encouraged to calibrate their scoring reliability on a regular basis and consider incorporating consensus scoring as possible to improve reliability and PAS score accuracy.

Study Limitations

The main limitations in this current investigation are the small sample size (i.e., MBSS) and prevalence of PAS scores. A considerable effort was made to avoid prevalence bias and include a balanced frequency of PAS scores (Table 2); however, higher, most severe PAS scores are typically associated with more severe patients in which MBSS protocols are not fully administered due to safety reasons. This resulted in a tradeoff between sample size and PAS score frequency, and though clinically representative it is not statistically balanced and may have potentially influenced rater’s performance in each scoring condition. Therefore, future studies with larger sample sizes and adequate demonstration of the full range of PAS scores are needed to further investigate the effect of PAS scoring conditions on rater reliability and score accuracy.

Conclusions

In this pilot investigation, no statistically significant differences in PAS rater reliability and score accuracy were found between the two scoring conditions. If findings from this pilot study are reproduced in larger sample sizes, the time and intensity involved in splicing and randomizing MBSS for scoring may not be necessary.

Appendix

See Table 9.

Table 9.

PAS scoring guidelines and operational definitions

When to score?	Apply your PAS score at the end of each swallowing event within a swallow task (i.e., larynx at rest, airway open). For example, if material enters the laryngeal vestibule, to a level above the vocal folds, and the swallow event is complete:
	Score 2 or 4 (depending on depth) if you no longer detect residue in the laryngeal vestibule
	Score 3 or 5 (depending on depth) if you detect residue in the laryngeal vestibule
What to report?	Score the maximum within each swallow task. For example, if a patient swallowed three times during a swallow task (e.g., cup sip nectar) assign the worst PAS score across all three swallows for that specific task
How to score penetrated or aspirated residue from a previous swallow task?	Residue from a previous swallow task should not be considered when assigning a PAS score for the current swallow task. Newly presented contrast material is bounded by the leading edge of the bolus. Any penetration or aspiration event that is the result of residue from the preceding swallow task AND occurs before the leading edge of the bolus reaches the laryngeal surface of the epiglottis should only contribute to the PAS score for the preceding swallow task
	For example: Swallow 1 receives a score of 3 with residue remaining above the level of the vocal folds. Before the leading edge of the bolus from Swallow 2 reaches the laryngeal surface of the epiglottis, residue from Swallow 1 progresses below the level of the vocal folds with no effort to eject. The PAS score for Swallow 1 becomes an 8
PAS = 2 Material enters the airway, remains above the vocal folds, and is ejected from the airway	Anatomical boundaries include the apex of the arytenoid cartilages. Score 1 if material progresses to but does not pass the apex of the arytenoid cartilages
	Score ≥ 2 if material passes beyond the apex of the arytenoid cartilages and onto the anterior laryngeal surface of the arytenoids
PAS = 6 Material enters the airway, passes below the vocal folds, and is ejected into the larynx or out of the airway	Anatomical boundaries include the undersurface of the true vocal folds
	Score 6 if material contacts the undersurface of the true vocal folds and is ejected into the laryngeal vestibule or beyond
	Score ≥ 7 if material adheres to the undersurface of the true vocal folds or progresses into the trachea
PAS =7 Material enters the airway, passes below the vocal folds, and is not ejected from the trachea despite effort	Timing of effort to eject (e.g., cough, throat clear) is observed within the duration of each swallow task segment (e.g., from the aspiration event until fluoro is turned off). Any attempt to clear aspirated material contributes to the score if it occurs within the defined segment, regardless of delay. Do not consider cued attempts (by the SLP) to clear bolus material when deriving your score