Abstract
Objective
The term “active larynx” is a nonspecific and subjective term used by otolaryngologists to describe laryngeal inflammation that can influence the timing of airway reconstruction. We sought to measure the reliability of visual assessments of laryngeal inflammation for later scale development.
Study Design
A cross‐sectional study.
Setting
Pediatric tertiary care center.
Methods
We created an image library from a direct laryngoscopy and bronchoscopy database. Blinded judges were asked to rate the characteristics of laryngeal inflammation (edema, erythema, cobblestoned appearance, and ventricular eversion; 5‐point Likert scale), the overall “activeness” of the larynx (10‐point scale), and whether laryngeal inflammation would influence a delay in reconstructive surgery (yes/no). A tentative scale was also constructed. Intraclass correlations with 2‐way random effects, and Fleiss's κ were used to evaluate interrater reliability. The convergent and discriminant validity of the tentative scale were measured.
Results
Three pediatric otolaryngologists reviewed 15 larynges for a total of 45 image ratings. Intraclass coefficients indicated substantial agreement for edema (0.76) and erythema (0.83) and moderate agreement for ventricular eversion (0.58). Cobblestoning had low agreement (intraclass correlation coefficient [ICC] < 0.20). The agreement was substantial for overall “activeness” (ICC 0.76) and moderate for whether inflammation would delay surgery (ICC 0.47). By Fleiss's κ, edema and erythema had moderate agreement (0.50 and 0.61, respectively), whereas all others had poor agreement. The convergent and discriminant validity of the tentative scale were reassuring.
Conclusion
While the reliability of laryngeal inflammation by visual assessment is variable, the creation of an active larynx scale appears feasible.
Keywords: laryngeal inflammation, laryngotracheal reconstruction, scale development
Laryngeal inflammation at the time of open airway reconstruction can alter the procedure's success. 1 Operating in an inflamed larynx can alter the chance of a successful long‐term outcome, and this inflammation can be an important determinant of surgical timing. The origin of the inflammation can be congenital or acquired, chronic or acute, and associated with a separate condition—such as gastroesophageal reflux disease or eosinophilic esophagitis 1 —or be undetermined in etiology. 2 , 3 Inflammation appears to be an etiologic factor in stenosis due to intubation, burns, and other disorders. 4 , 5 , 6 Inflammation can induce squamous metaplasia, contributing to stenosis and reducing graft epithelialization. 5 , 7
The etiology of airway inflammation is not always known, so nonspecific elements of laryngeal inflammation are visually assessed by the surgeon considering and planning airway reconstruction. This is sometimes termed the activeness of a larynx, or a patient is said to have an “active larynx.” 2 , 3 However, the activeness of a larynx is an inherently subjective assessment, potentially prone to variability or error. In other words, if 2 surgeons disagree on the activeness of a larynx, they may disagree on the optimal perioperative management or timing.
Despite rarely being mentioned by this name in the literature, 2 , 3 the active larynx may confound differences in laryngotracheal reconstruction (LTR) outcomes. Previous literature has demonstrated only moderate agreement on visual assessments of laryngomalacia 8 and the poor correlation of flexible laryngoscopic findings of reflux and pH‐impedance probe testing. 9 However, there is limited literature on the subjectiveness of the visual assessment of laryngeal inflammation in the context of LTR.
We sought to identify and measure the reliability of characteristics of an active larynx for otolaryngologists planning airway reconstruction. Specifically, we tested the consistency and validity of our methodology for the eventual creation of an active larynx scale. The goal of standardizing this measure is to improve clinical communication and decision‐making in patients requiring surgical intervention to correct airway pathologies. It will also foster future research on the effect of laryngeal inflammation on surgical outcomes.
Methods
Study Design
This study was a cross‐sectional evaluation of direct laryngoscopy and bronchoscopy (DLB) images using 3 judges via an electronic survey. The study was approved by the Institutional Review Board of Nationwide Children's Hospital (STUDY00000032). The study was designed after reviewing the literature and consulting with airway reconstruction experts for content validity.
Judges
Three fellowship‐trained pediatric otolaryngologists served as judges for the DLB image library. All 3 were blinded to all aspects of construction of the survey instrument including selection of images for the library and any associated patient information. The judges all have a demonstrated LTR practice at Nationwide Children's Hospital, a tertiary pediatric care center in Columbus, Ohio.
Survey Ratings
For the pilot study, deidentified DLB images from 15 procedures were used to create a library. All procedures were performed at Nationwide Children's Hospital, and each included a supraglottic and glottic view of the larynx (Figure 1A and B). The images were selected by the authors P.P. and A.M.M. to represent a diverse range of activeness levels and photos that were excessively blurry, not centered, or otherwise difficult to evaluate were excluded. The images were primarily selected from surgical logs in the medical record, as well as newly created tracheostomy and open airway reconstruction databases. Notably, image suggestions were not solicited from colleagues for fear of biasing those colleagues regarding those images in the questionnaire. Both views of each larynx were presented twice (Presentation A and Presentation B), with specific questions for each presentation. The 30 presentations (15 larynges, 2 presentations each) were randomized throughout the survey.
Figure 1.
Example presentation of larynx images: (A) larynx with the lowest total activeness rating by all judges; (B) larynx with the highest rating for edema item; (C) larynx with the highest rating for erythema item; (D) larynx tied for the highest rating for the cobblestoning item; (E) larynx with the highest rating for the ventricular eversion item, tied for the highest rating on the cobblestoning item, and the highest rating for total activeness. Mean scores are discussed in the “Results” section.
Presentation A (Figure 2) asked the respondent to analyze the larynx and rate several characteristics on a 5‐point Likert scale. The following characteristics were surveyed for edema, erythema, cobblestone mucosa, and eversion of laryngeal ventricles. Other features of inflammation, such as crusting, exudate, and the presence of nodules or polyps were initially included but removed after insufficient representation was found in our image library.
Figure 2.
Larynx Presentation A questions.
Presentation B (Figure 3) asked the respondent to analyze the overall “activeness” of the larynx on a scale of 0.0 to 10.0, indicating how active the larynx appeared to be. The respondent was also asked a binary question (yes/no) if they would delay performing airway reconstruction surgery based on how active the larynx was.
Figure 3.
Larynx Presentation B questions.
Statistical Analysis
Data were assessed for internal consistency, and a preliminary scale based on the tested characteristics was presented and tested. In total, there were 4 inflammation characteristic variables (edema, erythema, cobblestoning, and eversion of laryngeal ventricles), 1 variable for overall “activeness,” and 1 for whether the surgeon would delay airway reconstruction. Finally, 1 variable for a tentative active larynx scale was created summing the 4 characteristic variables (with 1 point for each step on the Likert scale), with a minimum value of 4 and a maximum value of 20.
Interrater agreement was evaluated using intraclass correlation coefficient (ICC) and 95% confidence interval (CI) using a 2‐way random‐effects model with both judges and pictures as random‐effects testing for absolute agreement. 10 ICC is comparable to Cohen's κ for assessing interrater reliability with multiple raters and categorical data and is in line with our previous work in scale development for airway pathology. The ICCs were interpreted with standard cutoffs for levels of agreement 11 , 12 : below 0.50 for poor agreement, between 0.50 and 0.75 for moderate agreement, between 0.75 and 0.90 for good agreement, and above 0.90 for excellent agreement. In addition to ICC, Fleiss's κ using quadratic weighting was also calculated to evaluate interrater reliability. 13 The same cutoffs were used for κ, as this would be similar to or more stringent than other cutoffs in the literature. 14 , 15 , 16 Pearson's R was used to evaluate interrater reliability for each pair of the 3 judges (judges 1 and 2, 2 and 3, and 1 and 3) for the overall activeness variable and the activeness scale.
Preliminary measures of construct validity were measured. Convergent validity was measured via correlation of the tentative activeness scale to the overall activeness rating, again using Pearson's R. Next, discriminant validity was tested. A 2‐sample t‐test was used to determine if activeness was significantly different for larynges that surgeons would delay surgery than those they would not. This was performed for both the tentative activeness scale and the overall activeness rating. All analyses were performed using Stata/IC 16 software (StataCorp LP).
Results
All judges reviewed all 15 DLBs, yielding 45 image‐ratings. The overall distribution for all image findings is demonstrated in Table 1. Notably, responses were skewed, with limited responses indicating “moderate,” “very,” or “extreme,” especially for cobblestoning and ventricular eversion. Figure 1A demonstrates the larynx with the lowest rating by the judges (average rating 4.67 on the tentative scale). Figure 1B demonstrates the larynx with the highest rating for the edema item, with an average score of 3.33 from this item (between “moderate” and “very”). Figure 1C demonstrates the larynx with the highest rating for the erythema item (average score 4). Figure 1D demonstrates the larynx tied for the highest score for cobblestoning (average score 2; tied with Figure 1E). Finally, the larynx in Figure 1E had the highest score for the ventricular eversion item (average score 2), tied for the highest score for cobblestoning, and had the highest overall rating (average rating 10).
Table 1.
Distribution of Image Ratings
| Characteristics | None | Slight | Moderate | Very | Extreme |
|---|---|---|---|---|---|
| Edema | 6 | 21 | 16 | 2 | 0 |
| Erythema | 14 | 18 | 8 | 4 | 1 |
| Eversion | 42 | 2 | 1 | 0 | 0 |
| Cobblestoning | 36 | 7 | 1 | 1 | 0 |
| Combined measures | 25th percentile | Median | 75th percentile | ||
|---|---|---|---|---|---|
| Tentative scale (range 4‐20) | 5 | 6 | 8 | ||
| Overall “Activeness” (range 0‐10) | 1.65 | 2.9 | 5 | ||
| Binary measure | Yes | No | |||
|---|---|---|---|---|---|
| Delay surgery? | 9 | 36 | |||
Table 2 demonstrates the interrater reliability of the ratings, through ICC, Fleiss's κ, and Pearson's correlation. By ICC, edema and erythema had a substantial agreement, while ventricular eversion had a moderate agreement and cobblestoning had a poor agreement. ICC was poor for whether the surgeon would delay airway reconstruction, but substantial for the overall assessment of activeness and the tentative scale created by summing the 4 characteristics. By Fleiss's κ, edema and erythema had a moderate agreement, while eversion and cobblestoning had a poor agreement. The question of whether to delay surgery also had low agreement. Both the overall assessment of activeness and the tentative scale had a strong correlation when comparing each pair of judges.
Table 2.
Interrater Reliability Measures
| Variable | ICC (95% CI) | Fleiss κ |
|---|---|---|
| Edema | 0.76 (0.34‐0.92) | 0.50 |
| Erythema | 0.83 (0.55‐0.94) | 0.61 |
| Eversion | 0.58 (0.02‐0.85) | 0.30 |
| Cobblestoning | 0.18 (−0.57 to 0.67) | 0.07 |
| Delay surgery? | 0.47 (−0.11 to 0.80) | 0.22 |
| Combined measure | ICC (95% CI) | Pearson's R * |
|---|---|---|
| Tentative scale | 0.78 (0.40‐0.92) | 0.72‐0.78 |
| Overall “Activeness” | 0.76 (0.34‐0.92) | 0.65‐0.71 |
Abbreviations: CI, confidence interval; ICC, intraclass correlation coefficient.
Pearson's R was reported as the range of lowest to highest correlations between each 2 judges.
The preliminary assessment of construct validity demonstrated strong convergent validity (R of 0.68). A significant difference was noted in the tentative activeness scale between larynges for which the surgeon would delay surgery and ones for which they would not (P < .001). A significant difference was also noted in the overall activeness rating for both decisions (P < .001). Convergent and discriminant validity measures are demonstrated in Table 3.
Table 3.
Construct Validity Measures
| Convergent validity | R |
|---|---|
| Correlation between tentative activeness scale and overall activeness rating | 0.68 |
| Discriminant validity | Delay surgery? | ||||
|---|---|---|---|---|---|
| Yes | No | ||||
| Measure | Mean | SD | Mean | SD | P value |
| Tentative scale (range 4‐20) | 8.78 | 0.70 | 6.28 | 0.25 | <.001 |
| Overall “Activeness” (range 0‐10) | 6.27 | 0.39 | 2.47 | 0.25 | <.001 |
Discussion
The focus of this study was to measure the reliability of visual indicators of laryngeal inflammation, improve clinical communication and decision‐making, and provide a framework for developing a scale to define an “active” larynx.
This study provides promising preliminary data toward the development of a scale for visually assessing the inflammation of a larynx when planning LTR. In particular, both ICC and Fleiss's κ indicated good interrater reliability for edema and erythema, but relatively poor agreement for ventricular eversion or cobblestone appearance of laryngeal mucosa. This is likely due to the poor representation of laryngeal eversion and cobblestoned mucosa in the image library, a limitation of the preliminary work that will be rectified in future inquiries. This limited image library was in part due to an effort to not use images from the respondents' own procedures in case of potential bias. In future multi‐center expansions, this concern will be less relevant. Not surprisingly, while the tentative scale has a range of possible scores from 4 to 20, the range of scores given by the raters was considerably less representative (4.67‐10), as shown in Figure 1A‐E. Additionally, the agreement was substantial for the overall active larynx rating and the tentative scale formed by summing the inflammatory characteristics together. But agreement was low to moderate for the binary (yes/no) question of whether the rater would delay surgery. This indicates that although raters agreed on the total score, they did not consistently agree on whether surgery should be delayed. Given that the raters were blinded to individual patient factors, this disagreement demonstrates how subjectivity can lead to inconsistencies in clinical decision‐making among surgeons. It reinforces the need for a more standardized assessment such as with the use of a scale to assess latent factors that might otherwise influence treatment decision‐making. 17
While the agreement was poor for the decision of whether to delay surgery due to laryngeal inflammation, there was a significant difference in the overall rating of activeness and in the sum of the characteristics measured between larynges for which surgeons would delay surgery and larynges for which surgeons would not. A formal analysis of the predictive capacity including exploratory factor analysis and evaluations of internal consistency is not possible with such a small study population. 17 , 18 The decision to proceed with surgery is multifactorial, and activeness may play a small role in the decision. As such, an activeness scale is unlikely to ever be predictive of surgery planning. However, this initial evaluation of convergent and discriminant validity is promising and warrants further investigation.
Although the term active larynx is anecdotally discussed frequently among airway surgeons, it is rarely seen in the literature. A recent consensus statement on outcomes to report for LTR nearly reached consensus for including “active/reactive larynx.” 2 Other guidance documents reference evaluating for inflammation whether inflammation is “active” or mature. 3 Numerous other reports discuss prophylactic treatments of reflux medications, azithromycin, topical or systemic steroids, or other anti‐inflammatory treatments. 1 , 19 , 20 This is not surprising given how inflammation is implicated in the formation and progression of stenosis in the context of airway reconstruction and otherwise. 1 , 4 , 5 , 6 , 7 Despite these numerous causes of airway inflammation, it is not clear that these causes have different appearances endoscopically. For example, diagnosis of laryngopharyngeal reflux by laryngoscopy findings has been limited by low positive predictive value and competing diagnoses. 9 , 21 Follow‐up studies on the active larynx might categorize larynges in the image library according to etiology to determine if these causes can be differentiated visually.
Despite airway inflammation being implicated in the success rate after LTR, literature on the active larynx has been limited. Multicenter outcome studies on LTR have primarily identified medical comorbidities and grade and anatomy of airway stenosis in predicting surgical success rate. 22 A standardized assessment of laryngeal inflammation could allow for the assessment of this potential surgical risk factor. While the current study is preliminary, it sets a foundation for such a standardized assessment, reducing unnecessary variation in surgical decision‐making.
In addition to the limited representation of more severe ventricular eversion or cobblestoned mucosa, there were several limitations in this study. The limited sample size of the preliminary study could lead to an increase in agreement due to overlapping management strategies and paradigms. Additionally, the use of static images for the survey may have affected assessments, given that a DLB is a dynamic evaluation. While video presentations may have improved the accuracy of ratings, lack of palpation with endoscopic instruments may have limited subtler assessments (eg, friability of mucosa). Additionally, the image library was created by selecting appropriate and clear images of laryngeal pathology, potentially biasing the results toward greater agreement among raters. Finally, subglottic and suprastomal pathology visible through the glottis may have biased raters toward specific management paradigms and similar scores. These concerns are being used to plan a larger, multi‐institutional survey project, evaluating the agreement of visual assessment of laryngeal inflammation for LTR decision‐making.
Conclusion
The active larynx is an important consideration in planning laryngeal reconstructive surgery. While the assessment of the active larynx is subjective, some characteristics can be reliably ascertained. Hence, the creation of an active larynx scale is feasible.
Author Contributions
Prasanth Pattisapu, design, conduct, analysis, presentation, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; Amy M. Manning, design, conduct, analysis, presentation, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; Michael J. Boutros, design, conduct, analysis, presentation, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; Megan McNutt, conduct, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; Tendy Chiang, conduct, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; Jonathan M. Grischkan, conduct, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; Meredith N. Lind, conduct, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript; John F.P. Bridges, design, writing, review and editing of manuscript drafts, approval of the final draft of the manuscript.
Disclosures
Competing interests
The authors have no conflicts of interest to disclose.
Funding source
None.
This article was presented at the AAO‐HNSF 2023 Annual Meeting & OTO Experience, Nashville, TN, September 30 ‐ October 4, 2023.
References
- 1. Jefferson ND, Cohen AP, Rutter MJ. Subglottic stenosis. Semin Pediatr Surg. 2016;25(3):138‐143. 10.1053/j.sempedsurg.2016.02.006 [DOI] [PubMed] [Google Scholar]
- 2. Balakrishnan K, Sidell DR, Bauman NM, et al. Outcome measures for pediatric laryngotracheal reconstruction: international consensus statement. Laryngoscope. 2019;129(1):244‐255. 10.1002/lary.27445 [DOI] [PubMed] [Google Scholar]
- 3. Bailey M, Hoeve H, Monnier P. Paediatric laryngotracheal stenosis: a consensus paper from three European centres. Eur Arch Otrhinolaryngol. 2003;260(3):118‐123. 10.1007/s00405-002-0526-2 [DOI] [PubMed] [Google Scholar]
- 4. Monnier P. Management of laryngotracheal stenosis in infants and children. Pediatr Airway Surg. Springer; 2010. [Google Scholar]
- 5. Welkoborsky HJ, Hinni ML, Moebius H, Bauer L, Ostertag H. Microscopic examination of iatrogenic subglottic tracheal stenosis: observations that may elucidate its histopathologic origin. Ann Otol Rhinol Laryngol. 2014;123(1):25‐31. 10.1177/0003489414521382 [DOI] [PubMed] [Google Scholar]
- 6. Jayawardena A, Lowery AS, Wootten C, et al. Early surgical management of thermal airway injury: a case series. J Burn Care Res. 2019;40(2):189‐195. 10.1093/jbcr/iry059 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Wiatrak BJ, Albert DM, Holmes DK, Cotton RT. Cartilage graft epithelialization. A preliminary study using a goat model. Arch Otolaryngol Head Neck Surg. 1993;119(7):777‐781. 10.1001/archotol.1993.01880190073015 [DOI] [PubMed] [Google Scholar]
- 8. Sivarajah S, Isaac A, Anderson S, et al. Validity of laryngomalacia classification systems: a multi‐institutional agreement study. Clin Otolaryngol. 2020;45(4):471‐476. 10.1111/coa.13530 [DOI] [PubMed] [Google Scholar]
- 9. de Bortoli N. How many cases of laryngopharyngeal reflux suspected by laryngoscopy are gastroesophageal reflux disease‐related. World J Gastroenterol. 2012;18(32):4363‐4370. 10.3748/wjg.v18.i32.4363 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420‐428. 10.1037//0033-2909.86.2.420 [DOI] [PubMed] [Google Scholar]
- 11. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155‐163. 10.1016/j.jcm.2016.02.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Pattisapu P, Abts MF, Bly RA, et al. Validation of the Seattle Suprastomal Safety Score (5S): a novel measure in pediatric tracheostomy‐dependent patients. Otolaryngol Head Neck Surg. 2022;166(5):970‐975. 10.1177/01945998211037254 [DOI] [PubMed] [Google Scholar]
- 13. Zhao X, Liu JS, Deng K. Assumptions behind intercoder reliability indices. Ann Int Commun Assoc. 2013;36(1):419‐480. 10.1080/23808985.2013.11679142 [DOI] [Google Scholar]
- 14. Altman DG. Practical Statistics for Medical Research . 1st ed. Chapman and Hall/CRC; 1999. [Google Scholar]
- 15. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159‐174. [PubMed] [Google Scholar]
- 16. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med. 2012;22(3):276‐282. [PMC free article] [PubMed] [Google Scholar]
- 17. DeVellis RF, Thorpe CT. Scale Development: Theory and Application. 5th ed. Sage Publications, Inc; 1991. [Google Scholar]
- 18. Basch E, Spertus J, Adams Dudley R, et al. Methods for developing patient‐reported outcome‐based performance measures (PRO‐PMs). Value Health. 2015;18(4):493‐504. 10.1016/j.jval.2015.02.018 [DOI] [PubMed] [Google Scholar]
- 19. Sidell DR, Meister KD, de Alarcon A, et al. International Pediatric Otolaryngology Group (IPOG) consensus recommendations: evaluation and management of congenital tracheal stenosis. Int J Pediatr Otorhinolaryngol. 2022;161:111251. 10.1016/j.ijporl.2022.111251 [DOI] [PubMed] [Google Scholar]
- 20. Wertz A, Ryan M, Jacobs I, Piccione J. Impact of pre‐operative multidisciplinary evaluation on laryngotracheal reconstruction outcomes. Laryngoscope. 2021;131(7):E2356‐E2362. 10.1002/lary.29338 [DOI] [PubMed] [Google Scholar]
- 21. Rosen R, Mitchell PD, Amirault J, Amin M, Watters K, Rahbar R. The edematous and erythematous airway does not denote pathologic gastroesophageal reflux. J Pediatr. 2017;183:127‐131. 10.1016/j.jpeds.2016.11.035 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Jacobs IN, Giordano T, Soaper A, et al. A multicenter study analyzing the impact of pre‐existing comorbidities on laryngotracheal reconstruction (LTR) outcomes. Int J Pediatr Otorhinolaryngol. 2023;172:111631. 10.1016/j.ijporl.2023.111631 [DOI] [PubMed] [Google Scholar]