The diagnostic validity of clinical airway assessments for predicting difficult laryngoscopy using a grey zone approach

Abstract

Objectives

The diagnostic validity of clinical airway assessment tests for predicting difficult laryngoscopy in patients requiring endotracheal intubation were evaluated using receiver operating characteristic (ROC) curve analysis and a grey zone approach.

Methods

In this prospective observational study, patients were evaluated during a pre-anaesthetic visit. Predictive airway assessment tests (i.e. Modified Mallampati [MMT] classification; upper lip bite test [ULBT]; mouth opening; sternomental distance; thyromental distance [TMD]; neck circumference; neck mobility; height to thyromental distance [HT/TMD]; neck circumference-to-thyromental distance [NC/TMD]) were performed on each patient and LEMON, Naguib, and MACOCHA scores were also calculated. In addition, laryngeal images were acquired and assessed for percentage of glottic opening (POGO) scores. A POGO score of zero was categorized as difficult laryngoscopy.

Results

The incidence of difficult laryngoscopy was 14.4% (35/243). Although seven predictive airway assessments (i.e. MMT classification, ULBT, mouth opening, HT/TMD, NC/TMD, and the LEMON and Naguib models) predicted difficult laryngoscopy by ROC analyses, a grey zone approach showed that the parameters were inconclusive in approximately 70% of patients. From all the tests, the HT/TMD ratio showed the highest sensitivity (80.0%) and ULBT had the highest specificity (95.2%).

Conclusion

Using the grey zone approach, all predictive airway assessment tests showed large inconclusive zones which may explain previous inconsistent results in the prediction of difficult laryngoscopy. Our results suggest that the usefulness of clinical airway evaluation tests for predicting difficult laryngoscopy remains controversial.

Clinical trial registration

ClinicalTrials.gov (NCT01719848)

Introduction

Inability to secure the airways can result in serious complications.^1–3 Difficult laryngoscopy, which has been defined as no visualization of any part of the vocal cords with conventional laryngoscope,⁴ is thought to be associated with difficult intubation,⁵ though the condition of ‘difficult tracheal intubation’ can also occur per se.^6–9 Studies have attempted to delineate the most appropriate clinical airway assessment for predicting difficult laryngoscopy but the outcomes are controversial,^9–12 and wide ranges in discriminative power for all airway examinations have been reported.^10–14 In addition, several meta-analyses have suggested that even a combination of airway examinations has limited ability to predict difficult laryngoscopy because of low sensitivity and low positive predictive value.^11,15,16

To find a more accurate predictive test, some studies have sought to identify a clear definition for difficult laryngoscopy^4,17 and identify risk factors by multivariate analysis from a derivation cohort while applying the same test to a validation cohort.^6,18 However, the usefulness of airway evaluations for predicting difficult laryngoscopy is unclear because of the fundamental problem in identifying a binary cut-off value.¹⁹ It is our opinion that in a clinical situation, the division of patients into those who may, and may not have, a difficult laryngoscopy without an overlap is difficult. Therefore, an inconclusive grey zone has been proposed to simulate the reality of clinical practice; the values in the grey zone correspond to predictions that are not precise enough for binary diagnostic decisions.²⁰ Instead of one cut-off value, the grey zone method proposes two cut-offs that constitute its borders.²¹ The first cut-off allows exclusion of diagnosis with near certainty (i.e. privilege sensitivity and negative predictive value) whereas the second cut-off includes the diagnosis with near certainty (i.e. privilege specificity and positive predictive value).²¹ Intermediate values included in the grey zone correspond to a prediction not precise enough for a diagnostic decision.

The aim of this prospective, observational study was to evaluate clinical airway assessments and scoring systems as predictors of difficult laryngoscopy using receiver operating characteristic (ROC) curve analysis with a grey zone approach. The findings of this study may clarify inconsistencies in the prediction of difficult laryngoscopy reported in previous studies.

Patients and methods

Study design

This prospective observational study was conducted at a tertiary care university hospital (Department of Anaesthesiology, Samsung Medical Centre, Seoul, Korea) between July 2012 and July 2013. Airway assessment examinations were performed preoperatively by an experienced, board certified anaesthesiologist (G.K.). Laryngeal images were obtained during endotracheal intubation using a fibreoptic scope (AV® scope; CAREtec, Kangwon-Do, South Korea) attached to a laryngoscope blade after adjusting the axis to align with the eyes (Figures 1A−C). English-type Macintosh laryngoscope blades, size 3, were used to expose the laryngeal view in the ‘sniffing position’ by using a hard, 7 cm pillow to elevate the head.²² The study was approved by the Institutional Review Board at Samsung Medical Centre, Seoul, Korea (SMC 2012-05-079-002) and registered at ClinicalTrials.gov (NCT01719848). All patients provided written informed consent.

Figure 1.

Photographs showing (A) an AV® scope preloaded with a Macintosh laryngoscope blade, size 3; (B) laryngeal exposure with endotracheal tube placement using the AV® scope; and (C) a laryngeal image obtained during endotracheal intubation using a AV scope.

Patient selection

Male and female patients aged 20–80 years undergoing general anaesthesia who required tracheal intubation were assessed for eligibility. Exclusion criteria included a history of difficult intubation, congenital disorders related to difficult laryngoscopy, upper airway disease and cervical spine disease with neurological symptoms.

The following predictive airway assessments were performed on each patient:^4,6

1. Modified Mallampati classification (MMT): performed with the patient in the sitting position with the head in extension, mouth fully opened, tongue out, and without phonation: Grade I: Faucial pillars, soft palate and uvula visible; Grade II: Faucial pillars, soft palate visible, but uvula masked by the base of the tongue; Grade III: Soft palate only visible; Grade IV: Soft palate not visible.

2. Upper Lip Bite Test (ULBT): Class I: Lower incisors biting the upper lip, making the mucosa of the upper lip totally invisible; Class II: The same biting manoeuvre revealing a partially visible mucosa; Class III: The lower incisors fail to bite the upper lip.

3. Sternomental distance (SMD): Distance measured in the seated position with the head fully extended on the neck and with the mouth closed (straight distance between the upper border of the manubrium-sterni and bony point of the mentum).

4. Thyromental distance (TMD): Distance from the thyroid cartilage to the mental prominence with the neck fully extended.

5. Mouth opening: interincisor or intergingival distance depending on the presence of teeth.

6. Neck circumference: measured at the level of the cricoid cartilage.

7. Neck mobility: estimated as the angle between the maxillary tooth occlusal surface and a horizontal line with the patient’s head fully extended.²³

The ratios of neck circumference-to-thyromental distance (NC/TMD)¹² and height-to-thyromental distance (HT/TMD)²⁴ were also calculated. Laryngeal views were graded by a blinded anaesthesiologist (E.K.) from saved images using percentage of glottic opening (POGO) score (i.e. 0 [glottis is not visible] to 100% [entire glottis is visible]).^17,25 A POGO score of zero was categorized as difficult laryngoscopy.

Three multivariate clinical models (Naguib,²⁶ LEMON method²⁷ and MACOCHA score ¹⁸) were also evaluated in the prediction of unanticipated difficult intubation. The Naguib score is regarded as the best score in operative settings:²⁶ the prediction (l) is determined by the following formula: l = 0.2262 – 0.4621 × TMD + 2.5516 × MMT score – 1.1461 × interincisor distance + 0.0433 × height.

Where the TMD distance, interincisor gap, and height were measured in cm and MMT score was graded as 0 or 1. A positive numerical value for l signifies difficult laryngoscopy and a negative value indicates easy laryngoscopy. The LEMON criteria are commonly used in the emergency department.²⁷ The MACOCHA score was created for patients admitted to intensive care units.¹⁸ Patients with difficult laryngoscopy are more likely to have higher LEMON and MACOCHA scores than those patients with a good laryngoscopic view.^18,27

Statistical analyses

All data were analysed using SPSS software (version 18 for Windows®; IBM Corp., Armonk, NY, USA) and MedCalc 12.7.2 (MedCalc Software bvba, Ostend, Belgium). Comparisons of continuous variables between difficult and easy laryngoscopy groups were conducted with independent t-test or Mann-Whitney U-test according to data normality. The MMT classification and ULBT were analysed using the χ²-test for trend.²⁸

The ability of each clinical airway test to predict difficult laryngoscopy was evaluated by ROC curve analysis.²¹ Sensitivity, specificity, positive predictive value, and negative predictive value were calculated for statistically significant variables on ROC curves. In addition, a grey zone was calculated to encompass values from each clinical airway test for which formal conclusions could not be obtained. The grey zone was defined as previously described.²¹ The optimal threshold was estimated using Youden's index (i.e. Sensitivity + Specificity − 1) to maximize both sensitivity and specificity. The mean and 95% confidence intervals (CIs) of Youden’s index were estimated using 1000 bootstrap replications. Therefore, the grey zone was defined as well as its 95% CI.²¹ In a second analytical approach, three classes of response were defined as negative, inconclusive and positive. Inconclusive grey zone was defined as the range of values with sensitivity < 90% and specificity < 90% (diagnostic tolerance of 10%). A P-value < 0.05 was considered to indicate statistical significance.

Sample size calculation was based on a pilot study of 35 patients that showed that the incidence of difficult laryngoscopy was 14.3% (n = 5) and that the area under the ROC curve for MMT classification was 0.66. To demonstrate that MMT classification could predict difficult laryngoscopy with an area under the ROC curve (AUC) of > 0.66, an alpha risk of 0.05 and a beta risk of 0.2, it was estimated that at least 180 patients with easy laryngoscopy and 30 patients with difficult laryngoscopy would be required for the main study.

Results

This study enrolled 263 patients (Figure 2). Of these, 20 were excluded for the following reasons: withdrew informed consent (n = 4); newly developed cervical spine disorder (n = 4); initially received alternative device (n = 2); laryngeal views not saved as video files due to mechanical error (n = 10). Therefore, 243 patients were included in this observational study.

Figure 2.

Flow diagram showing the patient (n = 263) progression through this prospective, observational study. Of the 263 patients who were enrolled, 20 were excluded, leaving 243 patients included in the analysis.

Thirty-five patients (14.4%) had a POGO score of zero and so were categorized as having difficult laryngoscopy and the remaining 208 patients were categorized as having easy laryngoscopy. Patient characteristics according to easy or difficult laryngoscopy are presented in Table 1. Although there was no difference in body mass index (BMI) between the two groups, patients with difficult laryngoscopy were statistically significantly older (P = 0.009) and there were more men than women (P = 0.001) compared with the easy laryngoscopy group. Patients with difficult laryngoscopy had smaller mouth openings (P = 0.009) and higher NC/TMD (P = 0.028) and HT/TMD ratios (P = 0.011). In addition, the distribution of ULBT was statistically significantly different between the two groups (P = 0.001).

Table 1.

Clinical characteristics of the 243 patients enrolled in this prospective study assessing the diagnostic validity of clinical airway assessments for predicting difficult laryngoscopy.

Characteristic	Easy laryngoscopy group n = 208	Difficult laryngoscopy group n = 35	Statistical significancea
Age, years	47.3 ± 13.5	53.8 ± 13.3	P = 0.009
Sex, male/female	70/138	23/12	P = 0.001
BMI, kg/m2	23.7 ± 3.2	23.8 ± 3.1	NS
MMT, I/II/III/IV	81/71/47/8 b	8/14/10/3	NS
ULBT, I/II/III	73/125/10	6/21/8	P = 0.001
SMD, cm	18.5 (17.0, 20.0)	18.0 (17.0, 19.0)	NS
TMD, cm	8.0 (7.0, 8.5)	8.0 (7.3, 8.5)	NS
NC, cm	36.6 (34.0, 39.2)	38.5 (35.0, 40.5)	NS
Neck mobility, °	45 (37, 60)	45 (30, 55)	NS
Mouth opening, cm	4.5 (4.1, 5.0)	4.2 (4.0, 4.6)	P = 0.009
NC/TMD	4.6 (4.3, 5.1)	5.0 (4.5, 5.4)	P = 0.028
HT/TMD	20.1 (19.1, 22.3)	21.3 (20.0, 22.9)	P = 0.011

Data presented as mean ± SD, n of patients or median (interquartile range).

^aBetween-group comparison; continuous variables were compared using independent t-test or Mann–Whitney U-test according to data normality. The MMT classification and ULBT were analysed using the χ²-test for trend.

^bData available from 207 patients.

MMT, Modified Mallampati classification: Grade 1: Faucial pillars, soft palate and uvula visible; Grade II: Faucial pillars, soft palate visible, but uvula masked by the base of the tongue; Grade III: Soft palate only visible; Grade IV: Soft palate not visible.

ULBT, Upper Lip Bite Test: Class I: Lower incisors biting the upper lip, making the mucosa of the upper lip totally invisible; Class II: The same biting manoeuvre revealing a partially visible mucosa; Class III: The lower incisors fail to bite the upper lip.

BMI, body mass index; SMD, sternomental distance; TMD, thyromental distance; NC, neck circumference; NC/TMD, neck circumference-to-thyromental distance; HT/TMD, height-to-thyromental distance; NS, no significant between-group difference (P ≥ 0.05).

Receiver operating characteristic curve analysis of airway assessments and established airway scores was performed for predicting difficult laryngoscopy (Table 2). Statistically significant predictors of difficult laryngoscopy were identified on ROC curve analysis and verified using the grey zone approach. Although seven predictive airway assessments (i.e. MMT classification, ULBT, mouth opening, HT/TMD, NC/TMD, and LEMON and Naguib models) predicted difficult laryngoscopy by ROC analyses, a grey zone approach showed that the parameters were inconclusive in approximately 70% of patients. For example, the grey zone included 174 patients (72%) between grade I or II MMT classification, 225 (93%) between grade I and II on ULBT, 171 (74%) between 4.1 and 5.5 for NC/TMD, 159 (65%) between 3.7 and 4.9 cm for mouth opening, 178 (73%) between 18.4 and 23.5 for HT/TMD, 169 (70%) between 0 and 1 for LEMON score and 168 patients (70%) between 0.93 and 6.95 for Naguib score (Table 2).

Table 2.

Receiver operating characteristic (ROC) curve analysis of airway assessments for predicting difficult laryngoscopy using a grey zone approach.

Parameter (n)	ROC Cut-off value	ROC AUC ± SE	95% CI	Statistical significancea	Lower, upper limits of grey zone	Patients in the grey zone n (%)	Patients with difficult laryngoscopy in the grey zone n (%)
BMI (242)	> 21.9 kg/m2	0.519 ± 0.050	0.454, 0.583	NS	16.4, 28.0 kg/m2	219 (90%)	33 (94%)
MMT (242)	≥ II	0.596 ± 0.049	0.531, 0.659	P = 0.0480	I, II	174 (72%)	22 (63%)
ULBT (243)	≥ III	0.644 ± 0.046	0.580, 0.704	P = 0.0016	I, II	225 (93%)	27 (77%)
SMD (243)	≤ 19.5 cm	0.566 ± 0.049	0.501, 0.630	NS	15.0, 20.5 cm	204 (84%)	32 (91%)
TMD (243)	≤ 37.6 cm	0.542 ± 0.052	0.477, 0.606	NS	6.2, 8.9 cm	191 (79%)	30 (86%)
NC (230)b	> 38.4 cm	0.590 ± 0.056	0.524, 0.655	NS	30.7, 41.7 cm	203 (88%)	27 (82%)
Neck mobility (243)	≤ 30°	0.592 ± 0.053	0.527, 0.654	NS	15, 59°	185 (76%)	29 (83%)
Mouth opening (243)	≤ 4.3 cm	0.637 ± 0.051	0.573, 0.697	P = 0.0076	3.7, 4.9 cm	159 (65%)	25 (71%)
NC/TMD (230)b	> 5.1	0.603 ± 0.050	0.536, 0.666	P = 0.0396	4.1, 5.5	171 (74%)	27 (82%)
HT/TMD (243)	> 19.9	0.636 ± 0.045	0.572, 0.696	P = 0.0027	18.4, 23.5	178 (73%)	29 (83%)
LEMON (243)	≥ 2	0.648 ± 0.052	0.585, 0.708	P = 0.0047	0, 1	169 (70%)	16 (46%)
Naguib (241)	> 4.55	0.678 ± 0.048	0.615, 0.736	P = 0.0002	0.93, 6.95	168 (70%)	25 (71%)
MACOCHA (243)	> 0	0.593 ± 0.049	0.528, 0.655	NS	0, 3	175 (72%)	22 (63%)

^aROC curve analysis.

^bData available from 33 patients with difficult laryngoscopy.

AUC, area under the ROC curve; SE, standard error; CI, confidence interval; BMI, body mass index; MMT, Modified Mallampati classification; ULBT, upper lip bite test; SMD, sternomental distance; TMD, thyromental distance; NC, neck circumference; SMD, sternomental distance; NC/TMD, neck circumference-to-thyromental distance; HT/TMD, height-to-thyromental distance; LEMON, Naguib and MACOCHA are scoring systems^18,25,26; NS, no significant difference (P ≥ 0.05).

The diagnostic validity profiles from the significantly predictive airway assessments are shown in Table 3. Low positive predictive values were observed for all airway tests. The ratio of HT/TMD showed the highest sensitivity (80.0%) and ULBT had the highest specificity (95.2%).

Table 3.

Diagnostic validity profiles of airway assessments for predicting difficult laryngoscopy (n = 243).

Airway assessment with cut-off value	TP (n)	TN (n)	FP (n)	FN (n)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
MMT ≥ II	27	81	127	8	77.1 (59.9, 89.6)	39.1 (32.4, 46.1)	17.6 (12.0, 24.6)	91.0 (83.0, 96.1)
ULBT ≥ III	8	198	10	27	22.9 (10.4, 40.1)	95.2 (91.3, 97.7)	44.4 (21.5, 69.2)	88.0 (83.0, 91.9)
Mouth opening ≤ 4.3 cm	22	122	86	13	62.9 (44.9, 78.5)	58.7 (51.6, 65.4)	20.4 (13.2, 29.2)	90.4 (84.1, 94.8)
NC/TMD > 5.1a	14	152	45	19	42.4 (25.5, 60.8)	77.2 (70.7, 82.8)	23.7 (13.6, 36.6)	88.9 (83.2, 93.2)
HT/TMD > 19.9	28	96	112	7	80.0 (63.1, 91.6	46.15 (39.2, 53.2)	20.0 (13.7, 27.6)	93.2 (86.5, 97.2)
LEMON ≥ 2	19	153	55	16	54.3 (36.6, 71.2)	73.6 (67.0, 79.4)	25.7 (16.2, 37.2)	90.5 (85.1, 94.5)
Naguib > 4.55b	21	149	57	14	60.0 (42.1, 76.1)	72.3 (65.7, 78.3)	26.9 (17.5, 38.2)	91.4 (86.0, 95.2)

Data presented as values (95% confidence interval) for sensitivity, specificity, PPV, and NPV.

^aData available from 230 patients.

^bData available from 241 patients.

TP, true positive; TN, true negative; FP, false positive; FN, false negative; PPV, positive predictive value; NPV, negative predictive value; MMT, Modified Mallampati classification; ULBT, upper lip bite test; NC/TMD, neck circumference-to-thyromental distance; HT/TMD, height-to-thyromental distance; LEMON and Naguib are scoring systems.^25,26

Discussion

In this prospective observational study, seven airway examinations and/or scoring systems (i.e. MMT classification, ULBT, mouth opening, NC/TMD, HT/TMD, and the LEMON and Naguib models) were found to be statistically significant predictors of difficult laryngoscopy by ROC analysis. However, on further analysis using the grey zone approach, all variables showed large inconclusive zones with a large proportion of patients in the grey zone, which may explain previous inconsistent results with regard to the prediction of difficult laryngoscopy. Out of 243 eligible patients, 35 (14.4%) were categorized as having difficult laryngoscopy. Although statistically significant differences were observed between the easy (n = 208) and difficult laryngoscopy (n = 35) groups in terms of age and sex, in our opinion these differences were not clinically relevant to the study outcome.

Difficult airways are generally encountered in unexpected clinical situations.¹ Although video laryngoscopic airway devices are increasingly used, direct laryngoscopy is most often used as the first device for intubation. The identification of patients in whom laryngoscopy and intubation may be difficult remains a challenge. Several studies in a variety of clinical fields have attempted to identify predictors of difficult laryngoscopy and intubation.^4,15,16 However, airway examinations for predicting difficult airway have been shown to have diverse reliabilities and various cut-off values have been used for the ROC curve analysis.^{8,10,11,29–31} Moreover, these cut-off values are difficult to establish^32–35 and it has been suggested that perfect prediction of truly difficult intubation is impossible because of its rarity.³⁵

Despite being widely used to evaluate diagnostic tests,¹⁹ ROC curve analysis has a fundamental problem in that it is based on a binary decision with a single cut-off value to differentiate patients with and without difficult laryngoscopy. This ‘black or white’ approach might not be suitable in actual clinical situations.³⁶ Therefore, two cut-off values separated by a grey zone have been proposed.²⁰ Recent studies suggest that defining values within a grey zone requires comprehensive evaluation using other clinical data.^37–39 In the present study, each airway assessment had a wide grey zone, which suggests that these predictive tests are inconclusive for predicting difficult laryngoscopy for approximately 70% of patients. Possible reasons for the wide grey zones include variation in operator expertise in airway assessment, laryngoscopy and/or intubation.^32,35,40 Many factors contribute to the problem of reliably predicting difficult laryngoscopy including differences in airway anatomy and morphology (e.g. large tongue, limited neck movement, limited mouth opening, small TMD distance), complex interactions among the various airway parameters used to determine airway difficulty,⁴¹ and the limited value of some tests for accurately assessing specific parts of the airway.³⁵

In addition to the mathematical issues in establishing predictability of difficult airways, airway assessments can be difficult to perform in an emergency situation.⁴ Indeed, most airway assessment examinations require the cooperation of conscious patients.^6,7 While the American Society of Anesthesiologists Task Force on Management of Difficult Airways generally recommend that a MMT score should be obtained from a cooperative patient in the sitting position with adequate tongue protrusion, only 32% of patients who require endotracheal intubation in the emergency department are able to follow commands.⁷ Furthermore, most patients who required endotracheal intubation in intensive care units are supine or recumbent and uncooperative.¹⁸ Unexpected difficult airways have been reported in patients with undiagnosed lingual tonsillitis, lingual thyroid, laryngeal papillomatosis and laryngeal web even after a thorough physical airway examination.^42–46 Although video laryngoscopic airway devices have been shown to improve laryngeal views, successful endotracheal intubations and successful first attempt intubations, the current findings suggest that the accuracy of airway assessments in emergency situations may be questionable.

This present study had several limitations. First, a POGO score of 0 was used to grade difficult laryngoscopy instead of ‘difficult intubation’. Difficult laryngoscopy has occasionally been considered ‘difficult tracheal intubation’,^6–9 but it is only one of the many reasons for difficult intubation. In addition, in some patients with a high POGO score (i.e. good laryngeal view), a guiding gum elastic bougie or multiple attempts were used in the process of endotracheal intubation. In these patients, it is suggested that the curve of the endotracheal tube may not have fitted with the alignment of the oral, pharyngeal, and laryngeal axes. Secondly, the present study used a single type of laryngoscope blade (Macintosh size 3) in all patients. Therefore, because the upper part of the glottis can potentially be covered by the curved Macintosh blade, the POGO scores might have been underestimated. This underestimation might have led to the higher incidence of difficult laryngoscopy in this present study compared with previous reports.¹⁶. Thirdly, all airway evaluations in the present study showed low positive predictive values because of high false positive rates. Generally, an airway assessment with a high sensitivity and a low false positive rate will predict difficult laryngoscopy with precision.⁴⁷ However, the current results suggest that a single type of airway assessment is not sufficient for determining patients who may have difficult laryngoscopy. Considering that positive and negative predictive values are affected by the prevalence of the condition (i.e. difficult laryngoscopy) in the population,^7,48 it is not possible to exclude the possibility that the relatively high incidence of difficult laryngoscopy in this present study (14.4%) influenced the results. Finally, information about the patients in the grey zone with potentially difficult laryngoscopy was not provided. Since difficult laryngoscopy could not be predicted by a single pre-existing airway test or score in approximately 70% of patients, further studies are required to validate these current results. Nevertheless, the current results are useful in that they reaffirm that there are problems in predicting difficult laryngoscopy.

In conclusion, this prospective observational study found that several airway examinations were statistically significant predictors of difficult laryngoscopy by ROC analysis, but using the grey zone approach all variables showed large inconclusive zones that may explain previous inconsistent results in the prediction of difficult laryngoscopy.

Declaration of conflicting interests

The authors declare that there are no conflicts of interest.

Funding

This study was supported solely by departmental resources. The authors have no commercial, proprietary, or financial interest in the products or companies described in this article.