Inter-rater and intra-rater reliability of the airway diameter measured by sonography

Abstract

Introduction

Sonography has gained a broad field of applications in current anesthetic practice. During airway management, it could have an important role, because it allows real-time measurement of anatomical aspects and a functional evaluation with high safety profile. However, assessment of the reliability and precision of these measurements is critical if sonography is used as a clinical diagnostic tool. The aim of the study was to estimate inter-rater and intra-rater reliability of the airway diameter measured by sonography in heathy volunteers.

Methods

We conducted a cross-sectional study during 2015 on healthy adult volunteers. Using sonography, transverse internal diameter of subglottic space was measured twice by two blinded sonographers.

Results

Agreement was classified as very good for intra-rater measurements (ICC = 0.84) and good for the inter-rater measurements (ICC = 0.71). The 95% CI of agreement limits for the intra-rater measurements were inferior to those for the inter-rater measurements.

Conclusions

Ultrasound measurement is a reliable and precise method to measure the internal subglottic diameter of the airway. Ultimately, this method may provide clinicians valuable information regarding airway diameter in adults and may help to guide treatment options.

Introduction

Ultrasonography has gained a broad field of applications in current anesthetic practice. It has an important role, because it allows real-time anatomical and functional evaluation with a high safety profile [1].

Like other imaging techniques, the reliability of ultrasound measurement depends on two properties: validity and reproducibility. While validity expresses the degree to which the phenomenon of interest is actually measured (proximity to truth or lack of bias), reproducibility indicates the extent to which the same values are obtained when a measure is achieved repeatedly under similar conditions [2]. Measurement variability and agreement among them may determine their usefulness in clinical practice [3].

Some authors have explored the importance of sonography as a tool in airway evaluation and management, primarily in pediatric population [4, 5]. Most have focused on adjusting the diameter of the tracheal tube to use or on comparing the measurement with the traditional methods, such as the formulas [4–7]. Considering that the airway is a dynamic structure and sonographic measurements depend on the evaluator, these may change over time or vary according to sonographer skills.

The objective of this study was to determine the variability and reproducibility of the sonographic measurement of the subglottic diameter in healthy volunteers, as an important step in defining its clinical applicability.

Materials and methods

After approval from the Institutional Review Board at Hospital Universitario San José (Popayán, Colombia) (Reference: 103-1.5.32, 09/10/2014), we conducted a cross-sectional study between March and December 2015.

Three members of the research team were trained to measure the subglottic diameter through ultrasound by an experienced radiology specialist. They were guided during three training rounds with four consecutive volunteers each one (12 volunteers in total). Each training-round was done during different times and these volunteers were not included as participants of the study. To measure the subglottic airway diameter, the radiologist and members of the research team followed the instructions and methodology published by Schramm et al. [4, 5]. During spontaneous ventilation, the minimal transverse diameter of the subglottic airway was measured by the trainer with full explanation of the procedure of image acquisition. Measurements were performed with a 6–13 MHz linear transducer connected to the portable ultrasound (Sonosite^® M-Turbo, Bothell, WA, USA) at 10 MHz as definite resolution. Initially, the transducer was located in the midline of the neck in sagittal position to identify the cricoid cartilage and the first tracheal ring. After that, the transducer was rotated transversely (short axis) to locate the cricoid cartilage, which is a hypoechoic arch, because it is an air-mucosal interface. Then, the transducer advanced in cephalic direction to just above the cricoid cartilage, an anatomical site corresponding to the subglottic space [7] (Fig. 1). The transverse internal diameter of this structure was measured (in millimeters) on the reference of the internal borders (Fig. 2).

Fig. 1.

a Long axis of the trachea showing thyroid cartilage, cricoid cartilage (yellow arrows), and tracheal rings (red arrows). b Short-axis sonogram of the anatomical site corresponding to the subglottic space

Fig. 2.

Assessment and measurement of the transverse internal diameter (dashed line)

For the study phase, healthy volunteers over 18 years of age with ASA physical status 1 or 2 were included. All volunteers agreed to participate and signed a written informed consent form prior to data collection. Exclusion criteria for this study were ASA physical status greater than 2 and a history of having predictors of difficult airway, given by: an obvious visual alteration of the external anatomy of the neck, circumference of the neck larger than 40 cm, previous exposure to radiation of the head and neck, or limitation of cervical mobilization (extension) [8]. In addition, volunteers with obvious airway malformations and/or medical conditions impeding or altering ultrasound measurement, such as cervical collar, tracheostomy, tracheomalacia, masses or tumors, were also excluded.

Volunteers were placed in supine position with the head in extension during spontaneous breathing. Measurements were performed with a 6–13 MHz linear transducer from a portable ultrasound (Sonosite^® M-Turbo, Bothell, WA, USA) using a deep of 2.7 and 3.3 cm for the long and short axes, respectively. Other settings were: focus on at level around 1.5 and 2 cm, gain level less than 50%, frequency 10 MHz, and tissue harmonics on.

Three researchers (second-year anesthesiology residents) performed the measurements. Two performed the procedure twice (time 1 and time 2), and a third one recorded and collected the results and duration for each assessment. Each measurement was blinded to the sonographer to avoid information bias. The roles of researchers were reassigned successively at random on each subject included. In addition, age, gender, and weight were collected for each participant.

Sample size was calculated to contrast null hypothesis (H₀: ρ = ρ₀) with the alternative hypothesis (H₁: ρ > ρ₀), where ρ is the value for the intra-class correlation coefficient (ICC) and ρ₀ is a value predetermined for the ICC. According to an a priori hypothesis, it was expected to estimate an intra-rater ICC = 0.8 and inter-rater ICC = 0.6, which were previously considered as a substantial reliability (consistency) [9]. Using the figures reported by Eliasziw et al. [10], a minimum of 35 subjects, two evaluators and two measurements for each, was enough to reach a statistical power superior to 80% with a significance level α = 0.05.

Data analysis was made using descriptive statistical summary measures [mean ± SD, median (IQR), frequencies, and proportions] according to their distribution and nature. Ultrasound measurements were expressed in millimeters and their normality was evaluated through the Shapiro–Wilkinson test. For each one, coefficient of variation and 95% CI were reported to describe measurement precision and are reported in units of the measurement itself.

To evaluate intra-rater and inter-rater reliability, ICC was calculated from a random effects model using analysis of variance (ANOVA) for repeated measurements [11]. In addition, we reported 95% CI for each ICC. Reliability was qualified with the ICC as: bad or null (ICC < 0.20), poor (ICC = 0.21–0.40), moderate (ICC = 0.41–0.60), good (ICC = 0.61–0.80), or very good (ICC = 0.81–1.00) [12, 13].

Bland–Altman plots were constructed to graphically analyze agreement between measurements. This method of displaying data plots the difference between two measurements for each subject against their mean [14, 15]. The 95% CI of agreement limits were calculated. A priori, clinical usefulness of the ultrasound airway measurement was predetermined if these limits of agreement range from ± 5 mm.

All statistical calculations were performed using R (R Foundation, Vienna, Austria) [16].

Results

The study recruited 100 participants of which 64 were females and 36 were males, with a mean age of 34 years (1 SD = 10.6 years) and median of 30 years (IQR = 25–42). The mean weight for the participants was 67 kg (1 SD = 12 kg). Table 1 shows statistical summaries and the distribution of each measurement performed by the evaluators at the time one, time two, and their variability.

Table 1.

Statistical summaries of each measurement

	Mean ± 1 SD	Variation coefficient	95% CI	Median (IQR)	P value*
Evaluator one Measurement one	8.12 ± 1.59	21.7	7.8–8.4	7.6 (2, 6, 8, 9)	0.000
Evaluator one Measurement two	8.04 ± 1.53	19	7.7–8.3	7.8 (6.9–9.0)	0.038
Evaluator two Measurement one	7.97 ± 1.23	15.5	7.7–8.2	7.7 (7.0–8.5)	0.024
Evaluator two Measurement two	7.86 ± 1.22	12	7.6–8.1	7.7 (7.0–8.45)	0.000

Time one and time two (n = 100)

*Kolmogorov–Smirnov normality test

Inter-rater and intra-rater reliability is shown in Table 2. Agreement was classified as very good for intra-rater measurements (ICC = 0.84) and good for intra-rater measurements (ICC = 0.71).

Table 2.

Intra-class correlation coefficient for each evaluator and each measurement (intra-rater and inter-rater reliability) (n = 100)

	Intra-class correlation coefficient (ICC)	95% CI
Evaluator 1 (Measurement 1 and 2) Intra-rater	0.84	0.78–0.89
Evaluator 2 (Measurement 1 and 2) Intra-rater	0.84	0.78–0.89
Measurement 1 (Evaluator 1 and 2) Inter-rater	0.71	0.58–0.81
Measurement 2 (Evaluator 1 and 2) Inter-rater	0.71	0.57–0.80

Bland–Altman analysis is presented in Fig. 3 and 95% CI agreement limits are shown in Table 3. The 95% CI agreement limits for the intra-rater measurements were inferior to those for the inter-rater measurements.

Fig. 3.

Bland–Altman plots for intra-rater agreement and inter-rater agreement (n = 100)

Table 3.

Bland-Altman analysis

	Difference	1 SD	95% agreement limits
Evaluator 1 (Measurement 1 and 2) Intra-rater	0.08	0.86	− 1.61 to 1.78
Evaluator 2 (Measurement 1 and 2) Intra-rater	0.10	0.67	− 1.21 to 1.43
Measurement 1 (Evaluator 1 and 2) Inter-rater	0.15	1.33	− 2.45 to 2.77
Measurement 2 (Evaluator 1 and 2) Inter-rater	0.17	1.30	− 2.38 to 2.74

Differences among measurements, variability, and 95% CI agreement limits for intra-rater (evaluator 1 and 2) and for inter-rater (measurement 1 and 2) (n = 100)

Discussion

The main results of the present study indicate that variability of independent measures of subglottic diameter is low. Second, very good intra-rater measurement agreement exists and agreement limits are narrow, within a 5-mm range.

Scientific interest exists in measuring the airway anatomic variability [17], which may also have clinical usefulness. Recent studies show low correlation between tomographic-based measurements of the airway with age (or other anthropometric variables of easy evaluation), demonstrating low performance of an isolated tracheal size clinical measurement [18].

Ultrasonography has demonstrated high correlation and agreement with airway measurements made by magnetic resonance imaging (precision = 0.33 mm), also with clinical applicability at the patient´s bedside [7]. However, operator dependence of ultrasound as an imaging modality has been highlighted as potentially problematic for both qualitative and quantitative anatomic measurements. Our study demonstrates that variability of ultrasound airway measurement (95% CI) has ranges < 0.6 cm for each time and for independent evaluators, which can be graded as low.

To our knowledge, this is the first study to evaluate intra-rater and inter-rater agreement of ultrasound repeated airway measurements. Our results support that, for trained evaluators, ultrasound agreement is very good. Several studies have been published regarding ultrasonography agreement in other anatomic sites: some static (liver, prostate) [19, 20] and other dynamic (cava vein) [21, 22]. Although airway diameter may be modified by respiratory dynamics [23] and considering that our protocol did not include a specific static measurement time, the agreement is good. It is important to highlight that machine settings should be adjusted properly to avoid an increased artifacts formation, leading to incorrect measures. In addition, every pathologic condition (i.e., tracheal stenosis and granulomas) may increase disturbing artifacts. In particular, secretions could interfere with ultrasound beam reflection and cause artifacts that would alter measure limits.

The influence of the orotracheal tube diameter on ventilatory mechanics is historically recognized (Hagen–Pousielle law). Dead-space addition leads to increased respiratory work [24]. Based on this premise, we recommend [25] using the highest possible internal diameter for the endotracheal tube, with low resistance and which allows overcoming the mechanical effect on the tracheal mucosa. Shapiro et al. demonstrated that in adults with spontaneous breathing, the respiratory work significantly increases when the orotracheal tube diameter decreases from 7 to 6 mm [26].

The agreement limits for ultrasound measurements are within a 5-mm range. This finding explains why the intra-rater or inter-rater reliability influence upon orotracheal tube diameter selection may be low, when considering that the minimal difference for changing a tube is 5 mm (superior or inferior).

Clinical applicability of ultrasound during orotracheal tube selection has been assessed by several authors: Kim EJ et al., Shibasaki M. et al, and Schramm C et al. evaluated the usefulness of airway ultrasonography measurement in selecting an appropriate orotracheal tube in pediatric populations. These authors documented narrow agreement limits and high predictive capability of ultrasound in selecting the orotracheal tube, superior to age- and height-based formulas [4–6].

Regarding clinical applicability, our study shows that subglottic diameter ultrasound measurements have high intra-rater and inter-rater reproducibility in adults. This study did not consider pediatric population, which might be a limitation but also an opportunity for further studies.

This study adds to the body of literature on using sonography to assess airway diameter. Our study demonstrates that with standardized positioning and a short prior training, sonography becomes a precise and reliable tool in assessing subglottic diameter.

Compliance with ethical standards

Conflict of interest

All the authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in this study were in accordance with the ethical standards of The Institutional Research Committee and with the 1964 Helsinki declaration and its later amendments.

Informed consent

All volunteers agreed to participate and signed a written informed consent form prior to data collection.

Sources funding

None.