Interobserver Reliability of the Respiratory Physical Examination in Premature Infants: A Multicenter Study

Abstract

Objective

To measure the inter-rater reliability of 7 visual and 3 auscultatory respiratory physical examination findings at 36–40 weeks’ postmenstrual age in infants born less than 29 weeks’ gestation. Physicians also estimated the probability that each infant would remain hospitalized for 3 months after the examination or be readmitted for a respiratory illness during that time.

Study design

Prospective, multicenter, inter-rater reliability study using standardized audio-video recordings of respiratory physical examinations.

Results

We recorded the respiratory physical examination of 30 infants at 2 centers and invited 32 physicians from 9 centers to review the examinations. The intraclass correlation values for physician agreement ranged from 0.73 (95% CI 0.57–0.85) for subcostal retractions to 0.22 (95% CI 0.11–0.41) for expiratory abdominal muscle use. Eight (27%) infants remained hospitalized or were readmitted within 3 months after the examination. The area under the receiver operating characteristic curve for prediction of this outcome was 0.82 (95% CI 0.78–0.86). Physician predictive accuracy was greater for infants receiving supplemental oxygen (0.90, 95% CI 0.86–0.95) compared with those breathing in room air (0.71, 95% CI 0.66–0.75).

Conclusions

Physicians often do not agree on respiratory physical examination findings in premature infants. Physician prediction of short-term respiratory morbidity was more accurate for infants receiving supplemental oxygen compared with those breathing in room air.

Premature infants are at risk for significant, persistent respiratory morbidity. Bronchopulmonary dysplasia (BPD) is the most common chronic respiratory condition associated with preterm birth and is a strong predictor of multiple adverse health outcomes, including impairments in lung function and neurodevelopmental delay^1–3; however, many former preterm infants who do not fulfill the diagnostic criteria for BPD also experience deficits in respiratory health through school age and into adulthood.^4–6 Better methods are needed to quantify the severity of lung disease in the neonatal period and to predict which infants are likely to experience long-term respiratory complications.

The Prematurity and Respiratory Outcomes Program (PROP) is a large, multicenter observational study of infants who are born at less than 29 weeks of gestation.^7,8 One of the many aims of PROP is to evaluate whether components of the respiratory physical examination assessed between 36 and 40 weeks’ postmenstrual age (PMA) in extremely preterm infants are predictive of respiratory outcomes at 1-year corrected age. A necessary step in the validation of a physical examination finding as a diagnostic test or surrogate study endpoint is assessment of interobserver agreement. This has not been measured for the respiratory examination in preterm infants.

Studies in older infants and children with acute respiratory illnesses, including asthma, bronchiolitis, croup, and pneumonia, reported levels of interobserver agreement that varied from slight to almost perfect.^9–18 Moreover, no visual or auscultatory finding consistently demonstrated high or low agreement in those studies.^9–18 The objective of the present study was to measure the interobserver reliability of several components of the respiratory physical examination in premature infants when examined between 36 and 40 weeks’ PMA. In addition, we assessed whether physicians accurately could predict an infant’s risk for short-term future respiratory morbidity based on findings from a single respiratory physical examination.

Methods

Infants with gestational ages between 23 and 28^6/7 weeks without hemodynamically significant congenital heart disease; structural abnormalities of the upper airway, lungs, or chest; or congenital malformations or syndromes were eligible for enrollment. The proportions of infants breathing room air and receiving various levels of respiratory support at the time of the examination were a priori selected to be similar to the respiratory support characteristics of the full PROP cohort at 36–40 weeks’ PMA. This was done to encourage generalizability of our findings to PROP and the broader US population of extremely preterm infants for which PROP is intended to be a representative sample. All infants were enrolled from the neonatal intensive care units (NICUs) at the Hospital of the University of Pennsylvania or the Children’s Hospital of Philadelphia. The institutional review boards at both institutions approved the protocol, and written informed consent was obtained from a parent or guardian of each infant.

Attending or senior fellow physicians in the divisions of neo-natology and pediatric pulmonology at the Children’s Hospital of Philadelphia and physician members of the PROP steering committee were invited to participate. Written informed consent was obtained from each physician.

Physical Examination Recordings and Playback

A 3-minute audio-video recording of a structured respiratory physical examination was produced for each enrolled infant (Video; available at www.jpeds.com). Wide angle, cross-table lateral, and close-up views of the chest, abdomen, and face and audio recordings of 8 lung and upper airway fields were captured in each recording (Table I). All recordings were filmed from the infant’s side at an approximately 45° angle except for a cross-table lateral view filmed level with the infant. The infants were filmed wearing a diaper only and while in a quiet awake or sleep state. All recordings were conducted in the NICU at the infant’s bedside. No artificial lighting was used, and ambient sounds such as those generated by monitoring equipment and medication pumps were not silenced. To protect the privacy of those present in the NICU at the time of the examination, however, audio was muted during the visual inspection portion of the recording when a stethoscope was not in use.

Table I.

Respiratory examination recording duration by section

Examination sections	Duration
Visual observation
Full view of infant	12 s
Close-up view of the chest and abdomen	12 s
Cross table lateral view of the chest and abdomen	12 s
Close-up view of the neck and upper chest	12 s
Close-up view of the head and face	12 s
Auscultation
Stethoscope held over the mouth and nose	12 s
Stethoscope held over the trachea	12 s
Anterior lung fields bilaterally	12 s each
Lateral lung fields bilaterally	12 s each
Posterior lung fields bilaterally	12 s each
Total examination duration	2 min, 36 s
Nonexamination duration (subject identification, video transitions)	24 s
Total recording duration	3 min

A JVC EX GZ-515 high-definition video camera (JVC, Wayne, New Jersey) mounted on a tripod and a Littmann 3200 electronic stethoscope (3M, Maplewood, Minnesota) were used to record all examinations. The Littmann 3200 captures cardiac and respiratory sounds with frequencies between 20 and 2000 Hz.¹⁹ The majority of breath sounds have frequencies of 200–250 Hz, except for wheezes and crackles, which have frequency spectra of 100–1000 Hz and 200–2000 Hz, respectively.^20,21 All audio-video editing was done with iMovie version 10.0.4 (Apple Inc, Cupertino, California).

All physicians reviewed the examination recordings using personal computers while under the supervision of a study team member or by accessing the password protected PROP study Web site. Video resolution for all recordings was set at 720 × 1280 px. Each physician was provided a set of Sony MDR-EX38iP in-ear stereo headphones (Sony, New York, New York) and was requested to set the audio playback volume in a range similar to actual physical auscultation. Physicians were permitted to review the examination recordings as many times as necessary. All findings were reported to the study team via the online data capture software REDCap.²²

Outcomes

The primary study outcomes were the physicians’ ratings of 7 visual (suprasternal, intercostal, and subcostal retractions; thoracoabdominal synchrony; head bobbing; nasal flaring; and expiratory abdominal muscle use) and 3 auscultatory (wheezing or noisy breathing, crackles, and stridor) respiratory physical examination findings. Each examination finding was assessed as present or absent except for synchronous or asynchronous thoracoabdominal movement. All infants were followed for 3 months after the recording to determine whether they remained hospitalized or were readmitted for a respiratory illness after discharge from the NICU. Physicians were asked to estimate the probability of this respiratory morbidity for each infant solely on the basis of the physical examination findings observed in the recording. No additional demographic or clinical information was provided.

Statistical Analyses

Physician and infant characteristics were summarized with standard descriptive statistics. The interobserver reliability for each examination finding was measured via the intraclass correlation coefficient (ICC). The ICC was calculated with a mixed-effects logistic regression model that included infants as a random effect and physicians as a fixed effect.²³ According to standard criteria, an ICC < 0.4 indicated “poor” reliability, 0.4–0.74 indicated “fair” reliability, and ≥0.75 indicated “excellent” reliability.²⁴ To assess the accuracy of the physicians’ predictions of future respiratory morbidity, we computed a single best linear unbiased prediction for each infant using a generalized linear model. Infants were included in the model as a first order random effect, and physicians were included as a second order random effect. A receiver operating characteristic (ROC) curve was then used to assess the accuracy of each infant-specific prediction relative to the infant’s actual outcome. The ROC analysis was conducted for the full cohort and stratified between the infants who were and were not receiving supplemental oxygen or positive airway pressure at the time of the examination.

Sample Size

We chose to record the respiratory physical examinations of a feasible number of 30 infants. The necessary number of physician raters was calculated accordingly. On the basis of a hypothesized ICC for each examination finding of 0.6, a sample size of 23 physicians would provide 80% power with a type I error of 0.05 to measure an ICC with a 95% CI of ±0.15. This confidence width was selected to distinguish with statistical significance between an ICC in the “fair” range and one in the “poor” or “excellent” range. To account for the possibility of incomplete data (ie, some physicians not reviewing all recorded examinations), the physician sample size was increased by 30% resulting in a target enrollment of 30 physicians.

Results

Between July 2013 and March 2015, 30 infants were enrolled and underwent recording of a respiratory physical examination at 36–40 weeks’ PMA (Table II). We invited 32 physicians from 9 academic centers to review the recorded examinations (Table II). Neonatology was the most common specialty (59.4%), and most physicians (78.1%) were attending neonatologists or pediatric pulmonologists rather than trainees in these subspecialties.

Table II.

Characteristics of study participants

Characteristics
Infant subjects (n = 30)
Gestational age, wk, median (range)	27.4 (23.3–29)
Birth weight, g, median (range)	846 (455–1260)
PMA at examination, wk, median (range)	38.6 (36–40)
Weight at examination, g, median (range)	2463 (1780–4600)
Respiratory support at examination, n (%)
Room air	16 (53.3)
Nasal cannula	7 (26.7)
Nasal CPAP	4 (13.3)
Invasive mechanical ventilation	2 (6.7)
Physician subjects (n = 32)
Neonatologists, n (%)	19 (59.4)
Pediatric pulmonologists, n (%)	13 (40.6)
Attending physicians, n (%)	25 (78.1)
In-practice ≥10 y, n (%)	13 (40.6)

CPAP, continuous positive airway pressure.

Comparison of In-Person and Recorded Examinations

Two study physicians (E.J., H.P.) completed in-person respiratory physical examinations on 5 infants (2 breathing room in air, 2 receiving noninvasive positive airway pressure, and 1 receiving mechanical ventilation) at the time of recording of the examination. The same physicians reviewed the recorded examinations 2–4 weeks after the in-person examinations. The reliability between the recorded and in-person examinations was excellent (ICC 0.84; 95% CI 0.71–0.96). The only points of disagreement between the 2 examinations were for visual findings.

Inter-Rater Reliability of the Examination Findings

Each physician reviewed a median of 20 (IQR 18–30) recorded examinations. Table III shows the inter-rater reliability of the 10 examination findings. The ICC values indicate that the interobserver reliability ranged from “fair” to “poor.” None of the findings showed “excellent” agreement. Subcostal retractions had the greatest inter-rater reliability and were the most frequently identified abnormal finding. Despite this observation, there was no significant correlation between how frequently the finding was reported as present and the level of inter-rater reliability. The level of interobserver reliability was also not consistently higher or lower when infants were stratified by the use of supplemental respiratory support (Table III).

Table III.

Inter-rater reliability of the examination findings

Examination findings*	Frequency of a “present” finding (%)	ICC (95% CI) all infants (n = 30)	ICC (95% CI) infants on respiratory support (n = 13)	ICC (95% CI) infants not on respiratory support (n = 17)
Visual
Subcostal retractions	78.2	0.73 (0.57–0.85)	0.48 (0.15–0.82)	0.77 (0.57–0.89)
Suprasternal retractions	38.3	0.69 (0.53–0.81)	0.69 (0.47–0.85)	0.75 (0.53–0.89)
Head bobbing	22.5	0.67 (0.50–0.81)	0.65 (0.40–0.85)	0.72 (0.48–0.88)
Nasal flaring	14.9	0.47 (0.29–0.67)	0.55 (0.28–0.79)	0.43 (0.19–0.71)
Thoracoabdominal synchrony	43.3†	0.31 (0.19–0.47)	0.48 (0.26–0.71)	0.27 (0.13–0.49)
Intercostal retractions	28.4	0.29 (0.17–0.45)	0.21 (0.07–0.47)	0.39 (0.20–0.64)
Expiratory muscle use	30.6	0.22 (0.11–0.41)	0.20 (0.06–0.47)	0.16 (0.04–0.46)
Auscultatory
Stridor	13.1	0.54 (0.35–0.71)	0.52 (0.25–0.78)	0.69 (0.41–0.88)
Crackles	27.7	0.35 (0.21–0.51)	0.30 (0.13–0.54)	0.19 (0.07–0.44)
Wheezy or noisy breathing	41.6	0.34 (0.20–0.50)	0.50 (0.28–0.72)	0.19 (0.07–0.40)

^*All findings were assessed as present or absent except synchronous or asynchronous thoracoabdominal movement.

^†This proportion refers to the frequency of asynchronous thoraco-abdominal movement.

Physician Prediction of Infant Future Respiratory Morbidity

A total of 8 infants (26.7%) remained hospitalized or were readmitted for a respiratory illness during the 3-month period following recording of the respiratory physical examination. Of those, 5 remained hospitalized, and 3 were discharged and readmitted. The reasons for readmission were an apparent life-threatening event due to suspected aspiration, a viral upper respiratory tract infection, and pneumonia. Six of the infants who remained hospitalized or were readmitted received supplemental respiratory support at the time of the recorded physical examination, and 2 were breathing in room air.

The median estimated probability of continued admission or readmission within 3 months was 30% (IQR 15%–50%) for the 8 infants with this outcome and 10% (IQR 5%–30%) for the 22 infants without the outcome. The area under the ROC curve for prediction of continued admission or readmission was 0.81 (95% CI 0.78–0.86). When stratified by the use of supplemental respiratory support at the time of the examination, the accuracy of physician prediction was significantly greater for infants receiving supplemental support (area under the ROC curve 0.90, 95% CI 0.86–0.95) compared with those breathing in room air (area under the ROC curve 0.71, 95% CI 0.66–0.75) (P value for comparison of ROC curves <.001).

Discussion

The structured respiratory physical examination has been used in the diagnosis and management of suspected pulmonary disease for nearly 200 years.^25,26 Despite this long history, a growing body of evidence suggests that the respiratory physical examination is not truly “objective.” Studies in children with acute respiratory conditions report wide variability in interobserver agreement for respiratory examination findings.^9–18 To our knowledge, however, none evaluated the interobserver reliability of the respiratory physical examination in premature infants. As components of the respiratory examination conducted at or near term corrected gestation in extremely preterm infants currently are under investigation as potential predictors of future respiratory morbidity, we undertook the present study to address this knowledge gap. Our results show that physicians often disagree on examination findings in this population. According to conventional interpretation of ICC values,²⁴ one-half of the evaluated findings had “fair” agreement and one-half had “poor” agreement. The level of interobserver agreement was not associated with how frequently an abnormal finding was identified. There was also no consistent difference in agreement when infants were stratified by the use of respiratory support.

The examination findings with the greatest interobserver reliability in this study (subcostal retractions, suprasternal retractions, and head bobbing) were visual. Only 1 of the 3 auscultatory findings (stridor) demonstrated “fair” agreement. In other studies conducted in young children, neither visual nor auscultatory findings consistently demonstrated better agreement.^{9,11–13,17,18} In a study of 158 children ages 3 months to 5 years who presented to a single emergency department with suspected croup, Chan et al⁹ found that the interobserver agreement between emergency department physicians and nurses was greatest for retractions, albeit only in the “moderate” range. The assessment of air entry demonstrated the lowest agreement of 6 examination findings included in an illness severity score.⁹ In contrast, among 56 infants younger than 1 year of age with symptoms suggestive of an acute lower respiratory tract infection, Margolis et al¹⁸ reported greater agreement for expiratory wheezing and audible wheezing detected without the use of a stethoscope than for retractions and use of accessory respiratory muscles.

One potential etiology for the fair-to-poor interobserver agreement found in the present study is intrapatient variability of examination findings. Studies in which the authors use respiratory plethysmography and esophageal pressure manometry indicate that respiratory mechanics can vary considerably from breath to breath in preterm infants with and without BPD.^27–29 Several physicians participating in this study reported that they observed certain findings to wax and wane throughout the examination, making it difficult for them to decide whether to classify the finding as “present” or “absent.” The use of additional descriptors for examination findings in this study may have allowed physicians to more precisely classify the frequency or severity of abnormal findings; however, doing so would also increase the opportunity for disagreement.

To our knowledge, previous studies that assessed interobserver reliability of the respiratory examination in pediatric patients were either conducted at single centers or compiled results from individual centers. We used audio-video recordings of examinations to measure interobserver agreement among physicians at 9 academic centers throughout the US. Consistent with the PROP study protocol, we did not provide definitions or examples of abnormal findings. Standardized definitions of examination findings can improve interobserver agreement in multicenter studies,^30,31 but these may be used infrequently in studies that include respiratory examination findings as outcomes. A systematic review of pediatric drug and vaccine trials that included wheeze as an adverse event reported that only 26% of the trials provided a definition of wheezing.³² The strong agreement between the in-person and recorded examinations in this study suggests that digital audio-video recordings may be a reliable means for future trials and multicenter studies to standardize assessment of respiratory findings.

We found that physicians using findings from a single respiratory physical examination predicted with good accuracy whether an infant would either remain hospitalized for at least 3 more months or be readmitted for a respiratory illness during that time.³³ When the cohort was stratified by respiratory support, the predictive accuracy for the infants receiving supplemental oxygen or positive airway pressure was excellent but was only fair for those breathing in room air. This finding, particularly in light of our observation that interobserver agreement did not vary significantly based on respiratory support, suggests that physician prediction may depend more heavily on the presence or absence of supplemental respiratory support than specific examination findings.

With increasing use of advanced imaging and point-of-care diagnostic technology, some consider the physical examination a “lost art.”^34,35 Others point to the limited accuracy of the physical examination for many conditions^36–39 and suggest that these technologies appropriately reduced the role of the physical examination in treatment decisions.⁴⁰ To be of clinical utility as a diagnostic test, physical examination findings must be precise (high interobserver reliability) and accurate compared with a corresponding “gold standard.” In this cohort of academic clinicians, of whom 40% have been in practice for more than 10 years, the interobserver reliability for most of the assessed findings was not sufficient for use as a diagnostic test. Although the clinicians’ ability to predict short-term respiratory morbidity from a single examination demonstrated moderate discriminatory power, the predictive accuracy was significantly influenced by the infant’s use of respiratory support. We conclude that in the absence of measures to improve interobserver agreement between physicians, the respiratory physical examination in preterm infants should not be used as a surrogate endpoint in multicenter studies.

Glossary

BPD

Bronchopulmonary dysplasia

ICC

Intraclass correlation coefficient

NICU

Neonatal intensive care unit

PMA

Postmenstrual age

PROP

Prematurity and Respiratory Outcomes Program

ROC

Receiver operating characteristic