MISCLASSIFICATION OF ACUTE RESPIRATORY DISTRESS SYNDROME AFTER TRAUMATIC INJURY: THE COST OF LESS RIGOROUS APPROACHES

Abstract

BACKGROUND

Adherence to rigorous research protocols for identifying acute respiratory distress syndrome (ARDS) after trauma is variable. To examine how misclassification of ARDS may bias observational studies in trauma populations, we evaluated the agreement of two methods for adjudicating ARDS after trauma: the gold standard, direct review of chest radiographs and review of dictated radiology reports, a commonly used alternative.

METHODS

This nested cohort study included 123 mechanically ventilated patients between 2005–2008, with at least one PaO2:FiO2 <300 within the first 8 days of admission. Two blinded physician investigators adjudicated ARDS by two methods. The investigators directly reviewed all chest radiographs to evaluate for bilateral infiltrates. Several months later, blinded to their previous assessments, they adjudicated ARDS using a standardized rubric to classify radiology reports. A kappa statistics was calculated. Regression analyses quantified the association between established risk factors as well as important clinical outcomes and ARDS determined by the aforementioned methods as well as hypoxemia as a surrogate marker.

RESULTS

The kappa was 0.47 for the observed agreement between ARDS adjudicated by direct review of chest radiographs and ARDS adjudicated by review of radiology reports. Both the magnitude and direction of bias on the estimates of association between ARDS and established risk factors as well as clinical outcomes varied by method of adjudication.

CONCLUSION

Classification of ARDS by review of dictated radiology reports had only moderate agreement with the gold standard, ARDS adjudicated by direct review of chest radiographs. While the misclassification of ARDS had varied effects on the estimates of associations with established risk factors, it tended to weaken the association of ARDS with important clinical outcomes. A standardized approach to ARDS adjudication after trauma by direct review of chest radiographs will minimize misclassification bias in future observational studies.

LEVEL OF EVIDENCE

Level III; Epidemiologic

BACKGROUND

The Acute Respiratory Distress Syndrome (ARDS) is associated with a nearly three-fold increase in odds of death in patients suffering traumatic injury.^1,2 Landmark randomized controlled trials (RCT) have established a standard approach for adjudication of ARDS in hypoxemic subjects using direct review of chest radiographs by physician investigators to identify diffuse bilateral infiltrates, in accordance with the original American–European Consensus Conference (AECC) criteria and the Berlin Definition of ARDS.^3–8 Inter-observer variability in chest radiograph interpretations is mitigated by two-physician consensus approach and review of serial images.⁹ In fact, a two-physician approach to lung injury scoring correlates well with clinical ARDS and ex-vivo lung pathology.¹⁰ Although it is labor-intensive, direct review of chest radiographs accurately identifies patients with pulmonary edema, and is the de facto gold standard for identifying ARDS in research studies.

A variety of alternative approaches to identify ARDS have been described in the trauma literature. Some study protocols screen dictated radiograph reports for patients with hypoxemia or use arterial hypoxemia without evaluation for bilateral infiltrates as a surrogate for ARDS, while others rely on ICD-9 codes or chart review by research coordinators.^2,11–14 The later two approaches are particularly problematic as ARDS is under-recognized by bedside clinicians.^15,16 Hypoxemia without review of chest radiography is a poor surrogate for ARDS after trauma and encompasses a clinically heterogeneous group of patients with cardiopulmonary dysfunction.¹⁷ It is common to state that ARDS cases were defined by AECC or Berlin Criteria, but important details outlining how chest imaging was evaluated are often omitted from the methods section of published manuscripts.^18–20

A lack of adherence to a standard approach for identifying ARDS cases may contribute to substantial variability in even the most basic epidemiologic measures. Recently reported estimates of the incidence of ARDS range from 5.2% to 31%.^{11,18,21–24} It is difficult to predict and adjust for the bias created by highly variable approaches to outcome ascertainment in observational studies. ^25–27 Substantially biased estimates of association can be driven by the misclassification of a few patients.²⁸

We hypothesized that relying on dictated radiology reports, rather than direct review of chest radiographs, would result in substantial misclassification of ARDS in a cohort of patients with severe traumatic injuries who required mechanical ventilation. We designed a protocol to evaluate the agreement between the two methods for ascertaining ARDS status: the gold standard, direct review of chest radiographs by two clinician investigators, and a commonly used alternative, a systematic approach to review of radiology reports of the same chest radiographs.

METHODS

This is a nested cohort study of consecutive patients enrolled between 2005–2008 in the Acute Coagulation and Inflammation of Trauma (ACIT) study, described in detail previously.^29,30 All adult patients who met criteria for full trauma team activation at San Francisco General Hospital, a level 1 trauma center, were eligible for enrollment. The Institutional Review Board of the University of California at San Francisco approved this study and informed consent was obtained from patients or their surrogates. Patients were followed for the duration of their hospital stay or up to 28 days after injury. Intubated subjects were eligible for this sub-study if they survived ≥ 6 hours after hospital arrival. During the study period, 233 patients met these criteria and of these, 123 had at least one PaO₂:FiO₂ of <300 within the first 8 days of admission and underwent ARDS adjudication. Studying ARDS after trauma is a major interest in this research group and our data collection protocols and analyses facilitate identification of patients who meet the Berlin Definition of ARDS Criteria.⁸ The timing of ARDS must be within one week of a known clinical insult and the 8-day study period insures that any radiographic abnormalities are acute and occur after trauma, an established risk factor for ARDS. We defined the gold standard outcome identification as ARDS per the Berlin definition using direct physician review of all chest radiographs for acute, bilateral infiltrates within 24 hours of documented hypoxemia (PaO₂:FiO₂ of <300), and exclusion of cardiogenic causes of hydrostatic pulmonary edema. Although the Berlin Criteria do not require echocardiogram evaluations or the use of pulmonary artery catheter wedge pressures to exclude fluid overload, the guidelines suggest that in ARDS pulmonary infiltrates are not fully explained by hydrostatic edema. As a quality assurance measure our group performs chart review of all radiographically identified ARDS cases to exclude subjects with a significant component of cardiogenic pulmonary edema. All patients with severe left ventricular dysfunction or volume overload noted on any echocardiogram performed in the first week of admission are excluded from analyses. For all ARDS cases, we review admission and discharge notes as well as dictated progress notes for the day before, the day of, and the day after alveolar infiltrates were present on chest radiographs looking for any indication of clinical suspicion of hydrostatic pulmonary edema. In the larger cohort, of 234 subjects enrolled between 2005 and 2011 only 5 subjects (2%) had clinician documentation or echocardiographic evidence of severe left ventricular dysfunction or volume overload. We excluded these patients from our ARDS analyses.

Direct Review of Chest Radiographs for Bilateral Infiltrates

Two senior physician investigators board-certified in Critical Care Medicine reviewed all chest radiographs from the first 8 days of admission for each patient with at least one PaO₂:FiO₂ of <300. The incidence of ARDS after trauma is highest in the first few days after injury and this time frame also insures that radiographic infiltrates are acute.³¹ The physician investigators were blinded to the treating physicians’ assessments. The investigators directly reviewed the images for the presence of bilateral infiltrates and classified subjects after review of all available films as definite ARDS, equivocal/difficult to interpret, or no ARDS.³² Films were considered positive for ARDS if there was radiographic evidence of bilateral opacities not fully explained by effusions, lobar/lung collapse, or nodules, in accordance with the Berlin Definition. If there was disagreement, the radiographs were re-evaluated and the clinicians discussed the case until a consensus was reached in all cases.

Review of Dictated Reports of Chest Radiographs

All chest radiographs in this study were obtained for clinical care and had associated radiology reports. Printed reports for each subject included the same patient demographic data and dates available during the direct review of chest radiographs, and were arranged in chronological order in research binders. Several months after initial adjudication by direct review of radiographs, the same two clinicians, blinded to the results of the previous classification, used a rubric to classify ARDS outcomes for the 123 subjects as positive, negative, or equivocal by reading each of the radiologist-dictated reports in the research binders (Table 1). If any dictated report qualified as having evidence of ARDS, this finding trumped all other readings and the subject was classified as “positive” for ARDS. If any dictated report qualified as equivocal for ARDS, that subject was classified as equivocal. If there was disagreement between the two-physician reviewers, the subject’s radiograph reports were re-evaluated and the clinicians discussed the case until a consensus was reached.

Table 1.

Classification System for ARDS in Dictated Chest Radiograph Reports

Positive

Mention of ALI or ARDS

Consistent with pulmonary edema

Bilateral infiltrates or opacities consistent with pneumonia

Bilateral infiltrates or opacities consistent with aspiration

Left and right-sided infiltrates or opacities as above but described in separate sentences.

Equivocal

Bibasilar opacities consistent with atelectasis or pneumonia

Possible pulmonary edema

Interstitial opacities or infiltrates

Ambivalent expressions alluding to a bilateral process including: “Cannot rule out…” “Might be…” “May represent…” “Consistent with ‘X’ but consider ‘A,’ ‘B,’ and ‘C’ diagnoses.

Negative

“Clear” film

Pleural effusions only

Atelectasis only

Clearly unilateral process- airspace consolidation and/or atelectasis

No pulmonary edema, without mention of consolidation, pneumonia, or aspiration

Definitions of abbreviations: ALI=Acute Lung Injury; ARDS = Acute Respiratory Distress Syndrome

Statistical Analysis

We described the demographic and clinical characteristics of the cohort with appropriate statistical measures. We tested the differences in demographic features and clinical risk factors between subjects with discordant outcome adjudications and those who had the same ARDS outcome classified under each approach using unordered Χ², t-test, and rank sum analyses as appropriate. Differences in factors that have been associated with ARDS after trauma and characteristics that might make radiographic interpretation more challenging, including body mass index (BMI), metabolic base deficit >5.0 on arrival, Injury Severity Score (ISS) ≥25, Chest Abbreviate Injury Score, and presence of rib fractures, were evaluated using appropriate statistical tests.^19,33–39

We calculated an unadjusted kappa statistic as a measure of inter-rater reliability. We performed sensitivity analyses to determine if the kappa statistic would improve under the following four approaches for classifying the interpretation of chest radiograph reports: 1.) All equivocal subjects classified as “positive.” 2.) All equivocal subjects classified as “negative.” 3.) A weighted kappa with all equivocal films assigned an agreement of 0.5. 4.) Exclusion of equivocal patients from the analysis.

To examine how misclassification of ARDS may influence findings from clinical studies conducted in this cohort of critically ill trauma patients, we performed linear and logistic regression analyses. For these analyses, we used data from the cohort of all intubated ACIT subjects enrolled from 2005–2008 and survived ≥ 6hrs (n=233) that gave rise to the 123 hypoxemic patients evaluated for ARDS by the two methods described above. Univariate logistic regression models were performed to evaluate the association between established risk factors and ARDS outcome ascertained in three ways: direct review of radiographs, review or radiology reports, and hypoxemia alone as a surrogate for ARDS. For each clinical outcome considered here, a model with the predictor defined as ARDS adjudicated by direct review of radiographs was compared to two other models: one with the independent variable of ARDS adjudicated by radiology report and a second with hypoxemia alone as a surrogate maker for ARDS. Univariate linear regression models were performed for hospital days, ICU days, and days of mechanical ventilation and excluded patients who died. Univariate linear regression analyses for ventilator free days were also performed. All linear regression models were then repeated adjusting for ISS >25 and age. A similar approach was used to compare results of univariate and multivariate logistic regression models for the outcome of 28-day mortality. Subjects adjudicated as “equivocal” ARDS status were excluded from the analyses to reduce misclassification bias.³² An α= 0.05 was considered significant. All analyses were performed by the authors with Stata version 13 (StataCorp, College Station, TX).

RESULTS

Demographics and Clinical Features

The cohort of hypoxemic patients who underwent two methods of ARDS adjudication was representative of a standard trauma population (Table 2). Subjects were predominantly male (80%), and the median age was 42 years. Subjects were severely injured (Median ISS = 29); 20% had severe chest injury (AIS>3), and most presented with a substantial base deficit (median −6.6). All cause 28-day mortality was 28%.

Table 2.

Demographic and Clinical Characteristics of 123 Hypoxemic Study Subjects with Adjudicated ARDS Status

Age (years): mean ± SD	42 ± 19
Male sex, n (%)	98(80)
Race, n (%)
White	66(54)
Black	33(27)
Asian	21(17)
Unknown or Other	3(2)
Latino Ethnicity: n (%)	21(17)
BMI: mean ± SD	27 ± 6
Blunt injury: n (%)	82 (67)
Injury Severity Score: median (IQR)	29 (21–43)
Head AIS† >3: n (%)	74 (60)
Chest AIS†>3: n (%)	25 (20)
Arrival Base Excess: median (IQR)	−6.6 (−9.6 – −4.3)
PRBC‡ Transfused first 24 hrs: median (IQR)	3 (0 –8)
Mortality at 28 days n (%)	35(29)

ARDS = Acute Respiratory Distress Syndrome

^†Abbreviated Injury Score

^‡Packed Red Blood Cell units

Agreement of ARDS Classification methods: Review of Radiology Report Compared to the Gold Standard of Direct Interpretation of Chest Radiographs

ARDS determined by two-physician direct review of chest radiographs was present in 70 (57%) of subjects with hypoxemia (PaO2:FiO2 <300). ARDS was present in 55 (45%) of hypoxemic subjects when classifying outcome status by radiology report. No subjects were classified as equivocal after direct review of radiographic images, but 4 (3%) of subjects were classified as equivocal for ARDS when reviewing radiology reports.

A kappa statistic was calculated using the raw data (Figure 1A.) The observed agreement was 72% and the kappa was 0.47, indicating only moderate agreement between the two methods (Figure 1B). In 89 subjects with classification agreement for ARDS between the two methods, 43 (48%) subjects were classified as negative and 46 (52%) subjects were classified as positive. Three of the four subjects classified as equivocal on review or radiology reports were classified as having ARDS by direct review of radiographs. Weighted kappa statistics calculated to handle disagreements in ARDS status among subjects who are classified as “equivocal” ARDS status by dictated reports did not substantially improve the agreement between the two methods.

Figure 1. Agreement of two adjudication methods for ARDS after traumatic injury.

Raw data is shown to demonstrate the agreement of two adjudication methods of ARDS after traumatic injury in intubated, hypoxemic patients (1A). From the raw data Kappa statistics can be calculated under a variety of schemata to handle the equivocal subjects (1B). In the weighted Kappa (†) calculation, equivocal subjects were assigned 0.5 agreement with either positive or negative findings by alternative method. The two adjudication methods show only moderate agreement under all classification schemata.

Clinical and Demographic Characteristics of Subjects with Discordant and Concordant ARDS Outcomes Adjudication

We did not identify any significant clinical or demographic differences between subjects who had agreement in outcome classification under the two systems, concordant ARDS classification, and those who had discordant ARDS outcomes (Table 3). Furthermore, we did not identify any significant differences in patient-specific factors that could make evaluation chest radiographs more challenging. Specifically, BMI, rate of blunt injury, presence of chest injury, severity of chest injury, and chronic lung diseases were no different between groups.

Table 3.

Clinical and Demographic Characteristics of Subjects by Agreement of ARDS Outcome Adjudication using Direct Radiograph Review and Radiology Report Review.

	Discordant (n=34)	Concordant (n=89)	P value
Age (years): mean ± SD	42 ± 18	42 ± 19	0.89
Male: n (%)	30 (88)	68 (76)	0.15
Race: n (%)			0.60
White	18 (53)	48 (54)
Black	7(21)	26 (29)
Asian	8 (23)	13 (15)
Other	1 (3)	2 (2)
Latino: n (%)	6 (18)	15 (17)	0.92
BMI♯: mean ± SD	29 ±6	27±5	0.12
Blunt: n (%)	20 (59)	62 (70)	0.25
ISS median IQR	29 (25–34)	27 (21–34)	0.39
AIS† Chest>3: n (%)	9 (26)	16 (18)	0.30
Any Rib Fractures: n (%)	18 (53)	45 (51)	0.81
PRBCS‡ in first 24h (units): Median IQR	2 (0–7)	3 (0–9)	0.65
EDϒ Arrival Base Deficit: Median IQR	−8.1 (−4.2–−10.4)	−6.6 (−4.5 – −9.2)	0.14
EDϒ Arrival Heart Rate: Median IQR	100 (81–119)	105 (88–121)	0.34

^♯Body Mass Index

^†Abbreviated Injury Score

^‡Packed Red Blood Cell units

^ϒEmergency Department

Logistic Regression Models to Evaluate the Effects of Misclassification Bias on Estimates of Association between ARDS and Established Clinical Risk Factors

In univariate logistic regression analyses, there was no discernable pattern to the magnitude and direction of bias of the association between clinical risk factors and ARDS determined by direct review of radiographs compared to ARDS determined by review of radiology reports (Table 4, Figure 2). For example, the odds of ARDS were higher in subjects who received massive transfusion when radiology reports were used to adjudicate ARDS compared to direct review of radiographs. Oppositely, the odds of ARDS were lower in subjects with ISS ≥ 25 when radiology reports were used to adjudicate ARDS compared to direct review of radiographs. Using hypoxemia alone as a surrogate for ARDS weakened the association between all analyzed risk factors and ARDS.

Table 4.

Effect of ARDS Adjudication Method on the Association of Known ARDS Risk Factors in Univariate Logistic Regression Models

	Odds of ARDS by Direct Review (95%CI)	Odds of ARDS by Radiology Report (95%CI)	Odds of ARDS by Hypoxemia Alone (95%CI)
Metabolic Base Deficit <−5.0 on ED* Arrival	3.1 (2.7–5.7)	3.2 (1.6–6.3)	2.7 (1.6–4.6)
Injury Severity Score ≥ 25	3.3 (1.7–6.1)	2.6 (1.3–5.0)	2.0 (1.2–3.4)
Massive Transfusionϒ	2.1 (1.1–3.9)	2.4 (1.2–4.8)	1.2 (0.7–2.4)

Effect of Adjudication Method on the Association of ARDS and Clinical Outcomes among Survivors in Unadjusted and Adjusted Linear Regression Models
	Direct Review	Radiology Report	Hypoxemia Alone
	Point Estimate for β Coefficient (95%CI)	Point Estimate for β Coefficient (95%CI)	Point Estimate for β Coefficient (95%CI)
Hospital Days	18.0 (7.4–28.6)	13.0 (1.0–25.0)	9.0 (−0.9–18.8)
Hospital Days Adjusted for ISS‡	7.5 (−2.7–17.7)	4.8 (−6.1–15.7)	1.7 (−7.3–10.7)
ICU Days	8.9 (5.9–11.8)	8.1 (4.8–11.5)	5.6 (2.8–8.4)
ICU Days Adjusted for ISS	5.5 (2.8–8.3)	5.6 (2.7–8.5)	3.2 (0.7–5.7)
Ventilator Days	8.6 (5.6–11.6)	8.4 (5.0–11.3)	5.4 (2.5–8.2)
Ventilator Days Adjusted for ISS	5.4 (2.6–8.3)	5.9 (2.9–8.9)	3.1 (0.6–5.7)
Ventilator-Free Days	−7.4 [−10.5–(−4.2)]	−7.3 [−10.8–(−3.9)]	−3.9 [6.9–(−0.9)]
Ventilator-Free Days Adjusted for ISS	−4.0 [−6.8–(−1.3)]	−4.6 [−7.6–(−1.7)]	−1.7 [−4.2–0.9)]

^*Emergency Department (ED)

^ϒ≥10units of packed red blood cells in 24 hours

^‡Injury Severity Score (ISS)

Figure 2. Association of ARDS and Clinical Outcomes in Linear Regression Models.

Graphical presentation of findings from linear regression models showing the point estimates and 95%CI of the β coefficients on the ARDS predictor variable for various clinical outcomes. All analyses excluded patients who died. Each style of marker represents a different method for adjudicating ARDS in hypoxemic trauma patients: direct review of chest radiographs, systematic evaluation of dictated reports of chest radiographs, and use of hypoxemia alone without consideration of chest radiograph findings. Multivariate models are adjusted for and injury severity score (ISS). The findings for the outcome Ventilator-Free Days were similar to ventilator days (Table 4B) but are not graphed here in order to streamline the visual presentation by limiting the range of values on the y-axis.

Linear Regression Models to Evaluate the Effects of Misclassification Bias on Estimates of Association between ARDS and Clinical Outcomes

Linear regression models were used to compare the estimates of association between ARDS defined by different adjudication methods and several clinical outcomes (Table 4, Figure 2). The unadjusted and adjusted estimates of the association between ARDS and hospital days were substantially weaker (18.0 vs. 13.0 and 7.5 vs. 4.8), when ARDS status was determined by review of dictated radiology reports compared to ARDS classified by direct review of radiographs (Table 4). Other clinical outcomes were less affected by the method of radiograph adjudication. In unadjusted models, the association between ICU days and ARDS was stronger when ARDS was adjudicated by direct review of radiographs (8.9 vs. 8.1). This difference was no longer apparent after adjusting for ISS, (5.5 vs. 5.6). In unadjusted models adjudication by direct review or radiology report resulted in similar associations between ARDS and duration of mechanical ventilation (Table 4). Notably, adjusting for ISS affected the estimate of association differently depending on the adjudication method used for the model. Specifically, the adjusted association between ventilator days and ARDS was stronger when ARDS was adjudicated by radiology report when compared to direct review. Adjusting for age did not significantly change the findings from the multivariate models described above (results not shown). In all adjusted and unadjusted models, the associations between ARDS and clinical outcomes were weaker when ARDS was defined by hypoxemia alone compared to ARDS determined by direct review of chest radiographs or radiology report.

There was no difference in the odds of 28-day mortality between classification strategies for ARDS. In the univariate logistic regression models the odds of death at 28-days were 1.1 (0.6–1.9) for subjects with ARDS by direct radiograph review, 1.3 (0.7–2.4) for subjects with ARDS by radiology report, and 0.8 (0.4–1.3) for subjects with hypoxia alone. Multivariate models adjusting for age and ISS showed similar patterns: 0.9 (0.4–1.7), 1.1 (0.5–2.2), and 0.6 (0.3–1.1) respectively.

DISCUSSION

We compared two protocols for ARDS adjudication in a cohort of severely injured trauma patients and found only moderate agreement between these two methods. The use of less rigorous classification systems had unpredictable effects on the association between ARDS and established risk factors as well as clinical outcomes. Our findings suggest that protocols that do not directly review chest radiographs introduce misclassification bias and may mask important findings or cause false associations to become statistically significant.

The Berlin definition of ARDS specifies that the radiographic findings must include, “Bilateral opacities—not fully explained by effusions, lobar/lung collapse, or nodules,” and the panel published example images of chest radiographs from subjects ARDS with their recommendations.⁸ Unlike research investigators with a directed task to evaluate chest radiographs for the findings described above, radiologists dictating reports for clinical care are evaluating the entire image and are unlikely to specifically address whether the opacities are fully explained by other findings. The expert panel identified more extensive radiographic involvement (3 or 4 quadrants) as an area for further investigation in defining severe ARDS. The careful attention to radiographic findings in the Berlin Definition supports the use of protocols that use investigators to directly review chest radiographs for ARDS adjudication.

The importance of rigorous case identification in the study of syndromes lacking a true gold standard is highlighted by large randomized controlled trials (RCTs) of treatments for the Acute Coronary Syndrome (ACS). These studies use research committees to directly review electrocardiogram tracings to identify myocardial infarction (MI). Data from an international RCT of eptifibatide in over 10,000 patients presenting with ACS showed that the use of a central adjudication process using systematic two-physician direct review of electrocardiogram (ECG) images was important to provide unbiased identification of suspected MIs.⁴⁰ ACS investigators do not rely on ECG reports generated for clinical purposes, but instead use direct investigator review of this data. We argue that ARDS research studies should similarly adhere to protocols using consensus opinion of investigators directly reviewing chest radiographs for evidence of ARDS.

Although implementing a rigorous algorithm for identifying ARDS is resource intensive, our findings support a more uniform approach to identifying ARDS cases in trauma cohorts in. Moreover, because the methods of ARDS ascertainment may influence study results, methods sections should explicitly describe how investigators adjudicated ARDS. Details including blinding status, consensus review process, and direct review of radiographs provide more useful information than stating adherence to AECC or Berlin Criteria.

To our knowledge, this is the first study comparing the agreement of two commonly used methods for ARDS adjudication in trauma cohorts: evaluation of dictated radiology reports and direct interpretation of chest radiographs in hypoxemic critically ill patients. The strengths of this study include a detailed description of the adjudication protocol that will be useful to investigators designing future observational studies. We provide concrete examples of how misclassification of ARDS through less resource intensive protocols causes variable bias of study conclusions about both predictors and outcomes.

This study has several limitations. This cohort is relatively small and was designed primarily to evaluate the agreement between the two methods descried here. It may not be adequately powered to identify differences between subjects with discordant classification and those with concordant classification. The small sample size likely contributes to the relatively large confidence intervals around the estimates of association. We acknowledge that inter-rater reliability may be problematic in direct review of chest radiographs for ARDS adjudication and may be more important in multi-site studies.^2,9,41 There is no standardized approach to adjudication by radiology report so we cannot speculate on the magnitude and direction of biases introduced by different rubrics that use similar methods to those tested here. Although it is possible that these findings are specific to our institution, the clinician investigators involved in this study, or the radiologists providing clinical care this is unlikely in light of previously published studies describing inter-rater reliability and rates of ARDS among acutely hypoxemic patients.^1,9,32

Although direct review of chest radiographs is the most rigorous method for ARDS adjudication it is resource intensive. There is some data to support incorporating automated tools to efficiently identify potential cases of ARDS. Cases could be confirmed by direct review of radiographs in a similar approach to the central events committee used in large studies of ACS and MI. Azzam et al tested a computer algorithm against the gold standard of blinded physician investigators directly reviewing films. Although the kappa statistic calculated from their results is 0.72, indicating substantial agreement, misclassification introduced by this tool could still be an important source of bias.⁴² Further studies are needed to test a protocol that combines the efficiency of computer algorithm with confirmatory adjudication against rigorous direct review adjudication of all intubated subjects with hypoxemia.

CONCLUSIONS

Classification of ARDS by review of dictated radiology reports has only moderate agreement with ARDS adjudicated by the current gold standard method, two-physician consensus after direct review of chest radiographs, (kappa = 0.47). The misclassification of outcomes using dictated reports alone introduced noise into the system, with unpredictable effects on the bias of the associations between ARDS and several clinical risk factors and outcomes. Using hypoxemia alone as a surrogate for ARDS weakens the estimates of association. Without rigorous adjudication protocols studies of ARDS after trauma may fail to identify valuable treatment strategies or generate informative prediction tools. Ultimately, continued efforts to develop sophisticated computer algorithms are needed to improve efficiency, precision, and accuracy of case identification for future observational and interventional studies in ARDS. However, while ARDS research awaits more facile tools, adhering to the de facto gold standard of consensus approach to direct review of chest radiographs reduces bias in observational studies.