Validation of the International Labour Office Digitized Standard Images for Recognition and Classification of Radiographs of Pneumoconiosis

Abstract

Rationale and Objectives

Chest radiographs are recommended for prevention and detection of pneumoconiosis. In 2011, the International Labour Office (ILO) released a revision of the International Classification of Radiographs of Pneumoconioses that included a digitized standard images set. The present study compared results of classifications of digital chest images performed using the new ILO 2011 digitized standard images to classification approaches used in the past.

Materials and Methods

Underground coal miners (N = 172) were examined using both digital and film-screen radiography (FSR) on the same day. Seven National Institute for Occupational Safety and Health-certified B Readers independently classified all 172 digital radiographs, once using the ILO 2011 digitized standard images (DR_ILO2011-D) and once using digitized standard images used in the previous research (DR_RES). The same seven B Readers classified all the miners’ chest films using the ILO film-based standards.

Results

Agreement between classifications of FSR and digital radiography was identical, using a standard image set (either DR_ILO2011-D or DR_RES). The overall weighted κ value was 0.58.Somespecific differences in the results were seen and noted. However, intrareader variability in this study was similar to the published values and did not appear to be affected by the use of the new ILO 2011 digitized standard images.

Conclusions

These findings validate the use of the ILO digitized standard images for classification of small pneumoconiotic opacities. When digital chest radiographs are obtained and displayed appropriately, results of pneumoconiosis classifications using the 2011 ILO digitized standards are comparable to film-based ILO classifications and to classifications using earlier research standards.

Chest radiographs are recommended for the detection and prevention of pneumoconiosis in workers involved in dusty trades, such as underground mining (1). In clinical practice and public health surveillance, digital chest radiographs (DR) presented on medical-grade monitors have largely replaced the conventional film-screen radiograph (FSR) technology. The International Labour Office (ILO) Guidelines for the Classification of the Pneumoconioses has been an invaluable tool for standardization of interpretations of chest radiographs for epidemiologic studies of the pneumoconioses (2). To enhance accuracy and precision in applying the ILO classification scoring system, readers are required to perform a side-by-side comparison of each individual worker’s radiograph to one or more prototypical chest images, which illustrate a variety of types and severity of radiographic abnormalities induced by dust inhalation. The ILO classification system includes a standard set of chest images for comparison purposes. Until recently, the ILO classification system only provided a set of standard images in the film-screen radiograph (FSR) format. However, in 2011, the ILO revised its guidelines to “extend the applicability of the Classification to digital radiographic images of the chest” (2). In the 2011 revision of the classification, the ILO includes a set of electronic image files (ILO Standard Digital Images [2011-D]) that was digitized from the film-based standards included in the 2000 revision of the classification.

Prior to the adoption of the ILO Standard Digital Images (2011-D), a series of research studies was undertaken to assess any potential impact of image modality on the outcomes of ILO classifications of chest radiographs (comparing FSR to DR displayed on medical-grade diagnostic monitors) (1,3–5). These investigations obtained both FSR and DR chest radiographs from study participants on the same day. FSR chest radiographs were interpreted using the ILO 2000 version of the classification system with the traditional film-based standard images. To enable classification of the DR chest radiographs displayed as soft copies on a medical-grade computer monitor, an existing set of the ILO standard films was scanned and digitized (see Franzblau et al. (3) for methods). The resulting image files (“research” digitized standards) were used as the ILO standards for classifying digital images in a number of previous investigations (3–6). These “research” digitized standards appear quite similar to the current set of standard films that are included in the ILO 2000 classification, but were digitized from a prior version of the ILO standard films that had been issued with a black label rather than the current white label. The methods used in these modality studies required that at least two National Institute for Occupational Safety and Health (NIOSH)-certified B Readers¹ interpret each participant’s DR chest radiograph presented as a soft copy image side-by-side with the digitized ILO standard radiographs, using two identical medical-grade monitors. In brief, these studies concluded that, with appropriate attention to image acquisition and soft copy display, both of the widely available digital radiography systems² can be equivalent to FSR in the visualization and classification of small interstitial lung opacities.

Despite the subjective similarity between the “research” digitized standards and the new ILO Standard Digital Images (2011-D), to our knowledge, there is no objective evidence that classifications acquired using either set would be equivalent. The objective of this study was to evaluate the equivalence of the “research” digitized standards and the ILO Standard Digital Images (2011-D) using paired digital and film chest radiographic examinations performed on the same day in miners from our previous studies. We searched for systematic differences in both the level of abnormality reported and the intra- and inter-reader variability.

MATERIALS AND METHODS

The present study used results from three groups of readings. As part of an earlier research investigation, 1401 miners participating in the NIOSH Enhanced Coal Workers’ Health Surveillance Program completed both digital and film chest radiography on the same day (4). For the purposes of that earlier investigation, all the radiographs were independently classified by at least two of eight B Readers, and 172 miners were found to have a profusion of small pneumoconiotic opacities >0/0 by at least one Reader. To investigate a potential effect of image modality on within- and between-reader variability, a follow-up study obtained additional readings for each of the 172 digital and film chest radiographs, using seven of the eight B Readers from the original study (6). Detailed information on study subjects, image acquisition and processing, and image interpretation is available in those previous reports (4,6). The present investigation used the group of 1204 ILO classifications of traditional film-screen radiographs (FSR) obtained during the earlier studies (seven B Readers and 172 miners) and the group of 1204 classifications of digitally acquired chest images performed using the “research” digitized standards (DR_RES image set) described previously. For the purposes of the present study, a third group of classifications was obtained for the 172 digital chest radiographs by the same seven B Readers (1204 interpretations), but now using the ILO Standard Digital Images (2011-D) (DR_ILO2011-D image set). Each B Reader performed the additional classification of the digital images in a manner blinded from their own and other readers’ previous interpretations. In summary, the present study used these three groups of 1204 B Reader classifications (total 3612 observations) to investigate any potential differences in pneumoconiosis outcomes related to the specific set of standard images used. The results of traditional classification, in which both radiographs and ILO standard images were film-based, were considered the “gold standard” and compared to the classifications of the digital images performed using either the DR_ILO2011-D set or the DR_RES standard image set. The central tendency of small opacity profusion, the prevalence of ILO Category 1/0 or greater, and measures of reader variation were assessed using previously described analytic methods (6). Within-reader variability of small opacity profusion category was compared by the standard image set used for classification (FSR vs. DR_ILO2011-D and DR_RES vs. DR_ILO2011-D), using Cicchetti–Allison weighted κ values and 95% confidence intervals (CIs). In addition, small opacity profusion categories classified using the DR_ILO2011-D set were compared to the DR_RES set and FSR by calculating a global interset weighted κ value and Spearman’s correlation coefficient, using all readings in aggregate. Finally, small opacity profusion classifications using DR_ILO2011-D, DR_RES, and FSR were examined using Bowker’s test of symmetry. The SAS statistical software version 9.3 (SAS Institute, Cary, NC) was used for all analyses. Data collection for this study was approved by the NIOSH institutional review board (HSRB 12-DRDS-NR02).

RESULTS

Image Quality

Compared to the group of film radiographs, the digital images were more often classified as “good” (ILO technical quality category 1) using either the DR_RES set (prevalence ratio [PR], 1.46; 95% CI, 1.35–1.57) or the DR_ILO2011-D set (PR, 1.54; 95% CI, 1.43–1.66) (Table 1). For the digital radiographs, the proportion rated “good” quality was similar, irrespective of the digital standard image set used for the classifications (P = .13).

TABLE 1.

Image Quality Ratings for 3612 Classifications by Image Modality and Standard Set

Quality	FSR [n (%)]	DRRES [n (%)]	DRILO2011-D [n (%)]
Good	536 (44.5)	792 (65.8)	827 (68.7)
Acceptable, no defects	410 (34.1)	332 (27.6)	337 (28.0)
Acceptable, some defects	228 (18.9)	74 (6.1)	39 (3.2)
Unacceptable	15 (1.2)	0 (0.0)	1 (0.1)
Missing/no indication	15 (1.2)	6 (0.5)	0 (0.0)

DR_ILO2011-D, digital radiograph interpreted using digitized ILO reference standards (2011-D); DR_RES, digital radiograph interpreted using the “research” digitized standards; FSR, film-screen radiograph interpreted using 2000 ILO film-based reference standard; ILO, International Labour Organization.

Each chest radiograph from 172 miners was classified by seven National Institute for Occupational Safety and Health (NIOSH) B Readers for a total of 1204 classifications.

Profusion of Small Opacities

A total of 3552 classifications of images with acceptable technical quality had complete information on small opacity profusion. Most (63.4%) of the readings showed a small opacity profusion category of 0/0 for all three groups of readings (Fig 1).

Figure 1.

Profusion of small opacities for 1204 classifications by image type and in the case of digital images by reference standard used. Small opacity profusion designations are standard International Labour Organization (ILO) designations. Inset panel is data presented without “0/0” classifications for viewing purposes. DR_RES, digital radiograph interpreted using the “research” digitized standards; DR_ILO2011-D, digital radiograph interpreted using digitized ILO reference standards (2011-D); FSR, film-screen radiograph interpreted using 2000 ILO film-based reference standard.

Table 2 summarizes the frequencies and percentages of small opacity profusion scores by image modality and standard set used for the reading group. Distribution of small opacity profusion on digital images appears similar to FSR when either the research (DR_RES) or ILO comparison set (DR_ILO2011-D) was used in the reading (Table 2). However, readers interpreting digital radiographs found somewhat more radiographs showing small opacities (≥0/1) when using DR_ILO2011-D compared to FSR or DR_RES (41.6% vs. 37.2% and 31.0% respectively; DR_ILO2011-D vs. FSR PR, 1.12; 95% CI, 1.01–1.24, P = .03 [Fig 3, left column]; DR_ILO2011-D vs. DR_RES PR, 1.34; 95% CI, 1.21–1.50, P < .0001).

TABLE 2.

Small Opacity Profusion by Image Type and Standard Set

Profusion	FSR [n (%)]	DRRES [n (%)]	DRILO2011-D [n (%)]
0/0	743 (62.8)	817 (69.0)	691 (58.4)
0/1	133 (11.2)	109 (9.2)	133 (11.2)
1/0	136 (11.5)	121 (10.2)	158 (13.3)
1/1	93 (7.9)	61 (5.2)	85 (7.2)
1/2	28 (2.4)	26 (2.2)	46 (3.9)
2/1	24 (2.0)	19 (1.6)	33 (2.8)
2/2	19 (1.6)	16 (1.4)	21 (1.8)
2/3	6 (0.5)	10 (0.8)	10 (0.8)
3/2	1 (0.1)	5 (0.4)	6 (0.5)
3/3	1 (0.1)	0	1 (0.1)

DR_ILO2011-D, digital radiograph interpreted using digitized ILO reference standards (2011-D), DR_RES, digital radiograph interpreted using the “research” digitized standards; FSR, film-screen radiograph interpreted using 2000 ILO film-based reference standard; ILO, International Labour Organization.

One hundred seventy-two images were classified by seven B readers, for a total of 1204 possible classifications for each method. Small opacity profusion categories are ILO designations

Figure 3.

Intrareader small opacity profusion by International Labour Organization (ILO) small opacity profusion classification (left column), standard classification method agreement (center), and primary shape and size classification by image type (right column) of radiographs from 172 miners. K, weighted κ values and 95% confidence intervals. Primary shape and size classifications are standard ILO designations: p, q, and r indicate small rounded opacities; s, t, and u indicate small irregularly shaped opacities.

When DR_RES and DR_ILO2011-D readings were compared, the overall weighted κ value was 0.58 (95% CI, 0.54–0.62) (Table 3), indicating moderate to good agreement between the two digitized standards. Agreement between FSR and either DR_ILO2011-D or DR_RES was identical, with overall weighted κ value of 0.58 (Table 3). When compared by modality and standard set, correlation of readings measured by Spearman’s correlation coefficient (r_s) indicated statistically significant strong correlation between all three groups of results: DR_RES versus DR_ILO2011-D r_s = 0.70 (P < .0001); FSR versus DR_ILO2011-D r_s = 0.68 (P < .0001); FSR versus DR_RES r_s = 0.66 (P < .0001). When comparing DR_ILO2011-D readings to FSR and DR_RES readings, small opacities tended to be classified at a slightly higher profusion score when readers used DR_ILO2011-D, indicated by the rejection of Bowker’s test of symmetry null hypothesis (DR_RES vs. DR_ILO2011-D P < .0001 and FSR vs. DR_ILO2011-D P = .01).

TABLE 3.

Intrareader Weighted Kappa Values and 95% Confidence Intervals (CIs) Comparing ILO Small Opacity Profusion

	Reader 1	Reader 2	Reader 3	Reader 4	Reader 5	Reader 6	Reader 7	Overall
FSR-DRRES*	0.48 (0.37–0.59)	0.39 (0.29–0.49)	0.72 (0.56–0.87)	0.67 (0.59–0.74)	0.43 (0.31–0.54)	0.56 (0.45–0.67)	0.72 (0.58–0.86)	0.58 (0.54–0.62)
FSR-DRILO2011-D	0.38 (0.29–0.47)	0.40 (0.30–0.51)	0.58 (0.41–0.75)	0.57 (0.47–0.67)	0.56 (0.46–0.65)	0.57 (0.48–0.67)	0.70 (0.60–0.81)	0.58 (0.54–0.61)
DRRES-DRILO2011-D	0.29 (0.20–0.39)	0.46 (0.37–0.55)	0.54 (0.35–0.73)	0.64 (0.54–0.75)	0.62 (0.49–0.74)	0.66 (0.58–0.75)	0.62 (0.49–0.76)	0.58 (0.54–0.62)

DR_ILO2011-D, digital radiograph interpreted using digitized ILO reference standards (2011-D), DR_RES, digital radiograph interpreted using the “research” digitized standards; FSR, film-screen radiograph interpreted using 2000 ILO film-based reference standard; ILO, International Labour Organization.

Date represented as κ Value (95% CI).

^*Data reported in Laney et al. 2011 (6).

Small Opacity Shape and Size

Table 4 summarizes the readers’ designations of size and shape of the primary small pneumoconiotic opacities. The overall proportions of digital images reported as showing rounded and irregular opacities did not differ by standard sets, DR_RES or DR_ILO2011-D (P = .50). However, readers classifying digital images using DR_ILO2011-D found a marginally higher proportion of irregular opacities compared to the readings of FSR (PR, 1.10; 95% CI, 1.00–1.18, P = .05; Fig 3, right column). Also, the smallest size categories of opacities (p and s type; width <1.5 mm) were reported more frequently when using DR_ILO2011-D compared to FSR (DR_ILO2011-D vs. FSR: PR, 1.47; 95% CI, 1.28–1.68, P < .0001; Fig 3, right column). When comparing the digital readings, a borderline significant difference in the proportion of p and s opacities was noted between the two digitized standard sets (DR_ILO2011-D vs. DR_RES: PR, 1.13; 95% CI, 1.00–1.28, P = .04).

TABLE 4.

Small Opacity Shape and Size by Image Modality and Standard Set

Shape and Size	FSR [n (%)]	DRRES [n (%)]	DRILO2011-D [n (%)]
Rounded	328 (74.4)	245 (66.4)	338 (68.6)
p	108 (24.5)	107 (29.0)	187 (37.9)
q	160 (36.3)	102 (27.6)	121 (24.5)
r	60 (13.6)	36 (9.8)	30 (6.1)
Irregular	113 (25.6)	124 (33.6)	155 (31.4)
s	72 (16.3)	88 (23.9)	108 (21.9)
t	36 (8.2)	34 (9.2)	44 (8.9)
u	5 (1.1)	2 (0.5)	3 (0.6)

DR_ILO2011-D, digital radiograph interpreted using digitized ILO reference standards (2011-D), DR_RES, digital radiograph interpreted using the “research” digitized standards; FSR, film-screen radiograph interpreted using 2000 ILO film-based reference standard; ILO, International Labour Organization.

One hundred seventy-two images were classified by seven B readers, for a total of 3612 possible classifications for each method. Primary shape and size designations are standard ILO designations. Readers did not record shape and size for images with 0/0 profusion.

When DR_RES and DR_ILO2011-D primary size and shape readings were compared, the overall κ value was 0.57 (95% CI, 0.50–0.63) (Table 5), indicating moderate to good agreement between the two digitized standards. Agreement between FSR and DR_ILO2011-D was moderate, with an overall κ value of 0.47 (95% CI, 0.41–0.54) (Table 5).

TABLE 5.

Simple κ Values and 95% Confidence Intervals (CIs) Comparing Primary Shape and Size Designation by Image Modality and Standard Set

	κ Value	95% CI
FSR-DRRES*	0.51	(0.44–0.58)
FSR-DRILO2011-D	0.47	(0.41–0.54)
DRRES-DRILO2011-D	0.57	(0.50–0.63)

DR_ILO2011-D, digital radiograph interpreted using digitized ILO reference standards (2011-D), DR_RES, digital radiograph interpreted using the “research” digitized standards; FSR, film-screen radiograph interpreted using 2000 ILO film-based reference standard; ILO, International Labour Organization.

^*Data reported in Laney et al. 2011 (6).

Within-reader Variability

Intrareader variability of small opacity profusion from this study is presented using κ values, in Table 3 and Figure 2, along with the values previously reported by Laney et al. (6). Intrareader variability did not differ by image modality and standard set and appeared to be relatively similar to previously reported values. The κ values ranged from 0.38 to 0.70 for FSR versus DR_ILO2011-D and from 0.29 to 0.66 for DR_RES versus DR_ILO2011-D (Table 3). Only three κ values had 95% CIs that did not intersect the mean value for the comparison type: FSR versus DR_ILO2011-D reader 1 and reader 2, DR_RES versus DR_ILO2011-D reader 1.

Figure 2.

Intrareader variability of small opacity profusion from this study and from the values previously reported by Laney et al. (FSR-DR_RES) (6) is presented using weighted κ values for classification method comparisons. Background shading represents κ value interpretation scale, darker shading signifies decreases in agreement: κ = 0.01–0.20 slight agreement, 0.21–0.40 fair agreement, 0.41–0.60 moderate agreement, and 0.61–0.80 substantial agreement (7). Not shown: κ < 0 less than chance agreement and κ = 0.81–0.99 almost perfect agreement. DR_RES, digital radiograph interpreted using the “research” digitized standards; DR_ILO2011-D, digital radiograph interpreted using digitized ILO reference standards (2011-D); FSR, film-screen radiograph interpreted using 2000 ILO film-based reference standard.

DISCUSSION

In November 2011, the ILO issued a new edition of the International Classification of Radiographs of Pneumoconioses including a set of digitized standard images (2011-D), which, for the first time, sanctioned the application of the ILO pneumoconiosis classification to digitally acquired chest radiographs. We compared the results of classifications of coal miner digital chest radiographs, interpreted using these new ILO 2011-D standard images, to a “gold standard” traditional classification of the miner’s film-screen radiograph taken on the same day. To reduce reader effects, all seven of the B Readers performed all classifications. The results showed a high degree of concurrence in the recognition of small pneumoconiotic opacities between digitally acquired radiographs interpreted using the current ILO Standard Digital Images (2011-D) and traditional film-screen radiographs interpreted using the ILO film-based standards (Spearman’s correlation coefficient r_s = 0.68). Intrareader variability for the seven readers, measured by the κ statistic, was relatively similar to previously reported values and did not appear to be affected by the use of the new 2011-D standard set. These findings help to validate the use of the newly released ILO digitized standard images (2) for classification of small pneumoconiotic opacities.

Both similarities and differences between FSR and DR imaging modalities reported in earlier comparisons, which used the DR_RES standards, were largely observed in the present study using theDR_ILO2011-D referent standards. As mentioned previously, we observed no major difference between FSR and DR_ILO2011-D. The present findings appear to confirm previous reports that small pneumoconiotic opacities are more often classified as irregularly shaped on digital radiographs compared to FSR. The available research data suggest that the visualization of small linear shadows may be enhanced through the use of DR, although, even for digital images, reader agreement on the shape of small opacities remains low. Clinical experience does seem to confirm a modestly enhanced visualization of linear/irregular small opacities as reported here and elsewhere. Adding credence to this view is our observation that the smallest opacity types (s and p) were recorded significantly more frequently using DR, compared to FSR. However, these are subjective interpretations of the study findings and we are unable to make definitive inferences based on our results regarding whether DR provides enhanced visualization of pneumoconiotic shadows.

We did observe some minor differences between classifications of coal miner digital radiographs using the DR_ILO2011-D referent standards compared to readings using the DR_RES set of standards developed for previous research studies. In general, when the results of the two digital classifications differed, the readings using the DR_ILO2011-D more closely reflected the FSR result than the readings using DR_RES (Tables 2 and 3 and Fig 1).

This study has a number of advantages. It was based on repeat interpretations of a large number of radiographs by seven experienced B Readers who were blinded to their previous readings and the readings of others. The methods in this article are well established and reported in previous peer-reviewed publications. A number of limitations also exist for this study. Figure 3 demonstrates that the seven B Readers have many different patterns of reading. Most of the radiograph interpretations in this study indicated that no disease was visibly present, (Fig 1) which may affect calculations of agreement both within and between readers. Pneumoconiosis classification using digital chest radiographs is a new procedure, and the readers who performed the earlier research readings may have become progressively more familiar with it. Given the time difference between the two groups of digital readings used in this study (at least 2.5 years), some readers may have evolved or modified their natural reading patterns during that time. It is possible that the differences observed between the two digital readings (DR_ILO2011-D and DR_RES) may in part be related to alterations in reading behavior over time.

In conclusion, our results help to validate the classification of digital chest radiographs of pneumoconiosis using the new ILO digitized standard images. The body of scientific research conducted to date supports the use of digital radiography in the recognition of small interstitial lung opacities. The results of the present study demonstrate that, when chest radiographs are obtained and displayed using appropriate equipment, methods, and conditions, the results of classification of digital chest radiographs performed using 2011-D standards should be quite comparable to results based on the previous digitized research standards (DR_RES) and also to the film-based ILO 2000 classification. The current challenge for the ILO is to complete development and testing of a set of reference standards for the classification of digital chest images that are digitally acquired.