Reliability and validity of expert assessment based on airborne and urinary measures of nickel and chromium exposure in the electroplating industry

Abstract

The reliability and validity of six experts’ exposure ratings were evaluated for 64 nickel-exposed and 72 chromium-exposed workers from six Shanghai electroplating plants based on airborne and urinary nickel and chromium measurements. Three industrial hygienists and three occupational physicians independently ranked the exposure intensity of each metal on an ordinal scale (1–4) for each worker's job in two rounds: the first round was based on responses to an occupational history questionnaire and the second round also included responses to an electroplating industry-specific questionnaire. Spearman correlation (r_s) was used to compare each rating's validity to its corresponding subject-specific arithmetic mean of four airborne or four urinary measurements. Reliability was moderately-high (weighted kappa range=0.60–0.64). Validity was poor to moderate (r_s= -0.37–0.46) for both airborne and urinary concentrations of both metals. For airborne nickel concentrations, validity differed by plant. For dichotomized metrics, sensitivity and specificity were higher based on urinary measurements (47–78%) than airborne measurements (16–50%). Few patterns were observed by metal, assessment round, or expert type. These results suggest that, for electroplating exposures, experts can achieve moderately-high agreement and (reasonably) distinguish between low and high exposures when reviewing responses to in-depth questionnaires used in population-based case-control studies.

Introduction

Exposure assessment in population-based studies often requires experts to review study subjects’ responses to questionnaires designed to collect occupational information to provide exposure estimates for use in epidemiologic analyses ^1-2. The reliability and validity of these expert ratings are important to characterize because exposure misclassification can mask exposure-disease associations. Several studies have evaluated factors that can affect the reliability of experts’ ratings within population-based studies ^3-14. Of these, only five studies evaluated the experts’ validity^{4, 8-9, 11, 14}, Validity studies for industry-based studies are somewhat more plentiful¹⁰; for example, one of the earliest evaluated semi-quantitative estimates of methylene chloride and styrene for jobs in a small polyester factory¹⁵. A previous review of these studies found that the experts’ validity, based on kappas or intraclass correlation coefficients, varied widely from poor to excellent, with a median of ~0.6¹⁰. Most validity studies have compared the ratings to airborne measurements despite the fact that multiple routes of exposure are relevant for many agents. To date, we identified only three studies that have evaluated experts’ ratings compared to urinary measurements ^{11, 16-17}.

No study has evaluated the reliability and validity of experts’ ratings in the electroplating industry. Previous studies of electroplating workers have often reported poor/no correlations between air and urine metal concentrations ^18-20, although some studies have observed moderate to high correlations (range: r = 0.48–0.96, median: r = 0.68) ^21-24. The relationship between air and urinary measurements likely varies due to the extent of dermal exposure, the use of personal protective equipment, and personal behaviors such as smoking that may transfer the contaminants from hand to mouth, as well as the time of day the urinary measurements are collected in relation to the air measurements ^{21-22, 25-26}. Subject-specific variations in uptake and metabolic/excretion rates could also reduce the correlation with the post work shift urinary concentrations. Therefore, our primary objective was to characterize the reliability and validity of experts’ ratings of nickel and chromium exposures within the context of the a case-control study design for workers’ current jobs within an electroplating setting in relation to both airborne and urinary measurements of exposure. Our secondary objective was to evaluate the experts’ ratings in relation to the availability of participants’ responses to two types of questionnaires typically used in population-based studies (an occupational history questionnaire (OH) and an electroplating industry-specific questionnaire (EIQ)) and by type of expert (industrial hygienists vs. occupational physicians).

Materials and methods

Study subjects and self-reported occupational information

We recruited 64 nickel-exposed workers and 72 chromium-exposed workers from six electroplating plants (nickel-exposed workers from plants 1–3; chromium-exposed workers from plants 4–6) in Shanghai, China. Subjects were selected based on their willingness to participate from those who had held their current job for at least six months and were expected to remain in their current job for at least six more months. Because this study was meant to mimic the type of information available to experts within a case-control study, each study subject completed an OH only for his/her current job. The OH included open-ended questions on job title, employment dates, products made or services provided by employer, primary work tasks and activities, tools and equipment used, and chemicals and materials used. Each subject also completed an EIQ for the same job, which asked more detailed questions (predominantly with categorical responses) about specific tasks, time spent in work locations within the plants, proximity to the source of metal aerosols, use of personal protective equipment, presence of ventilation systems and the subjects’ impression of operating efficiencies, and contact with the liquids from the plating tank. The OH and EIQ can be obtained from the corresponding author. Participation was voluntary and undertaken according to protocols approved by the Institutional Review Board of National Cancer Institute and the Shanghai Centers for Disease Control (hereafter, Shanghai CDC).

Air and urinary measurements

Personal airborne and urinary nickel and chromium measurements were collected on four occasions per subject over two seasons (summer and winter), from June 2002 through August 2003. The airborne samples collected total particulates in the workers’ breathing zones on mixed cellulose ester filters (pore size = 0.8 μm) with a 37-mm cassette using a portable sampling pump and analyzed according to National Institute for Occupational Safety and Health (NIOSH) Method 7300 ²⁷. Each worker provided one spot (~50 ml) urine sample at the end of the work shift on each measurement day, using NIOSH Method 8310 ²⁸. These methods measure all insoluble and soluble forms, and all valences, of aerosols of the respective metals.

Expert exposure ratings

Three industrial hygienists and three occupational physicians affiliated with the Shanghai CDC or the US National Cancer Institute estimated subject-specific average (arithmetic mean) exposure intensity over a work shift to nickel or to chromium (depending on the plant in which the subject worked) for each subject's current job in two separate assessment rounds. The experts were asked to estimate the arithmetic mean because it is generally considered the most relevant metric for chronic health effects²⁹. In the first assessment round, the experts provided ratings based solely on the OH responses (OH round). In the second assessment round, the experts provided ratings based on both the OH and EIQ responses (OH/EIQ round). In each round, each expert independently assigned an intensity rating for nickel or chromium using an ordinal scale of 1 (very low) to 4 (high) without access to the exposure measurements. The categories were not anchored to specific exposure levels. The experts had no prior measurements available for these worksites with which to anchor their assignments, however, the experts were advised to consider the categories as approximately <10%, 10–50%, 51–100%, and <100% of the occupational exposure limit (OEL) in place at the time of the evaluation.

Statistical analyses

To evaluate whether the experts were able to accurately estimate the arithmetic mean exposure of each study subject, all comparisons were made to the arithmetic mean of the subject-specific inhalation and urinary concentrations. The subject-specific inhalation concentration was calculated as the arithmetic mean of the four airborne measurements. Similarly, the subject-specific urinary concentration was calculated as the arithmetic mean of the four urinary measurements. All analyses were conducted using Stata 11.1 (StataCorp, College Station, TX, USA).

Descriptive analyses

Descriptive statistics of the subject-specific inhalation and urinary concentrations were calculated, overall and by plant. The Spearman correlation (r_s) between the airborne and urinary subject-specific means was also calculated for each metal, overall and by plant. Intraclass correlation coefficients (ICCs) that indicate the contrast in exposures between subjects (between-subject variance/(between-subject variance+ within-subject variance) were calculated from variance components obtained from random-effects models with subject ID included as the random effect.

Inter-expert reliability

Agreement between each pair of the six experts (15 pairs) was calculated using two metrics: proportion of agreement and weighted kappa (κ_w). We reported the mean and range of each kappa metric observed across the 15 pairs. To interpret the kappa values, we arbitrarily categorized kappa values <0.2 as poor, 0.2–0.4 as fair, >0.4–0.6 as moderate, >0.6–0.8 as moderately-high, and >0.8 as high based on categories originally proposed by Landis and Koch (1977)³⁰.

Validity of expert ratings

We evaluated the validity of expert ratings (ordinal scale, 1–4) compared to the subject-specific arithmetic means of the airborne and urinary concentrations (used as an approximate the ‘gold standard’ of each subject's average exposure) using the Spearman correlation measure (r_s), which is a non-parametric comparison that does not assume a linear relationship between the expert ratings and the measured subject-specific arithmetic means. For each metal and sample media, we report the mean and range of the correlations observed across the six experts. We also calculated the Spearmen correlation coefficient between the subject-specific means for each metal and sample media and the arithmetic mean of the six experts’ ratings (‘group rating’; continuous scale, range 1–4), the mean of the three industrial hygienists’ ratings, and the mean of the three occupational physicians’ ratings. 95% confidence intervals (CI) were calculated based on Fisher's transformation. To interpret the correlations, we used the cut points described above.

Sensitivity and specificity of a two-category scale

The sensitivity and specificity of the group rating compared to the subject-specific mean concentrations of airborne and urinary nickel and chromium exposure were calculated based on a two-category scale. For this calculation, group ratings ≤2.5 (the mid-point of the 4-category scale) and subject-specific means ≤ median were categorized as ‘low exposed’ and group ratings >2.5 and subject-specific means > median were categorized as ‘high exposed’.

The reliability and validity analyses described above were stratified by exposure agent, assessment round, and type of expert. Some analyses, identified in the results, were also stratified by plant. Measures of central tendency (mean, range) are reported because we focused on the performance of a ‘typical’ expert rather than a specific expert.

Results

Measurement data

For nickel, the overall arithmetic mean (AM) of the subject-specific means was 7.4 μg m^-3 for air and 30.1 μg g^-1 creatinine for urine (Table 1). Plant 1 had the highest airborne nickel AM; in contrast, plant 3 had the highest urinary nickel AM. For chromium, the overall AM of the subject-specific means was 3.0 μg m^-3 for air and 76.3 μg g^-1 creatinine for urine. Plant 5 had the highest airborne and urinary chromium AMs. Poor correlations were observed between the air and urine concentrations overall for nickel (r_s = -0.28) and for chromium (r_s = 0.09) and by plant for both metals (r_s range = -0.39–0.30).

Table 1.

Descriptive statistics of the subject-specific arithmetic means of airborne and urinary nickel and chromium exposure concentrations and the correlations between the airborne and urinary concentrations, overall and by plant.

Agent	Plant	N subjects	Air (μg m−3)a					Urine (μg g−1 creatinine)a					Spearman correlation
			AM	SD	GM	GSD	Range	AM	SD	GM	GSD	Range
Nickel	1	30	11.7	4.4	10.9	1.43	5.5–22.7	22.9	7.3	21.8	1.38	13.4–38.2	0.11
	2	23	3.5	6.7	1.9	2.37	0.76–32.0	35.4	28.5	29.6	1.78	7.6–155	0.06
	3	11	3.8	1.7	3.4	1.61	1.2–6.9	38.7	13.1	36.9	1.37	24.3–68.0	0.30
	Overall	64	7.4	6.4	4.8	2.72	0.76–32.0	30.1	19.5	26.6	1.60	7.6–155	−0.28
Chromium	4	36	2.0	0.5	1.9	1.26	1.3–3.1	65.0	30.4	59.7	1.49	28.3–175	0.22
	5	24	4.8	5.9	3.4	2.04	1.7–26.2	107	78.4	85.5	1.93	33.2–344	−0.05
	6	12	2.3	1.1	2.1	1.56	1.2–5.0	49.6	13.1	48.1	1.31	33.7–72.4	−0.39
	Overall	72	3.0	3.6	2.4	1.71	1.2–26.2	76.3	54.5	64.9	1.69	28.3–344	0.09

AM: arithmetic mean; GM: geometric mean; GSD: geometric standard deviation; N: number of subjects; SD: standard deviation.

^aDescriptive statistics reported here are based on the subject-specific means, calculated as the arithmetic means of the 4 air and 4 urinary measurements collected on each subject.

Between- and within-subject variance components are reported in Table 2, overall and by plant. For airborne concentrations, ICCs ranged from <0.001 to 0.57, indicating poor to moderate contrast in airborne concentrations among subjects overall and within individual plants. For urinary concentrations, ICCs ranged from <0.001 to 0.26, indicating nearly all variability in urinary concentrations was within-subject variability.

Table 2.

Within- and between-subject variance components for airborne and urinary concentrations of nickel and chromium, overall and by plant.

Agent	Plant	Air			Urine
		Within-subject variance	Between-subject variance	ICC	Within-subject variance	Between-subject variance	ICC
Nickel	1	0.775	0.273	0.260	0.922	<0.001	<0.001
	2	0.638	0.704	0.525	1.07	<0.001	<0.001
	3	0.715	0.233	0.246	0.837	<0.001	<0.001
	Overall	0.718	0.940	0.566	0.984	<0.001	<0.001
Chromium	4	0.544	<0.001	<0.001	0.700	<0.001	<0.001
	5	0.642	0.538	0.456	0.926	<0.001	<0.001
	6	0.445	0.349	0.440	0.457	0.163	0.263
	Overall	0.566	0.400	0.414	0.763	<0.001	<0.001

ICC, intraclass correlation coefficient (between-subject variance/(between-subject variance+within-subject variance)

Expert reliability

The overall means of the expert ratings were similar for both metals and for both assessment rounds (Table 3). Based on weighted kappa, moderately-high agreement among experts was observed for both metals in the OH (κ_w: nickel = 0.60, chromium = 0.64) and OH/EIQ (κ_w: nickel = 0.60, chromium = 0.61) rounds. Agreement was only fair to moderate for both metals and assessment rounds when evaluated based on the proportion of agreement (means 0.47−0.57). For all measures, the agreement was somewhat higher for chromium than for nickel. No differences were observed by assessment round for either metal. The industrial hygienists had, on average, somewhat higher agreement amongst themselves than that observed amongst the occupational physicians for nickel but not for chromium (Supplementary Table S1 and S2).

Table 3.

Measures of agreement by metal and assessment round

Agent/round	Mean of all Ratingsa (SD)	Agreement between any two expertsb
		Proportion of agreement (%)		Weighted kappa (κw)
		Mean	Range	Mean	Range
Nickel
OH	2.7 (1.0)	47.6	28.1–68.8	0.60	0.08–0.82
OH/EIQ	2.7 (0.9)	47.4	25.0–67.2	0.60	0.13–0.85
Chromium
OH	2.6 (0.9)	55.6	34.7–91.7	0.64	0.48–0.93
OH/EIQ	2.6 (0.9)	56.7	37.5–79.2	0.61	0.39–0.97

OH, occupational history round; OH/EIQ, occupational history and electroplating industry-specific questionnaire round; SD, standard deviation.

^aArithmetic mean of 6 experts × 64 ratings for nickel and 6 experts × 72 ratings for chromium.

^bArithmetic mean and range of the given statistic from all 2-expert combinations (n=15) of the 6 experts.

Validity of expert ratings

For nickel, the mean Spearman correlation between each expert's rating in the OH round and the subject-specific AM was -0.30 (r_s range: -0.48–0.04) for air and 0.38 (r_s range: 0.27–0.47) for urine. For chromium, the mean Spearman correlation between each expert's rating in the OH round and the subject-specific AM was -0.02 (r_s range: -0.11–0.06) for air and 0.04 (r_s range: - 0.06–0.15) for urine. Similarly poor correlations in the EIQ round were observed (not shown). The Spearman correlations between the subject-specific means and the group ratings are shown in Table 4 for the OH round. Overall, poor to moderate correlations (r_s = -0.37–0.46) were observed and did not vary by assessment round or type of expert. However, plant-specific differences were observed. For airborne nickel, we observed a good correlation for plant 1 (r_s = 0.70), but poor correlations for plant 2 (r_s = 0.15) and plant 3 (r_s = 0.19). For urinary nickel, poor correlations were observed in all three plants. For chromium, poor correlations (-0.67– -0.04) were observed for both airborne and urinary concentrations in all three plants. For nickel, the industrial hygienists’ ratings had higher validity than the occupational physicians’ ratings in all plants based on the airborne measurements and in plants 2 and 3 based on urinary concentrations. For chromium, ratings from both types of experts had similarly low validity for both airborne and urinary concentrations.

Table 4.

For the OH round, Spearman correlation coefficients between the group rating (arithmetic mean of the six experts’ ratings) and the subject-specific arithmetic mean of airborne and urinary metal exposures, overall and by plant and type of expert

Measurement	Type of Expert	Nickel Spearman correlation (95% CI)				Chromium Spearman correlation (95% CI)
		Overall	Plant 1	Plant 2	Plant 3	Overall	Plant 4	Plant 5	Plant 6
Air	Groupa	−0.37 (−0.56 – −0.14)	0.70 (0.55 – 0.81)	0.15 (−0.10 – 0.38)	0.19 (−0.06 – 0.42)	−0.03 (−0.26 – 0.20)	−0.41 (−0.59 – −0.20)	−0.29 (−0.49 – −0.06)	−0.04 (−0.27 – 0.19)
	Industrial hygienists	−0.38 (−0.57 – −0.15)	0.75 (0.62 – 0.84)	0.38 (0.15 – 0.57)	0.55 (0.35 – 0.70)	−0.01 (−0.24 – 0.22)	−0.35 (−0.54 – −0.13)	−0.29 (−0.49 – −0.06)	−0.05 (−0.28 – 0.18)
	Occupational physicians	−0.30 (−0.51 – −0.06)	0.62 (0.44 – 0.75)	−0.11 (−0.35 – 0.14)	0.19 (−0.06 – 0.42)	0.05 (−0.18 – 0.28)	−0.38 (−0.56 – −0.16)	−0.27 (−0.47 – −0.04)	0.01 (−0.22 – 0.24)
Urine	Groupa	0.46 (0.24 – 0.63)	0.24 (−0.01 – 0.46)	0.04 (−0.21 – 0.28)	0.23 (−0.02 – 0.45)	0.03 (−0.20 – 0.26)	−0.14 (−0.36 – 0.10)	−0.06 (−0.29 – 0.17)	−0.67 (−0.78 – −0.52)
	Industrial hygienists	0.46 (0.24 – 0.63)	0.15 (−0.10 − 0.38)	0.07 (−0.18 – 0.31)	0.57 (0.38 – 0.72)	0.06 (−0.17 – 0.29)	−0.10 (−0.32 – 0.14)	−0.03 (−0.26 – 0.20)	−0.46 (−0.66 – 0.26)
	Occupational physicians	0.43 (0.21 – 0.61)	0.34 (0.10 – 0.54)	0.03 (−0.22 – 0.27)	0.16 (−0.09 – 0.39)	−0.002 (−0.23 – 0.23)	−0.11 (−0.33 – 0.13)	−0.09 (−0.32 – 0.15)	−0.77 (−0.85 – 0.66)

CI, confidence intervals based on Fisher's transformation; OH, occupational history.

^aIndustrial hygienists + Occupational physicians.

We explored the relationship between the group ratings and the subject-specific means further visually. Figure 1 shows the scatter plot between the group rating and the subject-specific means of airborne nickel exposure in the OH and OH/EIQ rounds for plants 1, 2, and 3 (Figures 1a–c) and each plant's corresponding distribution of the airborne nickel subject-specific AMs (Figures 1d–f). We found that the distribution of the subject-specific AMs for airborne nickel exposure was much wider and more evenly spread for plant 1 (Figure 1d), where we also observed moderately-high validity, than for plants 2 and 3 (Figures 1e and 1f), where we observed poor validity. For urinary nickel and for airborne and urinary chromium, we observed narrow distributions for the subject-specific AMs in each plant (shown in supplemental Figures S1, S2, and S3) that were similar to the distributions shown for plants 2 and 3 for airborne nickel (Figures 1e and 1f).

Figure 1.

Scatter plot, best-fit linear line, and Spearman correlation statistic between the average rating and the subject-specific arithmetic mean (AM) of airborne nickel exposure in the OH and OH/EIQ rounds for Plants 1, 2, and 3 (a–c) and each plant's corresponding distribution of the airborne nickel subject-specific AMs (d–f). OH = assessment based on the occupational history questionnaire only; OH/EIQ = assessment based on both the occupational history questionnaire and electroplating industrial questionnaire.

For nickel, the two-category scale derived for the group rating had higher sensitivity and specificity when compared to the urinary measurements (sensitivity = 75% in OH round, 78% in OH/EIQ round; specificity = 66% in both rounds) than when compared to the airborne measurements (sensitivity = 25% in OH round, 28% in OH/EIQ round; specificity = 16% in both rounds). For chromium, the two-category scale also had slightly higher sensitivity and specificity with the urinary measurements (sensitivity = 50% in OH round, 47% in OH/EIQ round; specificity = 58% in both rounds) than with the airborne measurements (sensitivity = 47% in OH round, 39% in OH/EIQ round; specificity = 50% in both rounds). No consistent pattern was observed across assessment rounds for these metrics. The two metals showed somewhat opposite patterns, with 9–12% higher sensitivity than specificity for nickel and 8–11% lower sensitivity than specificity for chromium.

Discussion

In this study, experts’ ratings for nickel- and chromium-exposed workers based on OH and OH/EIQ responses were evaluated against repeated airborne and urinary measurements. The experts’ moderately-high agreement amongst themselves (κ_w = 0.60–0.64) was comparable to the median agreement reported in previous studies ^{9-10, 12-14, 31-32}. Despite their moderately-high reliability, the experts’ ratings had only poor to moderate validity overall in relation to both airborne and urinary measurements and was lower than most previous validity studies ^{9, 11, 17, 33}. For example, Hertzman et al. ¹⁷ found much higher correlations (> 0.65) between experienced workers’ estimates of exposure and urinary chlorophenate levels in a large cohort study of the lumber industry. However, Tielemans et al. ¹¹ observed similarly poor to fair agreement (kappa < 0.4) between estimates based on expert review of job-specific questionnaires (which has similarities to the EIQ used here) and urinary measurements (i.e., methylhippuric, hippuric and chromium) in a population-based study. The validity observed here was also much lower than previously reported for two other Shanghai industries (textile industry, r_s = 0.30–0.65; foundry industry, r_s = 0.65–0.85) evaluated using nearly the same study design but based solely on airborne measurements ⁹. The group rating provided minimal improvement to the experts’ validity in this study, unlike previous studies that observed improvements in the validity when the estimates of multiple raters were averaged ^{9, 12, 34-35}. Moderate to moderately-high validity, however, was observed when the experts’ four category rating scale was dichotomized.

The poor overall validity observed here was likely, at least in part, a function of the limited exposure distribution between subjects within the same plant. Our plant-stratified analyses found much higher validity for Plant 1, which was the only plant where the subject-specific means were relatively evenly distributed across a relative wide range of exposure levels. All other plants had subject-specific means that were skewed and/or were narrowly clustered in the low exposure range (See Figure 1). Thus, the experts assessed workers’ exposures on a four-category scale when, for some plants in this study, there was little contrast (ICCs ranging from 0 to 0.5)among the exposures of most of the assessed workers. This finding is consistent with our previous study that observed poorer validity in the textile industry (with a highly skewed exposure distribution that clustered at low concentrations) than in the foundry industry (with more evenly distributed exposure concentrations) ⁹. This finding may also point to the difficulty of asking experts to provide exposure ratings in the absence of any exposure measurements on which to anchor their estimates, which has previously been shown to improve experts’ validity^{8, 10, 15}, The availability of at least some measurements for these workplaces in advance of the rater evaluations may have revealed the lack of exposure contrast. Although evaluating whether the experts’ ratings were better associated with air or urine concentrations was our primary objectives, our findings are inconclusive because of the overall poor validity observed here. The experts’ ratings had better sensitivity and specificity for the urinary concentrations than the airborne concentrations, which provides some, but not conclusive, evidence that the experts may have considered all routes of exposure. This result was somewhat surprising because dermal exposure is more difficult to assess, has not been as well studied as airborne exposure, and little is known about contribution of various dermal exposure determinants ³⁶. These analyses also demonstrated that assessing exposure using a two-category scale (low and high) is easier than using a four-category scale when between-worker difference in exposure is small. However, the sensitivities (nickel: 75– 78%; chromium: 47–50%) and specificities (nickel: 66%; chromium: 58%) based on the urinary concentrations remained only moderate to moderately-high for both metals in both assessment rounds, suggesting that the experts were only partly successful in distinguishing exposures based on a two category scale in this setting. The poor validity on both the four and two-category scales suggests that even a limited, a priori, characterization of exposure may help determine appropriate semi-quantitative categories for the exposure range and the number of exposure categories to use for expert assessment in a workplace or study before requesting experts to provide their exposure estimates. For example, syntheses of the data in the published literature ^37-39 or from inspection measurements ^40-44 and other exposure databases ⁴⁵ can provide useful information on exposure variability and help anchor the experts’ ratings to a concentration scale.

The use of the electroplating industry questionnaire (EIQ) responses had little impact on the overall measures of reliability or validity of the experts’ ratings in this study, likely (at least in part) due to the limited contrast in exposure concentrations between participants within the same plant. However, our finding is consistent with our similar evaluations in the textile and foundry industries ⁹ and with previous studies that showed that additional information did not improve raters’ reliability and validity ^{12, 34., 46}, although some improvements were observed by Tielemans et al. ¹¹ with the use of similar types of questionnaires. The advantage of the EIQ, however, is that within-job differences are systematically captured and thus more easily used in programmable decision rules ⁴⁷.

No consistent patterns were observed in the reliability and validity of the experts’ ratings by type of rater, similar to our previous findings in the textile and foundry industries ⁹. Industrial hygienists had somewhat higher reliability than the occupational physicians for both metals in most comparisons. However, both types of experts generally had similar poor-to-fair validity with airborne and urinary nickel and chromium exposures overall. Plant-specific differences were observed, with the industrial hygienists’ ratings having higher validity than the occupational physicians’ ratings in some comparisons. The similarity of the two types of raters likely relates to these occupational physicians’ regular work site visits and familiarity with exposure monitoring and may not be generalizable to occupational physicians without this expertise. For instance, several studies have reported that substantial field experience is critical to provide valid exposure estimates ^{10, 16, 34, 48}.

This study has several limitations. As in Friesen et al. ⁹, this study design straddled the population-based and industry-based study designs by using questionnaires designed for a typical case-control study but evaluating the performance in a single industry for a single year. Thus, this study cannot predict how well these experts would have performed if asked to estimate these same exposures across a wide time span or across multiple industries with a wide range of exposures. In addition, the small amount of contrast in exposures in these plants limited our ability to assess the validity of experts’ ratings in the electroplating industry. The contrast in case-control studies is likely to vary by the agent, time period and population being assessed. In addition, the airborne and urinary measurements may be ‘alloyed’ gold standards because the measurements did not account for the metal's solubility, valence, form, or particle size and four measurements may not be sufficient to fully characterize the subjects’ average exposures.

Conclusion

In this study, despite moderately-high reliability among experts, experts ratings had low validity in relation to airborne and urinary exposure measurements which was likely attributable to low contrast in exposures. As a result, our evaluations of the influence of responses to industry-specific questionnaires on the validity of the experts’ ratings in relation to airborne and urinary measurements were inconclusive. However, this study provides some insight into the challenges for designing a study to evaluate the validity of exposure ratings. Validity was generally better when there was a wider and more even distribution of exposure and poor when there was low contrast. In addition, this study showed that the experts were reasonably able to distinguish between high and low exposures, but not finer categories. Both findings indicate that prior knowledge of the variability in exposure may be necessary for setting exposure categories for experts’ ratings. In population-based studies, however, information on historical exposure and its variability is generally limited or nonexistent and thus we will likely need to rely on syntheses of publicly available data to anchor experts’ estimates, which do not necessarily reflect the working conditions of the job being assessed. Our results also point to the continued need for methods to increase the ability of experts to evaluate exposures. For instance, training experts in the interpretation of exposure distributions and determinants⁴⁹ and developing transparent, programmable decision rules to systematically capture the exposure differences that can be reviewed and refined by multiple experts⁴⁷ may improve the accuracy of expert judgment.