. 2016 Jan 4;60(4):467–478. doi: 10.1093/annhyg/mev095

Table 2.

Agreement among ordinal occupational diesel exposure estimates from two experts and from CT models for 350 jobs, by exposure metric and strata defined by CT models’ assessment confidence

Strata^a	Comparison group	Probability		Intensity		Frequency
		% Agree.	κ _w	% Agree.	κ _w	% Agree.	κ _w
A. High conf.	Expert 1 versus 2	90.2	0.86	78.0	0.84	72.4	0.84
	CT prob >0	90.5	0.56	71.6	0.72	62.2	0.71
	Expert 1 versus CT	83.7	0.82	88.6	0.85	69.9	0.84
	CT prob >0	79.7	^b	74.3	0.60	73.0	0.79
	Expert 2 versus CT	85.4	0.83	80.5	0.81	81.3	0.90
	CT prob >0	75.7	^b	85.1	0.71	54.1	0.69
B. Discordant prob.	Expert 1 versus 2	68.8	0.42	77.1	0.51	72.9	0.66
	Expert 1 versus CT	83.3	0.46	89.6	0.57	87.5	0.43
	Expert 2 versus CT	75.0	0.32	72.9	0.24	72.9	0.39
C. Low frequency conf.	Expert 1 versus 2	79.3	0.50	69.5	0.57	67.1	0.67
	Expert 1 versus CT	82.9	0.24	80.5	0.52	34.1	0.17
	Expert 2 versus CT	85.4	0.22	67.1	0.23	40.2	0.22
D. Low prob./intensity conf.	Expert 1 versus 2	68.0	0.63	64.9	0.47	56.7	0.68
	Expert 1 versus CT	61.9	0.09	67.0	0.34	62.9	0.44
	Expert 2 versus CT	63.9	0.20	64.9	0.27	51.5	0.28

κ _w, weighted kappa; Agree., agreement; CT, classification tree; conf., confidence; prob., probability rating.

^aDefinitions of strata are provided in section on Reliability Assessment.

^bRestricting the jobs to CT probability >0 turns the CT probability metric into a 3 category rather than 4 category assessment. As a result, weighted kappa’s between the CT estimate and the expert estimate cannot be calculated.