Table A3.

Formulae used to calculate the G and Φ generalizability coefficients in Table 3, derived from references [20,21]. Abbreviations: A, aid or calibration level; J, temporomandibular joint; P, patient; R, reader; S, specialty (binary, radiologist or non-radiologist clinician); colon (:), nested relation, (e.g., J:P means joint is nested within the patient variable); cross (x), crossed relation (e.g., RxA means each Reader provides data at all levels of the aid variable).

Coefficient Meaning	Formula
J:P × R × A Design—Pooling Across Use or Disuse of Aid, Separately for Radiologist and Non-Radiologist Readers
Between readers of the same calibration (R random, P and A fixed)	$G\left(R\right)=\frac{J:P+AJ:P}{J:P+AJ:P+R+RJ:P+ARJ:P}$
Same reader with vs. without aid (A random, P and R fixed)	$G\left(A\right)=\frac{J:P+RJ:P}{J:P+RJ:P+A+AJ:P+ARJ:P}$
Between readers with different calibration (A and R random, P fixed)	$G\left(R,A\right)=\frac{J:P}{J:P+A+R+AR+AJ:P+RJ:P+ARJ:P}$
Between readers with different calibration, averaging 2 readings per reader (A and R random, P fixed, A/2)	$G\left(R,A\right)=\frac{J:P}{J:P+\frac{A}{2}+R+\frac{AR}{2}+\frac{AJ:P}{2}+RJ:P+\frac{ARJ:P}{2}}$
Between readers with different calibration, averaging 2 readers at a time (A and R random, P fixed, R/2)	$G\left(R,A\right)=\frac{J:P}{J:P+A+\frac{R}{2}+\frac{AR}{2}+AJ:P+\frac{RJ:P}{2}+\frac{ARJ:P}{2}}$
Between readers of the same calibration, averaging two readers at a time (R random, P and A fixed, R/2)	$G\left(R\right)=\frac{J:P+AJ:P}{J:P+AJ:P+\frac{R}{2}+\frac{RJ:P}{2}+\frac{ARJ:P}{2}}$
Right-left joint agreement when scored by different readers with different calibration (J, A and R random)	$\Phi \left(J,R,A\right)=\frac{P}{P+J:P+R+A+PA+PR+AR+PAR+AJ:P+RJ:P+ARJ:P}$
Right-left joint agreement when scored by the same reader with the same calibration (J random, A and R fixed)	$G\left(J\right)=\frac{P+PA+PR+PAR}{P+PA+PR+PAR+J:P+AJ:P+RJ:P+ARJ:P}$
J:P × R:S Design—Pooling Across Radiologist and Non-Radiologist Readers, Separately for Each Aid
Between radiologists and non-radiologists (R and S random, P fixed)	$G\left(R,S\right)=\frac{J:P}{J:P+R:S+S+J:PS+JR:PS}$
Between readers of the same specialty (R random, S and P fixed)	$G\left(R\right)=\frac{J:P+J:PS}{J:P+J:PS+R:S+JR:PS}$
Right-left joint agreement when scored by radiologist and non-radiologist (J, S and R random)	$\Phi \left(J,R,S\right)=\frac{P}{P+J:P+R:S+S+PS+PR:S+J:PS+JR:PS}$
Right-left joint agreement when scored by the same reader (J random, R and S fixed)	$G\left(J\right)=\frac{P+PS+PR:S}{P+PS+PR:S+J:P+J:PS+JR:PS}$