TABLE 5.

Most consistent individual representations based on group and finding level standard deviation of AUC across systems

Finding	Proportion	Document LIRE	Controlled Vocabulary	Document MIMIC	Controlled Vocabulary Filter Only	N-Grams
All Findings	-	0.010✦	0.012	0.013	0.014	0.051
Potentially Clinically Important Findings	-	0.007✦	0.035	0.024	0.031	0.076
any degeneration	0.896	0.021	0.031	0.024	0.041	0.020✦
facet degeneration	0.762	0.018✦	0.039	0.022	0.038	0.064
disc height loss	0.507	0.033	0.050	0.010✦	0.057	0.103
any stenosis*	0.480	0.023	0.032	0.013✦	0.035	0.051
disc bulge	0.435	0.019	0.022	0.009✦	0.015	0.040
foraminal stenosis*	0.400	0.016✦	0.058	0.035	0.077	0.042
central stenosis*	0.351	0.030✦	0.046	0.038	0.056	0.043
any osteophyte	0.332	0.034	0.099	0.031✦	0.094	0.108
listhesis grade 1	0.324	0.032	0.014	0.009✦	0.014	0.076
disc degeneration	0.322	0.033	0.052	0.092	0.049	0.026✦
scoliosis	0.274	0.017✦	0.051	0.025	0.043	0.068
osteophyte anterior column	0.271	0.025	0.084	0.020✦	0.083	0.088
spondylosis	0.217	0.040	0.156	0.036	0.164	0.015✦
fracture	0.212	0.019✦	0.028	0.029	0.039	0.029
disc protrusion	0.197	0.021	0.011✦	0.018	0.016	0.053
disc desiccation	0.189	0.024	0.052	0.020✦	0.051	0.090
nerve root displaced/compressed*	0.169	0.020	0.018	0.015	0.011✦	0.021
lateral recess stenosis*	0.163	0.017✦	0.038	0.019	0.034	0.063
annular fissure	0.099	0.015✦	0.064	0.057	0.065	0.063
nerve root contact	0.097	0.071	0.035	0.044	0.014✦	0.090
disc extrusion*	0.079	0.007✦	0.024	0.015	0.023	0.076
endplate edema*	0.059	0.059✦	0.164	0.059✦	0.091	0.126
hemangioma	0.049	0.073	0.048	0.007✦	0.048	0.105
disc herniation	0.038	0.046✦	0.079	0.092	0.065	0.094
spondylolysis	0.032	0.107	0.113	0.087	0.108	0.044✦
listhesis grade 2*	0.028	0.047✦	0.128	0.184	0.099	0.184

For each representation, we trained our model on reports from three systems and evaluated on the fourth, iteratively, for each finding. For each finding, we calculated the standard deviation of the AUC across the four systems. We calculated group-level consistency by averaging the AUC across all findings, and across all potentially clinically important findings for each system as a test set and then calculated the standard deviation across the systems. Table shows the most consistent representation ordered left to right based on the All Findings row (1^st row). The first column indicates the finding/group. For the findings, the second column indicates prevalence in the test set represented as a proportion. ✦ indicates the most consistent representation for that finding and group. Finally * indicates findings that were potentially clinically important. AUC = Area Under the Curve.