TABLE 4.

Validation performance based on small development datasets of n = 100 for the main simulation scenarios (reported as the average performance on a large validation dataset for 200 simulated development datasets)

	Calibration intercepts/Calibration slopes
	Per outcome category			Per outcome dichotomy		Model‐specific			Single number metrics
Model	$Y=1$	$Y=2$	$Y=3$	$Y>1$	$Y>2$	LP1	LP2		rMSPE	ORC
MLR truth scenario 1: balanced outcome, equidistant means
MLR	0.00/0.78	0.03/0.31	−0.02/0.80	0.00/0.78	−0.02/0.80	0.02/0.73	−0.03/0.76		0.104	0.727
CL‐PO	0.00/0.86	0.03/0.55	−0.02/0.86	0.00/0.86	−0.02/0.86	0.02/0.84	0.00/0.84		0.080	0.728
AC‐PO	0.00/0.84	0.03/0.73	−0.02/0.85	0.00/0.84	−0.02/0.85	0.02/0.83	−0.04/0.83		0.077	0.728
SLM	0.00/0.85	0.03/0.56	−0.02/0.85	0.00/0.85	−0.02/0.85	0.02/1.04	−0.02/0.82		0.085	0.728
MLR truth scenario 2: imbalanced outcome, equidistant means
MLR	−0.01/0.80	0.03/0.45	0.01/0.73	0.01/0.80	0.01/0.73	0.02/0.82	−0.01/0.58		0.104	0.724
CL‐PO	0.00/0.80	0.02/0.64	0.00/0.96	0.00/0.80	0.00/0.96	0.00/0.84	−0.04/0.84		0.079	0.726
AC‐PO	−0.01/0.84	0.03/0.82	0.01/0.83	0.01/0.84	0.01/0.83	0.02/0.83	−0.01/0.83		0.077	0.725
SLM	−0.01/0.84	0.02/0.70	0.01/0.88	0.01/0.84	0.01/0.88	0.02/0.92	0.02/0.82		0.085	0.725
MLR truth scenario 3: balanced outcome, nonequidistant means
MLR	0.03/0.85	−0.03/0.56	0.02/0.79	−0.03/0.85	0.02/0.79	−0.04/0.83	0.03/0.67		0.105	0.725
CL‐PO	−0.01/1.07	−0.03/0.60	0.06/0.76	0.01/1.07	0.06/0.76	0.04/0.89	−0.04/0.89		0.110	0.725
AC‐PO	0.02/1.05	−0.02/0.76	0.03/0.75	−0.02/1.05	0.03/0.75	−0.03/0.87	0.03/0.87		0.108	0.725
SLM	0.03/0.86	−0.03/0.65	0.02/0.90	−0.03/0.86	0.02/0.90	−0.03/0.89	0.00/0.86		0.109	0.722
MLR truth scenario 4: imbalanced outcome, nonequidistant means
MLR	0.02/0.83	0.01/0.69	0.00/0.70	−0.02/0.83	0.00/0.70	0.00/0.84	0.00/0.54		0.101	0.724
CL‐PO	−0.02/0.95	0.03/0.94	0.02/0.73	0.02/0.95	0.02/0.73	0.02/0.86	−0.02/0.86		0.095	0.724
AC‐PO	0.00/0.99	0.02/1.20	−0.01/0.65	0.00/0.99	−0.01/0.65	0.01/0.84	−0.02/0.84		0.097	0.724
SLM	0.01/0.85	0.02/0.83	−0.01/0.86	−0.01/0.85	−0.01/0.86	0.01/0.89	−0.01/0.90		0.096	0.721
CL‐PO truth scenario 1: balanced outcome
MLR	0.01/0.79	0.00/0.38	0.02/0.80	−0.01/0.79	0.02/0.80	0.00/0.75	0.01/0.73		0.108	0.726
CL‐PO	0.00/0.87	0.01/0.75	0.01/0.86	0.00/0.87	0.01/0.86	0.03/0.86	−0.03/0.86		0.080	0.728
AC‐PO	0.01/0.85	0.01/1.01	0.01/0.85	−0.01/0.85	0.01/0.85	0.00/0.84	0.00/0.84		0.079	0.728
SLM	0.01/0.85	0.01/0.75	0.01/0.88	−0.01/0.85	0.01/0.88	0.00/0.68	0.00/0.83		0.089	0.727
CL‐PO truth scenario 2: imbalanced outcome
MLR	0.02/0.84	−0.01/0.63	0.03/0.73	−0.02/0.84	0.03/0.73	−0.02/0.86	0.03/0.54		0.103	0.724
CL‐PO	0.02/0.87	0.00/0.85	0.00/0.87	−0.02/0.87	0.00/0.87	0.03/0.87	−0.03/0.87		0.080	0.726
AC‐PO	0.02/0.92	0.00/1.12	0.01/0.76	−0.02/0.92	0.01/0.76	−0.01/0.85	0.01/0.85		0.081	0.725
SLM	0.02/0.87	−0.01/0.92	0.03/0.88	−0.02/0.87	0.03/0.88	−0.02/0.91	0.01/0.84		0.087	0.725
CL‐PO truth scenario 3: highly imbalanced outcome
MLR	−0.02/0.77	0.07/0.69	−0.04/0.46	0.02/0.77	−0.04/0.46	0.05/0.80	0.01/0.23		0.100	0.723
CL‐PO	−0.03/0.82	0.06/0.83	−0.04/0.83	0.03/0.82	−0.04/0.83	−0.02/0.82	−0.01/0.82		0.075	0.726
AC‐PO	−0.03/0.87	0.06/1.00	−0.05/0.64	0.03/0.87	−0.05/0.64	0.05/0.81	−0.08/0.81		0.077	0.726
SLM	−0.03/0.81	0.06/0.83	−0.01/0.72	0.03/0.81	−0.01/0.72	0.05/0.82	0.01/0.75		0.085	0.725

Abbreviations: AC‐PO, adjacent category logit model with proportional odds; CAD, coronary artery disease; CL‐PO, cumulative logit model with proportional odds; ECI, estimated calibration index; LP, linear predictor; MLR, multinomial logistic regression; ORC, ordinal C statistic; rMSPE, root mean squared prediction error; SLM, stereotype logit model.