TABLE 5.

Validation performance based on small development datasets of n = 500 for the main simulation scenarios (reported as the average performance on a large validation dataset for 200 simulated development datasets)

	Calibration intercepts/Calibration slopes
	Per outcome category			Per outcome dichotomy		Model‐specific			Single number metrics
Model	$Y=1$	$Y=2$	$Y=3$	$Y>1$	$Y>2$	LP1	LP2		rMSPE	ORC
MLR truth scenario 1: balanced outcome, equidistant means
MLR	0.01/0.97	−0.01/0.67	0.00/0.97	−0.01/0.97	0.00/0.97	−0.01/0.95	0.01/0.97		0.047	0.738
CL‐PO	0.01/1.00	−0.01/0.72	0.01/1.00	−0.01/1.00	0.01/1.00	0.01/0.98	−0.01/0.98		0.038	0.738
AC‐PO	0.01/0.98	−0.01/0.95	0.00/0.98	−0.01/0.98	0.00/0.98	−0.01/0.98	0.01/0.98		0.034	0.738
SLM	0.01/0.98	−0.01/0.87	0.00/0.99	−0.01/0.98	0.00/0.99	−0.01/0.99	0.00/0.98		0.037	0.738
MLR truth scenario 2: imbalanced outcome, equidistant means
MLR	0.00/0.95	0.00/0.83	0.00/0.95	0.00/0.95	0.00/0.95	0.00/0.95	0.00/0.93		0.044	0.736
CL‐PO	0.02/0.92	−0.01/0.75	−0.01/1.10	−0.02/0.92	−0.01/1.10	0.01/0.96	−0.01/0.96		0.038	0.737
AC‐PO	0.00/0.96	0.00/0.97	0.00/0.96	0.00/0.96	0.00/0.96	0.00/0.96	0.00/0.96		0.033	0.737
SLM	0.00/0.96	0.00/0.95	0.00/0.97	0.00/0.96	0.00/0.97	0.00/0.96	0.00/0.96		0.036	0.737
MLR truth scenario 3: balanced outcome, nonequidistant means
MLR	0.00/0.97	−0.01/0.88	0.01/0.97	0.00/0.97	0.01/0.97	−0.01/0.97	0.01/0.94		0.045	0.738
CL‐PO	−0.03/1.19	−0.01/0.72	0.05/0.84	0.03/1.19	0.05/0.84	0.01/0.98	−0.01/0.98		0.082	0.736
AC‐PO	0.00/1.17	−0.01/0.92	0.01/0.83	0.00/1.17	0.01/0.83	−0.01/0.98	0.01/0.98		0.081	0.736
SLM	0.00/0.98	−0.01/0.92	0.01/1.00	0.00/0.98	0.01/1.00	−0.01/0.98	0.00/0.98		0.073	0.735
MLR truth scenario 4: imbalanced outcome, nonequidistant means
MLR	0.01/0.97	−0.01/0.94	0.00/0.94	−0.01/0.97	0.00/0.94	−0.01/0.98	0.00/0.87		0.043	0.735
CL‐PO	−0.01/1.08	0.00/1.10	0.02/0.83	0.01/1.08	0.02/0.83	0.01/0.98	0.00/0.98		0.067	0.733
AC‐PO	0.01/1.14	0.00/1.42	0.00/0.75	−0.01/1.14	0.00/0.75	−0.01/0.97	0.00/0.97		0.072	0.733
SLM	0.01/0.98	0.00/0.99	0.00/0.99	−0.01/0.98	0.00/0.99	−0.01/0.98	−0.01/0.97		0.060	0.730
CL‐PO truth scenario 1: balanced outcome
MLR	0.02/0.94	0.00/0.90	−0.02/0.96	−0.02/0.94	−0.02/0.96	−0.01/0.94	−0.02/0.96		0.048	0.738
CL‐PO	0.02/0.97	0.00/0.96	−0.02/0.96	−0.02/0.97	−0.02/0.96	0.02/0.97	0.02/0.97		0.034	0.738
AC‐PO	0.02/0.96	0.00/1.30	−0.02/0.96	−0.02/0.96	−0.02/0.96	−0.01/0.96	−0.02/0.96		0.036	0.738
SLM	0.02/0.95	0.01/1.19	−0.02/0.97	−0.02/0.95	−0.02/0.97	−0.01/0.97	−0.03/0.96		0.039	0.738
CL‐PO truth scenario 2: imbalanced outcome
MLR	0.00/0.97	−0.01/0.97	0.03/0.91	0.00/0.97	0.03/0.91	0.00/0.98	0.03/0.84		0.046	0.737
CL‐PO	0.00/0.97	−0.01/0.97	0.03/0.96	0.00/0.97	0.03/0.96	0.00/0.97	−0.03/0.97		0.034	0.738
AC‐PO	0.00/1.03	−0.01/1.27	0.03/0.85	0.00/1.03	0.03/0.85	0.00/0.96	0.03/0.96		0.038	0.737
SLM	0.00/0.97	−0.01/1.07	0.03/0.94	0.00/0.97	0.03/0.94	0.00/0.98	0.03/0.95		0.038	0.738
CL‐PO truth scenario 3: highly imbalanced outcome
MLR	−0.02/0.96	0.02/0.96	0.01/0.87	0.02/0.96	0.01/0.87	0.02/0.98	0.00/0.64		0.041	0.739
CL‐PO	−0.02/0.96	0.02/0.97	0.01/0.97	0.02/0.96	0.01/0.97	−0.01/0.96	−0.02/0.96		0.033	0.739
AC‐PO	−0.02/1.04	0.02/1.17	0.01/0.74	0.02/1.04	0.01/0.74	0.02/0.96	−0.01/0.96		0.037	0.739
SLM	−0.02/0.96	0.02/0.99	0.01/0.97	0.02/0.96	0.01/0.97	0.02/0.97	0.02/0.97		0.035	0.739

Abbreviations: AC‐PO, adjacent category logit model with proportional odds; CAD, coronary artery disease; CL‐PO, cumulative logit model with proportional odds; ECI, estimated calibration index; LP, linear predictor; MLR, multinomial logistic regression; ORC, ordinal C statistic; rMSPE, root mean squared prediction error; SLM, stereotype logit model.