TABLE 3.
Apparent performance based on a large dataset of n = 200 000 for the main simulation scenarios
| Calibration intercepts/Calibration slopes | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Per outcome category | Per outcome dichotomy | Model‐specific | Single number metrics | ||||||||||||
| Model |
|
|
|
|
|
LP1 | LP2 | ECI | rMSPE | ORC | |||||
| MLR truth scenario 1: balanced outcome, equidistant means | |||||||||||||||
| MLR | 0.00/1.00 | 0.00/0.99 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.002 | 0.741 | |||||
| CL‐PO | 0.00/1.02 | −0.01/0.75 | 0.00/1.02 | 0.00/1.02 | 0.00/1.02 | 0.00/1.00 | 0.00/1.00 | 0.006 | 0.012 | 0.741 | |||||
| AC‐PO | 0.00/1.00 | 0.00/0.99 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.001 | 0.741 | |||||
| SLM | 0.00/1.00 | 0.00/0.99 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.001 | 0.741 | |||||
| MLR truth scenario 2: imbalanced outcome, equidistant means | |||||||||||||||
| MLR | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.002 | 0.740 | |||||
| CL‐PO | 0.01/0.96 | −0.01/0.79 | −0.01/1.14 | −0.01/0.96 | −0.01/1.14 | 0.00/1.00 | 0.00/1.00 | 0.010 | 0.016 | 0.740 | |||||
| AC‐PO | 0.00/1.00 | 0.00/1.01 | 0.00/0.99 | 0.00/1.00 | 0.00/0.99 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.002 | 0.740 | |||||
| SLM | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.002 | 0.740 | |||||
| MLR truth scenario 3: balanced outcome, nonequidistant means | |||||||||||||||
| MLR | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.002 | 0.741 | |||||
| CL‐PO | −0.03/1.21 | −0.01/0.75 | 0.03/0.86 | 0.03/1.21 | 0.03/0.86 | 0.00/1.00 | 0.00/1.00 | 0.049 | 0.075 | 0.738 | |||||
| AC‐PO | 0.00/1.19 | 0.00/0.95 | 0.00/0.84 | 0.00/1.19 | 0.00/0.84 | 0.00/1.00 | 0.00/1.00 | 0.046 | 0.074 | 0.738 | |||||
| SLM | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.063 | 0.738 | |||||
| MLR truth scenario 4: imbalanced outcome, nonequidistant means | |||||||||||||||
| MLR | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.002 | 0.737 | |||||
| CL‐PO | −0.02/1.11 | 0.01/1.13 | 0.03/0.85 | 0.02/1.11 | 0.03/0.85 | 0.00/1.00 | 0.00/1.00 | 0.032 | 0.058 | 0.735 | |||||
| AC‐PO | 0.00/1.17 | 0.00/1.47 | 0.00/0.76 | 0.00/1.17 | 0.00/0.76 | 0.00/1.00 | 0.00/1.00 | 0.059 | 0.064 | 0.736 | |||||
| SLM | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.047 | 0.733 | |||||
| CL‐PO truth scenario 1: balanced outcome | |||||||||||||||
| MLR | 0.00/0.99 | 0.00/1.38 | 0.00/0.99 | 0.00/0.99 | 0.00/0.99 | 0.00/1.00 | 0.00/1.00 | 0.006 | 0.014 | 0.740 | |||||
| CL‐PO | 0.00/1.00 | 0.00/1.01 | 0.00/0.99 | 0.00/1.00 | 0.00/0.99 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.003 | 0.740 | |||||
| AC‐PO | 0.00/1.00 | 0.00/1.38 | 0.00/0.99 | 0.00/1.00 | 0.00/0.99 | 0.00/1.00 | 0.00/1.00 | 0.006 | 0.014 | 0.740 | |||||
| SLM | 0.00/0.99 | 0.00/1.38 | 0.00/0.99 | 0.00/0.99 | 0.00/0.99 | 0.00/1.00 | 0.00/1.00 | 0.006 | 0.014 | 0.740 | |||||
| CL‐PO truth scenario 2: imbalanced outcome | |||||||||||||||
| MLR | 0.00/1.00 | 0.00/1.09 | 0.00/0.99 | 0.00/1.00 | 0.00/0.99 | 0.00/1.00 | 0.00/1.00 | 0.005 | 0.013 | 0.740 | |||||
| CL‐PO | 0.00/1.00 | 0.00/1.01 | 0.00/0.99 | 0.00/1.00 | 0.00/0.99 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.003 | 0.740 | |||||
| AC‐PO | 0.00/1.07 | 0.00/1.34 | 0.00/0.88 | 0.00/1.07 | 0.00/0.88 | 0.00/1.00 | 0.00/1.00 | 0.012 | 0.018 | 0.740 | |||||
| SLM | 0.00/1.00 | 0.00/1.09 | 0.00/0.99 | 0.00/1.00 | 0.00/0.99 | 0.00/1.00 | 0.00/1.00 | 0.006 | 0.013 | 0.740 | |||||
| CL‐PO truth scenario 3: highly imbalanced outcome | |||||||||||||||
| MLR | 0.00/1.00 | 0.00/1.02 | 0.00/0.98 | 0.00/1.00 | 0.00/0.98 | 0.00/1.00 | 0.00/1.00 | 0.004 | 0.009 | 0.742 | |||||
| CL‐PO | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.00/1.00 | 0.000 | 0.002 | 0.742 | |||||
| AC‐PO | 0.00/1.08 | 0.00/1.22 | 0.00/0.77 | 0.00/1.08 | 0.00/0.77 | 0.00/1.00 | 0.00/1.00 | 0.015 | 0.017 | 0.742 | |||||
| SLM | 0.00/1.00 | 0.00/1.02 | 0.00/0.98 | 0.00/1.00 | 0.00/0.98 | 0.00/1.00 | 0.00/1.00 | 0.004 | 0.009 | 0.742 | |||||
Abbreviations: AC‐PO, adjacent category logit model with proportional odds; CAD, coronary artery disease; CL‐PO, cumulative logit model with proportional odds; ECI, estimated calibration index; LP, linear predictor; MLR, multinomial logistic regression; ORC, ordinal C statistic; rMSPE, root mean squared prediction error; SLM, stereotype logit model.