. 2019 Jan 16;38(11):2074–2102. doi: 10.1002/sim.8086

Table 6.

Performance measures: definitions, estimates and Monte Carlo standard errors

Performance

Measure

Definition

Estimate

Monte Carlo SE of Estimate

Bias

E [\hat{θ}] - θ

\frac{1}{n_{sim}} \sum_{i = 1}^{n_{sim}} {\hat{θ}}_{i} - θ

\sqrt{\frac{1}{n_{sim} (n_{sim} - 1)} \sum_{i = 1}^{n_{sim}} {({\hat{θ}}_{i} - \bar{θ})}^{2}}

EmpSE

\sqrt{Var (\hat{θ})}

\sqrt{\frac{1}{n_{sim} - 1} \sum_{i = 1}^{n_{sim}} {({\hat{θ}}_{i} - \bar{θ})}^{2}}

\frac{\hat{EmpSE}}{\sqrt{2 (n_{sim} - 1)}}

Relative % increase in precision (B vs A) ^a

100 (\frac{Var ({\hat{θ}}_{A})}{Var ({\hat{θ}}_{B})} - 1)

100 ({(\frac{{\hat{EmpSE}}_{A}}{{\hat{EmpSE}}_{B}})}^{2} - 1)

200 {(\frac{{\hat{EmpSE}}_{A}}{{\hat{EmpSE}}_{B}})}^{2} \sqrt{\frac{1 - Corr {({\hat{θ}}_{A}, {\hat{θ}}_{B})}^{2}}{n_{sim} - 1}}

MSE

E [{(\hat{θ} - θ)}^{2}]

\frac{1}{n_{sim}} \sum_{i = 1}^{n_{sim}} {({\hat{θ}}_{i} - θ)}^{2}

\sqrt{\frac{\sum_{i = 1}^{n_{sim}} {[{({\hat{θ}}_{i} - θ)}^{2} - \hat{MSE}]}^{2}}{n_{sim} (n_{sim} - 1)}}

Average ModSE^a

\sqrt{E [\hat{Var} (\hat{θ})]}

\sqrt{\frac{1}{n_{sim}} \sum_{i = 1}^{n_{sim}} \hat{Var} ({\hat{θ}}_{i})}

\sqrt{\frac{\hat{Var} [\hat{Var} (\hat{θ})]}{4 n_{sim} \times {\hat{ModSE}}^{2}}}

Relative % error in ModSE^a

100 (\frac{ModSE}{EmpSE} - 1)

100 (\frac{\hat{ModSE}}{\hat{EmpSE}} - 1)

100 (\frac{\hat{ModSE}}{\hat{EmpSE}}) \sqrt{\frac{\hat{Var} [\hat{Var} (\hat{θ})]}{4 n_{sim} \times {\hat{ModSE}}^{4}} + \frac{1}{2 (n - 1)}}

Coverage

\Pr ({\hat{θ}}_{low} \leq θ \leq {\hat{θ}}_{upp})

\frac{1}{n_{sim}} \sum_{i = 1}^{n_{sim}} 1 ({\hat{θ}}_{low, i} \leq θ \leq {\hat{θ}}_{upp, i})

\sqrt{\frac{\hat{Cover.} \times (1 - \hat{Cover.})}{n_{sim}}}

Bias‐eliminated coverage

\Pr ({\hat{θ}}_{low} \leq \bar{θ} \leq {\hat{θ}}_{upp})

\frac{1}{n_{sim}} \sum_{i = 1}^{n_{sim}} 1 ({\hat{θ}}_{low, i} \leq \bar{θ} \leq {\hat{θ}}_{upp, i})

\sqrt{\frac{\hat{B‐E Cover.} \times (1 - \hat{B‐E Cover.})}{n_{sim}}}

Rejection % (power or type I error)

Pr(p _i ≤ α)

\frac{1}{n_{sim}} \sum_{i = 1}^{n_{sim}} 1 (p_{i} \leq α)

\sqrt{\frac{\hat{Power} \times (1 - \hat{Power})}{n_{sim}}}

Monte Carlo SEs are approximate for Relative % increase in precision, Average ModSE, and Relative %error in ModSE.

$\hat{Var} [\hat{Var} (\hat{θ})] = \frac{1}{n_{sim} - 1} \sum_{i = 1}^{n_{sim}} {\hat{Var} ({\hat{θ}}_{i}) - \frac{1}{n_{sim}} \sum_{j = 1}^{n_{sim}} \hat{Var} ({\hat{θ}}_{j})}^{2}$ .