Table 3. Definitions of Common Performance Measures, their Estimates, Monte Carlo Standard Errors (MCSE), and Number of Simulation Repetitions n_sim to Achieve a Desired MCSE*.

Performance measure	Definition	Estimate	MCSE	n sim
Bias	$\text{E}(\widehat{\theta })-\theta$	$\left({{\displaystyle \sum }}_{i=1}^{n_{\text{sim}}}{\widehat{\theta }}_i/n_{\text{sim}}\right)-\theta$	$\sqrt{S_{\widehat{\theta }}^2/n_{\text{sim}}}$	$S_{\widehat{\theta }}^2/{\text{MCSE}}_{\ast }^2$
Relative bias	$\{\text{E}(\widehat{\theta })-\theta \}/\theta$	$\left\{\left({{\displaystyle \sum }}_{i=1}^{n_{\text{sim }}}{\widehat{\theta }}_i/n_{\text{sim }}\right)-\theta \right\}/\theta$	$\sqrt{S_{\widehat{\theta }}^2/\left({\theta }^2{n}_{\text{sim}}\right)}$	$S_{\widehat{\theta }}^2/\left({\text{MCSE}}_{\ast }^2{\theta }^2\right)$
Mean square error (MSE)	$\text{E}\left\{{(\widehat{\theta }-\theta )}^2\right\}$	${{\displaystyle \sum }}_{i=1}^{n_{\text{sim}}}{\left({\widehat{\theta }}_i-\theta \right)}^2/n_{\text{sim}}$	$\sqrt{S_{{(\widehat{\theta }-\theta )}^2}^2/n_{\text{sim}}}$	$S_{{(\widehat{\theta }-\theta )}^2}^2/{\text{MCSE}}_{\ast }^2$
Root mean square error (RMSE)	$\sqrt{\text{E}\left\{{(\widehat{\theta }-\theta )}^2\right\}}$	$\sqrt{{{\displaystyle \sum }}_{i=1}^{n_{\text{sim}}}{\left({\widehat{\theta }}_i-\theta \right)}^2/n_{\text{sim}}}$	$\sqrt{S_{{(\widehat{\theta }-\theta )}^2}^2/\left(4n_{\text{sim}}\widehat{\text{MSE}}\right)}$	$S_{{(\widehat{\theta }-\theta )}^2}^2/\left(4\widehat{\text{MSE}}{\text{ MCSE}}_{\ast }^2\right)$
Empirical variance	$\text{Var}(\widehat{\theta })$	$S_{\widehat{\theta }}^2$	$S_{\widehat{\theta }}^2\sqrt{2/\left(n_{\text{sim}}-1\right)}$	$1+2{\left(S_{\widehat{\theta }}^2\right)}^2/{\text{MCSE}}_{\ast }^2$
Empirical standard error	$\sqrt{\text{Var}(\widehat{\theta })}$	$\sqrt{S_{\widehat{\theta }}^2}$	$\sqrt{S_{\widehat{\theta }}^2/\left\{2\left(n_{\text{sim}}-1\right)\right\}}$	$1+S_{\widehat{\theta }}^2/\left(2{\text{MCSE}}_{\ast }^2\right)$
Coverage	Pr(CI includes θ)	${{\displaystyle \sum }}_{i=1}^{n_{\text{sim }}}𝟙\left({\text{CI}}_i\text{includes}\theta \right)/n_{\text{sim }}$	$\sqrt{\widehat{\text{Cov}}(1-\widehat{\text{Cov}})/n_{\text{sim}}}$	$\widehat{\text{Cov}}(1-\widehat{\text{Cov}})/{\text{MCSE}}_{\ast }^2$
Power (or Type I error rate)	Pr(Test rejects H0)	${{\displaystyle \sum }}_{i=1}^{n_{\text{sim}}}𝟙({\text{Test}}_i\text{rejects }{H}_0)/n_{\text{sim }}$	$\sqrt{\widehat{\text{ Pow }}(1-\widehat{\text{ Pow }})/n_{\text{sim }}}$	$\widehat{\text{Pow}}(1-\widehat{\text{Pow}})/{\text{MCSE}}_{\ast }^2$
Mean CI width	E(CIupper − CIlower)	${{\displaystyle \sum }}_{i=1}^{n_{\text{sim }}}\left(\text{C}{\text{I}}_{i,\text{upper}}-\text{C}{\text{I}}_{i,\text{lower}}\right)/n_{\text{sim }}$	$\sqrt{S_W^2/n_{\text{sim }}}$	$S_W^2/{\text{MCSE}}_{\ast }^2$
Mean of generic statistic G	E(G)	${{\displaystyle \sum }}_{i=1}^{n_{\text{sim}}}{G}_i/n_{\text{sim}}$	$\sqrt{S_G^2/n_{\text{sim}}}$	$S_G^2/{\text{MCSE}}_{\ast }^2$

Note. Table adapted from Table 6 in Morris et al. (2019)

E(X) and Var(X) are the expected value and variance of a random variable X, respectively. Summation is denoted by ${{\displaystyle \sum }}_{i=1}^n{x}_i=x_1+x_2+\cdots +x_{n-1}+x_n$.

$\widehat{\theta }$ is an estimator of the estimand θ, and ${\widehat{\theta }}_i$ is the estimate obtained from simulation i

𝟙 (CI_i includes θ) and 𝟙 (Test_i rejects H₀) are 1 if the respective event occurred in simulation i and 0 otherwise

$\widehat{\text{MSE}}\text{, }\widehat{\text{Cov}}$, and $\widehat{\text{Pow}}$ denote the estimated MSE, coverage, and power, respectively. MCSE_* denotes the desired MCSE when calculating the number of repetitions n_sim.

The sample variance of the estimates is $S_{\widehat{\theta }}^2={{\displaystyle \sum }}_{i=1}^{n_{\text{sim }}}{\{{\widehat{\theta }}_i-\left({{\displaystyle \sum }}_{i=1}^{n_{\text{sim }}}{\widehat{\theta }}_i/n_{\text{sim }}\right)\}}^2/\left(n_{\text{sim }}-1\right)$

The sample variance of the square errors is $S_{{(\widehat{\theta }-\theta )}^2}^2={{\displaystyle \sum }}_{i=1}^{n_{\text{sim}}}{\left[{\left({\widehat{\theta }}_i-\theta \right)}^2-\left\{{{\displaystyle \sum }}_{i=1}^{n_{\text{sim}}}{\left({\widehat{\theta }}_i-\theta \right)}^2/n_{\text{sim}}\right\}\right]}^2/\left(n_{\text{sim}}-1\right)$

The sample variance of the CI widths is $S_W^2={{\displaystyle \sum }}_{i=1}^{n_{\text{sim}}}{[\left(\text{C}{\text{I}}_{i,\text{upper}}-\text{C}{\text{I}}_{i,\text{lower}}\right)-\left\{{{\displaystyle \sum }}_{i=1}^{n_{\text{sim}}}\left(\text{C}{\text{I}}_{i,\text{upper }}-\text{C}{\text{I}}_{i,\text{lower}}\right)/n_{\text{sim }}\right\}]}^2/\left(n_{\text{sim }}-1\right)$

The sample variance of a generic statistic G is $S_G^2={{\displaystyle \sum }}_{i=1}^{n_{\text{sim}}}{\{G_i-\left({{\displaystyle \sum }}_{i=1}^{n_{\text{sim}}}{G}_i/n_{\text{sim}}\right)\}}^2/\left(n_{\text{sim}}-1\right)$ with G_i the statistic obtained from simulation i. For example, G may be a measure of predictive performance.