On summary ROC curve for dichotomous diagnostic studies: an application to meta-analysis of COVID-19

Abstract

In a systematic review of a diagnostic performance, summarizing performance metrics is crucial. There are various summary models in the literature, and hence model selection becomes inevitable. However, most existing large-sample-based model selection approaches may not fit in a meta-analysis of diagnostic studies, typically having a rather small sample size. Researchers need to effectively determine the final model for further inference, which motivates this article to investigate existing methods and to suggest a more robust method for this need. We considered models covering several widely-used methods for bivariate summary of sensitivity and specificity. Simulation studies were conducted based on different number of studies and different population sensitivity and specificity. Then final models were selected using several existing criteria, and we compared the summary receiver operating characteristic (sROC) curves to the theoretical ROC curve given the generating model. Even though parametric likelihood-based criteria are often applied in practice for their asymptotic property, they fail to consistently choose appropriate models under the limited number of studies. When the number of studies is as small as 10 or 5, our suggestion is best in different scenarios. An example for summary ROC curves for chemiluminescence immunoassay (CLIA) used in COVID-19 diagnosis is also illustrated.

1. Introduction

Summarizing information on test performance metrics, such as sensitivity, specificity, and diagnostic odds ratios (DOR), is an important part of a systematic review of a medical test performance on clinical outcomes. Through a meta-analysis on clinical studies of diagnostic tests, we may investigate hypotheses about the test performance that cannot be answered by an individual study. Sotiriadis et al. [33] recommended a guideline for systematic review of diagnostic test accuracy studies, as a counterpart to Cochrane Handbook [21] and PRISMA [32] widely used in general systematic review.

Most diagnostic tests are used to separate patients into two groups as test positive and test negative (i.e. $T+/T-$). The fundamental diagnostic result can be a real value (continuous output), in which case the classification boundary between the two groups must be determined by a threshold value, usually based on a receiver operating characteristic (ROC) curve [35]. Accordingly, there are four possible outcomes from a dichotomized test when a gold standard is available. If the true disease status of a subject is positive ( $D+$), a $T+$ classification is called a true positive (TP), while a $T-$ result is called a false negative (FN). Conversely, given a true negative disease status of a subject ( $D-$), a $T-$ classification results in a true negative (TN) and a $T+$ result gets a false positive (FP).

When a gold standard exists, test accuracy is estimated as the proportion of diseased individuals to be ‘test positive’ (sensitivity) and of non-diseased individuals to be ‘test negative’ (specificity) [14,23,24]. For systematic reviews of dichotomous diagnostic studies, we have ‘data’ merely consisting of nTP, nFN, nFP, and nTN for each study involved, summing up the number of subjects classified as TP, FN, FP, or TN, respectively. The corresponding results are usually reported based on a particular threshold, as used in clinical practice. It is improper to simply use the sums across studies of these four numbers to derive summary estimates of sensitivity, specificity and DOR, where the summary statistics would be dominantly affected by several studies in the largest study sizes [17].

Another naive summary is to pool sensitivity and specificity separately using standard meta-analyses for proportions. However, Walter and Jadad [38] and Moses et al. [26] showed that sensitivity and specificity are often negatively correlated, usually because of different thresholds among studies to define $T+$ and $T-$. Even though this ‘separate summary’ method is sometimes recommended [8,34], ignoring the correlation between sensitivity and specificity would result in an inaccurate inference or even misleading in the claims.

Chappell et al. [8] and Trikalinos et al. [34] discussed that helpful ways about summarizing medical test studies include ‘separate summary’, ‘summary point’, and ‘summary line’. Some advised procedures regarding when to use which kind of summary representations were also provided in their studies. Various models have been proposed in the literature for a meaningful summary ‘point’ or ‘line’ across all studies [22,26,28,29] The methods of Reitsma et al. [28] and Rutter and Gatsonis [29] have become almost the de facto standard for a ‘summary point’ or a ‘summary line’, and Harbord et al. [20] showed their equivalence when no covariates are included. If more studies are available (usually with a number larger than 30), some sophisticated extensions attempt to incorporate other sources of heterogeneities, such as disease prevalence [11], latent subgroups [30] or measurement errors [19]. In view of so many alternative methods, a natural and important question is how to select a suitable model. Recently, Doebler et al. [15] integrated a wide range of models into a unified parametric linear mixed model framework after transformation, utilizing likelihood-maximization to estimate model parameters. It covers the models of Chu et al. [10], Reitsma et al. [28], Rutter and Gatsonis [29], and Holling et al. [22] as special cases. Furthermore, some likelihood-based criteria can be used for selecting a ‘best’ model, among which the Akaike information criterion (AIC) with nice asymptotic properties is the most often used method [5]. A related criterion, Bayesian information criterion (BIC) or Schwarz information criterion [31], is often used, too. When a set of candidate models is considered, we may choose the model with the smallest AIC or BIC value, and make statistical inference based on it. As discussed in Section 2.2, in the cases considered here, AIC and BIC will always select the same model. Hence in this paper we only discuss on AIC.

Nevertheless, the practical essence of such meta-analyses is restricted to a small sample size N more often than not, where N is the final number of studies for systematic review. For the same medical test design, there are usually not many compatible studies and hence N is too limited to apply the relative asymptotic theories. Several studies proposed conditional AIC (cAIC) in linear mixed models, which is a tailored model selection method for small N [18,25,36]. In addition, an empirical likelihood (EL) method analogous to the work of Owen [27] can also be used for small N in practice. In this article, we focus on the issue of model selection for ‘summary line’ situations. The key questions here include: (a) whether AIC gives an acceptable result, (b) which selection criterion (e.g. AIC, cAIC, or EL) has better performance, and (c) does there exist a summary method that performs satisfactorily under various situations especially when N is small?

The rest of the article is organized as follows. Section 2 first reviews several commonly used models and then describes some existing model selection criteria. The effectiveness of these model selection methods are compared through simulations in Section 3. An example of its application to coronavirus disease-2019 (COVID-19) diagnosis is given in Section 4. Finally, we conclude with some discussion in Section 5.

2. Model families and selection criteria

This section introduces the fundamentals of this study. We shall reviews commonly used models and describe several existing model selection criteria. Then all these methods will be investigated using simulations in Section 3.

2.1. Models to be considered

This section briefly describes the two model families under $t_{\alpha }$ transformation as illustrated in [15] and the relations with other approaches. The details of models can be found in the original literature.

A class of monotonic transformation functions controlled by $\alpha \in [0,1]$ for $x\in [0,1]$ is defined as follows [15]:

$$t_{\alpha }(x)=\alpha \log (x)-(2-\alpha )\log (1-x).$$

Let $p_i$ and $q_i$ be the unobserved true sensitivity and false positive rate (1-specificity) for the ith study, respectively. With a pair of transformation parameters $({\alpha }_p,{\alpha }_q{)}^{\mathrm{\prime }}$, the two transformed variables $(t_{{\alpha }_p}(p_i),t_{{\alpha }_q}(q_i){)}^{\mathrm{\prime }}$ are then assumed to follow a bivariate normal distribution with mean $\mathit{\mu }=({\mu }_p,{\mu }_q{)}^{\mathrm{\prime }}$ and covariance matrix

$${\mathbf{\Sigma }}_i=\left(\begin{array}{cc}{\sigma }_p^2& \sigma \\ \sigma & {\sigma }_q^2\end{array}\right)+\left(\begin{array}{cc}{d}_{ip}^2& 0\\ 0& d_{iq}^2\end{array}\right).$$ (1)

Following Doebler et al. [15], we consider two families of models by setting $d_{ip}^2$ and $d_{iq}^2$ in (1) to different values. The first family of models uses fixed $d_{ip}^2=d_{iq}^2=0$ while the second family of models takes study heteroscedasticity into account with $d_{ip}^2=\widehat{\mathrm{Var}}(\frac{{\mathrm{nTP}}_i}{{\mathrm{nTP}}_i+{\mathrm{nFN}}_i})$ and $d_{iq}^2=\widehat{\mathrm{Var}}(\frac{{\mathrm{nFP}}_i}{{\mathrm{nTN}}_i+{\mathrm{nFP}}_i})$.

It has been pointed out that $t_{\alpha }$ respectively corresponds to logit transformation with $\alpha =1$ and log transformation with $\alpha =2$. Moreover, $t_{\alpha }$ is also approximately proportional to the complementary logarithmic function when α is around 0.6. On the other hand, $t_{\alpha }(x)$ can be regarded as $\log (1-x)$ and complementary $\log (1-x)$ if $\alpha =0$ and $\alpha =1.4$, respectively.

When ${\alpha }_p={\alpha }_q=2$, the first family of models corresponds to the Lehmann family or proportional hazard models of Holling et al. [22]. If $\mathrm{logit}(p_i)$ and $\mathrm{logit}(q_i)$ are assumed to follow a bivariate normal distribution, the first family of models with ${\alpha }_p={\alpha }_q=1$ coincides with the summary ROC method of Moses et al. [26] but based on different parameterizations. Hereafter, the corresponding model is called MSL method.

When ${\alpha }_p={\alpha }_q=1$, the second family of models is equivalent to bivariate models [28] and the hierarchical summary ROC method (HSROC) of [29]. Furthermore, the second family of models approaches the complementary logarithmic models of [10] when ${\alpha }_p={\alpha }_q=1.4$.

Aside from $({\alpha }_p,{\alpha }_q{)}^{\mathrm{\prime }}$, each family has five common parameters $\mathit{\theta }=({\mu }_p,{\mu }_q,{\sigma }_p^2,{\sigma }_q^2,\sigma {)}^{\mathrm{\prime }}$ involved. Maximum likelihood (ML) or restricted maximum likelihood (REML) methods can be used to estimate model parameters. For a larger N, $({\alpha }_p,{\alpha }_q{)}^{\mathrm{\prime }}$ can also be estimated as additional parameters.

2.2. Model selection criteria

The work of [15] reviewed in the previous subsection generalized several widely used models. For $N\ge 20$, they also showed that it is possible to recover $({\alpha }_p,{\alpha }_q{)}^{\mathrm{\prime }}$ by treating it as free parameters. Nevertheless, they admitted that it is hard to estimate $({\alpha }_p,{\alpha }_q{)}^{\mathrm{\prime }}$ for $N\le 10$, and they suggested these two quantities should be fixed for a small N. In practice, treating $({\alpha }_p,{\alpha }_q{)}^{\mathrm{\prime }}$ as fixed or free parameters does not respectively make the problem easier or harder; an analyst still needs a way to determine suitable values of $({\alpha }_p,{\alpha }_q{)}^{\mathrm{\prime }}$ for transformation. Additionally, although they proposed two useful families of models, little has been known about how to select among them especially when the sample size N is small. Therefore, a good model selection strategy is important.

Model selection can be viewed as a selection of both the model assumptions and the estimated parameters, which amounts to a choice of underlying probabilistic mechanism. Model selection (or variable selection) in linear regression and generalized linear models has been studied extensively [5,12,16]. Unfortunately, the models considered here have no variables to be selected and the key structure, the covariance matrix ${\mathbf{\Sigma }}_i$, is heavily affected by $({\alpha }_p,{\alpha }_q{)}^{\mathrm{\prime }}$. Selection of the covariance structure in a linear mixed model is still a very open research area. Yet, the two families considered here are linear mixed models only after transformation, which raises another challenge for us. Therefore, the results would be doubtful if one directly applies the existing model selection methods to select among the two families.

In what follows, a ‘model’ indicates a triplet of ${\alpha }_p$, ${\alpha }_q$, and a model family index (i.e. 1st or 2nd). We shall consider several model selection criteria, and compare their performance based on simulations. The first one is AIC [1], which was also inspected in Doebler et al.[15]. Let $l_M(\hat{\mathit{\theta }})$ be the log-likelihood for a model M and $\hat{\mathit{\theta }}$ be the corresponding estimates of model parameters, and k be the number of parameters. Then AIC is defined as $-2l_M(\hat{\mathit{\theta }})+2k$ with k being the number of parameters in the model M. Since each model has 5 parameters, minimizing AIC amounts to choosing the model having the largest $l_M(\hat{\mathit{\theta }})$, where $\hat{\mathit{\theta }}$ is obtained by ML or REML. BIC defined as $-2l_M(\hat{\mathit{\theta }})+k\log (N)$, will also choose the maximum-likelihood model, since $k\log (N)$ are the same for all models. Thus AIC and BIC are equivalent for the considered cases. Moreover, we use the term AIC simply because Doebler et al. [15] also suggested this criterion.

While AIC has been shown to have nice asymptotic properties for model selection with a large N [5] the focus in this work is the small N problem. Although there is a corrected version of AIC with penalty $2k(k+1)/(N-k-1)$ instead of $2k$ for small N [6], it selects an identical model as AIC for models considered here.

The second and third criteria are based on cAIC in the literature [18,25,36]. Let ${\mathbf{y}}_i=(\frac{{\mathrm{nTP}}_i}{{\mathrm{nTP}}_i+{\mathrm{nFN}}_i},\frac{{\mathrm{nFP}}_i}{{\mathrm{nTN}}_i+{\mathrm{nFP}}_i}{)}^{\mathrm{\prime }}$; $i=1,2,\dots ,N$ be the vector of observed sensitivities and 1-specificities. For a specific model M, define ${\hat{\mathbf{z}}}_i(M)={\mathrm{E}}_M({\mathbf{z}}_i|\hat{\mathit{\theta }},\mathbf{Z})$, where

$${\mathbf{z}}_i=\mathbf{t}({\mathbf{y}}_i)\equiv {(t_{{\alpha }_p}\left(\frac{{\mathrm{nTP}}_i}{{\mathrm{nTP}}_i+{\mathrm{nFN}}_i}\right),t_{{\alpha }_q}\left(\frac{{\mathrm{nFP}}_i}{{\mathrm{nTN}}_i+{\mathrm{nFP}}_i}\right))}^{\mathrm{\prime }},$$

and $\mathbf{Z}=({\mathbf{z}}_1^{\mathrm{\prime }},\dots ,{\mathbf{z}}_N^{\mathrm{\prime }}{)}^{\mathrm{\prime }}$. Thus ${\hat{\mathbf{z}}}_i(M)$ is the empirical best linear unbiased predictor of $\mathbf{t}({\mathit{\mu }}_i)\equiv (t_{{\alpha }_p}(p_i),t_{{\alpha }_q}(q_i){)}^{\mathrm{\prime }}$ based on the model M. Then the conditional AIC (cAIC) is defined as $l_M^{\ast }(\hat{\mathit{\theta }})$+ penalty, where $l_M^{\ast }(\hat{\mathit{\theta }})=l_M(\hat{\mathit{\theta }})-l_M(\widehat{\mathit{\theta }|}\hat{\mathbf{Z}}(M))$ with $l_M(\widehat{\mathit{\theta }|}\hat{\mathbf{Z}}(M))$ being the log-likelihood for a model M evaluated at $\hat{\mathit{\theta }}$ and the observations $\mathbf{Z}$ replaced by $\hat{\mathbf{Z}}(M)\equiv ({\hat{\mathbf{z}}}_1(M{)}^{\mathrm{\prime }},\dots ,{\hat{\mathbf{z}}}_N(M{)}^{\mathrm{\prime }}{)}^{\mathrm{\prime }}$, and the penalty in cAIC was discussed in Vaida and Blanchard [36], Liang et al. [25], and Greven and Kneib [18]. In particular, Vaida and Blanchard [36] assumed $\mathit{\theta }$ to be known, while Liang et al. [25] and Greven and Kneib [18] took the uncertainty of estimation into consideration. The major difference between Liang et al. [25] and Greven and Kneib [18] is that the former calculated the penalty approximately, while the latter provided an exact method. In the simulation studies, we will compare the performance of Vaida and Blanchard [36] and Greven and Kneib [18], and refer to them as cAIC-VB and cAIC-GK, respectively. Also note that a model in the first family does not consider the random effect, hence cAIC reduces to AIC in this case.

The fourth criterion is EL approach [27], which was primarily a method for constructing a confidence region for mean parameters, and Baggerly [3] pointed out its connection to goodness-of-fit measures. Denote ${\mathbf{t}}^{-1}(\mathbf{x})\equiv (t_{{\alpha }_p}^{-1}(x_p),t_{{\alpha }_q}^{-1}(x_q){)}^{\mathrm{\prime }}$ for an arbitrary $\mathbf{x}=(x_p,x_q{)}^{\mathrm{\prime }}$, and hence ${\mathbf{t}}^{-1}({\mathit{\mu }}_M)\equiv (t_{{\alpha }_p}^{-1}({\mu }_p),t_{{\alpha }_q}^{-1}({\mu }_q){)}^{\mathrm{\prime }}$ is the (back-transformed) mean of summarized sensitivity and 1-specificity for a model M. For a model with the mean parameters being ${\mathit{\mu }}_M=({\mu }_p,{\mu }_q{)}^{\mathrm{\prime }}$, its EL is given by

$$L({\mathit{\mu }}_M)=max\prod _{i=1}^N{w}_i({\mathit{\mu }}_M)$$ (2)

under constraints $\sum _i{w}_i({\mathit{\mu }}_M)({\mathbf{y}}_i-{\mathbf{t}}^{-1}({\mathit{\mu }}_M))=\mathbf{0}$, $w_i({\mathit{\mu }}_M)>0$ and $\sum _i{w}_i({\mathit{\mu }}_M)=1$. EL for the saturated model is

$$L({\mathit{\mu }}^{\ast })=max\prod _{i=1}^N{w}_i$$

under the constraints $w_i>0$ and $\sum _i{w}_i=1$. To assess the hypothesis that ${\mathit{\mu }}_M$ is the mean of N independent data ${\mathbf{y}}_1,\dots ,{\mathbf{y}}_N$, we should first find the weight $w_i({\mathit{\mu }}_M)$ of each datum with (2). Then,

$$\begin{array}{rl}R& =-2\log \left(\frac{L({\mathit{\mu }}_M)}{L({\mathit{\mu }}^{\ast })}\right)\\ & =-2\sum _{i=1}^N\log (w_i({\mathit{\mu }}_M))+2N\log (N)\end{array}$$

can be obtained, where $L({\mathit{\mu }}^{\ast })=N^{-N}$ is a constant and R has a chi-square limiting distribution with a degree of freedom equal to the rank of $\mathrm{Var}(\mathbf{y})$ [27]. Thus, a larger value of R indicates a model's deficiency.

3. Comparison of selection criteria

To investigate the model selection methods above, we conduct several simulation studies. Each model selection criterion is applied to the simulated data, and we compare the models selected by different criteria.

3.1. Simulation setup

We considered meta-analyses of N = 5, 10 or 50 primary studies in the diagnostic test, and generated the corresponding ${\mathbf{x}}_i$=(nTP ${}_i$,nFN ${}_i$,nFP ${}_i$,nTN ${}_i$) ${}^{\mathrm{\prime }}$ for each primary study according to the model given in the previous subsection. Two scenarios of sensitivity and specificity were generated. The first scenario has $\mathit{\mu }=(t_{{\alpha }_p}(0.9),t_{{\alpha }_q}(0.9){)}^{\mathrm{\prime }}$ while the second has $\mathit{\mu }=(t_{{\alpha }_p}(0.95),t_{{\alpha }_q}(0.8){)}^{\mathrm{\prime }}$. To generate data for the ith study, $m_{0i}$ non-disease participants and $m_{1i}$ diseased participants were drawn independently and identically from a Poisson distribution with mean 76 and 49 respectively. That is, on average we had 49 diseased and 76 non-disease participants in each primary study. The mean numbers for participants are based on a survey by Bachmann et al. [2].

Once $(t_{{\alpha }_p}(p_i),t_{{\alpha }_q}(q_i){)}^{\mathrm{\prime }}$ was drawn from a bivariate normal distribution, we straightforward obtained ${\mathbf{x}}_i=$(nTP ${}_i$,nFN ${}_i$,nFP ${}_i$,nTN ${}_i$) ${}^{\mathrm{\prime }}$. Based on data values of ${\mathbf{x}}_i$; $i=1,2,\dots ,N$, the standard estimation procedures for a model (a triplet of ${\alpha }_p$, ${\alpha }_q$ and index of family) was applied, and all competing model selection criteria introduced in the previous subsection were also used. For simplicity, candidate models were restricted to those only within $({\alpha }_p,{\alpha }_q{)}^{\mathrm{\prime }}\in \{0,0.6,1,1.4,2{\}}^2$ in combination with the two model family indices. Therefore, there are 50 candidate models under this setting. For each criterion, a ‘best’ model was chosen, i.e. the model with the smallest criterion's value was selected among all the candidate models.

The ‘true’ models considered here were simply ${\alpha }_p={\alpha }_q\in \{0,0.6,1,1.4,2\}$ with a index of family. After data generation from each ‘true’ model, a final model was determined based on each selection criterion. We assessed a model selection criterion as follows.

Let M be the true model, and $M^{\ast }$ be the selected model. Let C be the theoretical ROC curve in $(0,1)\times (0,1)$ space based on M, and $C_M^{\ast }$ be the estimates of C based on $M^{\ast }$. Denote an indicator function as δ. Then, the successful recovery rate $\frac{1}{B}{\delta }_{(M=M^{\ast })}$, and the mean integrated absolute deviation between $C_{M^{\ast }}$ and C, MIAE( $C_{M^{\ast }}$), are used for assessment. The simulation experiment was replicated B=500 times for each combination of scenario, N, and ${\alpha }_p={\alpha }_q$.

3.2. Results

We shall merely report results based on REML estimators, since [15] concluded that the ML estimator of the covariance is always biased. In fact, results based on ML estimator give the same conclusion.

The successful recovery rate of each model selection criterion is given in Tables 1 and 2 for the two scenarios. Most of criteria fail to find the true model, no matter which sample size (N) is used. cAIC-VB seems consistently to find the true model with at least $60\mathrm{\%}$ success when ${\alpha }_p={\alpha }_q=0$ under the first family. Under the first family and ${\alpha }_p={\alpha }_q$ taken a value of 0.6, 1, or 1.4, cAIC-GK outperforms others, despite of a low success rate never larger than $15\mathrm{\%}$. When ${\alpha }_p={\alpha }_q=2$ under the second family, EL does a better job with a success rate ranging from $30\mathrm{\%}$ to $90\mathrm{\%}$. AIC outperforms others when ${\alpha }_p={\alpha }_q=0$ under the second family. Note that about $10\mathrm{\%}$ to $15\mathrm{\%}$ calculation of the criterion cAIC-GK will not converge. Even though without convergence we can still have a value for cAIC-GK, the successful recovery rate may be underestimated. EL also has about $3\mathrm{\%}$ failure of convergence.

Table 1.

Percentages of selecting the correct model in 500 simulations for each model selection criterion in Scenario 1. (a), (c) and (e) under family = 1; (b), (d) and (f) under family = 2; (a) and (b): N = 5; (c) and (d): N = 10; (e) and (f): N = 50.

(a)
	AIC	cAIC-VB	cAIC-GK	EL
$\alpha =0$	4.8	92.0	4.0	0.0
$\alpha =0.4$	0.4	0.0	2.0	0.0
$\alpha =1$	0.0	0.0	1.6	0.0
$\alpha =1.6$	1.6	0.0	7.0	0.0
$\alpha =2$	0.4	0.0	1.6	26.4
(b)
	AIC	cAIC-VB	cAIC-GK	EL
$\alpha =0$	56.8	4.4	3.2	1.2
$\alpha =0.6$	0.8	0.0	0.0	0.0
$\alpha =1$	0.4	0.0	0.4	0.4
$\alpha =1.4$	0.8	0.0	0.0	3.6
$\alpha =2$	0.0	0.0	1.2	46.0
(c)
	AIC	cAIC-VB	cAIC-GK	EL
$\alpha =0$	5.2	98.0	2.0	0.0
$\alpha =0.6$	0.4	0.0	0.4	0.0
$\alpha =1$	2.0	0.0	4.8	0.0
$\alpha =1.4$	3.6	0.0	7.8	0.0
$\alpha =2$	0.0	0.0	0.4	22.0
(d)
	AIC	cAIC-VB	cAIC-GK	EL
$\alpha =0$	52.8	0	0.8	0.4
$\alpha =0.6$	1.2	0	0.4	0.0
$\alpha =1$	0.0	0	0.8	0.0
$\alpha =1.4$	0.0	0	0.0	2.8
$\alpha =2$	0.0	0	0.4	60.8
(e)
	AIC	cAIC-VB	cAIC-GK	EL
$\alpha =0$	15.2	100	1.2	0.0
$\alpha =0.6$	5.2	0.0	6.8	0.0
$\alpha =1$	3.6	0.0	11.2	0.0
$\alpha =1.4$	1.6	0.0	24.2	0.0
$\alpha =2$	0.0	0.0	13.2	2.4
(f)
	AIC	cAIC-VB	cAIC-GK	EL
$\alpha =0$	53.6	0.0	1.2	0.4
$\alpha =0.6$	1.6	0.0	0.4	0.0
$\alpha =1$	0.0	0.0	0.4	0.0
$\alpha =1.4$	0.0	0.0	0.0	0.0
$\alpha =2$	0.0	0.0	0.0	90.2

Table 2.

Percentages of selecting the correct model in 500 simulations for each model selection criterion in Scenario 2. (a), (c) and (e) under family = 1; (b), (d) and (f) under family = 2; (a) and (b): N = 5; (c) and (d): N = 10; (e) and (f): N = 50.

(a)
	AIC	cAIC-VB	cAIC-GK	EL
$\alpha =0$	3.6	62.6	7.2	0.4
$\alpha =0.6$	0.0	0.0	3.2	0.0
$\alpha =1$	0.4	0.0	1.8	0.0
$\alpha =1.4$	3.2	0.0	6.8	1.2
$\alpha =2$	0.4	0.0	2.8	22.8
(b)
	AIC	cAIC-VB	cAIC-GK	EL
$\alpha =0$	22.4	0.0	0.4	1.2
$\alpha =0.6$	0.0	0.0	0.8	0.0
$\alpha =1$	0.4	0.0	0.4	0.0
$\alpha =1.4$	0.0	0.0	1.0	0.4
$\alpha =2$	0.0	0.0	0.8	32.4
(c)
	AIC	cAIC-VB	cAIC-GK	EL
$\alpha =0$	6.0	60.8	4.8	0.0
$\alpha =0.6$	3.2	0.0	6.0	0.0
$\alpha =1$	1.6	0.0	4.0	0.0
$\alpha =1.4$	6.8	0.0	11.2	0.4
$\alpha =2$	0.0	0.0	1.6	12.0
(d)
	AIC	cAIC-VB	cAIC-GK	EL
$\alpha =0$	8.0	0.0	0.8	0.0
$\alpha =0.6$	0.0	0.0	1.0	0.0
$\alpha =1$	0.2	0.0	0.8	0.0
$\alpha =1.4$	0.2	0.0	4.0	1.8
$\alpha =2$	0.0	0.0	1.2	51.2
(e)
	AIC	cAIC-VB	cAIC-GK	EL
$\alpha =0$	23.2	66.4	16.0	0.0
$\alpha =0.6$	14.8	0.0	9.1	0.0
$\alpha =1$	4.0	0.0	10.8	0.0
$\alpha =1.4$	3.2	0.0	24.4	0.0
$\alpha =2$	0.0	0.0	29.6	2.6
(f)
	AIC	cAIC-VB	cAIC-GK	EL
$\alpha =0$	47.2	0.0	0.0	0.8
$\alpha =0.6$	0.0	0.0	0.0	0.0
$\alpha =1$	2.0	0.0	4.0	0.0
$\alpha =1.4$	3.4	0.0	6.8	0.0
$\alpha =2$	0.0	0.0	0.0	88.6

Then MIAE of each model selection criterion is given in Figures 1 and 2 for the two scenarios. The trend between α is quite clear: the MIAE has larger variation among models in the second family. This should be due to the second family is more flexible than the first family. Among the four criteria, EL seems to have the smallest MIAE, i.e. the ROC curve most similar to the true one for most cases. AIC and cAIC-VB are better than EL for some values of α if N = 50.

Figure 1.

MIAE of different candidate models or selected models for Scenario 1 under first (left panel) and second (right panel) families. (a) and (b): N = 5; (c) and (d): N = 10; (e) and (f): N = 50. Different colors stand for different α: 0 in cyan, 0.6 in black, 1 in red, 1.4 in green, and 2 in gray. The numbers in the parentheses on the horizontal axis represent ( ${\alpha }_p$, ${\alpha }_q$, family index).

Figure 2.

MIAE of different candidate models or selected models for Scenario 2 under first (left panel) and second (right panel) families. (a) and (b): N = 5; (c) and (d): N = 10; (e) and (f): N = 50. Different colors stand for different α: 0 in cyan, 0.6 in black, 1 in red, 1.4 in green, and 2 in gray. The numbers in the parentheses on the horizontal axis represent ( ${\alpha }_p$, ${\alpha }_q$, family index).

Closer inspection of Figures 1 and 2 reveals that, even more surprisingly, the ‘true’ models usually are not the best, and there is an ever victor outperforming all the other models. The model with ${\alpha }_p=1.4$, ${\alpha }_q=1.4$, and index $=1$ almost always has the smallest MIAE, except for $\alpha =0$ and N = 50 cases in Scenario 2.

4. Application of sROC curve to COVID-19 diagnosis

As stated in the official statistics [39], at the end of year 2020 there have been 82404103 confirmed cases of COVID-19, including 1801382 deaths, reported to World Health Organization (WHO). The outbreak was first identified in December 2019 and it developed into a global pandemic over more than 200 countries by March 2020. Symptoms of COVID-19 are highly variable and can be very mild or severe. Commonly reported symptoms usually last around 9 to 10 days after symptom onset, but people remain infectious for up to two weeks even without showing symptoms [7]. Hence, accurate and rapid diagnostic tests is important for the policy of disease control.

Although detection with real-time polymerase chain reaction (RT-PCR) has quite high accuracy, the method is too expensive and time-consuming to be widely available. In contrast, immunoassays detect a few antibodies and have been used for rapid and low-cost tests. Our bodies could make the antibodies to resistance to disease from viruses or bacteria. The antibodies are also called the immunoglobulins. The types of immunoglobulin/antibody include IgA, IgG, IgM, IgE and IgD, and different types of them fight for different diseases. We could find out whether a person is infected through a whole blood or serum test with these antibodies.

Intensive researches are emerging for the rapid and highly accurate detection, and chemiluminescence immunoassay (CLIA) is one of such assays. Other molecular and immunoassay-based techniques, including enzyme-linked immunosorbent assay (ELISA) with a long history, are referred to [37]. It has been shown that CLIA is more time-saving with competitive sensitivity, specificity and accuracy compared to ELISA in the clinical applications (e.g. [9,13]). Bastos et al. [4] have reviewed quite a few studies reporting the diagnostic accuracy of serological tests for COVID-19. Here, we focus on the sROC curve of COVID-19 diagnosis using CLIA by detecting specific IgG and IgM. There were 9 studies for IgG and 10 studies for IgM, whose nTP, nFN, nFP, and nTN can be easily found in Tables 3 and 4 of [4].

We used the one having the smallest MIAE in the simulations, i.e. ${\alpha }_p={\alpha }_q=1.4$ and index $=1$, to estimate the sROC curves for both antibodies, and the results are shown in Figures 3(d) and 4(d). This method is referred to as MIM in what follows. AIC selected ${\alpha }_p=0$, ${\alpha }_q=1.5$ and index $=2$ for CLIA (IgG), while it selected ${\alpha }_p=1$, ${\alpha }_q=1.5$ and index $=2$ for CLIA (IgM). We also compared the sROC based on the HSROC model of [29] ( ${\alpha }_p={\alpha }_q=1$, index $=2$) and the MSL model of [26] ( ${\alpha }_p={\alpha }_q=1$, index $=1$).

Figure 3.

Estimated sROC curves with the confidence region (red line) and prediction region (dashed line) for CLIA(IgG) using (a) the model selected by AIC, (b) MSL method, (c) HSROC model, and (d) the model with ${\alpha }_p={\alpha }_q=1.4$, and index $=1$, where gray circles (with the area proportional to the sample size) are the sensitivity and 1-specificity observed from individual studies and the black triangle indicates the summary sensitivity and 1-specificity.

Figure 4.

Estimated sROC curves with the confidence region (red line) and prediction region (dashed line) for CLIA(IgM) using (a) the model selected by AIC, (b) MSL method, (c) HSROC model, and (d) the model with ${\alpha }_p={\alpha }_q=1.4$, and index $=1$, where gray circles (with the area proportional to the sample size) are the sensitivity and 1-specificity observed from individual studies and the black triangle indicates the summary sensitivity and 1-specificity.

A positive IgM antibody means that the patient is in the early stage of infection, while a positive IgG antibody means that the patient is in the late stage of infection or recovery. Therefore, a higher sensitivity of IgG is expected due to the sampled subjects, which is confirmed from the overall patterns in Figures 3 and 4. In theses two problems, MIM seems to fall somewhere between the other three. As can be easily seen, MIM has a narrow confidence region for the summary sensitivity and specificity, compared to MSL and HSROC. It also has a slightly higher summary specificity and a lower summary sensitivity. This reflects that MIM puts more weights on studies around the top-left corner in plots, while MSL and HSROC were influenced by more small-sample studies.

5. Discussion and conclusion

According to the results of the successful recovery rate and MIAE, we can answer the first two key questions. AIC is clearly not acceptable here. It finds the true model with an about $20\mathrm{\%}$ to $60\mathrm{\%}$ successful rate only in very restricted situations, and in most cases its ability to find the true model is even worse than a random guessing ( $2\mathrm{\%}$ in expectation). Surprisingly, its performance does not improve when N becomes larger. The second key question cannot be completely solved by considering only the four model selection criteria. Although some of them are better for a certain α in the simulations, we never know the ‘true’ α in practice. There does not exist an overall best criteria among the four. Thus we could not find a better model merely based on these selection criteria.

We are curious about what is the MIAE if we use the ‘true’ α and family index to estimate its ROC curve. As expected, ‘true’ models are at the top of the list. The $\alpha =0$ cases in the first family are exceptions, where the ‘true’ model has a very large MIAE. We can also compare the performance of some special cases often used in terms of MIAE. MSL method with a triplet (1,1,1) and HSROC with a triplet (1,1,2) usually perform quite well and give quite similar MIAE if the true $\alpha \le 1$. Complementary logarithmic model with a triplet (1.4,1.4,2) and proportional hazard model with a triplet (2,2,1) are slightly better than the previous two models if the true $\alpha >1$.

Now we turn to the third key question, regarding a summary method that performs satisfactorily under various situations especially when N is small. According to Figures 1 and 2, the model with ${\alpha }_p={\alpha }_q=1.4$ and index $=1$ always has the smallest MIAE, except for $\alpha =0$ and N = 50 cases in Scenario 2. Since we rarely have a diagnostic meta-analysis with a sample size as large as 50, always using this model is the best strategy.

We suggest to construct the sROC curve based on the $t_{\alpha }$ model with ${\alpha }_p={\alpha }_q=1.4$ and index $=1$. It is the best in terms of MIAE, which gives the ROC curve most similar to the true one. The recommended complementary logarithmic model is close related Chu et al. [10]. However, very few meta-analyses of diagnostic studies applied the method even though several years have passed since it was proposed in 2010. A lack of software for practitioners would be a main reason. In addition, unlike the popular logistic transformation in [22,26,29], the complementary logarithmic link is asymmetric with respect to 0.5, making it more difficult to imagine the behavior of extremely large or small sensitivity and specificity after transformation. We provided our implementation based on R language in Appendix, hopefully to increase the chance of it being used.

Appendix.

We illustrate how to use our implementation based on R language through the COVID-19 data.

First of all, install and load the package mode on Comprehensive R Archive Network (CRAN), which already gave the model fitting for the second class of models. We modified their code so as to fit models of both classes using a single function reitsma.default. Our codes are in ‘func.R’, which can be read into R using source.

Next, input the dataset. Note that its column names should be in such an order: ‘TP’, ‘FN’, ‘FP’ and ‘TN’.

Below are codes for finding the model with the smallest AIC among all the ( ${\alpha }_p$, ${\alpha }_q$, family) considered. The main function reitsma.default has four essential inputs: dataset, alphasens ( ${\alpha }_p$), alphafpr ( ${\alpha }_q$), and use1stModel (TRUE or FALSE to use the first class of models).

We also implemented a function srocPlot for plotting as that in Figures 3 and 4, which takes three inputs: (1) model, (2) title, and (3) list of sensitivities and specificities. The function freq2ratio can be used to calculate sensitivity and specificity for each study.

Funding Statement

Financial support for this study was provided in part by a grant from National Science Council, Taiwan [grant number NSC102-2118-M038-002], and Ministry of Science and Technology, Taiwan [grant numbers MOST104-2314-B039-037, MOST107-2118-M-110-004-MY3 and MOST108-2628-M-008-005-MY3]. The funding agreement ensured the authors' independence in designing the study, interpreting the results, writing, and publishing the report.

Disclosure statement

No potential conflict of interest was reported by the author(s).