A composite likelihood method for bivariate meta-analysis in diagnostic systematic reviews

Abstract

Diagnostic systematic review is a vital step in the evaluation of diagnostic technologies. In many applications, it involves pooling pairs of sensitivity and specificity of a dichotomized diagnostic test from multiple studies. We propose a composite likelihood method for bivariate meta-analysis in diagnostic systematic reviews. This method provides an alternative way to make inference on diagnostic measures such as sensitivity, specificity, likelihood ratios and diagnostic odds ratio. Its main advantages over the standard likelihood method are the avoidance of the non-convergence problem, which is non-trivial when the number of studies are relatively small, the computational simplicity and some robustness to model mis-specifications. Simulation studies show that the composite likelihood method maintains high relative efficiency compared to that of the standard likelihood method. We illustrate our method in a diagnostic review of the performance of contemporary diagnostic imaging technologies for detecting metastases in patients with melanoma.

1 Introduction

Conducting diagnostic reviews is a vital step in the evaluation of diagnostic technologies^1,2. The majority of diagnostic papers report estimates of sensitivity and specificity and represent data summaries as 2 × 2 tables based on dichotomized test results compared to the gold-standard determination of disease status³. The procedure of pooling pairs of sensitivity and specificity is not straightforward because of two important characteristics of this type of data. The first is that the estimated sensitivities and specificities are typically negatively correlated across studies⁴. One of the reasons for such a negative correlation is that the classification of disease status is typically based on a continuum of (at least partly) measurable traits. Suppose the higher trait is associated with positive result. Then higher threshold leads to lower sensitivity but higher specificity. When the threshold varies across studies, the study-specific sensitivity and specificity are negatively correlated⁵. The second important characteristic of the data is the substantial between-study heterogeneity in sensitivities and specificities^6,7,8. Such heterogeneity may arise due to differences in study population characteristics, variability of assessment, and other factors.

A simple method for conducting diagnostic reviews that has been frequently used in practice is to construct a summary receiver operating characteristic (sROC) curve from the studies using simple linear regression^9,6. However, serious limitations of this method have been pointed out by several authors^10,8,11. Specifically, the assumptions of simple linear regression are usually not met and the resulting inference may not be valid. Furthermore, the sROC approach converts each pair of sensitivity and specificity values into a single measure of accuracy, the diagnostic odds ratio, which does not distinguish the ability to detect individuals with disease from the ability to identify healthy individuals. The conation of sensitivity and specificity makes it difficult to determine the optimal use of a test and therefore diminishes the practical utility of this method in clinical practice⁴.

Two statistical methods that have been proposed for conducting diagnostic reviews have overcome the limitations of the sROC approach. One is the hierarchical summary receiver operating characteristic (HSROC) model^12,8. The other is the bivariate mixed effects models^{13,14,4,15,11}. Importantly, Harbord et al. reported that the HSROC model and the bivariate mixed effects model are very closely related, and are even identical in the absence of covariates¹⁶.

Among the bivariate mixed effects models, the bivariate general linear mixed effects model and the bivariate generalized linear mixed effects model (BGLMM) are commonly used^{13,14,4,15,11,17}. The performances of these two models have been compared by extensive simulation studies^18,19, and the conclusion is that the BGLMM is preferred due to less bias and better coverage probability performance, especially for studies with small sample sizes, or with sensitivities or specificities close to 1. However, two practical issues in the standard likelihood inference have been reported^18,19. The first is a non-convergence or non positive definite covariance matrix problem¹⁹. Such problems are caused mainly by the maximum likelihood estimate of the correlation being close to ±1, and are even more severe when the number of studies is small or moderate. The second practical issue is computational difficulty caused by a double-integral in the likelihood function. Although modern computational methods such as Laplace or adaptive Gaussian quadrature approximation are easy to implement in software such as NLMIXED in SAS (SAS Institute Inc., Cary, NC) and ADMB (Automatic Differentiation Model Builder)²⁰, these approximations may still have non-negligible approximation errors. These computational errors often result in unstable or unreproducible estimates (e.g., results sensitive to initial values)¹⁹. To our best knowledge, there is no satisfactory solution to these practical problems. More importantly, the standard likelihood inference of BGLMM relies on the bivariate normality assumption on the logit sensitivity and specificity, which may not be appropriate. One scenario is that the logit sensitivity and specificity may follow distributions with heavier tails. Another scenario is that the correlation between sensitivity and specificity may be non-homogeneous across studies. Under these situations, the inference based on the standard likelihood method may lead to biased estimates of diagnostic accuracies and their standard errors.

In this paper, we propose an alternative inference procedure for better computational performance and model robustness. The idea is to construct a composite likelihood (CL) function by using an independent working assumption between sensitivity and specificity^21,22. Such a CL has been used in longitudinal data analysis and multivariate survival data analysis to account for the correlations between observations^23,24. There are three immediate advantages of using this CL method. First, the non-convergence or non positive definite covariance matrix problem is resolved since there is no correlation parameter involved in the CL. Secondly, because the two-dimensional integration involved in the standard likelihood is substituted by one-dimensional integrals, the approximation errors are substantially reduced. Thirdly, the inference based on the CL only relies on the marginal normality of logit sensitivity and specificity. Hence the proposed method can be more robust than the standard likelihood inference to mis-specifications of the joint distribution assumption.

This article is organized as follows. In Section 2, we describe the proposed CL method. In Section 3, we conduct simulation studies to compare the CL method with the standard likelihood method where their biases, coverage probabilities and relative efficiencies are investigated. We illustrate the CL method in Section 4 with a diagnostic review of contemporary diagnostic imaging technologies for detecting metastases in patients with melanoma. We provide a brief discussion in Section 5.

2 Statistical Methodology

We consider a diagnostic review with m studies. For the ith study, denote n_i11, n_i00, n_i01, and n_i10 as the number of true positives, true negatives, false positives, and false negatives, respectively, i = 1, …, m. Let n_i1 = n_i11 + n_i10 and n_i0 = n_i01 + n_i00 be the number of diseased and healthy subjects, respectively, and Se_i and Sp_i be the study-specific sensitivity and specificity, respectively.

To account for the heterogeneity between studies and the correlation between Se_i and Sp_i, the following bivariate generalized linear mixed model (BGLMM) approach is commonly used in diagnostic reviews^15,11. The BGLMM can be formulated in two stages. Specifically, the first stage of the BGLMM is

$$n_{i11}|S{e}_i~\text{Binomial}(n_{i1},S{e}_i),n_{i00}|S{p}_i~\text{Binomial}(n_{i0},S{p}_i).$$ (1)

If Se_i and Sp_i are known, the number of true positives n_i11 and the number of true negative n_i00 are assumed to follow independent binomial distributions. This conditional independence assumption is generally reasonable because the true positives and true negatives are from two different groups of patients. At the second stage, to take the heterogeneity between studies and the correlation between Se_i and Sp_i into consideration, a random effects model is assumed,

$$g(S{e}_i)=X_i^T{\beta }_1+{\mu }_{i1},g(S{p}_i)=Z_i^T{\beta }_2+{\mu }_{i2}.$$ (2)

Here g(·) is a known link function such as a logit function, X_i and Z_i are vectors of study-level covariates, possibly overlapping, related to Se_i and Sp_i, respectively, and the intercepts (μ_i1, μ_i2) are often assumed to follow a bivariate normal distribution with mean zero and covariate matrix Σ,

$$\mathrm{\Sigma }=\left(\begin{array}{cc}\hfill {\tau }_1^2& \hfill \rho {\tau }_1{\tau }_2\\ \hfill \rho {\tau }_1{\tau }_2& \hfill {\tau }_2^2\end{array}\right),$$

where ${\tau }_1^2$ and ${\tau }_2^2$ capture the between-study heterogeneity in sensitivities and specificities, respectively, and ρ describes the correlation between the random effects Se_i and Sp_i in the transformed scale.

To keep the notation simple and make our discussion concrete, we assume X_i = Z_i = 1 and choose a logit link function. In this case, β₁ and β₂ are the respective overall sensitivity and specificity (in a logit scale). For diagnostic accuracy, there are several measures available to help clinicians in decision making. The most popular one is the pair of overall sensitivity and specificity, i.e., exp(β₁)/{1+exp(β₁)} and exp(β₂)/{1+exp(β₂)}, respectively. Alternatively, likelihood ratios, LR+ and LR−, have been suggested in the literature²⁵, where LR+ = sensitivity/(1 − specificity) = Pr(+|D)/Pr(+|D̄) = exp(β₁){1+exp(β₂)}/{1+exp(β₁)}, and LR− = (1 − sensitivity)/specificity = Pr(−|D)/Pr(−|D̄) = {1 + exp(β₂)}/[exp(β₂) {1 + exp(β₁)}]. Likelihood ratios quantify the extent to which a test result changes the probability of disease. Thus they can be used by clinicians to make decisions in treating patients, conducting further testing, or not evaluating patients further because the prevalence of the disease is low²⁵. If a single measure of diagnostic accuracy is preferred, a commonly used one is the diagnostic odds ratio (dOR), where dOR = {sensitivity/(1-sensitivity)} × {specificity/(1-specificity)} = exp(β₁ + β₂). The value of dOR ranges from zero to infinity, with a higher value indicating better discriminatory power. A value of 1 is expected for tests with no difference detected between the diseased group and the healthy group²⁶. We note that these descriptive measures are all functions of β₁ and β₂ only.

For simplicity of notation, denote ${\theta }_1=({\beta }_1,{\tau }_1^2),{\theta }_2=({\beta }_2,{\tau }_2^2)$ and θ = (θ₁, θ₂). The log likelihood function of (θ₁, θ₂, ρ) is

$$\text{log}L({\theta }_1,{\theta }_2,\rho )=\sum _{i=1}^m\text{log Pr}(n_{i00},n_{i11}|n_{i0},n_{i1})=\sum _{i=1}^m\text{log}\int \int \text{Binomial}(n_{i00}|n_{i0};S{p}_i)\times \text{Binomial}(n_{i11}|n_{i1};S{e}_i)\varphi (S{e}_i,S{p}_i;{\theta }_1,{\theta }_2,\rho )dS{e}_{i}dS{p}_i,$$ (3)

where ϕ(·, ·; θ₁, θ₂, ρ) is the bivariate logit normal distribution indexed by (θ₁; θ₂;ρ) and Binomial(·|·;·) is the binomial distribution. The integral in equation (3) does not have a closed form and has to be evaluated by numerical methods such as adaptive Gaussian quadrature²⁷. In practice, the package NLMIXED in SAS version 9.1 (SAS Institute Inc., Cary, NC) can be used to maximize the approximation to the log likelihood function in equation (3). However, the standard maximum likelihood inference (hereafter referred to as standard likelihood method or SL method) faces the computational difficulites as described in the Introduction section. These problems are due to the two-dimensional integrals in the likelihood function and the need of estimating the correlation parameter ρ¹⁹.

Now we propose an alternative inference procedure. We note that the commonly used descriptive measures of diagnostic test (e.g., overall sensitivity, specificity, diagnostic likelihood ratios and diagnostic odds ratio) are all functions of β₁ and β₂ only, and do not involve the correlation parameter ρ. In addition, the computational problems are mainly caused by the two-dimensional integrals. We propose to construct a pseudolikelihood by using an independent working assumption. Specifically, setting ρ = 0 in equation (3), we obtain the pseudolikelihood

$$\text{log}{L}_p({\theta }_1,{\theta }_2)=\text{log}{L}_1({\theta }_1)+\text{log}{L}_2({\theta }_2),$$ (4)

where

$$\text{log}{L}_1({\theta }_1)=\sum _{i=1}^m\text{log Pr}(n_{i11}|n_{i1};{\theta }_1)=\sum _{i=1}^m\{\text{log}\int \text{Binomial}(n_{i11}|n_{i1},S{e}_i)\varphi (S{e}_i;{\theta }_1)dS{e}_i\},\text{log}{L}_2({\theta }_2)=\sum _{i=1}^m\text{log Pr}(n_{i00}|n_{i0};{\theta }_2)=\sum _{i=1}^m\{\text{log}\int \text{Binomial}(n_{i00}|n_{i0},S{p}_i)\varphi (S{p}_i;{\theta }_2)dS{p}_i\},$$

and ϕ(·; θ_j) is the logit normal distribution indexed by θ_j (j = 1, 2). We note that only one-dimensional integrals are involved in the pseudolikelihood. Hence the approximation errors can be reduced. In addition, the non-convergence or non positive definite covariance matrix problem is alleviated since there is no correlation parameter involved in the pseudolikelihood. More importantly, in contrast to the bivariate normality assumption made by the standard likelihood method, the pseudolikelihood relies on the marginal normality of logit sensitivity and specificity. Hence the pseudolikelihood based inference may be more robust than the standard likelihood inference to misspecifications of the joint distribution assumption.

Such pseudolikelihood L_p(θ₁, θ₂) is a type of CL where (weighted) marginal or conditional densities are multiplied together to form the CL^28,21,29,22. Since each component of the CL function, i.e., log L_j(θ_j) (j = 1, 2), is a true log marginal likelihood, the corresponding score equation can be shown to be unbiased. Consequently, the maximum CL estimator (θ̃₁, θ̃₂) is consistent and asymptotically normal.

By a standard argument using asymptotic theories, we can show that the estimator (θ̃₁, θ̃₂) is asymptotically normal with mean zero and covariance matrix

$$\mathrm{\Sigma }=\left(\begin{array}{cc}\hfill I_{11}^{-1}\hfill & \hfill I_{11}^{-1}{I}_{12}{I}_{22}^{-1}\hfill \\ \hfill {(I_{11}^{-1}{I}_{12}{I}_{22}^{-1})}^T\hfill & \hfill I_{22}^{-1}\hfill \end{array}\right),$$

where

$$I_{jj}=E\{-\frac{{\partial }^2\text{log}{L}_j({\theta }_j)}{\partial {\theta }_j^2}\}\text{and}{I}_{12}=E\left[\right\{\frac{\partial \text{log}{L}_1({\theta }_1)}{\partial {\theta }_1}\left\}{\left\{\frac{\partial \text{log}{L}_2({\theta }_2)}{\partial {\theta }_2}\right\}}^T\right]$$

for j = 1, 2. Consequently, (θ̃₁, θ̃₂) is approximately normal with zero mean and covariance matrix Σ̂/m where Σ̂ is

$$\mathrm{\Sigma ̂}=\left(\begin{array}{cc}\hfill {Î}_{11}^{-1}\hfill & \hfill {Î}_{11}^{-1}{Î}_{12}{Î}_{22}^{-1}\hfill \\ \hfill {({Î}_{11}^{-1}{Î}_{12}{Î}_{22}^{-1})}^T\hfill & {Î}_{22}^{-1}\hfill \end{array}\right),$$ (5)

where

$${Î}_{11}=\frac{1}{m}\sum _{i=1}^m\frac{{\partial }^2\text{log Pr}(n_{i11}|n_{i1};{\theta }_1)}{\partial {\theta }_1^2},{Î}_{22}=\frac{1}{m}\sum _{i=1}^m\frac{{\partial }^2\text{log Pr}(n_{i00}|n_{i0};{\theta }_2)}{\partial {\theta }_2^2}$$

and

$${Î}_{12}=\frac{1}{m}\sum _{i=1}^m\left\{\frac{\partial \text{log Pr}(n_{i11}|n_{i1};{\theta }_1)}{\partial {\theta }_1}\right\}{\left\{\frac{\partial \text{log Pr}(n_{i00}|n_{i0};{\theta }_2)}{\partial {\theta }_2}\right\}}^T.$$

It is worth noting that the asymptotic results of the maximum CL estimates follow the standard maximum likelihood³⁰ under model mis-specifications. Specifically, the variance takes the form of a sandwich form³¹ as that in generalized estimating equation²³, which often arises when the information equality does not hold.

The maximum CL estimates can be obtained by conducting two separate univariate meta-analyses with a random effect model based on data {(n_i11, n_i1) : i = 1, …, m} or {(n_i00, n_i0) : i = 1, …, m}. Such univariate random effect model can be easily fitted in most of statistical software. The covariance matrix can be easily calculated using the above formulas, which only involve one-dimensional integrations. We note that although the correlation parameter ρis set at 0, the off-diagonal matrices in Σ̂ still properly account for the covariance between the estimated overall sensitivity and specificity. In contrast, such covariance cannot be properly considered if investigators conduct meta-analysis by univariate meta-analyses only. In other words, although both the CL method and the univariate meta-analysis provide the same and valid inference on sensitivity and specificity alone, they provide different inferences on functions of sensitivity and specificity such as diagnostic likelihood ratios and diagnostic odds ratio. The CL method can correctly account for the covariance between the estimated sensitivity and specificity whereas the univariate method cannot due to the ignored covariance between the estimated sensitivity and specificity. In Section 1 of the Supplementary Materials, we numerically demonstrate the advantage of the proposed method over the univariate analysis in estimating functions of sensitivity and specificity, where the univariate method leads to confidence intervals with incorrect coverage probabilities. We consider the CL method as a method between bivariate and univariate meta-analyses, inheriting the ability of inferring functions of overall parameters (such as diagnostic likelihood ratios and diagnostic odds ratio) with correct variance/covariance estimates while not suffering from their limitations.

3 Simulation Study

We conduct simulation studies to evaluate the finite sample performance of the CL method and compare it to that of the SL method. The data are generated from a two-stage procedure as specified by equations (1) and (2). We consider five different scenarios including settings with or without study level covariates, normal and t distributions for random effects, and logit and complementary log-log (c-log-log) link functions. Table 1 provides a description of these five different scenarios. For the scenario with study-level covariates, we consider two covariates: a binary covariate (e.g. 1 for regional cancer and 0 for distant cancer), and a continuous covariate sampled from a uniform distribution (e.g., QUADAS score with range of [1, 14]). We consider three different sizes of meta-analysis with a relatively small number (m = 10), a moderate number (m = 25), and a relatively large number (m = 50) of studies. The number of subjects in each study is randomly sampled from an application of meta-analysis for diagnostic tests of metastases, which will be described in details in Section 4. We consider two settings of sensitivity and specificity: (Se, Sp)=(0.70, 0.80) and (Se, Sp)=(0.90, 0.95), reflecting scenarios of low accuracy and high accuracy tests. Five thousand datasets are generated for each simulation setting. For each generated dataset, we apply both the CL and SL methods to obtain the estimates of sensitivity (Se), specificity (Sp), LR+ and LR−. Specifically, the CL method is implemented in R (R Development Core Team, Version 2.14.1) by using the glmmML package³². The model-based standard errors of the CL estimates are obtained by equation (5). To obtain the SL estimates, we use the adaptive Gaussian quadrature method in the SAS NLMIXED procedure (SAS Institute Inc., Cary, NC). The model-based standard errors of the SL estimates are obtained by the inverse of the negative hessian matrix of the log likelihood. Programming codes are provided in the Appendix.

Table 1.

Configurations of five different simulation scenarios: ℳ₁ ~ ℳ₅.

Model	Covariates	Link function	Random effects distribution	Correlation structure
ℳ1	No	logit	bivariate normal	fixed
ℳ2	No	logit	bivraiate t	fixed
ℳ3	No	c-log-log	bivariate t	fixed
ℳ4	No	logit	bivariate normal	mixture
ℳ5	Yes	logit	bivariate normal	fixed

In Table 2 we show the empirical bias (BIAS), empirical standard error (SE), average of model-based standard error estimates (MBSE) and the coverage probability (CP) of the 95% confidence intervals for the estimates when data is generated from ℳ₁ with (Se, Sp)=(0.90, 0.95). When the number of studies is moderate (m = 25) or relatively large (m = 50), both the CL and SL methods provide approximately unbiased estimates (relative bias < 3%), MBSEs close to true SEs, and CPs close to nominal levels, with the CP from the CL method being slightly better than that from the SL method. When the number of studies is relatively small (m = 10), both methods have approximately 10% relative bias and under-estimated standard errors in the estimation of LR+ and LR−. The CP of the CL method is still acceptable with the range of [87.8%, 90.7%] and is not influenced by the degree of correlation. In contrast, the CP of the SL method deteriorates as the magnitude of correlation increases, and has a range of [73.0%, 90.3%]. When the SL method is applied, around 5 ~ 11% of the simulated replicates have the non-convergence problem (i.e., number of iterations reaches the default number of 200 iterations while the relative gradient convergence criterion < 1e − 10 is not satisfied), or non-positive definite covariance matrix problem. The fitting results under non-convergence were excluded when summarizing simulation results from replications. Simulation results with lower sensitivity, i.e., (Se, Sp)=(0.70, 0.80), are summarized in Table S1 of the Supplemental Material. Similar findings are obtained with the CL method achieving better CPs compared to the SL method.

Table 2.

Summary of 5, 000 simulations with data generated from ℳ₁: bias (BIAS), standard errors (SE), model-based standard errors (MBSE) and coverage probabilities (CP) of estimates. True values of model parameters are: sensitivity = 90.0, specificity = 95.0, LR+ = 18.00, and LR− = 0.11. All entries for sensitivity and specificity are multiplied by 100.

			SL method†				CL method*
m	ρ		BIAS	SE	MBSE	CP(%)	BIAS	SE	MBSE	CP(%)
10	0.0	Se	−0.5	3.5	3.2	90.3	−0.5	3.6	3.2	89.9
		Sp	−0.3	2.0	1.8	90.1	−0.3	1.9	1.8	90.9
		LR+	1.82	16.49	8.61	87.2	1.63	9.84	7.92	88.3
		LR−	0.01	0.04	0.03	90.3	0.01	0.04	0.04	90.5
	−0.3	Se	−0.5	3.6	3.3	81.5	−0.5	3.5	3.2	89.9
		Sp	−0.3	2.0	1.8	81.5	−0.3	2.0	1.8	89.9
		LR+	1.88	12.23	8.38	79.2	1.46	9.06	7.41	87.8
		LR−	0.01	0.04	0.03	81.7	0.01	0.04	0.03	90.7
	−0.6	Se	−0.5	3.6	3.2	75.2	−0.6	3.5	3.2	89.5
		Sp	−0.3	2.0	1.8	74.6	−0.3	2.0	1.8	90.0
		LR+	1.68	8.71	7.72	73.0	1.51	8.96	7.15	88.4
		LR−	0.01	0.04	0.03	75.4	0.01	0.04	0.03	90.5
25	0.0	Se	−0.2	2.2	2.1	93.1	−0.2	2.2	2.1	93.4
		Sp	−0.1	1.2	1.2	93.2	−0.1	1.2	1.2	92.6
		LR+	0.56	4.87	4.51	91.7	0.64	4.91	4.51	91.9
		LR−	0.00	0.02	0.02	92.9	0.00	0.02	0.02	93.6
	−0.3	Se	−0.2	2.2	2.1	92.8	−0.1	2.2	2.1	93.0
		Sp	−0.1	1.2	1.2	92.6	−0.1	1.2	1.2	93.6
		LR+	0.54	4.70	4.40	91.2	0.51	4.66	4.35	92.2
		LR−	0.00	0.02	0.02	92.6	0.00	0.02	0.02	93.0
	−0.6	Se	−0.2	2.2	2.1	90.6	−0.2	2.2	2.1	92.6
		Sp	−0.1	1.2	1.2	91.0	−0.1	1.2	1.2	93.8
		LR+	0.57	4.61	4.30	89.2	0.47	4.36	4.24	92.5
		LR−	0.00	0.02	0.02	90.8	0.00	0.02	0.02	92.7
50	0.0	Se	−0.2	1.6	1.5	93.9	−0.1	1.5	1.5	94.0
		Sp	−0.1	0.9	0.8	94.4	−0.1	0.8	0.8	94.4
		LR+	0.23	3.20	3.10	93.8	0.30	3.21	3.11	93.9
		LR−	0.00	0.02	0.02	93.9	0.00	0.02	0.02	94.0
	−0.3	Se	−0.1	1.6	1.5	94.1	−0.1	1.6	1.5	93.7
		Sp	−0.1	0.8	0.8	94.7	−0.1	0.8	0.8	94.5
		LR+	0.23	3.13	3.03	94.0	0.30	3.11	3.04	93.8
		LR−	0.00	0.02	0.02	93.9	0.00	0.02	0.02	93.5
	−0.6	Se	−0.1	1.6	1.5	94.0	−0.1	1.5	1.5	94.7
		Sp	−0.1	0.8	0.8	94.5	−0.1	0.8	0.8	94.7
		LR+	0.22	3.04	2.95	93.8	0.23	2.99	2.95	93.5
		LR−	0.00	0.02	0.02	93.8	0.00	0.02	0.02	94.7

SL method^†: standard maximum likelihood method based on the BGLMM

CL method^*: proposed composite likelihood method

One interesting finding seen from Table 2 is that the empirical SE of the estimates from the SL method does not decrease as the correlation increases (in magnitude) and the empirical SE from the SL method is very close to that from the CL method. This suggests that the efficiency gain in the joint analysis of sensitivity and specificity when conducting diagnostic reviews may not be large. Such an observation has been previously reported by Simel and Bossuyt²⁵. To investigate the relative efficiency of the CL method compared to the SL method, we plot the relative efficiency (defined by the sample variance of the SL estimates, divided by that of the CL estimates) against the correlation ρ, as shown in Figure 1. All four panels in Figure 1 show that the relative efficiency of the CL method is at least 90%. Note that when the number of studies is 25, the CL method tends to be more efficient than the SL method; whereas when the number of studies is 50, the SL method tends to be more efficient.

Figure 1.

Relative efficiency (RE) of maximum composite likelihood estimator of sensitivity (upper left panel), specificity (upper right panel), LR+ (lower left panel) and LR− (lower right panel) compared to the maximum likelihood estimator under various values of correlation ρ.

In Table 3 we evaluate the robustness of the CL and SL methods under various model mis-specifications when the number of studies is 50. Under the setting ℳ₂ where the study-specific sensitivity and specificity (in logit) are generated from bivariate t distribution with 4 degree of freedom, both methods produce satisfactory inferences with unbiased estimates and CP close to the nominal level. Interestingly, under the setting ℳ₃ where c-log-log (instead of logit) link is used, the CP of the CL method remains satisfactory, whereas the CP of the SL method deteriorates rapidly as the magnitude of correlation increases. This suggests that the SL method (which requires to estimate the correlation) is not sensitive to the heavy tail distribution under logit link, but is very sensitive under the asymmetric c-log-log link function. In contrast, the CL method is quite robust to the heavy tail distribution under both link functions. Under the setting ℳ₄, we consider the heterogeneity in correlation where the correlation takes one value in half of the studies and takes a different value in the remaining half. Under this setting, the likelihood of the SL method is mis-specified, whereas the likelihood of the CL method is not because the CL method does not assume homogeneous correlation across studies. As expected, the CL method leads to unbiased estimates with CP close to the nominal level, whereas the SL method underestimates the standard errors and has poor CPs (range of CP: [80.7%, 83.8%]).

Table 3.

Estimates of bias (BIAS), standard errors (SE), model-based standard errors (MBSE) and coverage probabilities (CP) of estimates in 5, 000 simulations, with study size = 50 for each simulation. The data are generated from models ℳ₂ ~ ℳ₄, respectively. True values of model parameters are: sensitivity = 70.0, specificity = 80.0, LR+ = 3.50, and LR− = 0.38. All entries for sensitivity and specificity are multiplied by 100.

			SL method†				CL method*
	ρ		BIAS	SE	MBSE	CP(%)	BIAS	SE	MBSE	CP(%)
ℳ2	0.0	Se	−0.2	4.0	3.9	93.8	−0.2	4.0	3.9	94.1
		Sp	−0.3	3.1	3.0	94.2	−0.3	3.1	3.1	94.8
		LR+	0.02	0.59	0.57	93.6	0.01	0.57	0.57	93.9
		LR−	0.01	0.05	0.05	94.0	0.01	0.05	0.05	94.7
	−0.3	Se	−0.3	4.1	3.9	93.7	−0.2	4.0	3.9	94.0
		Sp	−0.3	3.1	3.0	94.0	−0.4	3.1	3.1	94.5
		LR+	0.01	0.53	0.52	93.5	−0.01	0.52	0.52	93.1
		LR−	0.01	0.05	0.05	93.9	0.01	0.05	0.05	94.5
	−0.6	Se	−0.2	4.0	4.0	94.0	−0.4	4.0	4.0	94.0
		Sp	−0.3	3.1	3.1	94.2	−0.2	3.1	3.0	93.9
		LR+	0.02	0.46	0.46	93.5	0.01	0.47	0.46	93.8
		LR−	0.00	0.04	0.04	94.1	0.01	0.04	0.04	94.5
ℳ3	0.0	Se	−0.9	6.3	6.1	90.9	−0.7	6.3	6.4	93.3
		Sp	−1.2	5.7	5.4	89.0	−1.1	5.6	5.7	93.9
		LR+	0.02	1.15	1.03	86.5	0.06	1.14	0.78	78.7
		LR−	0.02	0.09	0.08	92.5	0.02	0.09	0.08	91.5
	−0.3	Se	−0.7	6.1	4.4	62.6	−0.7	6.4	6.4	93.6
		Sp	−1.1	6.0	3.8	61.0	−1.1	5.6	5.7	93.6
		LR+	0.07	1.18	0.66	55.9	0.02	1.04	0.72	81.8
		LR−	0.01	0.08	0.06	63.2	0.01	0.08	0.07	91.5
	−0.6	Se	−0.8	6.5	3.6	49.4	−0.7	6.4	6.4	93.5
		Sp	−1.0	5.9	3.0	49.5	−1.0	5.7	5.7	93.2
		LR+	0.04	1.00	0.51	45.7	0.03	0.98	0.67	81.4
		LR−	0.01	0.07	0.05	49.7	0.01	0.07	0.06	92.2
ℳ4	(0.0, −0.3)	Se	−0.3	4.6	3.1	81.0	0.0	3.2	3.2	94.1
		Sp	−0.2	3.4	2.4	81.7	−0.1	2.5	2.4	93.7
		LR+	0.04	0.64	0.44	82.0	0.03	0.45	0.44	93.6
		LR−	0.01	0.06	0.04	80.9	0.00	0.04	0.04	93.8
	(0.0, −0.6)	Se	−0.3	4.6	3.1	81.1	−0.1	3.2	3.2	94.3
		Sp	−0.2	3.3	2.4	83.4	−0.1	2.5	2.4	94.2
		LR+	0.04	0.58	0.42	83.8	0.02	0.42	0.42	94.1
		LR−	0.00	0.06	0.04	81.0	0.00	0.04	0.04	94.7
	(−0.3, −0.6)	Se	−0.3	4.6	3.1	81.1	−0.1	3.2	3.2	94.0
		Sp	−0.2	3.4	2.4	81.8	−0.1	2.5	2.4	94.0
		LR+	0.04	0.57	0.39	82.1	0.02	0.40	0.40	93.6
		LR−	0.00	0.05	0.04	80.7	0.00	0.04	0.04	94.0

SL method^†: standard maximum likelihood method based on the BGLMM

CL method^*: proposed composite likelihood method

Table 4 summarizes the simulation results when study-level covariates are available and the number of studies is 30 or 50 (i.e., setting ℳ₅). In this case, the regression coefficients are parameters of interest. Similar to the findings from Table 2, both methods provide unbiased estimates and CPs close to the nominal level. The CL method has up to 23.4% of efficiency loss. In this setting, the SL method is recommended if there is no convergence problem, where the CL method can be considered as an alternative.

Table 4.

Summary of 5, 000 simulations with data generated from ℳ₅: bias (BIAS), standard errors (SE), model-based standard errors (MBSE), coverage probabilities (CP) and relative efficiency (RE) of estimates of regression coefficients. True values of coefficients are: β₁₀ = 1.71, β₁₁ = −1.27, β₁₂ = 0.00, β₂₀ = 1.91, β₂₁ = 1.26, and β₂₂ = 0.00.

			SL method†					CL method*
m	ρ		BIAS	SE	MBSE	CP(%)	RE(%)	BIAS	SE	MBSE	CP(%)	RE%
30	0.0	β10	0.00	0.32	0.30	92.1	100.0	0.00	0.37	0.33	92.6	78.8
		β11	0.00	0.45	0.41	91.4	100.0	−0.01	0.51	0.46	91.2	76.8
		β12	0.00	0.38	0.36	92.2	100.0	0.00	0.44	0.40	91.6	75.8
		β20	0.00	0.30	0.28	92.2	100.0	0.00	0.31	0.28	92.1	99.3
		β21	0.01	0.46	0.43	91.9	100.0	0.00	0.46	0.43	92.4	101.3
		β22	0.00	0.40	0.37	92.0	100.0	0.01	0.39	0.37	92.3	102.0
	−0.3	β10	0.00	0.32	0.30	91.8	100.0	0.00	0.37	0.33	91.9	77.9
		β11	0.00	0.45	0.41	91.4	100.0	0.00	0.50	0.46	91.3	80.5
		β12	0.00	0.38	0.36	92.1	100.0	−0.01	0.44	0.40	91.7	74.7
		β20	0.00	0.31	0.28	92.1	100.0	0.00	0.31	0.28	92.3	100.0
		β21	0.01	0.46	0.42	91.8	100.0	0.00	0.46	0.43	92.2	103.1
		β22	0.00	0.40	0.37	92.5	100.0	0.00	0.40	0.37	92.5	99.5
	−0.6	β10	0.00	0.32	0.29	91.0	100.0	0.00	0.37	0.33	92.1	75.3
		β11	0.00	0.44	0.41	90.4	100.0	−0.02	0.50	0.46	91.6	78.2
		β12	0.00	0.38	0.35	91.3	100.0	0.00	0.44	0.40	91.8	73.6
		β20	0.00	0.31	0.28	90.5	100.0	0.00	0.31	0.28	92.0	98.7
		β21	0.01	0.46	0.42	90.7	100.0	0.00	0.46	0.43	91.8	100.9
		β22	0.00	0.39	0.37	91.3	100.0	0.00	0.40	0.37	92.6	97.0
50	0.0	β10	0.00	0.24	0.23	93.5	100.0	0.00	0.28	0.26	93.2	78.1
		β11	−0.01	0.33	0.32	93.6	100.0	−0.01	0.38	0.36	93.0	77.5
		β12	0.01	0.29	0.28	93.4	100.0	−0.01	0.33	0.31	93.1	76.3
		β20	−0.01	0.23	0.22	93.6	100.0	0.00	0.24	0.22	93.3	94.1
		β21	0.01	0.35	0.33	93.6	100.0	0.00	0.35	0.33	93.3	98.3
		β22	0.00	0.30	0.29	93.2	100.0	−0.01	0.30	0.29	94.3	102.7
	−0.3	β10	0.00	0.24	0.23	93.5	100.0	0.00	0.27	0.26	93.7	79.8
		β11	−0.01	0.33	0.32	93.6	100.0	0.00	0.37	0.36	93.6	79.2
		β12	0.00	0.29	0.28	93.5	100.0	−0.01	0.33	0.31	93.5	79.1
		β20	0.00	0.23	0.22	93.8	100.0	0.00	0.23	0.22	93.8	99.1
		β21	0.01	0.35	0.33	93.4	100.0	0.00	0.35	0.33	93.0	100.0
		β22	0.00	0.30	0.29	93.5	100.0	0.00	0.30	0.29	93.5	100.0
	−0.6	β10	0.00	0.24	0.23	93.3	100.0	0.00	0.28	0.26	93.5	76.8
		β11	0.00	0.33	0.32	93.5	100.0	−0.01	0.38	0.36	93.2	76.6
		β12	0.00	0.29	0.28	93.4	100.0	−0.01	0.33	0.31	93.3	77.1
		β20	0.00	0.23	0.22	93.6	100.0	0.01	0.23	0.22	93.2	99.1
		β21	0.01	0.35	0.33	93.2	100.0	0.01	0.35	0.33	93.6	101.7
		β22	0.00	0.30	0.28	93.5	100.0	0.00	0.30	0.29	93.6	99.3

SL method^†: standard maximum likelihood method based on the BGLMM

CL method^*: proposed composite likelihood method

In summary, simulation studies suggest that the CL method outperforms the SL method when the number of studies is relatively small, and when the correlation is heterogeneous across studies. The CL method is also more robust than the SL method under various model mis-specification settings considered. When study-level covariates are available, the CL method can be less efficient than the SL method, which should be used as an alternative when the SL method encounters convergence problem.

4 Applications

Melanoma is a malignant tumor of melanocytes, the cells that produce the skin pigment, melanin. Less common than other types of skin cancer, melanoma is much more dangerous when not found early, and causes the majority (75%) of deaths related to skin cancer³³. Although sentinel lymph node biopsy is the acknowledged gold standard for the pathological staging of melanoma in patients whose lymph nodes are clinically negative, imaging technology has also been used in some clinical settings for preoperative lymph node assessment and postoperative surveillance. Imaging technology can be used for the early detection of melanoma metastasis, and provides a cost-effective surveillance approach³⁴. Currently, the diagnostic imaging technologies most commonly used for melanoma include ultrasonography (US), computed tomography (CT), positron emission tomography (PET) and a combination of the latter two technologies (PET-CT). It is critical to evaluate the performance of these contemporary diagnostic imaging technologies when used for patients with melanoma.

Xing et al.³⁵ conducted a diagnostic review based on 98 published studies of 10, 528 patients carried out between January 1, 1990 and June 30, 2009. The number of studies for each diagnostic imaging technology and each type of cancer (i.e., regional and distant) are cross-tabulated in Table S3 in the Supplemental Material. To apply the composite likelihood method, we fit a sequence of nested meta-regression models where the smallest model includes a variable for stage of cancer (i.e., 1 for regional and 0 for distant), and three dummy variables for types of imaging modalities with PET-CT as the reference group. Larger models were considered sequentially by adding interaction terms between types of cancer and imaging modalities. To select models fitted by CL method, modifications of Akaike's information criterion (AIC) and Bayesian information criterion (BIC) can be used^36,37. Specifically, the composite likelihood version of AIC is defined as $\mathrm{C}\mathrm{L}-\mathrm{A}\mathrm{I}\mathrm{C}=-2\text{log}{L}_c+2d_s^{*}$, where $d_s^{*}=\text{trace}\left\{Ĵ{(Ĥ)}^{-1}\right\}$, Ĵ is the estimated covariance matrix of ∂L_c(θ₁, θ₂)/∂(θ₁, θ₂) evaluated at (θ̃₁, θ̃₂) and Ĥ is −∂² log L_c(θ₁, θ₂)/∂(θ₁, θ₂)² evaluated at (θ̃₁, θ̃₂). It is easy to show that $d_s^{*}$ equals to the number of parameters in the model. The composite likelihood version of BIC is defined as $\mathrm{C}\mathrm{L}-\mathrm{B}\mathrm{I}\mathrm{C}=-2\text{log}{L}_c+d_s^{*}\text{log}(m)+2\gamma d_s^{*}\text{log}(P)$, where P is the number of model parameters, and γ is a tuning parameter and is taken as 0 when P is relatively small compared to the number of studies as suggested in Gao and Song³⁷. The results of fitting the sequence of nested models are summarized in Table 5.

Table 5.

Model selection using the CL-AIC and CL-BIC when analyzing the data in Xing et al. (2011).

Model	$d_s^{*}$	−2logCL	CL-AIC	CL-BIC
baseline	12	1050	1074	1106
+I(Regional)*I(US)	14	1050	1078	1115
+I(Regional)*I(CT)	14	1048	1076	1113
+I(Regional)*I(PET)	14	1050	1078	1114
+I(Regional)*I(US)+I(Regional)*I(CT)	16	1048	1080	1122
+I(Regional)*I(US)+I(Regional)*I(PET)	16	1050	1082	1124
+I(Regional)*I(CT)+I(Regional)*I(PET)	16	1048	1080	1122
+I(Regional)*I(US)+I(Regional)*I(CT)+I(Regional)*I(PET)	18	1048	1084	1131

baseline: meta-regression model with study-level covariates of I(Regional) + I(US) + I(CT) + I(PET).

Both CL-AIC and CL-BIC suggest the use of the baseline model with 12 model parameters. However, model assumptions such as normality and equal variance need to be investigated. QQ-plots of the logit sensitivity and specificity are produced in Figure S2 of the supplementary material, and Shapiro-Wilk test³⁸ for normality is conducted. Both the plots and the test suggest the normality assumption is appropriate. To evaluate the equal variance assumption across different subgroups (i.e., different diagnostic technology and types of cancer), we apply Bartlett's test³⁹. This test suggests that homogeneity assumption is appropriate for logit specificity, but not for logit sensitivity (p < 0.001). To study the sensitivity of the results from the baseline model (as recommended by both CL-AIC and CL-BIC) on the equal variance assumption, we conduct a separate analysis for each subgroups, and find that the results from the subgroup analyses are generally similar to those from the baseline meta-regression model.

Figure 2 presents the results from the subgroup analyses with the CL method on the overall diagnostic sensitivity, specificity and diagnostic odds ratio, and the associated 95% confidence intervals for the four diagnostic imaging modalities. The results from the SL method are displayed as the dashed lines in Figure 1 for comparison. We note that for the subgroup with regional cancer, there are only 5 studies with PET-CT, and 3 studies with CT. The inference based on the SL method is sensitive to the choice of initial values, and singular covariance matrix is encountered. Thus, the confidence intervals for these two subgroups are not provided. In general, the results from the CL method are consistent with those from the SL method. For the surveillance of regional lymph node metastasis, US has the highest sensitivity (64%; 95% CI=40% to 82%), specificity (98%; 95% CI=95% to 99%) and diagnostic odds ratio (77.3; 95% CI=22.8 to 262.0) among all four imaging modalities. For the surveillance of distant lymph node metastasis, PET-CT has the highest sensitivity (85%; 95% CI=68% to 94%), specificity (94%; 95% CI = 86% to 97%) and diagnostic odds ratio (83.8; 95% CI=23.2 to 303.1). The results from the CL method suggest that US is a more accurate imaging modality for diagnosing regional lymph node involvement and PET-CT is the preferred imaging modality to diagnose distant metastasis. On the other hand, the confidence intervals for sensitivity, specificity and diagnostic odds ratio overlap substantially. Hence more studies are required to draw more definitive conclusions in regard to these imaging modalities and their roles for detection of metastasis.

Figure 2.

Estimated accuracies and 95% confidence intervals of four diagnostic imaging technologies when identifying the pathological stage of distant and regional lymph nodes in patients with melanoma. Solid segments: results from the CL method; dashed segments: results from the SL method.

Since the estimated bivariate summary measures are often correlated, separate confidence intervals that do not account for such a correlation may be misleading¹⁶. Figure 3 presents the summary points and 95% confidence region for sensitivity versus 1 minus specificity without stratification on stages of metastasis (i.e. regional or distant) using the CL method. Specifically, the parametric representation of the boundary of the elliptical Wald-type confidence region for sensitivity and specificity (in logit scale) is obtained as⁴⁰,

$$S_1={Ŝ}_1+s_{S_1}\sqrt{(2f_{2,n-2;\alpha })}\text{cos}\varphi \text{and}{C}_1={Ĉ}_1+s_{C_1}\sqrt{(2f_{2,n-2;\alpha })}\text{cos}(\varphi +\text{arccos}r),$$

where s_S1 and s_C1 are the estimated standard errors of Ŝ₁ and Ĉ₁, r is the estimate of their correlation, ϕ runs from 0 to 2π, and f_2,n−2;α is the upper 100α% point of the F distribution with degrees of freedom 2 and n − 2, and n is the number of studies. Similar as shown in Figure 1, the wide ranges of those confidence regions suggest that more studies are needed to increase the precision of those estimates, and to reach definitive conclusions comparing those imaging modalities.

Figure 3.

Summary points and 95% confidence regions of sensitivity versus 1-minus-specificity for four diagnostic imaging modalities. Filled circle: summary point; solid line: boundary of 95% confidence region for the summary point.

5 Discussion

In this paper, we proposed a composite likelihood method for the bivariate analysis of sensitivity and specificity in diagnostic reviews. The main advantages of this method over the standard likelihood method are the avoidance of the non-convergence problem, computational simplicity and some robustness to model mis-specifications such as distributions with heavy tails and non-homogeneous correlations across studies. Furthermore, our simulation studies suggested that the composite likelihood method maintains a high relative efficiency compared to the standard likelihood method. Other bivariate random effects models have been considered in the literature. For example, Sarmanov beta-binomial models have been studied by Chu et al.⁴¹ and Chen et al.⁴². The composite likelihood with working independence assumption can be used to deal with the limitations of constrained correlation parameter space^43,44. It is worth mentioning that the composite likelihood approach is analogous to two other approaches under specific applications: the generalized estimating equation (GEE) by Liang and Zeger²³ for the inference of marginal models for longitudinal data with an independence working correlation structure, and the pseudo-partial likelihood function by Lin²⁴ for the inference of marginal models for multivariate survival data when failure times within the same unit are treated as independent random variables. We believe our method can be a useful alternative to the standard likelihood method for bivariate analysis in diagnostic reviews.