Exact inference for the risk ratio with an imperfect diagnostic test

SUMMARY

The risk ratio quantifies the risk of disease in a study population relative to a reference population. Standard methods of estimation and testing assume a perfect diagnostic test having sensitivity and specificity of 100%. However, this assumption typically does not hold, and this may invalidate naive estimation and testing for the risk ratio. We propose procedures that control for sensitivity and specificity of the diagnostic test, given the risks are measured by proportions, as it is in cross-sectional studies or studies with fixed follow-up times. These procedures provide an exact unconditional test and confidence interval for the true risk ratio. The methods also cover the case when sensitivity and specificity differ in the two groups (differential misclassification). The resulting test and confidence interval may be useful in epidemiological studies as well as in clinical and vaccine trials. We illustrate the method with real-life examples which demonstrate that ignoring sensitivity and specificity of the diagnostic test may lead to considerable bias in the estimated risk ratio.

INTRODUCTION

The risk ratio or relative risk (RR) quantifies the risk in a study population (say, in those exposed to a risk factor) relative to a reference population. Naive inference assumes that the applied diagnostic procedure is perfect (its sensitivity and specificity is 100%), i.e. no misclassification occurs. Unfortunately, this assumption usually does not hold, and ignoring this may result in misleading conclusions. Our aim is to present an exact unconditional test and confidence interval for RR controlling for sensitivity and specificity of the diagnostic test.

We address the situation, in which

• risks are quantified by the proportion of those having the condition (disease, recovery, etc.), which is typical in cross-sectional epidemiological studies but may also occur in other study designs;

• two independent random samples are drawn from the two populations;

• there is no classification error in the group definition (i.e. in the exposure);

• sensitivity and specificity of the diagnostic procedure are known (rather than estimated in the same or in another study).

In this case RR is defined as the ratio of two proportions, RR = p₂/p₁, where p₂ is the proportion of diseased in the study group and p₁ is that in the reference group. Many authors use the term ‘prevalence ratio’ (PR) for this measure preserving the term ‘risk ratio’ for the incidence ratio [1–3], while others call it ‘prevalence risk ratio’ (PRR) [4, 5]. In the following text we will use the term risk ratio (RR). In cross-sectional studies and in therapeutic or vaccine trials with fixed-length follow-up this is the most natural measure to compare the groups. Guidelines for vaccine studies define vaccine efficacy as 1 − RR, where the reference group is placebo. Despite this, in many studies odds ratios (OR) are calculated just because logistic regression has become a standard analysis tool readily available in most statistical software systems, although using OR instead of RR is repeatedly criticized by statisticians [1, 6–8].

While the impact of misclassification on the results of statistical analyses has been studied since the 1950s in the biomedical as well as in the social sciences [9–16], no exact test or confidence interval for the true PR has been proposed. In the next section we describe the proposed procedures, then we present two applications, finally we summarize the properties of the method. R code for the procedures is available at www2.univet.hu/users/jreiczig/RR_SeSp.

METHODS

Let us denote for population i (i = 1, 2) the true prevalence by p_i, the sensitivity by Se_i, and the specificity by Sp_i. Then the probability of a positive diagnosis (also called observed or apparent prevalence) in the ith population is p_ia = p_i·Se_i + (1 − p_i)·(1 − Sp_i). This implies that taking independent samples of sizes n₁ and n₂ from the two populations, the number of test positives x₁ and x₂ follow the binomial distribution with parameters n₁, p_1a and n₂, p_2a. Thus, the relative frequencies x₁/n₁ and x₂/n₂ are estimates of p_1a and p_2a, therefore we will denote them by and . What we will make use of in the following text is that the parameters p_1a and p_2a are in a one-to-one relationship with the true prevalences p₁ and p₂, i.e. a hypothesis about p₁ and p₂ can be mapped onto a corresponding hypothesis about p_1a and p_2a, and tested using their estimates and . The general equation, which describes the relationship between the parameters p_i and p_ia, and allows for Se₁≠Se₂ and/or Sp₁≠Sp₂ is

1

If Se₁ = Se₂ and Sp₁ = Sp₂, which can often be assumed in real-life applications, the equation simplifies to

2

where Sp denotes the common specificity; and for a perfect diagnostic test, i.e. for Se₁ = Se₂ = Sp₁ = Sp₂ = 1, it reduces to

3

Solving the equations (1)–(3) for RR, the following expressions are obtained:

4

5

6

The point estimates for RR can be obtained by replacing p_1a and p_2a by and in equations (4)–(6). Note that the parameter space for the true prevalences (p₁, p₂) is the unit square, whereas that for (p_1a, p_2a) is a rectangle within the unit square, namely [1 − Sp₁, Se₁] × [1 − Sp₂, Se₂]. Note also that the estimates (, ) form a point in the sample space (which is actually a grid of points), rather than in the parameter space. In formula

Assume now that we want to test for H₀: RR = RR₀, where RR = p₂/p₁ is the true risk ratio. This H₀ is a composite hypothesis, corresponding to a line segment in the parameter space of the true prevalences p₁ and p₂, namely the set of points in the unit square satisfying the equation p₂ = p₁RR₀. If we map this onto the parameter space of the observed binomials p_1a and p_2a, it will form another line segment. Its position depends on RR₀ as well as on the sensitivities and specificities according to equations (1)–(3), but it is always located within the rectangle [1 − Sp₁, Se₁] × [1 − Sp₂, Se₂]. Figure 1 illustrates the position of this line segment for H₀: RR = 2 depending on the sensitivities Se₁, Se₂, and specificities Sp₁, Sp₂.

Fig. 1.

Lines corresponding to the hypothesis H₀: RR = 2 in the two-dimensional space of the parameters p_1a and p_2a of the observed binomial variables. The position of the line depends on sensitivity (Se₁ and Se₂) and specificity (Sp₁ and Sp₂) of the test in the two populations.

Testing for H₀ is equivalent to testing whether the parameters p_1a and p_2a of the observed independent binomial variables are located on the line segment corresponding to H₀. As this is also a composite hypothesis, it can be tested applying the intersection-union principle [17], which means that a critical (or rejection) region for a composite H₀ can be obtained by constructing an appropriate critical region for each element of H₀, and taking the intersection of these regions. The steps of constructing this critical region follow the logic of Reiczigel et al. [18].

• (1)
  For each simple hypothesis h₀ ∈ H₀, i.e. for each point (p_1a, p_2a) on the line segment representing H₀, we construct a critical region (rejection region) C_h0 in the sample space, consisting of those points, which have probability under h₀ less than or equal to the probability of the observed point (, ). In formula

  

  where P_h0() denotes the probability of a point (or a set of points) in the sample space, given h₀ is true. C_h0 can be considered as the two-dimensional generalization of that proposed by Sterne [19] for a single binomial proportion. Let P_h0(C_h0) denote the probability of C_h0 under h₀, and let M = max{P_h0(C_h0), h₀ ∈ H₀}.
• (2)Next, for each h₀ ∈ H₀ we determine the subset S_h0 of the sample space consisting of the points with the smallest probability under h₀ so that P_h0(S_h0)⩽M but adding any further point to S_h0 would result in P_h0(S_h0) > M. Let C_H0 denote the intersection of all these subsets, i.e. . It is easy to see that C_H0 contains (, ) on its boundary.
• (3)Finally, the P value is defined as the highest probability of C_H0 under H₀ (i.e. for all simple hypotheses h₀ in H₀). In formula, the P value is equal to max{P_h0(C_H0), h₀ ∈ H₀}.

Figure 2 illustrates how the resulting critical region for H₀: RR = 2 depends on sensitivities and specificities, given the two observed prevalences are  = 0·575 (n₁ = 40) and  = 0·667 (n₂ = 36).

Fig. 2.

Critical regions in the sample space for H₀: RR = 2 with observed proportions  = 0·575 (n₁ = 40) and  = 0·667 (n₂ = 36), depending on the sensitivities and specificities. Black dots form the critical region, the black square represents the observed data. The line shows the location of H₀ in the parameter space.

Confidence intervals for the true RR can be constructed by inverting the above test. That is, lower and upper confidence limits to a given confidence level (1 − α) are defined as the smallest and largest true RR₀ not rejected by the test, i.e.

where p_RR0 denotes the P value from testing for H₀: RR = RR₀. Computationally, L and U are determined by increasing RR₀ in small steps and performing the test. Step size may depend on the required precision. In our implementation of the algorithm the default is a multiplicative increment with step size 0·001, i.e. RR₀ is increased or decreased as RR_0,next = 1·001*RR₀ or 0·999*RR₀. Figure 3 illustrates the procedure for observed proportions  = 0·575 (n₁ = 40),  = 0·667 (n₂ = 36), and Se₁ = Se₂ = 0·91, Sp₁ = Sp₂ = 0·8.

Fig. 3.

Illustration of the confidence interval construction for observed proportions  = 0·575 (n₁ = 40),  = 0·667 (n₂ = 36), and Se₁ = Se₂ = 0·91, Sp₁ = Sp₂ = 0·8. The confidence limits represent the smallest and largest true risk ratio (RR) not rejected by the test.

One-tailed testing, i.e. H₀: RR = RR₀ against H₁: RR > RR₀ (or H₁: RR < RR₀) is also possible, although there are different options to define this. Perhaps the simplest method is that one side of the line representing H₀ (i.e. the intersection of the critical region with that half-plane) is removed from the critical region. One-sided confidence intervals (CI) can be derived by inverting this one-sided test.

APPLICATIONS

Example 1

Everhart et al. [20] studied the seroprevalence of Helicobacter pylori infection in adults in the United States. The analysis was carried out stratified by age and ethnic group. The infection status was determined by an IgG ELISA assay having 91% sensitivity and 96% specificity in all groups. For illustration we now compare the group of the youngest (20–29 years) and the oldest (⩾70 years), in which the observed seroprevalence was 16·7% and 56·9%, respectively. For these groups, the ratio of observed prevalences is 56·9/16·7 = 3·41 (95% CI 3·00–3·88), whereas the correction for sensitivity and specificity results in the true PR of 4·17 (95% CI 3·58–4·96).

Example 2

Suwancharoen et al. [21] conducted a serological survey of leptospirosis in livestock in Thailand using the microscopic agglutination test (MAT) to determine serostatus of the examined animals. Five animal species were included in the study: cattle, buffaloes, pigs, sheep and goats. Infection status of each species was measured by seroprevalence, and all other species were compared to cattle as the reference group by calculating the PRs (which are same as risk ratios). In this study the MAT test was assumed to be perfect, as this test is usually regarded to be the gold standard test. However, other studies found that the sensitivity of the MAT test is far below 100% [22–25]. Cumberland et al. [22] found the sensitivity to be 30% for first acute-phase specimens, 63% for second acute-phase specimens, and 76% for convalescent specimens. At the same time specificity was 99%, 98%, and 97%, respectively. Limmathurotsakul et al. [25] estimated the sensitivity of MAT by a Bayesian analysis and found it to be 49·8%. These findings indicate the need for an adjustment of the PR estimates.

If we consider the most optimistic scenario of these, i.e. Se = 76% and Sp = 97% [22], we obtain the adjusted PRs shown in Table 1. The difference between the unadjusted and adjusted PR is far from negligible, for example in case of sheep the adjusted ratio is less than half of the unadjusted one. In case of buffaloes, adjusted and unadjusted 95% CIs do not even overlap.

Table 1.

Prevalence ratios reported in Suwancharoen et al. [21] and adjusted prevalence ratios assuming sensitivity = 0·76 and specificity = 0·97 (same in all groups), as reported by Cumberland et al. [22]

Species	N	Observed prevalence	Unadjusted PR (95% CI)	True prevalence	Adjusted PR (95% CI)
Cattle	9288	9·9%	(reference group)	9·4%	(reference group)
Buffaloes	1376	30·5%	3·08 (2·79–3·41)	37·6%	3·99 (3·45–4·60)
Pigs	1898	10·8%	1·09 (0·95–1·26)	10·7%	1·13 (0·90–1·42)
Sheep	1110	4·7%	0·47 (0·36–0·62)	2·3%	0·24 (0·06–0·49)
Goats	516	7·9%	0·80 (0·60–1·09)	6·8%	0·72 (0·37–1·17)

PR, Prevalence ratio; CI, confidence interval.

DISCUSSION

If sensitivity and/or specificity of the diagnostic test is <100%, the observed and true prevalence may differ from each other, influencing also estimation and testing of relative risk measured by the PR. We proposed an exact unconditional test and CIs for the true PR. The method can be applied even if the sensitivities and specificities differ in the two groups, for example if patients are diagnosed by different methods or some sort of differential misclassification occurs.

Taking sensitivity and specificity into account may either increase or decrease the P value compared to the one obtained without considering sensitivity and specificity. For instance, consider testing for H₀: RR = 2 with observed prevalences  = 0·48 (n₁ = 50) and  = 0·62 (n₂ = 50). Assuming Se₁ = Se₂ = Sp₁ = Sp₂ = 1 results in P = 0·0251, for Se₁ = Se₂ = 0·8 and Sp₁ = Sp₂ = 1 the P value increases to P = 0·0474, whereas for Se₁ = Se₂ = 0·6 and Sp₁ = Sp₂ = 1 a smaller P value of 0·0155 is obtained.

It may occur that the observed data contradict the given sensitivity and specificity. Let us assume, for example, that a certain diagnostic test is known to have Se = 0·8 and Sp = 1 and using this test we observe 90 positives out of 100. This observation is very unlikely even if the true prevalence is 100%, since under these conditions the observed variable has a binomial distribution with n = 100, P = 0·8, and the probability that it is ⩾90 is as low as 0·0057. The same problem may arise if the number of positives is much less than the expected minimum assuming the given sensitivity and specificity. In such cases one should consider the possibility that sensitivity and/or specificity data are incorrect.

The proposed method can be further developed in several directions. First, one can take into account misclassification also in the group definition, i.e. in the exposure. Brenner et al. [26] investigated this for the incidence ratio in a cohort study. Going further, similar methods could be worked out for models with several predictors, of which the categorical ones may also be affected by misclassification. Some results are known on correcting the OR obtained from logistic regression [27, 28] but similar results still lack for the PR.

There has been a long debate whether the PR or the OR is more appropriate to quantify the risk in the study group relative to the control group in a study design in which both of them can be calculated [1, 7, 29, 30]. Savu et al. [8] stated that RR or PR is more intuitively interpretable than the OR. In spite of this, many studies report OR estimates, even if the design permits calculation of RR or PR. Petersen & Deddens [31] emphasized that in cross-sectional studies, in particular when the disease is not rare, it is preferable to use PR instead of OR. Anyway, it is worth noting that our method can quite easily be adapted to testing the OR or the risk difference (RD), as the hypotheses H₀: OR = OR₀ and H₀: RD = RD₀ also correspond to a subset of the parameter space [0,1] × [0,1].

Another direction of potential improvement of the proposed methods is to extend them to the case when sensitivity and specificity are not taken as known but estimated from other samples, which may increase the variance of the RR estimate. This is analogous to the problem solved by Lang & Reiczigel [32] for estimating prevalence.

DECLARATION OF INTEREST

None.