Testing and Quantifying Site-Level Variability in Diagnostic Sensitivity of an Anchor Variable

Abstract

In multi-site clinical research, diagnostic assessments can vary across sites even when standardized criteria and instruments are used, leading to inconsistent disease classification. This issue is examined in settings with an anchor variable that confidently identifies disease when positive but provides no information when negative. A random effects model is introduced for site-specific sensitivity, along with likelihood-based methods for estimation and hypothesis testing. The approach addresses two objectives: testing whether diagnostic sensitivity varies across sites, and quantifying the magnitude of such variability. Validation data is incorporated to establish parameter identifiability. Laplace approximation and the Expectation-Maximization (EM) algorithm are further engaged to address the computational challenge caused by an intractable integral in the likelihood function. Likelihood ratio and score tests are constructed to account for the boundary constraint that arises when the null hypothesis places the variance component at zero. Simulation studies demonstrate the good performance in finite samples, with accurate parameter estimates and appropriate test size and power. Application to a multi-site Huntington disease cohort for diagnosing mild cognitive impairment reveals differences in diagnostic sensitivity across sites, with tests providing strong evidence of heterogeneity. This framework offers a principled approach for testing and quantifying site-level variability in diagnostic sensitivity, supporting more consistent inference in multi-site studies.

1 |. INTRODUCTION

Mild cognitive impairment (MCI) is one of the earliest indicators of Huntington disease, a devastating neurodegenerative disorder that leads to progressive motor and cognitive decline^1,2. Detecting MCI in its early stages offers the potential to intervene before irreversible damage occurs. However, diagnosing MCI is not straightforward. Even with standardized cognitive tests, site-level differences in diagnostic practices—how these tests are administered, interpreted, and scored—can lead to inconsistent diagnoses across multi-site studies. Such inconsistency can result in patients being misclassified, potentially excluding them from clinical trials or delaying access to appropriate interventions³.

To address the diagnostic inconsistency, we propose a statistical framework that accounts for site-level variability while leveraging partial diagnostic information. Let $Y$ denote a binary variable indicating true MCI status: $Y=1$ if the patient has MCI, and $Y=0$ otherwise. Let $\mathbf{\text{X}}$ denote a vector of covariates—such as age, genetic information, or other relevant biomarkers—that are predictive of MCI status. Determining true MCI status for all patients requires comprehensive cognitive assessment and clinical judgment, which is resource-intensive. However, for patients whose cognitive test scores meet a strict diagnostic criterion, we can definitively confirm MCI status. We define an anchor variable $S$—a binary diagnostic indicator observed for every patient—where $S=1$ confirms the patient has MCI based on meeting this strict criterion, and $S=0$ means the patient’s MCI status remains uncertain and requires additional clinical evaluation to determine. Under this setting, where MCI status is confirmed for some patients but remains uncertain for others (known as “positive-only” data^4,5), the anchor variable has perfect positive predictive value: $\text{Pr}(Y=1\mid S=1)=1$, but provides no information about $Y$ when $S=0$. We refer to patients with $S=Y=1$ as anchor-positive cases, and those with $S=0$ as unlabeled patients. We assume $\mathbf{\text{X}}$ is fully observed for all patients. The observed data consist of $(\mathbf{\text{X}},S)$ for all patients, with $Y$ observed when $S=1$, supplemented by a validation subset of patients with $S=0$ for whom $Y$ is also observed. We refer to the data $(\mathbf{\text{X}},S)$ observed for all patients as the main data, and the subset where $Y$ is additionally observed for patients with $S=0$ as the validation data.

The anchor variable framework we use builds on the work of Zhang et al. (2020)⁶, who introduced a model for predicting $Y$ using $\mathbf{\text{X}}$, where the sensitivity of the anchor variable, $\text{Pr}(S=1\mid Y=1)$, was assumed constant across all observations. Specifically, they assumed $\text{Pr}(S=1\mid Y=1,\mathbf{\text{X}})=\text{Pr}(S=1\mid Y=1)=c$, where $c\in (0,1)$ is a fixed constant, and used a logistic regression model to relate $Y$ and $\mathbf{\text{X}}$ :

$$\text{logit}\text{pr}\left(Y=1∣\mathbf{\text{X}},\mathit{\beta }\right)={\mathbf{\text{X}}}^{\text{T}}\mathit{\beta }.$$

While this model provides a foundation for leveraging partial diagnostic information, it does not account for site-level variability in diagnostic practices. In multi-site studies, differences in clinician judgment, testing protocols, or patient populations can lead to systematic variation in how effectively sites identify true MCI cases, meaning the sensitivity $\text{Pr}(S=1\mid Y=1)$ may differ across sites rather than remaining constant. However, allowing sensitivity to vary across sites introduces identifiability challenges, as the additional site-specific parameters may not be uniquely determined without additional information.

In this paper, we extend the model of Zhang et al. (2020)⁶ by allowing the sensitivity of the anchor variable, $\text{Pr}(S=1\mid Y=1)$, to vary across sites through a random effect structure that captures systematic differences in diagnostic practices. Like Zhang et al. (2020)⁶, we use validation data consisting of patients with $S=0$ for whom true MCI status is observed. However, allowing sensitivity to vary by site introduces additional parameters and establishing identifiability conditions beyond those needed for the constant sensitivity case. We show that under our model structure with validation data, all parameters, including those governing site-to-site variability, remain identifiable. Our approach has two objectives: to test whether site-level variability exists in diagnostic sensitivity, and to quantify the magnitude of such variability. By explicitly modeling these site-specific differences, our framework can detect when diagnostic practices are inconsistent across a multi-site study, alerting researchers to potential problems in data quality or the need for improved standardization of diagnostic protocols.

The rest of this paper is organized as follows. Section 2 formulates the model, establishes identifiability, and develops the likelihood function, which involves an intractable integral over random effects and partially observed MCI status. Section 3 addresses this computational challenge by developing a Laplace approximation and an Expectation-Maximization (EM) algorithm to obtain the maximum likelihood estimator (MLE) of the model parameters, then uses MLE and its inference results to construct test statistics for assessing whether site-level variability exists. We evaluate our approach through simulation studies (Section 4) and an application to a multi-site study of MCI in Huntington disease (Section 5). Section 6 concludes with a discussion of potential methodological extensions and directions for future research. Technical details are provided in the Supplementary Material.

2 |. MODELING AND MAXIMUM LIKELIHOOD ESTIMATOR

Let $i=1,\dots ,I$ denote sites and $j=1,\dots ,n_{ij}$ denote patients within site $i$. As described in Section 1, we observe main data consisting of $\left({\mathbf{\text{X}}}_{ij},S_{ij}\right)$ for all patients, where ${\mathbf{\text{X}}}_{ij}$ is a vector of fully observed covariates and $S_{ij}$ is the anchor variable, and validation data consisting of true MCI status $Y_{ij}$ for a subset of patients with $S_{ij}=0$. Note that for patients with $S_{ij}=1$ in the main data, we have $Y_{ij}=1$ by the definition of the anchor variable.

We model the relationship between MCI status and covariates using a logistic regression:

$$\text{logit}\text{pr}\left(Y_{ij}=1\mid {\mathbf{\text{X}}}_{ij}\right)={\mathbf{\text{X}}}_{ij}^{\text{T}}\mathit{\beta }\text{,}$$ (1)

where ${\mathbf{\text{X}}}_{ij}$ contains patient-specific covariates—such as age, genetic information, or other relevant biomarkers—that are related to MCI status. We do not include site indicators in ${\mathbf{\text{X}}}_{ij}$ because site membership should not directly affect the probability of having MCI.

To account for site-level variation in diagnostic sensitivity, we introduce a random effects model for the anchor variable:

$$\text{logit}\text{pr}\left(S_{ij}=1\mid Y_{ij}=1\right)=a+b_i,b_i~N\left(0,\tau \right),$$ (2)

where $a$ is the fixed intercept, $b_i$ is the site-specific random effect, and $\tau$ quantifies the magnitude of site-to-site variability in diagnostic sensitivity. For notational simplicity, we use $P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)$ to denote $\text{pr}\left(Y_{ij}=1\mid {\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)$ and let $f_{\mathbf{\text{X}}}(\cdot )$ denote the probability density function of ${\mathbf{\text{X}}}_{ij}$.

Our objectives are to test whether site-level variability exists and to quantify its magnitude. Testing whether $\tau =0$ addresses whether different sites exhibit varying sensitivities in diagnosing MCI. A non-zero value of $\tau$ indicates that site-specific factors influence the likelihood of a positive diagnosis. We test the hypotheses:

$$H_0:\tau =0\text{vs}{H}_1:\tau >0.$$

Under $H_0$, diagnostic sensitivity is constant across all sites, while $H_1$ indicates systematic differences. We use the likelihood ratio test (LRT) and the asymptotically equivalent score test. Constructing these test statistics requires the likelihood and score functions, which we develop in the following subsections. The computational methods needed to evaluate these functions and the details of the tests are presented in Section 3.

2.1 |. Likelihood for Main Data

We derive the likelihood for the main data $\left({\mathbf{\text{X}}}_{ij},S_{ij}\right)$ at site $i$. We start with the joint probability density function

$$f\left({\mathbf{\text{X}}}_{ij},S_{ij}\right)={\left\{\text{pr}\left(S_{ij}=1\mid Y_{ij}=1\right)\text{pr}\left(Y_{ij}=1\mid {\mathbf{\text{X}}}_{ij}\right)f_X\left({\mathbf{\text{X}}}_{ij}\right)\right\}}^{S_{ij}}\times {\left[\left\{1-\text{pr}\left(S_{ij}=1\mid Y_{ij}=1\right)\text{pr}\left(Y_{ij}=1\mid {\mathbf{\text{X}}}_{ij}\right)\right\}{f}_X\left({\mathbf{\text{X}}}_{ij}\right)\right]}^{1-S_{ij}}.$$

Treating the random effect $b_i$ as if it were observed, the full data likelihood for $\left({\mathbf{\text{X}}}_{ij},S_{ij}\right),j=1,\dots ,n_i$, is

$$L_i^F\left(\mathit{\beta },a,\tau \right)=\prod _{j=1}^{n_i}\left[{\left\{\text{expit}\left(a+b_i\right)P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)\right\}}^{S_{ij}}{\left\{1-\text{expit}\left(a+b_i\right)P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)\right\}}^{1-S_{ij}}{f}_X\left({\mathbf{\text{X}}}_{ij}\right)\right].$$

Since $b_i$ is actually unobserved, we obtain the observed data likelihood by integrating $L_i^F(\mathit{\beta },a,\tau )$ over the distribution of $b_i$. The resulting observed data likelihood is the likelihood for the main data:

$$L_i\left(\mathit{\beta },a,\tau \right)=\prod _{j=1}^{n_i}\left\{f_X\left({\mathbf{\text{X}}}_{ij}\right)P{\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)}^{S_{ij}}\right\}\times \int \prod _{j=1}^{n_i}\left[\text{expit}{\left(a+b_i\right)}^{S_{ij}}{\left\{1-\text{expit}\left(a+b_i\right)P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)\right\}}^{1-S_{ij}}\right]\varphi \left(b_i\mid 0,\tau \right)d{b}_i,$$

where $\varphi \left(b_i\mid 0,\tau \right)$ is the density function of $\text{Normal}(0,\tau )$. The corresponding loglikelihood is

$$l_i(\mathit{\beta },a,\tau )=\sum _{j=1}^{n_i}\left\{\text{log}{f}_X\left({\mathbf{\text{X}}}_{ij}\right)+S_{ij}\text{log}P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)\right\}+\text{log}\int \prod _{j=1}^{n_i}\left[\text{expit}{\left(a+b_i\right)}^{S_{ij}}{\left\{1-\text{expit}\left(a+b_i\right)P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)\right\}}^{1-S_{ij}}\right]\varphi \left(b_i\mid 0,\tau \right)d{b}_i.$$

Detailed derivations are provided in Section S1 of the Supplementary Material.

2.2 |. Combined Likelihood for Main and Validation Data

We construct a validation subset by randomly sampling additional patients with $S_{ij}=0$ and observing their true MCI status $Y_{ij}$. Let $m_i$ denote the number of validation observations at site $i$. We now derive the likelihood for both the main data $\left\{\left({\mathbf{\text{X}}}_{ij},S_{ij}\right):i=1,\dots ,I,j=1,\dots ,n_i\right\}$ and the validation data $\left\{\left({\mathbf{\text{X}}}_{ij},S_{ij}=0,Y_{ij}\right):i=1,\dots ,I,j=n_i+1,\dots ,n_i+m_i\right\}$.

The joint density of $({\mathbf{\text{X}}}_{ij},S_{ij},Y_{ij})$ is

$$f\left({\mathbf{\text{X}}}_{ij},S_{ij},Y_{ij}\right)=\left\{I\left(j\le n_i\right)\text{pr}{\left(S_{ij}=1\mid {\mathbf{\text{X}}}_{ij}\right)}^{S_{ij}\text{pr}{\left(S_{ij}=1\mid {\mathbf{\text{X}}}_{ij}\right)}^{1-S_{ij}}}+I\left(j>n_i\right)\text{pr}\left(S_{ij}=0\mid Y_{ij}=y_{ij}\right)\text{pr}\left(Y_{ij}=y_{ij}\mid {\mathbf{\text{X}}}_{ij}\right)\right\}f\left({\mathbf{\text{X}}}_{ij}\right).$$

Treating the random effect $b_i$ as if it were observed, the full data likelihood combining the main data and validation data is

$$L_i^F(\mathit{\beta },a,\tau )=\prod _{j=1}^{n_i+m_i}{f}_X\left({\mathbf{\text{X}}}_{ij}\right)\text{expit}{\left(a+b_i\right)}^{S_{ij}I\left(j\le n_i\right)}{\left\{1-\text{expit}\left(a+b_i\right)\right\}}^{Y_{ij}I\left(j>n_i\right)}\times P{\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)}^{S_{ij}I\left(j\le n_i\right)+Y_{ij}I\left(j>n_i\right)}{\left\{1-P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)\right\}}^{\left(1-Y_{ij}\right)I\left(j>n_i\right)}\times {\left\{1-\text{expit}\left(a+b_i\right)P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)\right\}}^{\left(1-S_{ij}\right)I\left(j\le n_i\right)},$$ (4)

Since $b_i$ is actually unobserved, we obtain the observed data likelihood by integrating over the distribution of $b_i$. This observed data likelihood is the likelihood for the main and validation data combined. The corresponding loglikelihood for site $i$ is

$$l_i(\mathit{\beta },a,\tau )=\sum _{j=1}^{n_i+m_i}\left[\text{log}{f}_X\left({\mathbf{\text{X}}}_{ij}\right)+\left\{S_{ij}I\left(j\le n_i\right)+Y_{ij}I\left(j>n_i\right)\right\}\text{log}P\left({\mathit{\text{X}}}_{ij},\mathit{\beta }\right)+\left(1-Y_{ij}\right)I\left(j>n_i\right)\text{log}\left\{1-P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)\right\}\right]+\text{log}\int \prod _{j=1}^{n_i+m_i}\text{expit}{\left(a+b_i\right)}^{S_{ij}I\left(j\le n_i\right)}{\left\{1-\text{expit}\left(a+b_i\right)\right\}}^{Y_{ij}I\left(j>n_i\right)}\times {\left\{1-\text{expit}\left(a+b_i\right)P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)\right\}}^{\left(1-S_{ij}\right)I\left(j\le n_i\right)}\varphi \left(b_i\mid 0,\tau \right)d{b}_i.$$

Detailed derivations are provided in Section S2 of the Supplementary Material. We also provided identifiability of the model parameters in Section S3 in the Supplementary Material.

2.3 |. Maximum Likelihood Estimator

With identifiability established in Section S3, we now develop the maximum likelihood estimator for $\mathit{\theta }$, defined as $\mathit{\theta }\equiv {\left({\mathit{\beta }}^{\text{T}},a,\tau \right)}^{\text{T}}$, based on the combined likelihood for the main data and the validation data from Section 2.2. This likelihood involves an integral over the random effects $b_i$ that does not have a closed form, making direct maximization computationally challenging. To facilitate estimation, we reformulate the problem using a missing data framework that treats the unobserved $Y_{ij}$ values (for patients with $S_{ij}=0$ not in the validation data) as missing data.

We introduce an indicator variable $R_{ij}$ where $R_{ij}=1$ if $Y_{ij}$ is observed and $R_{ij}=0$ if $Y_{ij}$ is unobserved. Under this formulation, the observed data can be represented as $\left({\mathbf{\text{X}}}_{ij},S_{ij},R_{ij},R_{ij}{Y}_{ij}\right)$, where $Y_{ij}$ is only available when $R_{ij}=1$. This structure satisfies the missing at random (MAR) assumption because the observation mechanism depends only on observed quantities: for patients with $S_{ij}=1$ in the main data, MCI status is definitively known so $R_{ij}=1$, while for patients with $S_{ij}=0$ in the main data, whether $Y_{ij}$ is observed depends only on selection into the validation data at rate $N_v/\left(N_v+N_m\right)$, independent of $b_i,{\mathbf{\text{X}}}_{ij}$, or the unobserved $Y_{ij}$.

Noting that the joint density of $(Y_{ij},S_{ij},{\mathbf{\text{X}}}_{ij},b_i,R_{ij})$ is

$$f_{Y,S,\mathbf{\text{X}},B,R}\left(y_{ij},s_{ij},{\mathbf{\text{x}}}_{ij},b_i,r_{ij}\right)={\left\{f_{Y,S,\mathbf{\text{X}},B,R}\left(y_{ij},s_{ij},{\mathbf{\text{x}}}_{ij},b_i,r_{ij}=1\right)\right\}}^{r_{ij}}\times {\left\{f_{Y,S,\mathbf{\text{X}},B,R}\left(y_{ij},s_{ij},{\mathbf{\text{x}}}_{ij},b_i,r_{ij}=0\right)\right\}}^{1-r_{ij}},$$

the loglikelihood for site $i$ is

$$l_i(\mathit{\theta })=\sum _{j=1}^{n_i+m_i}{r}_{ij}\text{log}\left\{\frac{\text{exp}\left(y_{ij}{\mathit{\text{x}}}_{ij}^T\mathit{\beta }\right)}{1+\text{exp}\left({\mathit{\text{x}}}_{ij}^T\mathit{\beta }\right)}\right\}+\text{log}\int \prod _{j=1}^{n_i+m_i}\left({\left[y_{ij}\frac{\text{exp}\left\{s_{ij}\left(a+b_i\right)\right\}}{1+\text{exp}\left(a+b_i\right)}+\left(1-y_{ij}\right)\left(1-s_{ij}\right)\right]}^{r_{ij}}\left.\times {\left[\frac{\text{exp}\left({\mathbf{\text{x}}}_{ij}^{\text{T}}\mathit{\beta }\right)}{1+\text{exp}\left({\mathbf{\text{x}}}_{ij}^{\text{T}}\mathit{\beta }\right)}\frac{\text{exp}\left\{s_{ij}\left(a+b_i\right)\right\}}{1+\text{exp}\left(a+b_i\right)}+\frac{1-s_{ij}}{1+\text{exp}\left({\mathbf{\text{x}}}_{ij}^{\text{T}}\mathit{\beta }\right)}\right]}^{1-r_{ij}}\right)\varphi \left(b_i\mid 0,\tau \right)d{b}_i+C,\right.$$

where $C$ is a constant that does not depend on the parameters. We obtain the maximum likelihood estimator $\widehat{\mathit{\theta }}$ by maximizing $\sum _{i=1}^I{l}_i(\mathit{\theta })$ with respect to $\mathit{\theta }$. The following theorem characterizes the asymptotic distribution of $\widehat{\mathit{\theta }}$.

Theorem 1. Assume $\tau >0$. The MLE $\widehat{\mathit{\theta }}$ satisfies

$${\left(N_m+N_v\right)}^{\frac{1}{2}}\left(\widehat{\mathit{\theta }}-\mathit{\theta }\right)=-{\left[E\left\{\frac{\partial {\mathbf{\text{S}}}_{ij}\left(\mathit{\theta }\right)}{\partial {\mathit{\theta }}^{\text{T}}}\right\}\right]}^{-1}\left\{{\left(N_m+N_v\right)}^{-1}\sum _{i=1}^I\sum _{j=1}^{n_i+m_i}{\mathbf{\text{S}}}_{ij}\left(\mathit{\theta }\right)\right\}+o_p\left(1\right),$$

where ${\mathbf{\text{S}}}_{ij}(\mathit{\theta })$ is the score function. Therefore,

$${\left(N_m+N_v\right)}^{1/2}\left(\widehat{\mathit{\theta }}-\mathit{\theta }\right)\to N\left(0,{\left[E\left\{\frac{\partial {\mathbf{\text{S}}}_{ij}\left(\mathit{\theta }\right)}{\partial {\mathit{\theta }}^{\text{T}}}\right\}\right]}^{-1}\text{var}\left\{{\mathbf{\text{S}}}_{ij}\left(\mathit{\theta }\right)\right\}{\left[E\left\{\frac{\partial {\mathbf{\text{S}}}_{ij}\left(\mathit{\theta }\right)}{\partial {\mathit{\theta }}^{\text{T}}}\right\}\right]}^{-1^{\text{T}}}\right),$$

in distribution.

This theorem provides the theoretical foundation for quantifying site-level variability: the estimated variance component $\widehat{\tau }$ directly measures the magnitude of variability in diagnostic sensitivity across sites, and its asymptotic distribution allows the construction of confidence intervals. Specific form of the score function and more details of the proof are provided in Section S4 of the Supplementary Material.

3 |. COMPUTATION AND VARIANCE COMPONENT TESTING

While we derived the maximum likelihood estimator of $\mathit{\theta }$ in Section 2.3, we cannot obtain the MLE using conventional optimization procedures. This is because the likelihood involves an intractable integral over the random effects $b_i$, and it also contains the treatment of the missing MCI status $Y_{ij}$ in the main data. This section shows how to bypass these computational challenges and address our two objectives: estimating $\tau$ and testing whether site-level variability exists.

We develop two computational approaches. The Laplace approximation (Section 3.1) approximates the intractable integral to obtain tractable expressions for the likelihood. The Expectation-Maximization (EM) algorithm (Section 3.2) handles the issue caused by the missing MCI status. With these methods in hand, we construct the likelihood ratio test (LRT) and score test in Section 3.3.

3.1 |. Laplace Approximation

The likelihood for the main and validation data developed in Section 2.2 involves an intractable integral over the random effects $b_i$. We use the Laplace approximation to obtain a tractable expression for this likelihood. This approximation is used for both estimating $\tau$ via the EM algorithm (Section 3.2) and constructing test statistics (Section 3.3).

The likelihood we seek to approximate is $E\left\{L_i^F(\mathit{\theta })\right\}$ where $L_i^F(\mathit{\theta })$ is defined as in (4), where the expectation is taken over the distribution of $b_i$. For notational convenience, we substitute $a_i=a+b_i$ where $b_i~\text{Normal}(0,\tau )$; note that $a_i$ is not a parameter but simply represents the site-specific sensitivity $a+b_i$. We have

$$\begin{gathered} E\left\{L_i^F(\mathit{\theta })\right\}=\int \text{exp}\left\{l_i^F(\mathit{\theta })\right\}\varphi \left(b_i\mid 0,\tau \right)d{b}_i \\ =(2\pi \tau {)}^{-\frac{1}{2}}\int \text{exp}\left\{l_i^F\left(\mathit{\theta }\right)-\frac{b_i^2}{2\tau }\right\}d{b}_i, \end{gathered}$$ (5)

where $l_i^F(\mathit{\theta })=\text{log}{L}_i^F(\mathit{\theta })$. We apply Laplace approximation by expanding $l_i^F(\mathit{\theta })$ around $b_i=0$ using a second-order Taylor expansion. We define $L_{i0}^F={\left.L_i^F(\mathit{\theta })\right|}_{b_i=0},l_{i0}^F={\left.l_i^F(\mathit{\theta })\right|}_{b_i=0},l_{i0}^{F^{\prime }}={\left.\frac{\partial l_i^F(\mathit{\theta })}{\partial a_i}\right|}_{b_i=0}$, and $l_{i0}^{F″}={\left.\frac{{\partial }^2{l}_i^F(\mathit{\theta })}{\partial a_i^2}\right|}_{b_i=0}$. The Taylor expansion gives

$$l_i^F\left(\mathit{\theta }\right)\approx l_{i0}^F+l_{i0}^{F^{\prime }}{b}_i+\frac{1}{2}{l}_{i0}^{F^{″}}{b}_i^2.$$

Substituting this approximation into (5), we obtain

$$\begin{gathered} E\left\{L_i^F(\mathit{\theta })\right\}=(2\pi \tau {)}^{-\frac{1}{2}}\int \text{exp}\left\{l_i^F(\mathit{\theta })-\frac{b_i^2}{2\tau }\right\}d{b}_i \\ \approx (2\pi \tau {)}^{-\frac{1}{2}}{e}^{l_{i0}^F}\int \text{exp}\left\{l_{i0}^{F^{\prime }}{b}_i-\frac{1}{2}\left(\frac{1}{\tau }-l_{i0}^{F^{″}}\right)b_i^2\right\}d{b}_i \\ =(2\pi \tau {)}^{-\frac{1}{2}}{e}^{l_{i0}^F}\text{exp}\left\{\frac{\tau {\left(l_{i0}^{F^{\prime }}\right)}^2}{2\left(1-\tau l_{i0}^{F^{″}}\right)}\right\}\int \text{exp}\left[-\frac{1}{2}\left(\frac{1}{\tau }-l_{i0}^{F^{″}}\right)\left\{b_i-\frac{l_{i0}^{F^{\prime }}}{1/\tau -l_{i0}^{F^{″}}}\right\}\right]d{b}_i \\ ={\left(1-\tau l_{i0}^{F^{″}}\right)}^{-\frac{1}{2}}{L}_{i0}^F\text{exp}\left\{\frac{\tau {\left(l_{i0}^{F^{\prime }}\right)}^2}{2\left(1-\tau l_{i0}^{F^{″}}\right)}\right\}\equiv L_{iA}\left(\mathit{\theta }\right), \end{gathered}$$

where $e^{l_{i0}^F}=\text{exp}\left(\text{log}{L}_{i0}^F\right)=L_{i0}^F$ and the last equality uses the fact that the integrand is the kernel of a normal distribution. We denote this approximate likelihood as $L_{iA}(\mathit{\theta })$, where the subscript $A$ indicates approximation.

The corresponding approximate loglikelihood is

$$l_{iA}\left(\mathit{\theta }\right)=-\frac{1}{2}\text{log}\left(1-\tau l_{i0}^{F^{″}}\right)+l_{i0}^F+\frac{\tau {\left(l_{i0}^{F^{\prime }}\right)}^2}{2\left(1-\tau l_{i0}^{F^{″}}\right)},$$ (6)

where the detailed forms of $l_{i0}^F,l_{i0}^{F^{\prime }}$, and $l_{i0}^{F^{″}}$ are provided in Section S5 of the Supplementary Material.

This approximate loglikelihood provides the quantities needed for both estimation and testing. The score function is

$$S_{\tau A}\left(\mathit{\theta }\right)=\sum _{i=1}^I\frac{\partial l_{iA}\left(\mathit{\theta }\right)}{\partial \tau }=\frac{1}{2}\sum _{i=1}^I\left\{\frac{l_{i0}^{F^{″}}}{1-\tau l_{i0}^{F^{″}}}+\frac{{\left(l_{i0}^{F^{\prime }}\right)}^2}{{\left(1-\tau l_{i0}^{F^{″}}\right)}^2}\right\}.$$ (7)

Under $H_0:\tau =0$, this simplifies to

$$S_{\tau A}\left(\mathit{\theta }\right)=\frac{1}{2}\sum _{i=1}^I\left\{l_{i0}^{F^{″}}+{\left(l_{i0}^{F^{\prime }}\right)}^2\right\}.$$

The Fisher information is approximated by

$$\begin{gathered} -\frac{1}{I}\sum _{i=1}^I\frac{{\partial }^2{l}_{iA}(\mathit{\theta })}{\partial {\tau }^2}=-\frac{1}{I}\frac{d{S}_{\tau A}(\mathit{\theta })}{d\tau } \\ =-\frac{1}{I}\sum _{i=1}^I\frac{1}{2}\left\{{\left(\frac{l_{i0}^{F^{″}}}{1-\tau l_{i0}^{F^{″}}}\right)}^2+\frac{2{\left(l_{i0}^{F^{\prime }}\right)}^2{l}_{i0}^{F^{″}}}{{\left(1-\tau l_{i0}^{F^{″}}\right)}^3}\right\}\approx I_{\tau }\left(\mathit{\theta }\right). \end{gathered}$$

These quantities will be used in Section 3.2 for the EM algorithm and in Section 3.3 for constructing test statistics.

3.2 |. EM Algorithm Implementation

The likelihood for the main data and the validation data is not easily maximized directly using standard optimization procedures because it involves missing $Y_{ij}$ values for patients with $S_{ij}=0$ not in the validation data. We use the EM algorithm to operationalize the maximization by treating these unobserved $Y_{ij}$ values as missing data.

To describe the EM algorithm, we introduce notation for tracking which $Y_{ij}$ values are observed or missing. Let $W$ denote all $Y_{ij}$ values (both observed and missing), $W^O$ denote the observed $Y_{ij}$ values, and $W^{+}$ denote the missing $Y_{ij}$ values. The $W$ notation differs from the indicator variable $R_{ij}$ used in Section 2: while $R_{ij}$ indicates whether each $Y_{ij}$ is observed, the $W$ notation explicitly represents the values themselves, which is more convenient for the EM derivations.

The EM algorithm requires specifying a likelihood as if all $Y_{ij}$ values and random effects $b_i$ were observed. We denote the likelihood under such a hypothetical scenario as $L^H(\mathit{\theta })$:

$$L^H(\mathit{\theta })=\prod _{j=1}^{n_i+m_i}{f}_X\left({\mathbf{\text{X}}}_{ij}\right){\left\{\text{expit}\left(a+b_i\right)P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)\right\}}^{S_{ij}{Y}_{ij}}{\left[\left\{1-\text{expit}\left(a+b_i\right)\right\}P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)\right]}^{\left(1-S_{ij}\right)Y_{ij}}\times {\left\{1-P\left({\mathbf{\text{X}}}_{ij},\mathit{\beta }\right)\right\}}^{\left(1-S_{ij}\right)\left(1-Y_{ij}\right)}.$$ (8)

(8) differs from $L_i^F(\mathit{\theta })$ in (4): (8) includes all $Y_{ij}$ values (even the missing ones for patients with $S_{ij}=0$ in the main data), while (4) includes only the observed $Y_{ij}$ values.

Since the random effects $b_i$ are unobserved in (8), we apply the Laplace approximation from Section 3.1. We define $L_{i0}^H={\left.L_i^H(\mathit{\theta })\right|}_{b_i=0},l_{i0}^H={\left.l_i^H(\mathit{\theta })\right|}_{b_i=0},l_{i0}^{H^{\prime }}={\left.\frac{\partial l_i^H(\mathit{\theta })}{\partial a_i}\right|}_{b_i=0}$, and $l_{i0}^{H^{″}}={\left.\frac{{\partial }^2{l}_i^H(\mathit{\theta })}{\partial a_i^2}\right|}_{b_i=0}$, where $l_i^H(\mathit{\theta })=\text{log}{L}_i^H(\mathit{\theta })$. From the procedure in Section 3.1, we obtain

$$E\left\{L_i^H(\mathit{\theta })\right\}\approx L_{iA}^W(\mathit{\theta })={\left(1-\tau l_{i0}^{H^{″}}\right)}^{-\frac{1}{2}}{L}_{i0}^H\text{exp}\left\{\frac{\tau {\left(l_{i0}^{H^{\prime }}\right)}^2}{2\left(1-\tau l_{i0}^{H^{″}}\right)}\right\},$$

where we denote $L_{iA}^W(\mathit{\theta })$ as the approximate full-$Y$ likelihood, which is an approximation of the likelihood obtained when all $Y_{ij}$ values are observed.

The corresponding approximate loglikelihood is

$$l_{iA}^W(\mathit{\theta })=-\frac{1}{2}\text{log}\left(1-\tau l_{i0}^{H^{″}}\right)+l_{i0}^H+\frac{\tau {\left(l_{i0}^{H^{\prime }}\right)}^2}{2\left(1-\tau l_{i0}^{H^{″}}\right)},$$

where the detailed forms of $l_{i0}^H,l_{i0}^{H^{\prime }}$, and $l_{i0}^{H^{″}}$ are provided in Section S6 of the Supplementary Material.

Remark 1. The approximate likelihoods $L_{iA}(\mathit{\theta })$ from Section 3.1 and $L_{iA}^W(\mathit{\theta })$ serve different purposes. The likelihood $L_{iA}(\mathit{\theta })$ is based on the observed data (only truly observed $Y_{ij}$ values), while $L_{iA}^W(\mathit{\theta })$ is based on the complete $Y$ data (as if all $Y_{ij}$ values were observed). The relationship between them is $L_{iA}\left(\mathit{\theta },W^o\right)=\int L_{iA}^W\left(\mathit{\theta },W^o,W^{+}\right)d\mu \left(W^{+}\right)$, where $W^o$ denotes observed $Y$ values and $W^{+}$ denotes missing $Y$ values. The EM algorithm maximizes $L_{iA}\left(\mathit{\theta },W^o\right)$ by iteratively computing $E\left\{\text{log}{L}_{iA}^W\left(\mathit{\theta },W^o,W^{+}\right)\mid W^o,{\widehat{\mathit{\theta }}}_{(k-1)}\right\}$ and maximizing with respect to $\mathit{\theta }$.

The EM algorithm proceeds as in Algorithm 1.

The parameters $\mathit{\theta }$ may be estimated sequentially or simultaneously. Sections S7 and S8 of the Supplementary Material provide the EM procedures for both approaches.

3.3 |. Likelihood Ratio Test and Score Test Statistics

With the Laplace approximation and EM algorithm in hand, we now construct the likelihood ratio test (LRT) and score test statistics for testing $H_0:\tau =0$ versus $H_1:\tau >0$. We denote the MLE under $H_0$ as ${\widehat{\mathit{\theta }}}_{H_0}$ and the MLE under $H_1$ as $\widehat{\mathit{\theta }}$. The LRT statistic is

$$T_l\equiv -2l\left({\widehat{\mathit{\theta }}}_{H_0}\right)+2l\left(\widehat{\mathit{\theta }}\right).$$

Because the null hypothesis places $\tau$ on the boundary of the parameter space, the LRT statistic does not follow the standard ${\chi }^2$ distribution. Instead, $T_l$ follows asymptotically a mixture distribution $0.5{\chi }_1^2+0.5{\chi }_0^2$ under $H_0$^7,8, where ${\chi }_0^2$ is a degenerate distribution with $\text{pr}\left(T_l=0\right)=0.5$.

Algorithm 1.

EM Algorithm

E-step: At the $k$-th iteration, compute the expected complete data loglikelihood

$Q\left(\mathit{\theta }\mid {\widehat{\mathit{\theta }}}_{(k-1)}\right)=E\left\{\mathtt{\text{log}}{L}_A^W(\mathit{\theta })\mid W^O,{\widehat{\mathit{\theta }}}_{(k-1)}\right\},$

where ${\widehat{\mathit{\theta }}}_{(0)}$ is an initial value.

M-step: Obtain the updated estimate by

${\widehat{\mathit{\theta }}}_{(k)}={\text{argmax}}_{\mathit{\theta }}Q\left(\mathit{\theta }\mid {\widehat{\mathit{\theta }}}_{(k-1)}\right)$

Repeat until convergence: $‖{\widehat{\mathit{\theta }}}_{(k)}-{\widehat{\mathit{\theta }}}_{(k-1)}‖\le ϵ.$

To determine the critical value at significance level $\alpha$, note that the upper tail probability comes solely from the ${\chi }_1^2$ component. We have $\text{pr}\left(T_l>c_{\alpha }\right)=0.5\text{pr}\left({\chi }_1^2>c_{\alpha }\right)=\alpha$, which implies $\text{pr}\left({\chi }_1^2>c_{\alpha }\right)=2\alpha$. Therefore, we reject $H_0:\tau =0$ if $T_l\ge {\chi }_{2\alpha ,1}^2$, where ${\chi }_{2\alpha ,1}^2$ is the ($1-2\alpha$)-th quantile of ${\chi }_1^2$, equivalently the ($1-\alpha$)-th quantile of the mixture distribution. The corresponding $p$-value is $\text{pr}\left({\chi }_1^2\ge T_l\right)/2$.

The score test provides an asymptotically equivalent alternative. Let $\mathbf{\text{S}}\left({\widehat{\mathit{\theta }}}_{H_0}\right)=\sum _{i=1}^I\sum _{j=1}^{n_i+m_i}{\mathbf{\text{S}}}_{ij}\left({\widehat{\mathit{\theta }}}_{H_0}\right)$ denote the score function evaluated under $H_0$, where $\mathbf{\text{S}}(\mathit{\theta })={\left({\mathbf{\text{S}}}_{\mathit{\beta }}(\mathit{\theta }{)}^{\text{T}},S_a(\mathit{\theta }),S_{\tau }(\mathit{\theta })\right)}^{\text{T}}$. We partition the Fisher information matrix according to ${\left({\mathit{\beta }}^{\text{T}},a\right)}^{\text{T}}$ and $\tau$ as

$$\mathbf{\text{I}}(\mathit{\theta })=\left(\begin{array}{ll}{\mathbf{\text{I}}}_{11}& {\mathbf{\text{I}}}_{12}\\ {\mathbf{\text{I}}}_{21}& I_{22}\end{array}\right),$$

where

$${\mathbf{\text{I}}}_{11}=\left(\begin{array}{cc}E\left\{{\mathbf{\text{S}}}_{\mathit{\beta }}(\mathit{\theta }){\mathbf{\text{S}}}_{\mathit{\beta }}(\mathit{\theta }{)}^{\text{T}}\right\}& E\left\{{\mathbf{\text{S}}}_{\mathit{\beta }}(\mathit{\theta })S_a(\mathit{\theta })\right\}\\ E\left\{S_a(\mathit{\theta }){\mathbf{\text{S}}}_{\mathit{\beta }}(\mathit{\theta })\right\}& E\left\{S_a(\mathit{\theta }{)}^2\right\}\end{array}\right),$$

$${\mathbf{\text{I}}}_{12}={\left(E{\left\{S_{\tau }(\mathit{\theta }){\mathbf{\text{S}}}_{\mathit{\beta }}(\mathit{\theta })\right\}}^{\text{T}},E\left\{S_{\tau }(\mathit{\theta })S_a(\mathit{\theta })\right\}\right)}^{\text{T}},$$

$$I_{22}=E\left\{S_{\tau }(\mathit{\theta }{)}^2\right\}.$$

Let $I_{\tau }=I_{22}-{\mathbf{\text{I}}}_{21}{\mathbf{\text{I}}}_{11}^{-1}{\mathbf{\text{I}}}_{12}$ denote the conditional Fisher information for $\tau$ evaluated at ${\widehat{\mathit{\theta }}}_{H_0}$. The score test statistic is

$$T_s\equiv I_{\tau }^{-1/2}{S}_{\tau }\left({\widehat{\mathit{\theta }}}_{H_0}\right),$$

which follows a standard normal distribution under $H_0$. We reject $H_0$ if $T_s>z_{\alpha }$ in a one-sided test at significance level $\alpha$.

Explicit forms of $T_l$ and derivations of the asymptotic distributions are provided in Section S9 of the Supplementary Material.

4 |. SIMULATIONS

We assess the finite sample performance of our proposed method through simulation studies that address three key questions: Does the method provide accurate parameter estimates? Do the LRT and score tests maintain appropriate size and power? How does performance depend on sample size and number of sites?

We generate data from the logistic model

$$\text{logit}\text{pr}\left(Y_{ij}=1\mid X_{ij}\right)={\beta }_0+{\beta }_1{X}_{ij},$$

where $X_{ij}~N(1,1)$ and $\mathit{\beta }={\left({\beta }_0,{\beta }_1\right)}^{\text{T}}=(1,-5{)}^{\text{T}}$, yielding a phenotype prevalence of approximately 22%. The anchor variables are generated from

$$\text{logit}\text{pr}\left(S_{ij}=1\mid Y_{ij}=1\right)=a+b_i,$$

where $b_i~N(0,\tau )$. All results across simulations are based on 200 Monte Carlo samples. We carry out the three simulations below.

• Simulation 1: $a=2,\tau =0.02$, and $I=100$,

• Simulation 2: $\tau =0.1,0.05,0.02$ (Cases 1, 2, and 3) with fixed $a=0$ and $I=100$,

• Simulation 3: $a=2$ and $\tau =0.02$, with smaller values of $I$ and $n_i$. Specifically, (i) we vary $n_i=30,40,100,500$ with a fixed $I=20$; (ii) we vary $I=10,20,50,100$ with a fixed $n_i=40$.

For each simulation, we randomly select 50% of patients with $S_{ij}=0$ to form the validation subset, mirroring the structure of the Huntington disease data in Section 5. The three simulation scenarios are designed to evaluate different aspects of the method. Simulation 1 represents moderate site-level variability; Simulation 2 resembles the Huntington disease data and assesses sensitivity to the variance component; and Simulation 3 investigates performance with limited data, following a reviewer’s suggestion.

Table 1 presents parameter estimates and sample standard deviations for Simulations 1 and 2. The estimates are generally accurate across all sample sizes, with standard deviations decreasing as expected with larger samples. Table 2 shows results for the small-sample settings in Simulation 3. Parameter estimates exhibited greater variability in these settings but generally stabilized as either $n_i$ or $I$ increased, indicating that the method remains robust for data sizes comparable to the real application.

TABLE 1.

Simulations 1 and 2. Biases (bias) and sample standard deviations (std) based on 200 simulations. The validation set is 50% of the total number of observations with $S_{ij}=0$, the prevalence rate is 22%, and $I=100$.

Simulation 1
		${\beta }_0=1$	${\beta }_1=-5$	$a=2$	$\tau =0.02$
$n_i=200$	bias (std)	0.0077 (0.0407)	−0.0109 (0.1033)	0.0121 (0.0559)	0.0085 (0.0346)
$n_i=500$	bias (std)	−0.0041 (0.0288)	0.0057 (0.0631)	0.0070 (0.0361)	−0.0008 (0.0170)
$n_i=1000$	bias (std)	−0.0013 (0.0202)	0.0023 (0.0436)	0.0040 (0.0286)	−0.0006 (0.0111)
Case 1 of Simulation 2
		${\beta }_0=1$	${\beta }_1=-5$	$a=0$	$\tau =0.1$
$n_i=200$	bias (std)	0.0026 (0.0551)	0.0044 (0.1169)	0.0037 (0.0427)	−0.0075 (0.0283)
$n_i=500$	bias (std)	0.0005 (0.0324)	−0.0104 (0.0731)	0.0205 (0.0404)	−0.0065 (0.0196)
$n_i=1000$	bias (std)	−0.0044 (0.0217)	0.0073 (0.0479)	0.0286 (0.0375)	−0.0059 (0.0145)
Case 2 of Simulation 2
		${\beta }_0=1$	${\beta }_1=-5$	$a=0$	$\tau =0.05$
$n_i=200$	bias (std)	0.0011 (0.0510)	−0.0079 (0.1026)	0.0038 (0.0383)	−0.0031 (0.0186)
$n_i=500$	bias (std)	0.0036 (0.0313)	−0.0055 (0.0704)	0.0056 (0.0284)	−0.0031 (0.0115)
$n_i=1000$	bias (std)	0.0020 (0.0205)	−0.0036 (0.0465)	0.0049 (0.0261)	−0.0016 (0.0092)
Case 3 of Simulation 2
		${\beta }_0=1$	${\beta }_1=-5$	$a=0$	$\tau =0.02$
$n_i=200$	bias (std)	0.0054 (0.0496)	−0.0047 (0.1138)	−0.0004 (0.0325)	−0.0005 (0.0142)
$n_i=500$	bias (std)	0.0013 (0.0359)	0.0005 (0.0737)	0.0009 (0.0272)	−0.0003 (0.0087)
$n_i=1000$	bias (std)	0.0010 (0.0221)	−0.0021 (0.0482)	0.0018 (0.0199)	−0.0006 (0.0055)

TABLE 2.

Simulations 3. Biases (bias) and sample standard deviations (std) based on 200 simulations. The validation set is 50% of the total number of observations with $S_{ij}=0$, the prevalence rate is 22%, and $I=100$.

Fixed $I=20$
		${\beta }_0=1$	${\beta }_1=-5$	$a=0$	$\tau =0.02$
$n_i=30$	bias (std)	0.0031 (0.2382)	−0.0490 (0.5153)	0.0719 (0.3353)	−0.0187 (0.0106)
$n_i=40$	bias (std)	0.0332 (0.2152)	−0.0795 (0.4675)	0.0271 (0.2650)	−0.0161 (0.0231)
$n_i=100$	bias (std)	0.0023 (0.1466)	−0.0238 (0.3111)	−0.0064 (0.1730)	−0.0074 (0.0339)
$n_i=500$	bias (std)	0.0008 (0.0649)	0.0093 (0.1408)	0.0103 (0.0814)	−0.0016 (0.0270)
Fixed ${\mathit{\text{n}}}_{\mathit{\text{i}}}=40$
		${\beta }_0=1$	${\beta }_1=-5$	$a=0$	$\tau =0.02$
$I=10$	bias (std)	0.0408 (0.3183)	−0.1243 (0.6716)	0.1905 (1.1091)	−0.0024 (0.2053)
$I=20$	bias (std)	0.0258 (0.2227)	−0.0265 (0.4810)	0.0147 (0.2699)	−0.0145 (0.0246)
$I=50$	bias (std)	−0.0195 (0.1405)	0.0112 (0.2887)	0.0240 (0.1499)	−0.0163 (0.0159)
$I=100$	bias (std)	0.0063 (0.0943)	−0.0020 (0.2087)	−0.0085 (0.1336)	−0.0142 (0.0218)

Table 3 presents empirical sizes and powers of the LRT and score test based on 1,000 and 2,000 replicates, respectively, including those under the additional small-sample settings. The score test maintains empirical size close to the nominal level, while the LRT is slightly conservative. Both tests achieve similar power with power increasing as the variance component $\tau$ increases. Figure 1 illustrates the relationship between power and $\tau$ for several sample sizes based on 1,000 simulations. Overall, the score test demonstrates better finite-sample performance than the LRT, maintaining appropriate size while achieving comparable power, even in small-sample cases.

TABLE 3.

Empirical size and power comparisons based on 1000 simulations and 2000 simulations. The validation set is 50% of the total number of observations with $S_{ij}=0,I=100$, and 22% prevalence. “LRT” is likelihood ratio test, “Score” is score test. Significance level 5%.

Number of simulations	Method	$n_i$	Size $\tau =0$	Power
				$\tau =0.02$	$\tau =0.05$	$\tau =0.07$	$\tau =0.1$
1000	LRT	30	0.0240	0.0530	0.0890	0.1340	0.1770
		100	0.0420	0.1360	0.4250	0.6230	0.8470
		200	0.0440	0.3400	0.8810	0.9730	0.9970
		500	0.0360	0.8750	1.0000	1.0000	1.0000
		1000	0.0420	0.9980	1.0000	1.0000	1.0000
	Score	30	0.0373	0.0522	0.0950	0.1494	0.1990
		100	0.0551	0.1502	0.4580	0.6310	0.8340
		200	0.0500	0.3500	0.8710	0.9730	0.9980
		500	0.0500	0.8700	1.0000	1.0000	1.0000
		1000	0.0490	0.9980	1.0000	1.0000	1.0000
2000	LRT	30	0.0280	0.0565	0.0955	0.1365	0.1930
		100	0.0430	0.1435	0.4290	0.6285	0.8500
		200	0.0435	0.3505	0.8825	0.9745	0.9980
		500	0.0385	0.8755	1.0000	1.0000	1.0000
		1000	0.0455	0.9990	1.0000	1.0000	1.0000
	Score	30	0.0378	0.0607	0.1060	0.1544	0.2180
		100	0.0586	0.1621	0.4582	0.6405	0.8435
		200	0.0507	0.3605	0.8715	0.9745	0.9980
		500	0.0491	0.8695	1.0000	1.0000	1.0000
		1000	0.0495	0.9990	1.0000	1.0000	1.0000

FIGURE 1.

Empirical power of the proposed tests based on 1,000 simulations. The case $\tau =0$ (empirical size) is reported in Table 3; hence the curves begin at $\tau =0.005$.

5 |. APPLICATION

We analyze data from the Prospective Huntington At-Risk Observational Study (PHAROS), which includes 664 individuals at risk for Huntington’s disease (defined as having ≥ 36 CAG repeats in the Huntington gene) across 20 study sites. The number of patients at each site ranges from 25 to 59. Our goal is to assess whether differences in the diagnosis of MCI exist across these sites, as variability in diagnostic practices may lead to inconsistent results. To investigate site-specific variability, we focus on two key covariates: age at baseline and CAG repeat length, which refers to the number of cytosine-adenine-guanine (CAG) repeats in the Huntington gene, a well-established genetic marker for Huntington disease⁹. Though debated, evidence suggests both CAG repeat length and age at baseline are associated with cognitive decline, which is why it is important to consider their potential effects when diagnosing mild cognitive impairment in Huntington disease².

In this study, we aim to distinguish between patients who are certain to have MCI and those who do not, with some patients falling into an uncertain category. We define $Y_{ij}=1$ if the $j$-th patient at the $i$-th site is certain to have MCI, and $Y_{ij}=0$ otherwise. For patients who meet the criteria for multiple-domain mild cognitive impairment (i.e., when scores on two or more cognitive tests, such as the Symbol Digit Modality Test and the Stroop Color Word Test, are 1.5 standard deviations below normative means), we set $S_{ij}=1$, implying that $Y_{ij}=1$—the patient is definitively diagnosed with MCI^10,2. (Normative means for the cognitive tests are summarized in Table 4.) For patients who meet the criteria for MCIt on at most one cognitive test, we define $S_{ij}=0$, indicating that their MCI diagnosis is uncertain, meaning $Y_{ij}$ could be either 0 or 1.

TABLE 4.

The cutoff values for SDMT and SCWT scores

	mean	std	cutoff
SDMT	60.9	11.3	43.9
SCWT - color	79.1	14.0	58.1
SCWT - word	108.2	15.3	85.2
SCWT - interference	46.1	10.4	30.5

In the validation set, where $S_{ij}=0$, we classify patients who do not meet the MCI criteria for any of the cognitive tests as $Y_{ij}=0$ (no MCI). If a patient’s score is below the cutoff on a single cognitive test, we classify $Y_{ij}=1$, indicating single-domain mild cognitive impairment. The full dataset comprises 664 patients: 65 with $S_{ij}=1$ and 599 with $S_{ij}=0$. Among the 599 patients with $S_{ij}=0,300$ have observed $Y_{ij}$ and constitute the validation set. The remaining 364 patients (65 with $S_{ij}=1$ and 299 with $S_{ij}=0$) form the main data. The validation rate is approximately 300/599 ≈ 50%. From the statistical point of view, a larger validation data is always more beneficial. However, the cost and feasibility of collecting validation data impose restriction. Consequently, no single guideline is applicable to all situations. The size of the validation subset should balance two considerations: larger subsets enhance estimation efficiency and testing power, but also demand greater resources for outcome adjudication. When feasible, a validation rate above 30% is recommended; however, the proposed approach remains applicable with lower proportions if required. Also, only subjects with $S=0$ should be selected randomly to ensure that the validation sample accurately represents the unlabeled population.

We generated 200 bootstrap samples to estimate the parameters, with the results and standard deviations provided in Table 5. For each bootstrap replicate, we independently resampled 300 patients with replacement from the validation set and 364 patients with replacement from the main data. The estimated coefficients for age at baseline (${\widehat{\beta }}_1=-0.8572$) and CAG repeat length (${\widehat{\beta }}_2=0.7946$) have $z$-values of −0.8572 and 0.7946, respectively, indicating that the covariate effects are not statistically significant. The negative coefficient for age suggests that older age may be associated with a lower likelihood of MCI, while the positive coefficient for CAG repeat length suggests that longer CAG repeats are linked to a higher likelihood of MCI. However, the lack of statistical significance indicates that these relationships are not strong enough to be conclusive. These results align with existing literature, where CAG repeat length is known to influence the progression of Huntington disease and its cognitive effects². The lack of significance for age may reflect that other unmeasured site-specific or biological factors may also be contributing to the variability in MCI diagnosis¹.

TABLE 5.

The estimates (est) and the sample standard deviations (std) based on 200 bootstrap samples. The validation set is 50% of the total number of observations with $S_{ij}=0$ and $I=20$.

	${\widehat{\beta }}_0$	${\widehat{\beta }}_1$	${\widehat{\beta }}_2$	$\widehat{a}$	$\widehat{\tau }$
est	−0.6205	−0.0159	0.0195	−0.4544	1.8229
std	0.0853	0.0185	0.0246	0.4703	0.6444

Next, we performed both the LRT and a score test to evaluate the null hypothesis $H_0:\tau =0$ versus the alternative $H_1:\tau >0$, where $\tau$ represents the variability in site-specific sensitivity for the anchor variable $S$. The LRT statistic is $T_l=18.03$, with a $p$-value of < 0.001, and the score statistic $T_s=90.53$, also yielding a $p$-value of < 0.001. The significant results support $H_1:\tau >0$, indicating substantial site-level variability in the sensitivity of the anchor variable.

Although all sites used the same criteria for diagnosing MCI and administered the same quantitative cognitive tests, the variability in $\tau$ suggests that differences in site practices, such as testing protocols or clinician judgment, may influence MCI diagnosis. For example, some sites may have more experienced clinicians who interpret test results more conservatively, leading to fewer diagnoses of MCI, while other sites may be more inclined to diagnose MCI based on borderline test scores. Additionally, variability in testing environments—such as differences in patient comfort, time of day, or distractions—could influence cognitive performance and, consequently, diagnosis.

Having established that substantial site-level variability exists, we now examine the site-specific effects $b_i$ to identify which sites have notably higher or lower sensitivity. We provide estimates of $b_i$ using three approaches that leverage different sources of information. We first obtain estimates ${\widehat{a+b}}_i$ following (S.3), estimate $a$ by the sample average $\widehat{a}=I^{-1}\sum _{i=1}^I{\widehat{a+b}}_i$, and compute initial estimates of $b_i$ as ${\widehat{b}}_i={\widehat{a+b}}_i-\widehat{a}$ for $i=1,\dots ,I$. Next, using the estimated parameters $\widehat{a}$ and $\widehat{\tau }$ from our proposed method, we compute empirical Bayes (EB) estimates of the random effects $b_i$ under two models. The first EB estimator, denoted ${\widehat{b}}_i^{EB1}$, uses only the sensitivity model logit $\text{pr}\left(S_{ij}=1\mid Y_{ij}=1\right)=a+b_i$ and incorporates the estimated variance component $\widehat{\tau }$ to shrink estimates toward the mean. Specifically, we compute ${\widehat{b}}_i^{EB1}$ by maximizing the log-likelihood for each site $i:l_i^{EB1}(b)=m_{i1}\text{log}\{\text{expit}(\widehat{a}+b)\}+\left(n_{i1}-m_{i1}\right)\text{log}\{1-\text{expit}(\widehat{a}+b)\}-b^2/(2\widehat{\tau })+C$, where $m_{i1}=\#\left\{j:Y_{ij}=1,S_{ij}=1\right\}$ and $n_{i1}=\#\left\{j:Y_{ij}=1\right\}$ and $C$ is a constant not involving $b$. The second EB estimator, denoted ${\widehat{b}}_i^{EB2}$, additionally incorporates covariate information through $\text{pr}\left(S_{ij}=1\mid {\mathbf{\text{X}}}_{ij}\right)=\text{pr}\left(S_{ij}=1\mid Y_{ij}=1\right)\text{pr}\left(Y_{ij}=1\mid {\mathbf{\text{X}}}_{ij}\right)$, using both the sensitivity model and the covariate-disease relationship. Specifically, we compute ${\widehat{b}}_i^{EB2}$ by maximizing the log-likelihood for each site $i:l_i^{EB2}(b)=\sum _{j=1}^{n_i+m_i}\left[S_{ij}\text{log}\{\text{expit}(\widehat{a}+b)\}+\left(1-S_{ij}\right)\text{log}\left\{1-\text{expit}(\widehat{a}+b)\text{expit}\left({\mathbf{\text{X}}}_{ij}^{\text{T}}\widehat{\mathit{\beta }}\right)\right\}\right]-b^2/(2\widehat{\tau })+C$, where $C$ does not contain $b$. Figure 2 displays boxplots comparing the three sets of estimates. The initial estimates ${\widehat{b}}_i$ show the raw site-to-site variation before accounting for uncertainty. The EB estimates ${\widehat{b}}_i^{EB1}$ and ${\widehat{b}}_i^{EB2}$ shrink extreme values toward zero, with ${\widehat{b}}_i^{EB2}$ showing the most shrinkage due to incorporating additional covariate information. In all cases, the spread of the estimated $b_i$ values indicates substantial variation in diagnostic sensitivity across the 20 sites, confirming the test results and highlighting the importance of accounting for site effects. Sites with negative estimated $b_i$ values have lower sensitivity and may require closer examination to identify whether operational, clinical, or population-level factors are affecting diagnostic performance.

FIGURE 2.

Boxplots and individual site-specific estimates for all 20 sites, ${\widehat{b}}_i$ is based on (S.3), ${\widehat{b}}_i^{EB1}$ is an empirical Bayes estimate with the sensitivity model, and ${\widehat{b}}_i^{EB2}$ is an empirical Bayes estimate incorporated with sensitivity and covariate information.

Overall, our findings highlight the substantial variability in the sensitivity of the anchor variable across sites, suggesting that site-specific differences, even when using the same diagnostic criteria and cognitive tests, may influence the diagnosis of MCI. These differences can lead to diagnostic inconsistencies that affect patient outcomes. For example, sites with higher sensitivity may identify more cases of MCI, potentially leading to earlier interventions, while sites with lower sensitivity could miss cases, delaying care. Addressing these site-specific effects can ultimately help achieve diagnostic consistency and lead to providing timely, appropriate care for Huntington disease patients.

6 |. CONCLUSION

In this paper, within the context of Huntington disease, we developed a statistical framework for analyzing multi-site clinical data where only positive cases can be confidently identified through an anchor variable. This setting leads to a positive-only structure, where $S=1$ indicates confirmed disease an d $S=0$ represents unknown disease status. To address this challenge, we used a validation subset to ensure identifiability and formulated a likelihood-based estimation procedure.

We introduced a random-effect structure to model site-level variability in the sensitivity of the anchor variable and developed variance component tests—the likelihood ratio test and the score test—to formally assess whether such variability exists. Simulation studies showed that the proposed estimation and testing procedures perform well in finite samples, with the score test maintaining empirical size close to the nominal level while the LRT is slightly conservative. Both tests achieve similar power.

Applying our methods to a multi-site Huntington disease cohort revealed differences in diagnostic sensitivity across sites, suggesting variation in screening or clinical interpretation practices. Instead of prescribing how diagnoses should be standardized, our framework offers a principled way to quantify, compare, and evaluate site-level variability in diagnostic sensitivity. This provides methodological guidance for identifying where diagnostic procedures may need reassessment and supports more consistent inference across multi-site studies.

Future work could expand the framework in two ways. First, sensitivity might be modeled to change with patient-level characteristics beyond site-level effects. Second, in the situation where the anchor variable can be a false alarm, i.e., when $\text{pr}(Y=0\mid S=1)>0$, site-level variation could be modeled not only for sensitivity, but also for specificity. These extensions represent valuable directions for future research.

Supplementary Material

Additional supporting information may be found in the online version of the article at the publisher’s website.