Evaluating the Bias, type I error and statistical power of the prior Knowledge-Guided integrated likelihood estimation (PIE) for bias reduction in EHR based association studies

Abstract

Objectives:

Binary outcomes in electronic health records (EHR) derived using automated phenotype algorithms may suffer from phenotyping error, resulting in bias in association estimation. Huang et al. [1] proposed the Prior Knowledge-Guided Integrated Likelihood Estimation (PIE) method to mitigate the estimation bias, however, their investigation focused on point estimation without statistical inference, and the evaluation of PIE therein using simulation was a proof-of-concept with only a limited scope of scenarios. This study aims to comprehensively assess PIE’s performance including (1) how well PIE performs under a wide spectrum of operating characteristics of phenotyping algorithms under real-world scenarios (e. g., low prevalence, low sensitivity, high specificity); (2) beyond point estimation, how much variation of the PIE estimator was introduced by the prior distribution; and (3) from a hypothesis testing point of view, if PIE improves type I error and statistical power relative to the naïve method (i.e., ignoring the phenotyping error).

Methods:

Synthetic data and use-case analysis were utilized to evaluate PIE. The synthetic data were generated under diverse outcome prevalence, phenotyping algorithm sensitivity, and association effect sizes. Simulation studies compared PIE under different prior distributions with the naïve method, assessing bias, variance, type I error, and power. Use-case analysis compared the performance of PIE and the naïve method in estimating the association of multiple predictors with COVID-19 infection.

Results:

PIE exhibited reduced bias compared to the naïve method across varied simulation settings, with comparable type I error and power. As the effect size became larger, the bias reduced by PIE was larger. PIE has superior performance when prior distributions aligned closely with true phenotyping algorithm characteristics. Impact of prior quality was minor for low-prevalence outcomes but large for common outcomes. In use-case analysis, PIE maintains a relatively accurate estimation across different scenarios, particularly outperforming the naïve approach under large effect sizes.

Conclusion:

PIE effectively mitigates estimation bias in a wide spectrum of real-world settings, particularly with accurate prior information. Its main benefit lies in bias reduction rather than hypothesis testing. The impact of the prior is small for low-prevalence outcomes.

1. Introduction

Electronic health records (EHR) provide massive information about patients and have become an important resource for association studies, where the interest lies in learning about the relationship between an outcome and a predictor. Automated phenotyping algorithms are often applied to extract patient outcomes of interest (e. g., diabetes, atrial fibrillation). However, the accuracy of the phenotyping algorithms can be limited and sometimes far from satisfactory [2]. Due to the imperfection of phenotyping algorithms, the extracted outcomes suffer from misclassification, which, when later used in statistical modeling can produce misleading analysis results [3,4].

A direct detrimental consequence of phenotyping error is bias in association parameter estimates. To reduce this bias, Huang et al. [1] proposed a prior knowledge-guided integrated likelihood estimation (PIE) method. Their key idea was to use the prior knowledge on the distribution of the sensitivity and specificity of the phenotyping algorithm to adjust the likelihood function. Concretely, PIE integrates the sensitivity and specificity over a prior distribution on the likelihood function to obtain the integrated likelihood (IL) and then maximizes the IL to get the estimator. The strength of PIE is that it does not require validation data but only prior knowledge of the performance of the phenotyping algorithm, which can be obtained from the literature or expert opinion. Huang et al. [1] showed the advantages of PIE using simulation studies. However, they only studied the estimation bias but no inference, and the simulation study therein was a proof-of-concept and only considered a limited scope of scenarios.

Our objective is to systematically study the performance of PIE under a wide spectrum of realistic scenarios, in terms of both point estimation and hypothesis testing, and to provide a comprehensive understanding and guidance on when to apply PIE. Specifically, we generate simulated data under various settings, including those with low prevalence in the outcome and predictor variables, and low sensitivities. We compare PIE, using prior distributions with different levels of accuracy, with a naïve method and the method with known sensitivity and specificity to illustrate its strength and weaknesses.

2. Methods

2.1. Algorithms

Our target was to study the association between one binary outcome (e. g., disease status, presence or absence of a condition) and one predictor (e. g., receiving a treatment) with the outcome derived from EHR data using some phenotype algorithm. Due to the misclassification in the phenotype algorithm, the observed outcome $S_i$ for the $i$-th patient, is not the true outcome of that patient, $Y_i$. The total number of patients is $n$. Assume that the true association between $Y_i$ and the predictor $x_i$ is described by a logistic regression model

$$\text{Pr}(Y_i=1)=\text{expit}({\beta }_0+{\beta }_1{}^{\star }x_i)$$ (1)

where $\text{expit}(x)=\frac{1}{1+e^x}$. The estimator given by (1) is unbiased but unachievable, which is referred to as the gold standard estimator (i.e., regression on the true outcomes). The misclassification rate of the surrogate outcome $S_i$ is characterized by the sensitivity, ${\alpha }_1=\text{Pr}(S_i=1\mid Y_i=1)$, and specificity, ${\alpha }_0=\text{Pr}(S_i=0\mid Y_i=0)$, of the phenotype algorithm. Note that we assume the misclassification rate is non-differential, that is the sensitivity and specificity are independent of the covariate. The association between $S_i$ and $x_i$ is then described as

$$\text{Pr}(S_i=1)=(1-{\alpha }_0)+({\alpha }_0+{\alpha }_1-1)\text{expit}({\beta }_0+{\beta }_1{}^{\star }x_i)$$ (2)

The naïve method ignores the misclassification and treats $S_i$ as the true outcome. To estimate the association, it fits a logistic regression

$$\text{Pr}(S_i=1)=\text{expit}({\gamma }_0+{\gamma }_1{}^{\star }x_i).$$

The corresponding estimator of the association, ${\widehat{\gamma }}_1$, is a biased estimator of the true association ${\beta }_1$ and attenuates to null [2].

With known true sensitivity and specificity, an unbiased estimated can be obtained by substituting ${\alpha }_1$ and ${\alpha }_0$ in model (2) by the true sensitivity and specificity of the algorithm, and then maximizing the likelihood function

$$L({\beta }_0,{\beta }_1,{\alpha }_0,{\alpha }_1)=\prod _{i=1}^n\text{Pr}{(S_i=1)}^{S_i}{(1-\text{Pr}(S_i=1))}^{1-S_i},$$

where $\text{Pr}(S_i=1)=(1-{\alpha }_0^{\ast })+({\alpha }_0^{\ast }+{\alpha }_1^{\ast }-1)\text{expit}({\beta }_0+{\beta }_1{}^{\star }x_i)$ with ${\alpha }_0^{\ast }$ and ${\alpha }_1^{\ast }$ being the true values of sensitivity and specificity. However, choosing the correct values of sensitivity and specificity is challenging, requiring substantial effort to conduct an adequately large validation study to obtain accurate and precise estimates. If the misclassification rates are incorrect, the resulting estimator will be biased.

Instead of fixing the sensitivity and the specificity at specific values, PIE proposed to “average” over a range of values by adopting integrated likelihood [5,6]. Concretely, given a prior distribution for the sensitivity and specificity (${\alpha }_0$, ${\alpha }_1$), the integrated likelihood is defined as

$$L_I({\beta }_0,{\beta }_1)=\int \int L({\beta }_0,{\beta }_1,{\alpha }_0,{\alpha }_1)\pi ({\alpha }_0,{\alpha }_1)d{\alpha }_{0}d{\alpha }_1.$$

The PIE estimator is obtained by maximizing $L_I({\beta }_0,{\beta }_1)$. PIE is equivalent to the method with known sensitivity and specificity when $\pi ({\alpha }_0,{\alpha }_1)=1$ at the true (${\alpha }_0$, ${\alpha }_1$), and 0 otherwise. PIE requires only a prior distribution of the sensitivity and specificity that may be easier to obtain (e. g., through literature search on the accuracy of the phenotype algorithm or expert opinion) than the precise true values in the data set under study and still reduces the bias of the estimator relative to the naïve approach.

Huang et al. [1] only studied the estimation bias of PIE but no evaluation on inference, and the simulation study therein was a proof-of-concept with a limited scope of scenarios. To systematically study the performance of PIE and provide a comprehensive guidance on when to apply PIE, we conducted simulation studies under a wide spectrum of settings and discussed the strength and weaknesses of PIE. In addition, we evaluate PIE using real-world EHR data from the Children’s Hospital of Philadelphia with COVID-19 infection as the outcome.

2.2. Simulation study

In simulation settings, we considered a binary $x_i$ generated from a Bernoulli distribution with mean 0.3. Prevalence of the outcome for observations with $x_i=0$ ranged from 5 % to 50 % and ${\beta }_1$ from 0 to 1.2. The sensitivity was fixed at ${\alpha }_1=0.65$. The specificity was fixed at ${\alpha }_0=0.99$ because it is usually large in practice [2]. We generated $x_i$ for $n$= 3000 patients. Then, we generated $Y_i$ from Bernoulli distribution for each $x_i$. Finally, we generated the observed surrogate outcome $S_i$ based on ${\alpha }_1$ and ${\alpha }_0$.

Huang et al. [1] demonstrated that uniform prior distributions performed well in terms of estimation bias compared with other alternatives. Therefore, we have primarily focused our evaluation on uniform distributions prior distributions. For sensitivity, we used uniform distributions centered at the truth $U({\alpha }_1^{\ast }-d,{\alpha }_1^{\ast }+d)$ with $d$ ranging from 0.05 to 0.2 to study the influence of the spread of the distribution on the PIE estimator. In addition, we studied the performance of skewed uniform distribution not symmetric around the truth $U({\alpha }_1^{\ast }-0.15,{\alpha }_1^{\ast }+0.05)$ and $U({\alpha }_1^{\ast }-0.05,{\alpha }_1^{\ast }+0.15)$. For specificity, we used the uniform distribution $U(0.95,0.9999)$. We focused on studying the different prior distributions of sensitivity and fixed the prior distribution of specificity because specificity is usually large in practice. In addition to uniform distribution, we also considered various logit normal and beta prior distributions with different means and spreads in power and type I error evaluation. The details regarding the logit normal and beta prior distributions are given in the Supplementary Material.

We investigated the performance of PIE under various prevalence rates and degrees of dispersion of the prior distribution (by varying $d$), along with the method with known sensitivity and specificity and the naïve estimator, in terms of the bias, type I error, and the power. We set $d=0.05$, 0.1, 0.15, 0.2. When studying bias, we fixed ${\beta }_1=\log 3$ and generated 200 datasets under each setting. When studying type I error, we fixed${\beta }_1=0$ and generated 200 datasets under each setting. When studying the power, we fixed prevalence to be 5 %, set ${\beta }_1=0.1$, 0.3,0.5, 0.7, 0.9,1.2, and generated 200 datasets under each setting. The number of replications was justified by the Monte Carlo error, i.e., the standard deviation of the Monte Carlo estimator [7]. The details were reported in the Results section. Additional simulations were conducted to show the insensitivity of the results to the choice of the fixed parameters and were presented in the Supplementary Material.

The type I error and power were calculated as follows. We first estimated the standard deviation of the estimator by bootstrap. Then we constructed the test statistic for each dataset as the estimator divided by the standard deviation. If the absolute value of the test statistic was greater than 1.96, then the null was rejected. We calculated the proportion of the rejected nulls out of the 200 datasets as the type I error when${\beta }_1=0$ and as 1 – power when ${\beta }_1\ne 0$.

The computing programs were run in R and the code can be found online at https://github.com/Penncil/PIE-evaluation.

2.3. Use case

We assessed the performance of PIE by utilizing a simulated dataset generated from summary statistics derived the real-world electronic health records (EHR) data in existing study [13]. Specifically, we generated a cohort consisting of patients who have been tested for SARS-CoV-2, diagnosed with COVID-19, or received a COVID-19 vaccine between September 2020 and April 2022 [10]. Our final simulated dataset comprised a total of 146,518 patients, along with various clinical data such as demographics, medical history, and laboratory results. Similar as the EHR data in the existing study, there are approximately 10 % of patients with missing data in the simulated cohort. For the main evaluation, we removed the 10 % of missing data from the whole population and focusing on the patients with complete data. To further assess the robustness of PIE in real-world scenarios, we examined the performance of PIE when missing data was imputed, with details given in the Supplementary Material.

The outcome of interest is COVID-19 infection (determined by institutional mechanisms) represented by a binary variable. The diagnosis code for defining COVID-19 infection can be find in Supplemental Material. The predictors were constructed based on ‘negative controls’ [11]. A negative control is pair of outcome and predictor that has been established to have no association. Contrarily, a positive control is a pair of outcome and predictor that is known to have positive association. We simulated three negative controls with outcomes being COVID-19 infection and predictors being Control H46-H48: Disorders of optic nerve and visual pathways, Control H53-H54: Visual disturbances and blindness, Control H30-H36: Disorders of choroid and retina, and Control H15-H22: Disorders of sclera, cornea, iris, and ciliary b, respectively. Then we constructed synthetic positive controls using the three negative controls with effect size setting as 1.5 and 4. The constructed synthetic positive controls preserved the characteristics of real-world data whilst have known effect size allowing us to evaluate the results from PIE. We considered multiple negative controls and magnitudes of effect size to ensure that PIE remained valid across different scenarios. As the true sensitivity and specificity were unknown in the dataset, we set the prior distribution of sensitivity as Uniform (0.79, 0.95) and the prior distribution of specificity as Uniform (0.95, 0.9999) based on expert knowledge. We randomly selected 3000 patients and estimate the effect size using PIE and the naïve method for 20 times. Then we summarized the results in a boxplot.

3. Results

Bias.

Fig. 1 showed boxplots of the estimators from 200 simulated data sets per simulation setting. The standard deviations of the Monte Carlo mean (average across the 200 estimators) were calculated using bootstrap to justify the number of replications in the simulation, which under all the settings are smaller than 0.02. The red horizontal line represented the true value. The method with known sensitivity and specificity, under all the settings, had the smallest bias, with a medium very close to the truth. The naïve method had the largest bias with substantial attenuation toward the null and performed worse at higher prevalence. The PIE method was closer to the truth compared with the naïve method, indicating a bias reduction. As the prior of sensitivity became more variable (i.e., larger spread), PIE shifted away from the truth, but still performed better than the naïve estimator. The influence of the prior of sensitivity on the PIE estimator was larger as the prevalence of the response becomes larger. The results also indicated a biasvariance trade-off. While the bias of PIE was reduced compared with the naïve method, its variance slightly increased. The variance of PIE was similar to that of the estimator with known sensitivity and specificity.

Fig. 1.

Estimation using the method with known sensitivity and specificity (red), the naïve (yellow), and the PIE (blue) algorithms over 200 simulated data. The red horizontal line is the true association, $log3$. The x-axis is the prevalence under unexposed. Sensitivity and specificity are fixed at 0.65 and 0.99, respectively. The prior distribution of the sensitivity is set to be six different uniform distributions. The prior distribution of specificity is fixed as Uniform (0.95, 0.9999). The sample size in each data is 3000. The predictor is generated from a Bernoulli distribution with a mean of 0.3.

Type I error and power.

As shown in Fig. 2, no specific pattern was found for type I errors under different settings and different estimators. All the algorithms had type I errors that fluctuated around 0.05. The Monte Carlo errors of the estimated type I errors were smaller than 0.02. When studying their bias (figures not presented), we found all three methods had similar small biases when the true effect size is zero, the naïve estimator has the smallest variance, and the variance of all three methods decreased as the prevalence of the outcome decreased. This is reasonable since the naïve estimator tends to shift toward the null. This result also indicated that nondifferential misclassification may not cause inflation in type I errors. Similar results apply to the power as shown in Fig. 3. The Monte Carlo errors of the estimated powers were small (<0.01) when the effect size was large (>0.7), and relatively large (around 0.03) when the effect size was moderate (around 0.5). The powers of PIE, the method with known sensitivity and specificity, and the naïve method were similar and smaller than the gold standard. The power increased as the effect size increased. The same pattern were observed when using various logit normal and beta distribution with different means and spreads. The details were given in the Supplementary Material.

Fig. 2.

Empirical type I error with confidence intervals using the method with known sensitivity and specificity, the naïve, and the PIE under 6 prior distributions calculated using 200 simulated data. The true association is 0. Prevalence of the outcome when unexposed is set to be 5 %, 10 %, 20 %, 30 %, 40 %, and 50 %. Sensitivity and specificity are fixed at 0.65 and 0.99, respectively. The prior distribution of the sensitivity is set to be six different uniform distributions. The prior distribution of specificity is fixed as Uniform (0.95, 0.9999). The sample size in each data is 3000. The predictor is generated from a Bernoulli distribution with a mean of 0.3.

Fig. 3.

Empirical powers using the gold standard (known outcome), the method with known sensitivity and specificity, the naïve, and the PIE under 6 prior distributions calculated using 200 simulated data. The true associations are set as 0.1, 0.3, 0.5, 0.7, 0.9, 1.2. Prevalence under unexposed is fixed at 5 %. Sensitivity and specificity are fixed at 0.65 and 0.99, respectively. The prior distribution of the sensitivity is set to be six different uniform distributions. The prior distribution of specificity is fixed as Uniform (0.95, 0.9999). The sample size in each data is 3000. The predictor is generated from a Bernoulli distribution with a mean of 0.3.

Use-case evaluation.

PIE outperforms the naïve approach in the majority of the settings (Fig. 4). The naïve approach underestimates the effect size and becomes worse as the effect size increases. PIE maintains a relatively accurate estimation across different strengths of relationships, particularly outperforming the naïve approach when dealing with large effect sizes.

Fig. 4.

Estimation of effect size of four positive controls using the naïve and the PIE. The positive controls are synthesized using negative controls H46-H48: Disorders of optic nerve and visual pathways, H53-H54: Visual disturbances and blindness, H30-H36: Disorders of choroid and retina, and H15-H22: Disorders of sclera, cornea, iris, and ciliary b. with effect size 1.5 and 4 (red dashed line). The boxplot presents results from 20 iterations of sampling with size 3000.

4. Discussion

Our investigation found that PIE can effectively reduce the estimation bias when the phenotype-derived outcome suffers from misclassification. PIE has a smaller bias compared with the naïve method, but similar type I error and power due to larger variance. This indicates that PIE may be more suitable when the main task is estimation rather than testing. The estimation bias of PIE is smaller when the prior distribution is more concentrated around the truth. The influence of an inaccurate prior is larger as the prevalence of outcome gets large.

PIE has been effectively applied in an existing comparative effectiveness study and can also be utilized in future trial emulation studies to estimate causal effects. For example, it has been used in research assessing vaccine effectiveness against COVID-19 infection and severe disease in children and adolescents [13]. In this context, immunization records, often stored across various unlinked sources, result in underreported vaccination data in patients’ EHRs. PIE addresses this challenge by accommodating the misclassified vaccination status through the integrated likelihood method, meanwhile accommodating the prior knowledge on the range of sensitivity. This approach reduces bias in the estimation of vaccine effectiveness by avoiding the need to fix sensitivity at particular values.

We acknowledge the necessity of evaluating the PIE method across a broader spectrum of EHR data in addition to the pediatric population. Future studies could benefit from exploring the application of PIE to diverse datasets from various healthcare settings. We plan to continue the expansion of the evaluation to extend the generalizability and applicability of the PIE method in real-world clinical settings.

The current PIE algorithm considers only nondifferential misclassifications, where the misclassification rate is the same under all the values of the predictor. Differential misclassification, where the sensitivity and specificity can be dependent on the exposure and other covariates, as pointed out in [12], may also exist in EHR-derived data and can cause estimation bias and type I error inflation. It is of interest to extend PIE to incorporate differential misclassifications, for example, by modeling the sensitivity and specificity as functions of the predictor. Another natural extension of PIE is its application in dealing with mismeasurement in survival outcomes.

Statement of Significance.

Summary	Description
Problem	Automated phenotype algorithms applied to electronic health records (EHR) may produce binary outcomes that are prone to phenotyping errors, leading to biased estimations in association studies.
What is Already Known	Huang et al. introduced the Prior Knowledge-Guided Integrated Likelihood Estimation (PIE) method to address estimation bias. However, their study primarily concentrated on point estimation without statistical inference. Additionally, the simulation study conducted to assess PIE was a proof-of-concept and only examined a restricted range of scenarios.
What this Paper Adds	This research aims to thoroughly evaluate the performance of PIE in several aspects: (1) its effectiveness across various operating characteristics of phenotyping algorithms in real-world situations; (2) examining the extent of variation introduced by the prior distribution on the PIE estimator beyond point estimation; and (3) assessing whether PIE enhances type I error control and statistical power compared to the naïve method from a hypothesis testing perspective.