Impact of Case-Mix Measurement Error on Estimation and Inference in Profiling of Health Care Providers

Abstract

Profiling analysis aims to evaluate health care providers by modeling each provider’s performance with respect to a patient outcome, such as unplanned hospital readmission. High-dimensional regression models are used in profiling to risk-adjust for patient case-mix covariates. Case-mix covariates typically ascertained from administrative databases are inherently error-prone. We examine the impact of case-mix measurement error (ME) on profiling models. The results show that even though the models’ coefficient estimates are biased, this does not affect the estimation of standardized readmission ratio (SRR). However, ME leads to increased variation in SRR estimates and degrades the ability to identify under-performing providers.

1. Introduction

Profiling or evaluation of health care providers, including hospitals, nursing homes, and dialysis facilities, with respect to a patient outcome such as 30-day unplanned hospital readmission is important to ensure adequate and safe health care delivery. Profiling dates back nearly a century (Codman 1916) and has evolved over time to serve several purposes. This includes 1) identifying providers with below standard performance by government agencies for regulatory or payment purposes, 2) conveying information to patients regarding the quality of care, and 3) providing feedback to providers for quality improvement. In the United States, a more systematic reporting of patient outcomes among providers has only appeared directly to consumers in the last decade by the Centers for Medicare and Medicaid Services (CMS). This includes condition-specific 30-day mortality (e.g., acute myocardial infarction, heart failure, pneumonia) and 30-day (all-cause) unplanned readmission rates; see Keenan et al. (2008), Krumholz et al. (2011), Lindenauer et al. (2011), and Horwitz et al. (2011, 2014).

In profiling analysis, there are two main goals: (a) report accurate estimates of provider effects relative to a reference, such as a national rate, and (b) identify providers that are “worse,” “better” or “not different” relative to the reference. We refer to goals (a) and (b) as estimation and inference goals, respectively. With respect to goal (a), each provider’s effect is summarized as a ratio in the form of a risk-standardized readmission ratio (SRR) for the patient outcome of 30-day unplanned readmission, for instance. Patient outcomes vary across providers due to variation in providers’ quality of care (provider effects) and variation in patient case-mix (patient-level effects, including demographics, comorbidities, and types of index hospitalization). With patients nested within providers, profiling models are hierarchical logistic regressions of the form outcome = provider effects + patient case-mix effects.

Because profiling models involve large populations of patients, such as all dialysis patients treated at dialysis facilities in the U.S. or all patients admitted to hospitals, the patient case-mix covariates are ascertained from administrative claims data. As such, measurement errors (MEs) in the case-mix covariates, such as inaccuracies in continuous variables and misclassified categorical variables (e.g., patient comorbidities) are expected and well-known. The impact of ME in the covariates is biased regression coefficient estimates, a problem that has been well-studied (Carroll et al. 2006). However, no methodological study has been conducted to understand the general effect of ME on the estimation of SRR and on the inferential objective of profiling. To date, one study has reported an empirical comparison of the random effects (RE)/CMS profiling model for 30-day readmission among patients with acute myocardial infarction, using variables based on claims data (imprecise) versus patient medical records (more precise). The study concluded that administrative claims data is suitable for profiling (Krumholz et al. 2011). Without knowledge of the statistical properties of profiling models under case-mix ME, the generalizability of conclusions from an empirical study is unclear. For instance, are the estimates of provider-specific SRRs biased and therefore invalid for public reporting when case-mix variables are measured with error? And, to what extent do case-mix MEs reduce profiling models’ ability to identify under-performing providers? We aim to fill this knowledge gap and provide more general insights on the impact of case-mix ME in profiling.

To investigate the impact of ME, our work here focuses on the outcome of 30-day unplanned hospital readmission; however, the methodological issues are general and applicable to other outcomes, such as all-cause or cause-specific in-hospital mortality. Also, we will consider both random effects (RE) and fixed effects (FE) profiling models proposed in the literature. We note that RE models have been the dominant approach and are the adopted method by CMS (CMS 2014; Ash et al. 2012; Krumholz et al. 2011). Motivations for hierarchical RE have largely been conceptual and center around the need to account for the nested structure of the data (e.g., patients nested within providers) and data sparsity (low frequency of the observed outcome); see Normand and Shahian (2007), Normand et al. (1997), and Jones and Spiegelhalter (2011) for conceptual motivations for the RE models. (Throughout this work, we refer to the terms “RE model” and “CMS model” interchangeably, and they refer to the CMS adopted model with random intercepts for providers as further described in Section 2.)

FE models for profiling was proposed by Kalbfleisch and Wolfe (2013) and He et al. (2013), which model provider effects as fixed effects. We note that the FE model of He et al. (2013) is a high-dimensional parameter model with a unique fixed intercept for each provider. In practice, the number of providers is in the hundreds or thousands. The advantages of RE versus FE models have been discussed and compared previously (Kalbfleisch and Wolfe 2013; Chen et al. 2017); thus, it is not the focus of this paper. However, we briefly note that although RE models can provide stable provider effect estimates (through shrinkage), they are biased toward the overall provider average and the bias is larger for smaller providers (Kalbfleisch and Wolfe 2013). RE models also have smaller overall average estimation error, but this gain is in the center of the distribution of the outcomes, reducing the power to identify underperforming providers. We consider the impact of ME on both RE and FE models. Finally, to avoid confusion, we note that there is another use of the term “fixed effects” models in the profiling literature, where a “naive” FE model refers to a logistic regression model with one overall intercept (e.g., see Austin, Alter and Tu 2003). This naive FE model is no longer in wide usage and is different from the FE model of He et al. (2013).

2. Methods

2.1. RE and FE models without measurement error

The RE model implemented by CMS for (30-day unplanned) all-cause or condition-specific hospital readmission is the following RE logistic regression model,

$$g\left({\mu }_{ij}^{\ast }\right)={\gamma }_i^{\ast }+{\beta }^{\ast \text{T}}{\tilde{Z}}_{ij},{\gamma }_i~N\left({\gamma }_0^{\ast },{\sigma }^{\ast 2}\right),$$ (1)

where ${\mu }_{ij}^{\ast }=p_{ij}^{\ast }= \text{Pr}\left(Y_{ij}=1|{\mathit{\beta }}^{\ast },{\gamma }_i^{\ast },{\tilde{Z}}_{ij}\right)$ is the expected readmission for patient index discharge j = 1, 2, …,n_i in provider i = 1, 2,…,F, and Y_ij = 1 if the jth index discharge at provider i results in a 30-day readmission, and equals 0 otherwise. Also, $g\left(p_{ij}^{\ast }\right)=\text{log}\left\{p_{ij}^{\ast }/\left(1-p_{ij}^{\ast }\right)\right\}$ is the logit function. Model (1) adjusts for patient case-mix variables, such as patient baseline and admission characteristics, denoted by the vector of r covariates ${\tilde{Z}}_{ij}=\left(V_{ij1},\dots ,V_{ij{r}_1},X_{ij1},\dots ,X_{ij{r}_2}\right)\equiv \left(V_{ij},X_{ij}\right)$ and their effects denoted by ${\mathit{\beta }}^{\ast \text{T}}={\left({\beta }_{v,1}^{\ast },\dots ,{\beta }_{v,r_1}^{\ast },{\beta }_{x,1}^{\ast },\dots ,{\beta }_{x,r_2}^{\ast }\right)}^{\text{T}}\equiv {\left({\mathit{\beta }}_v^{\ast },{\mathit{\beta }}_x^{\ast }\right)}^{\text{T}}$ (with r = r₁ + r₂).

In contrast, in the context of providers as dialysis facilities, Kalbfleisch and Wolfe (2013) and He et al. (2013) proposed modeling providers’ effects with fixed effects (γ_1,…, γ_F):

$$g\left({\mu }_{ij}^{\ast }\right)={\gamma }_i^{\ast }+{\mathit{\beta }}^{\ast \text{T}}{\tilde{Z}}_{ij},i=1,\dots ,F.$$ (2)

For clarity, we emphasize that the FE model (2) is a single simultaneous model for all F providers with a high-dimensional parameter space and not a separate logistic regression model for each provider. We also note that for the FE model (2), in the context of performance assessment of dialysis facilities, further adjustment through inclusion of a hospital RE was considered in He et al. (2013), although it was found that the contribution of the hospital effect was small. In sorder to compare RE and FE models and for the results to be more applicable to different provider settings, we consider the FE model in (2) in this work.

2.2. Measurement error in patient case-mix variables

In practice, to date, it is assumed that the patient case-mix variables, ${\tilde{Z}}_{ij},$ are measured without error and no methodological study has been conducted to examine the effect of ME on estimation and inference in profiling.

To examine the impact of ME on RE/FE profiling models (1) and (2), define the vector of observed case-mix variables Z_ij = (V_ij1,…, V_ijr1, W_ij1,…, W_ijr2) ≡ (V_ij, W_ij) and their corresponding coefficients by β^T = (β_υ,1,…, β_υ,r1, β_w,1,…, β_w,r2)^T ≡ (β_υ, β_w)^T. For generality, let r₁ of the case-mix variables, namely V_ijℓ, ℓ = 1,…, r₁, to be measured without ME, while a subset of r₂ case-mix variables (W_ijℓ, ℓ = 1,…, r₂) are assumed to be ascertained with ME (e.g., patient comorbidities from administrative claims data). Thus, the observable RE and FE profiling models in practice are, respectively:

$$g\left({\mu }_{ij}\right)={\gamma }_i+{\mathit{\beta }}^{\text{T}}{Z}_{ij},{\gamma }_i~N\left({\gamma }_0,{\sigma }^2\right),$$ (3)

and

$$g\left({\mu }_{ij}\right)={\gamma }_i+{\mathit{\beta }}^{\text{T}}{Z}_{ij},i=1,\dots ,F,$$ (4)

where μ_ij, γ_i, γ₀, and σ² are analogously defined as above in Section 2.1.

Throughout we consider the classical ME model, W_ijℓ = X_ijℓ + U_ijℓ, where U_ijℓ’s are independent measurement errors in the ℓth case-mix variable that is continuous. The degree of ME can be quantified by the reliability ratio $\lambda = \text{var}(X)/\text{var}(W)={\sigma }_x^2/\left({\sigma }_x^2+{\sigma }_u^2\right).$ In the case of error-prone categorical case-mix variables, the ME is in the form of misclassification. For example, with the common binary patient cormorbidities, consider the probability mass function of (X,W): p_ab ≡ Pr(W_ijℓ = a|X_ijℓ = b) with a,b = 0,1. The amount of ME for misclassification is summarized by the sensitivity and specificity of the case-mix variable W_ijℓ, respectively: sn_ℓ = Pr(W_ijℓ = 1|X_ijℓ = 1) and sp_ℓ = Pr(W_ijℓ = 0|X_ijℓ = 0).

It is well studied in the classical ME literature that for generalized linear models ME leads to biased estimates of coefficients (estimates of {γ_i, β} do not target {${\gamma }_i^{\ast },$ β*}). However, how the quantities of interest in profiling, namely SRR, are impacted by ME is unknown. Also, how inference (e.g., identifying under-performing providers) is impacted by ME is also not known. We summarize these estimation and inference procedures for RE and FE models below.

2.3. Standardized readmission ratio

The main estimate of interest reported in profiling is the provider-specific standardized readmission ratio (SRR). The SRR for provider i is estimated as

$$SR{R}_i=\frac{{\sum }_{j=1}^{n_i}{\widehat{p}}_{ij}}{{\sum }_{j=1}^{n_i}{\widehat{p}}_{M,ij}},$$ (5)

where $\widehat{{\gamma }_i}$ is the provider effect estimate, $\widehat{\mathit{\beta }}$ is the case-mix effect estimate, ${\widehat{p}}_{ij}=g^{-1}\left({\widehat{\gamma }}_i+{\widehat{\mathit{\beta }}}^{\text{T}}{Z}_{ij}\right)$ is the estimated probability of readmission for patient j in provider i and ${\widehat{p}}_{M,ij}=g^{-1}\left({\widehat{\gamma }}_M+{\widehat{\mathit{\beta }}}^{\text{T}}{Z}_{ij}\right).$ For the FE model, ${\widehat{\gamma }}_M$ in the denominator is taken to be the median of the ${\left\{{\widehat{\gamma }}_i\right\}}_{i=1}^F$ and for the RE model it is the estimated mean of the distribution of γ_i (namely ${\widehat{\gamma }}_0$). The numerator of SRR_i is the expected total number of readmissions for facility i and the denominator is the expected total number of readmissions for an “average” provider (taken over the population of all providers), adjusted for the particular case-mix of the same patients in facility i. A SRR_i that is significantly larger or smaller than 1 indicates that provider i is under- or overperforming relative to the reference norm.

When there is no ME in the patient case-mix variables, the SRR is

$$SR{R}_i^{\ast }=\frac{{\sum }_{j=1}^{n_i}{\widehat{p}}_{ij}^{\ast }}{{\sum }_{j=1}^{n_i}{\widehat{p}}_{M,ij}^{\ast }},$$ (6)

Where ${\widehat{p}}_{ij}^{\ast }=g^{-1}\left({\widehat{\gamma }}_i^{\ast }+{\widehat{\mathit{\beta }}}^{\ast \text{T}}{\tilde{Z}}_{ij}\right)$ and ${\widehat{p}}_{M,ij}^{\ast }=g^{-1}\left({\widehat{\gamma }}_M^{\ast }+{\widehat{\mathit{\beta }}}^{\ast \text{T}}{\tilde{Z}}_{ij}\right).$ In this case, $SR{R}_i^{\ast }$ estimates the true/theoretical quantity ${\widetilde{SRR}}_i^{\ast }={\sum }_{j=1}^{n_i}{p}_{ij}^{\ast }/{\sum }_{j=1}^{n_i}{p}_{M,ij}^{\ast },$ where $p_{ij}^{\ast }=g^{-1}\left({\gamma }_i^{\ast }+{\mathit{\beta }}^{*\text{T}}{\tilde{Z}}_{ij}\right)$ and $p_{M,ij}^{\ast }=g^{-1}\left({\gamma }_M^{\ast }+{\mathit{\beta }}^{\ast \text{T}}{\tilde{Z}}_{ij}\right),$ particularly for the FE model. (RE models provide biased estimates of SRR even in the absence of case-mix ME.) We illustrate in Section 4 that even though all model parameters (γ_M, ${\left\{{\gamma }_i\right\}}_{i=1}^F,$ and β) are estimated with bias, this bias does not affect the estimation of SRR in either model.

2.4. Estimation and inference procedures

The RE model is a standard generalized linear mixed effects model for which available software can be used, including SAS PROC GLIMMIX or R library lme4 function glmer. The CMS implementation uses SAS PROC GLIMMIX (Ross et al. 2010; Horwitz et al. 2011). For the FE model estimation, He et al. (2013) proposed an iterative algorithm that alternates between estimation of {γ_i} given β and estimation of β given {γ_i} using one-step Newton-Raphson updates. Iteration terminates when max $\Vert p_{ij}^{(t+1)}-p_{ij}^{(t)}\Vert <{10}^{-6}$ on successive steps t. See He et al. (2013) for details. R codes for fitting both RE and FE models are provided as supplemental materials at http://faculty.sites.uci.edu/nguyenlab/supplement/. We note that in this work, references to ‘RE’ and ‘FE’ methods refer specifically to the models described by (1 and 3) and (2 and 4) together with the corresponding inference procedures described below.

To understand the impact of ME on the ability to identify under-performing providers and providers not different from the national reference, we provide some details on the inference procedures for the RE and FE models. First, for the RE model, a bootstrap resampling of providers with replacement (500 samples) is used to obtain a 95% confidence interval (CI) for each ${\widetilde{SRR}}_i.$ The REs sampled from the posterior distribution of γ_i are used to estimate ${\widetilde{SRR}}_i$ in each bootstrap sample (Horwitz et al. 2011). Provider i is flagged as performing worse than expected, relative to the reference, if the lower confidence limit is above 1. Similarly, a provider is identified as performing better than the reference when the upper confidence limit is below 1. Providers with CI containing 1 are considered not different from the reference.

Steps of the RE inference procedure (Ash et al. 2012) are provided below. Note that the sampled provider random effect, ${\widehat{\gamma }}_i^{(b)\ast }$ for facility i, is sampled from the posterior distribution in step 3 in each bootstrap iteration, which is based on case-mix variables with ME. As illustrated in Section 4, this results in wider confidence intervals on average.

RE Inference: Provider-Specific Bootstrap Confidence Interval
0. Fit the RE model (3). The provider-specific estimates are denoted by ${\widehat{\gamma }}_i,i=1,2,\dots ,F,$
with overall mean ${\widehat{\gamma }}_0.$ Also, denote the variance and patient case-mix estimates by ${\widehat{\sigma }}^2$
and $\widehat{\mathit{\beta }},$ respectively. Calculate SRRi as given by (5).
1. Generate a bootstrap dataset by sampling F providers with replacement from the original
dataset. Denote the unique set of providers sampled by ${\mathbb{F}}^{(b)},$ where b indexes the
bootstrap dataset.
2. Fit the RE model (3) to the bootstrap dataset and treat each resampled provider as
distinct. Calculate:
(a) The patient case-mix effects, ${\widehat{\mathit{\beta }}}^{(b)}.$
(b) The mean and variance of the distribution of the provider effects, ${\widehat{\gamma }}_0^{(b)},$ and ${\widehat{\sigma }}^{2(b)}.$
(c) The provider-specific effects and variances, $\left\{{\widehat{\gamma }}_i^{(b)},{\widehat{ \text{Var} }}^{(b)}\left({\gamma }_i\right)\right\},$ i = 1, 2,…,F. (If a
provider is sampled more than once, then randomly select one set of the provider-
specific estimates and variances.)
3. Generate a provider RE from the provider-specific distribution from step 2(c) for each
unique provider, $i\in {\mathbb{F}}^{(b)}.$ The posterior distribution of each RE is approximated by a
normal distribution, ${\widehat{\gamma }}_i^{(b)\ast }~N\left({\widehat{\gamma }}_i^{(b)},{\widehat{\text{Var}}}^{(b)}\left({\gamma }_i\right)\right).$
4. Calculate SRRi for each unique provider i sampled in step 1:
$SR{R}_i^{(b)}={\sum }_{j=1}^{n_i}{\widehat{p}}_{ij}^{(b)}/{\sum }_{j=1}^{n_i}{\widehat{p}}_{M,ij}^{(b)}={\sum }_{j=1}^{n_i}{g}^{-1}\left({\widehat{\gamma }}_i^{(b)*}+{\widehat{\mathit{\beta }}}^{{(b)}^{\text{T}}}{Z}_{ij}\right)/{\sum }_{j=1}^{n_i}{g}^{-1}\left({\widehat{\gamma }}_0^{(b)}+{\widehat{\mathit{\beta }}}^{{(b)}^{\text{T}}}{Z}_{ij}\right),\text{\hspace{0.17em}for\hspace{0.17em}}i\in {\mathbb{F}}^{(b)}.$
5. Repeat the bootstrap procedure, step 1 - step 4, 500 times (b = 1,…,500) and form the
95% CI estimate of ${\widetilde{SRR}}_i$ for each provider i = 1, 2,…,F.

For the FE model, inference for ${\widetilde{SRR}}_i$ can be equivalently based on the (fixed effect) provider specific parameters γ_i (for each i = 1,…, F). More specifically, FE inference is based on testing the hypothesis H₀ : γ_i= γ_M (i.e., ${\widetilde{SRR}}_i$ = 1), accounting for the high-dimensional FE parameters. He et al. (2013) proposed a method based on resampling under the null hypothesis to evaluate the p-value, the “probability that a given provider would experience a number of readmissions as least as extreme as that observed if the null hypothesis is true, accounting for the provider’s patient case-mix.” Details are provided in the FE inference procedure below (He et al. 2013). In step 2, note that the Bernoulli draws $Y_{ij}^{(b)}$ under the null hypothesis are based on the median provider and patient case-mix effect estimates, ${\widehat{\gamma }}_M$ and $\widehat{\mathit{\beta }},$ estimated from case-mix variables with ME. This results in higher variance in the resampled test statistic, $Y_{i\cdot }^{(b)},$ under the null. As illustrated in Section 4, this results in reduced sensitivity to detect under-performing providers.

FE Inference: Provider-Specific Hypothesis Testing
1. Estimate FE model (4) and obtain estimates $\widehat{\mathit{\beta }}$ and ${\widehat{\gamma }}_M.$
2. For the ith provider, draw B = 500 samples, ${\left\{Y_{ij}^{(b)}:j=1,2,\dots ,n_i\right\}}_{b=1}^B$ under the null
hypothesis, where each observation is independently drawn from a Bernoulli distribution:
$Y_{ij}^{(b)}~Ber\left({\overline{p}}_{ij}\right),$ where ${\overline{p}}_{ij}=\text{exp}\left({\widehat{\gamma }}_M+{\widehat{\mathit{\beta }}}^{\text{T}}{Z}_{ij}\right)/\left\{1+\text{exp}\left({\widehat{\gamma }}_M+{\widehat{\mathit{\beta }}}^{\text{T}}{Z}_{ij}\right)\right\}.$
3. Calculate the total number of readmissions in the resampled data: $Y_{i\cdot }^{(b)}={\sum }_{j=1}^{n_i}{Y}_{ij}^{(b)}.$
4. Calculate the p-value for testing H0 : γi = γM as follows. Compute $S{L}_i^{+}\equiv B^{-1}{\sum }_{b=1}^B\left[0.5\mathbb{I}\left(Y_{i\cdot }^{(b)}=O_i\right)+\mathbb{I}\left(Y_{i\cdot }^{(b)}>O_i\right)\right],$
where Oi is the observed number of readmissions for provider i in the original
data and $\mathbb{I}(\cdot )$ denotes the indicator function; similarly, compute $S{L}_i^{-}\equiv B^{-1}{\sum }_{b=1}^B\left[0.5\mathbb{I}\left(Y_{i\cdot }^{(b)}=O_i\right)+\mathbb{I}\left(Y_{i\cdot }^{(b)}<O_i\right)\right].$
The p-value is $P=2\times \text{min}\left\{S{L}_i^{+},S{L}_i^{-}\right\}.$
5. Repeat steps 2 – 4 for each provider, i = 1, 2,…,F.

3. Simulation studies

We consider 4 simulation models to address two specific aims: Determine the impact of patient case-mix ME on (1) estimation of SRR and (2) inference procedures in both RE and FE models. The studies provide answers to the following questions: (1) Does the inclusion of error-prone case-mix variables bias the estimation of provider-specific SRRs and as a consequence invalidate their interpretation/use? (2) How does the inclusion of error-prone case-mix variables affect inference, specifically, the ability to identify under-performing providers? The 4 simulation models (SM1-SM4) described below includes a simple set-up with two continuous variables (SM1), a more general dependence structure with 15 variables (SM2), a simple misclassification variable (SM3), and a more general model with 30 case-mix covariates based on the USRDS data that includes both ME on continuous variables and discrete variables.

3.1. Simulation model 1: Two continuous case-mix variables

We begin with the following simple, but instructive, basic profiling model without ME:

$$g\left({\mu }_{ij}^{\ast }\right)={\gamma }_i^{\ast }+{\gamma }_M^{\ast }+{\beta }_{v,1}^{\ast }{V}_{ij1}+{\beta }_{x,1}^{\ast }{X}_{ij1},$$ (SM1*)

with i = 1,…,F = 1, 000 providers and two continuous case-mix variables, V_ij₁ and X_ij₁, measured without error. For simplicity, we take both covariates to be normally distributed with means μ_V = 0, μ_X = 2, and variances ${\sigma }_x^2={\sigma }_v^2=1,$ and with correlation ρ(V_ij₁, X_ij₁) = 0.3. The variable V_ij1 is assumed to be measured without error. The unobserved variable is X_ij1 and the corresponding observed variable is $W_{ij1}=X_{ij1}+U_{ij1},\text{ where }{U}_{ij1}~N\left(0,{\sigma }_u^2\right)$ as described in Section 2.2. Thus, the observable profiling model is

$$g\left({\mu }_{ij}\right)={\gamma }_i+{\gamma }_M+{\beta }_{v,1}{V}_{ij1}+{\beta }_{w,1}{W}_{ij1}.$$ (SM1)

We considered two levels of ME by setting ${\sigma }_u^2={0.75}^2$ (lower ME: 100 × (1 – λ)% = 36%) and ${\sigma }_u^2=2^2$ (higher ME: 80%). We parametrized model (SM1*) to include ${\gamma }_M^{\ast }$ so that the baseline rates of readmission (BRR) can be varied. BRR of 14.3%, 27.3%, and 41.7% are referred to as low, medium and high, corresponding to ${\gamma }_M^{\ast }=\text{log}\left(1/6\right),\text{\hspace{0.17em}log}\left(3/8\right),$ and log(5/7), respectively.

Provider-specific effects, ${\gamma }_i^{\ast },$ were set as follows. Among the 1,000 providers, 2.5% were under-performers (worse: ${\mathcal{I}}_W=\left\{1,\dots ,25\right\}$), 2.5% were over-performers (better: ${\mathcal{I}}_B=\left\{26,\dots ,50\right\}$), and the remaining 95% of providers, with effects not different (ND: ${\mathcal{I}}_{ND}=\left\{51,\dots ,1000\right\}$) from the national reference, were generated from a N(0,σ²) distribution with σ² = 0.2². Providers that were truly different from the national reference were generated as ${\gamma }_i^{\ast }$ ~ Uniform(0.4, 1.5) for $i\in {\mathcal{I}}_W$ and ${\gamma }_i^{\ast }$ ~ −Uniform(0.4,1.5) for $i\in {\mathcal{I}}_B.$ (We note here that when all provider effects were generated from N(0, σ²) and true outlying providers were defined as the 2.5% in the tails of the distribution, the results were similar.)

Because the impact of ME on profiling will depend on the provider effect size (P-ES) and the patient case-mix effect size (CM-ES), we considered two P-ES and three CM-ES settings. The need for this is clear when one considers the extreme cases. For example, if providers have large effects on whether patients have unplanned 30-day readmission relative to the contribution from patient factors, then ME on the patient case-mix variables will have negligible effects on the outcome. On the other hand, if patient risk factors and comorbidities are the dominating effects on readmission (relative to a small P-ES), then ME can have a large effect on the outcome. Therefore, we considered (P-ES 1): ${\gamma }_i^{\ast }$ ~ Uniform(0.4,1.5) for $i\in {\mathcal{I}}_W$ and ${\gamma }_i^{\ast }$ ~ −Uniform(0.4,1.5) for $i\in {\mathcal{I}}_B;$ (P-ES 2): ${\gamma }_i^{\ast }$ ~ Uniform(0.6,1.5), for $i\in {\mathcal{I}}_W$ and ${\gamma }_i^{\ast }$ ~ −Uniform(0.6,1.5) for $i\in {\mathcal{I}}_B$ where provider signals have been increased. For patient case-mix effect size (CM-ES) we considered the following three increasing CM-ES settings: ${\beta }_{v,1}^{\ast }={\beta }_{x,1}^{\ast }$ = 0.5, 1, and 2 (CM-ES1, CM-ES2, and CM-ES3, respectively).

We also considered the case where worse and better performing provider effects were generated as fixed constants across Monte Carlo datasets so that each provider’s average bias in estimation can be characterized (see Section 4.1). For this, under the P-ES1 setting, the under-performing provider effects were set to ${\gamma }_i^{\ast }$ ∈ {0.40, 0.46,…, 1.5}, for $i\in {\mathcal{I}}_W.$ For over-performing providers, ${\gamma }_i^{\ast }$ ∈ {−1.50, −1.45,…, −0.40}, for $i\in {\mathcal{I}}_B.$ Similarly, under the P-ES2 setting: ${\gamma }_i^{\ast }$ ∈ {0.60, 0.64,…, 1.50}, for $i\in {\mathcal{I}}_W$ and ${\gamma }_i^{\ast }$ ∈ {−1.50, −1.46,…, −0.60}, for $i\in {\mathcal{I}}_B.$

The generated data consisted of provider volume ranging from 42 to 210 patients on average. More specifically, the number of patients were generated from a truncated Poisson distribution, where $n_i={\sum }_{h=1}^{1000}{m}_{ih}1\left\{m_{ih}\le 7\right\},$ were m_ih ~ Poisson(15). This process mimics the sparse data structure of patient discharges across U.S. dialysis facilities; see He et al. (2013) for details. From n_i, we defined small, medium and large sized providers by tertile (small: 42–103; medium: 104–126: large: 127–210 patients on average). Two hundred datasets, each with 1,000 providers, were generated for each simulation study scenario.

3.2. Simulation model 2: More general dependence structure

We extended the simulation model (SM1*) to include 10 error-free case-mix variables and 5 variables with ME, a generalization that allowed for investigation of a more general dependence structure among all case-mix variables as is typical in real data applications. More specifically,

$$g\left({\mu }_{ij}^{\ast }\right)={\gamma }_i^{\ast }+{\gamma }_M^{\ast }+{\beta }_{v,1}^{\ast }{V}_{ij1}+\cdots +{\beta }_{v,10}^{\ast }{V}_{ij10}+{\beta }_{x,1}^{\ast }{X}_{ij1}+\cdots +{\beta }_{x,5}^{\ast }{X}_{ij5},$$ (SM2*)

with i = 1,…, F = 1,000 providers. As with model (SM1*), the observed variables with error are $W_{ij\mathcal{l}}=X_{ij\mathcal{l}}+U_{ij\mathcal{l}},$ where $U_{ij\mathcal{l}}~N\left(0,{\sigma }_u^2\right),$ for $\mathcal{l}$ = 1,…,5, leading to the observable model

$$g\left({\mu }_{ij}\right)={\gamma }_i+{\gamma }_M+{\beta }_{v,1}{V}_{ij1}+\cdots +{\beta }_{v,10}{V}_{ij10}+{\beta }_{w,1}{W}_{ij1}+\cdots +{\beta }_{w,5}{W}_{ij5}.$$ (SM2)

The case-mix vector, ${\tilde{Z}}_{ij}=\left({\tilde{Z}}_{ij1},\dots ,{\tilde{Z}}_{ij15}\right)=\left(V_{ij},X_{ij}\right)$ in (SM2*), was generated from a multivariate normal distribution with means 0, variances 1, and correlation ${\rho }_{\mathcal{l}{\mathcal{l}}^{\prime }}\equiv \rho \left({\tilde{Z}}_{ij\mathcal{l}},{\tilde{Z}}_{ij{\mathcal{l}}^{\prime }}\right),1\le \mathcal{l},{\mathcal{l}}^{\prime }\le 15.$ Using the magnitude of correlations observed in USRDS data as a guide, we considered a more general dependence among variables that was generated in 3 blocks with different correlation structures: $0.01\le {\rho }_{\mathcal{l}{\mathcal{l}}^{\prime }}\le 0.05$ in block 1 for variables ${\tilde{Z}}_1$ to ${\tilde{Z}}_5;0.05\le {\rho }_{\mathcal{l}{\mathcal{l}}^{\prime }}\le 0.1$ in block 2 for variables ${\tilde{Z}}_6$ to ${\tilde{Z}}_{10};$ and $0.1\le {\rho }_{\mathcal{l}{\mathcal{l}}^{\prime }}\le 0.25$ in block 3 for variables ${\tilde{Z}}_{11}$ to ${\tilde{Z}}_{15}.$ Variables across blocks were also correlated in the range of $0.01\le {\rho }_{\mathcal{l}{\mathcal{l}}^{\prime }}\le \mathrm{0.25.}$ Provider effects were the same as in model (SM1*). Low, medium and high CM-ES were set as: (CM-ES1) β_x,l = 0.5, (CM-ES2) β_x,l = 1, and (CM-ES3) β_x,l = 2, for ℓ = 1,…, 5. Effects of the 10 error-free covariates were ${\beta }_{v,1}^{\ast }=\cdots ={\beta }_{v,5}^{\ast }=0.25$ and ${\beta }_{v,6}^{\ast }=\cdots ={\beta }_{v,10}^{\ast }=\mathrm{0.5.}$

3.3. Simulation model 3: Simple case-mix misclassification

Simulation model 3 considers a simple error-prone categorical case-mix variable, where the ME is in the form of misclassification. Model 3 is as defined above for (SM1*):

$$g\left({\mu }_{ij}^{\ast }\right)={\gamma }_i^{\ast }+{\gamma }_M^{\ast }+{\beta }_{v,1}^{\ast }{V}_{ij1}+{\beta }_{x,1}^{\ast }{X}_{ij1},$$ (SM3*)

and

$$g\left({\mu }_{ij}\right)={\gamma }_i+{\gamma }_M+{\beta }_{v,1}{V}_{ij1}+{\beta }_{w,1}{W}_{ij1},$$ (SM3)

where (X_ij1,W_ij1) are binary covariates with joint distribution given by p_ab = Pr(W_ijℓ = a|X_iji = b) with a, b = 0,1. For low ME, we set p₀₀ = 0.42, p₀₁ = 0.18, p₁₀ = 0.12, and p₁₁ = 0.28; thus, the case-mix classification sensitivity (sn) and specificity (sp) are sn = 0.7 and sp = 0.6. For the higher ME, we set p₀₀ = 0.15, p₀₁ = 0.25, p₁₀ = 0.5, and p₁₁ = 0.1, resulting in lower sn = 0.167 and sp = 0.375. The covariate, V_ij1, measured without error is correlated with X_ij1 and generated through (V_ij1|X_ij1 = 0) ~ N(0,1) and (V_ij1|X_ij1 = 1) ~ N(1,1.5²).

3.4. Simulation model 4: USRDS data characteristics

We also considered a fourth simulation study (SM4*/SM4) more tailored to the assessment of dialysis facilities. Paralleling works for profiling all-cause readmission for hospitals (CMS 2014; Horwitz et al. 2011), assessment of dialysis facilities included the following 30 patient case-mix covariates: Variables 1–4: age, body mass index (BMI), length of index hospitalization (days), time on dialysis (years); variable 5: sex; variables 6–7: high risk index hospitalization, diabetes as the cause of ESRD; and variables 8–30: 23 past-year comorbidities, which includes amputation status; chronic obstructive pulmonary disease; cardiorespiratory failure/shock; coagulation defects and other specified hematological disorders; drug and alcohol disorders; end-stage liver disease; fibrosis of lung or other chronic lung disorders; hemiplegia, paraplegia, paralysis; hip fracture/dislocation; major organ transplants; metastatic cancer; other hematological disorders; other infectious disease and pneumonias; other major cancers; pancreatic disease; psychiatric comorbidity; respirator dependence; rheumatoid arthritis and inflammatory connective tissue disease; seizure disorders; septicemia/shock; severe cancer; severe infection; and ulcers (CMS 2014). Our focus on ME is on the 23 past-year comorbidities, although for simplicity of exposition we also include variables 6–7 as potentially error-prone; thus, Xij = X_ij1,…,X_ij25.

To generate the binary covariates, X_ij, we consider underlying latent continuous variables L_ij = (L_iji,…, L_ij25) ~ N₂₅ (μ_L, ∑_L), with ∑_L = (σℓ, ℓ’) and μ_L= (μ₁,…, μ₂₅) chosen to be the observed covariance matrix and the means/prevalences for binary covariates based on USRDS data, respectively. The binary covariates X_ijℓ, ℓ = 1,…, 25, were then generated through the process: X_ijℓ = 1{L_ijℓ < Z_ℓσ_ℓ,ℓ + μ_ℓ}, where 1{A} is the indicator function for event A, z_ℓ = Φ⁻¹(μ_ℓ) and Φ⁻¹() is the standard normal inverse CDF. This process generates binary covariates (e.g., patient comorbidities) with prevalences equal to the corresponding observed prevalences in the USRDS data (namely μ_L). Next, the observed binary covariates with ME, W_ij1,…, W_ij25, were then generated based on the joint distributions of (X_ijℓ, W_ijℓ) for a range of misclassification sensitivities and specificities (su_ℓ, sp_ℓ), as described in Section 2.2. The range of sn_ℓ, sp_ℓ used was from 0.5 to 0.8.

The remaining 4 continuous variables and sex (V_ij1,…, V_ij5) were generated from a N₄(μ_V, ∑_V) and a Ber(0.48) distribution, where μ_z and ∑_z were the observed mean and covariance based on the USRDS data. The distribution of facility effect sizes, $\left\{{\gamma }_i^{\ast }\right\},$ was modeled as ${\gamma }_i^{\ast }$ ~ N(0, 0.2²) for facilities $i\in {\mathcal{I}}_{ND}$ and outlying facilities were generated as ${\gamma }_i^{\ast }$ ~ Uuiform(0.6, 2) for $i\in {\mathcal{I}}_W$ and ${\gamma }_i^{\ast }$ ~ Uniform(−2, −0.6) for $i\in {\mathcal{I}}_B$ under the P-ES2 setting. For the P-ES1 setting, the distributions were ±Uniform(0.4, 2). The patient case-mix effects, ${\mathit{\beta }}^{\text{T}\ast }=\left({\beta }_1^{\ast },\dots ,{\beta }_{30}^{\ast }\right),$ were set to be proportional to the estimates based on the USRDS data.

4. Results

4.1. Estimation of SRR

We present results based on the simple simulation case with two continuous case-mix variables (SM1* and SM1) in more details since the results were similar across simulation studies. First, the model coefficients from both RE and FE models are biased. This includes the biased estimated coefficients for case-mix variables measured with or without ME (${\widehat{\beta }}_x,{\widehat{\beta }}_z$) and provider effect estimates (${\widehat{\gamma }}_M,{\widehat{\gamma }}_i,i=1,\dots ,50;$ Figure 1). Results given are for the case with β_x = β_z = 2. As expected, the magnitude of the bias in the provider-specific estimates, ${\widehat{\gamma }}_i,$ increases with larger case-mix effect size (CM-ES1 to CM-ES3) and with higher level of ME (low [36%] vs. high [80%] ME) for a given CM-ES); see Figure 2 for the FE model. More generally, the magnitude of estimation bias depends on the provider effect size, case-mix effect size and the level of ME. For instance, when the CM-ES is small relative to the provider-specific effect size, the impact of ME is not detectable (e.g., see row 1 of Figure 2). (Results are similar for the RE model; see supplemental Figure S1.) We note that although high-dimensional FE models with covariates measured with error has not been examined in the ME literature to date, biased coefficients are expected similar to ME in generalized linear models.

Figure 1:

Bias case-mix coefficients estimates (${\widehat{\beta }}_z,{\widehat{\beta }}_x;$ row 1), overall “mean” provider effect estimates (${\widehat{\gamma }}_M;$ row 2), and provider-specific estimates (${\widehat{\gamma }}_i;$ row 3) due to measurement error for FE and RE models. (FE: fixed effects; RE: random effects; ME: measurement error)

Figure 2:

Bias of provider-specific estimates (${\widehat{\gamma }}_i$) of fixed effects (FE) models as a function of case-mix effect size (CM-ES 1 to CM-ES 3 corresponding to β_z= β_x = 0.5, 1, and 2) and the level of measurement error (ME: low or high ME).

However, biased model coefficients due to ME do not affect the estimation of SRR. Figure 3 shows the average SRR estimates for truly outlying providers (under- and over-performing providers under P-ES1) with true SRR ranging from 0.6 to 1.4. For the FE model (Figure 3), the SRR estimates under ME (dashed line) coincides with SRR estimates when using the true case-mix variables without ME (dotted line). Because FE models provide (asymptotically) unbiased estimates, these coincide with the true SRR (solid gray line). For the RE model (Figure S2), the results show that SRR estimates under RE models with (dashed) and without (dotted) ME also coincide. However, we note that these RE estimates do not target the true SRR, particularly in the tails of the distribution of SRR, because RE are generally biased shrinkage estimates (Kalbfleisch and Wolfe, 2013), regardless of whether case-mix variables are measured with or without error. The results hold with low (36%) or high (80%) amount of ME. (The pattern of results are similar for larger provider effect size, P-ES2; not shown.)

Figure 3:

Provider-effect estimates, SRR_i, for fixed effects (FE) models with and without measurement error (column 1 - low ME, column - high ME) and across case-mix effect sizes (CM-ES).

Thus, biased estimates of case-mix effects and provider-specific estimates do not translate to biased SRR estimates. However, we note that the variance in the estimates of SRR are higher under case-mix ME (results not shown). Increased variation as a consequence of covariate ME in regression models is also known in the classical ME setting (Carroll et al. 2006).

4.2. Inference: Identifying outlying providers

4.2.1. Overall impact of measurement error on profiling

To describe the impact of case-mix ME on profiling, we focus on the primary profiling goal of correctly identifying providers that under-perform (sensitivity “worse”: SEN-W) relative to the reference standard, and specificity (SPEC). SPEC refers to the correct identification of providers that are not different (ND) compared to the reference. Because provider assessment policies focus on identifying under-performing providers, we focus on the results for SEN-W and SPEC-ND. (Sensitivity “better” was similar to SEN-W in our studies due to symmetry.)

Overall, the studies consistently found that although the estimation of SRR is not affected by ME, both FE and RE models have substantially reduced power to detect truly underperforming providers. Because the overall patterns of results are similar across studies, we present in more detail the results for model SM1 with 2 continuous case-mix covariates. Figure 4 illustrates the general patterns of the impact of ME on the both FE and RE models. Row 1 of Figure 4 shows the average SEN-W for identifying under-performing providers as a function of case-mix effect size (β = 0.5,1, 2) for the case of low ME. The results support the following with respect to both FE and RE models: (1) The presence of case-mix ME does negatively impact profiling. (2) The reduction in the ability to identifying under-performing providers due to ME depends on the CM-ES. (3) These results are consistent across the two P-ES settings (low: black, high: gray curves) and also for the high level of ME (not shown). For example, average SEN-W was 74.3% and 62.0% for FE and RE models without ME, respectively. These SEN-W averages dropped to 69.2% and 52.3%, respectively, for FE and RE models with ME. Furthermore, note that when the case-mix effect is smallest (i.e., β = 0.5), average sensitivities were not different between the models using case-mix variables with and without ME. However, this should not be assumed to be the case in practice because case-mix variables, such as comorbidities, can have large effects on a patient’s likelihood of readmission.

Figure 4:

Row 1: Overall average flagging performance/sensitivity for identifying underperforming providers as a function of case-mix effect size (CM-ES: β= 0.5,1, 2), averaged over 200 simulated datasets. Given are results for FE and RE models when patient case-mix variables are ascertained without measure error (w/o ME; dotted), with ME (w/ME; dashed), and for provider effect sizes (P-ES1, P-ES2: low, high). Row 2: Distribution of sensitivity for identifying truly under-performing providers for the case of β = 2.

The specificity (SPEC-ND), i.e., rate of correctly identifying the providers that do not differ from the reference standard, as a function of CM-ES is summarized in Figure S3 for the case of low ME. As expected, the reduction in sensitivity for identifying outlying providers in the presence of ME led to a slight increase in SPEC-ND for FE model (e.g., 93.2% vs. 93.9%: w/o vs. w/ME at P-ES1) and RE model (97.2% vs. 98.2%: w/o vs. w/ME at P-ES1).

We note that the overall average flagging performance/sensitivity patterns (with and without ME) as a function of case-mix effect size are similar when examined separately for providers with small, medium, and large volume. This is illustrated in Figure S4 for both FE and RE models under low provider effects size (P-ES1).

Finally, we also examined the confidence interval length and the test level (coverage probability [CP]) for the RE and FE inference procedures, respectively. For the FE model, Figure 5 (row 1) shows the CPs for the hypothesis testing procedure (H₀ : γ_i = γ_M) under no ME and under ME with increasing CM-ES in both high and low levels of ME. CPs do not target the 0.95 level and are further below 0.95 for higher ME and larger CM-ES, as expected. For the RE model, the inference procedure based on 95% CI yielded wider CI length on average under case-mix ME (Figure 5, row 2).

Figure 5:

Coverage probabilities for hypothesis testing procedure use in fixed effects models (row 1) based on 1,000 Monte Carlo datasets. Average length of 95% confidence interval (CI) for SRR based on random effects inference procedure (row 2). (CM-ES: case-mix effect sizes 1–3 are β= 0.5,1, 2; ME: measurement error; SRR: standardized readmission ratio)

4.2.2. More general dependence/correlation and case-mix misclassification

To examine whether the adverse impact of ME holds under a more general profiling model, we consider a model containing 15 continuous case-mix variables with dependence/correlation within and across blocks of variables. The correlation structure consists of unequal correlations among patient case-mix variables (0.01 ≤ ρ_rr’ ≤ 0.25) selected to be similar to USRDS data described in Section 3.1. As illustrated in Figure S5 - column 1, the patterns of results for both RE and FE models were similar to the 2 variables case described above; however, in this more realistic model, the difference in average SEN-W between the models with and without ME is much greater. For example, at CM-ES β = 2 under low ME, the SEN-W for FE models with vs. without ME were 36.0% vs. 55.3% (and similarly, for RE model with vs. without ME: 6.7% vs. 27.9%). The results were similar for high level of ME (FE model with vs. without ME: 27.3% vs. 56.3%; RE model with vs. without ME: 2.1% vs. 25.4%).

Also, Figure S5 - column 2, illustrates the impact of ME in categorical variables, i.e., misclassification of categorical case-mix variables. The impact of misclassification on profiling in this case is similar to the results described earlier for two continuous covariates (Figure 4).

4.2.3. Simulation model based on USRDS data

Profiling results for simulated data modeled after USRDS data are summarized in Figure 6 for baseline readmission rate (BRR) of 27.3% (medium). The results are similar for lower (14.3%) and higher (41.7%) BRR. The general patterns of the impact of ME on the ability to identify truly under-performing providers (SEN-W) is similar to the those described earlier, although the loss in power is more severe. For example, the overall average SEN-W was 85.9% for the FE model without ME and was reduced to an average of 64.1% for case-mix measured with error. For the RE model, the effect of ME was more severe with an average SEN-W of 79.1% vs. 37.8% for models with and without ME, respectively. Not surprisingly, the direction of SPEC-ND is reversed (with ME > without ME) because the majority (190, 000 = 200 × 1000 × 0.95) of providers are truly ND relative to the reference standard.

Figure 6:

Performance of profiling models under USRDS data simulation studies: Fixed effects (FE) model (row 1) and random effects (RE) model (row 2).

5. Discussion

In this work we presented the first systematic study of the impact of measurement error on patient case-mix variables in profiling models. We found that estimates of SRR are valid under ME; therefore, their use is acceptable even under case-mix variables measured with error. However, ME increases variation in the SRR estimates and degrades the power of profiling methods to detect under-performing providers. Although beyond the scope of this work, careful studies to examine/document the degree of ME for all case-mix covariates (e.g., all comorbidities) is warranted to assess their impact. Furthermore, investigation of novel methods to improve the profiling performance under ME is needed. Finally, we note that our work focused on simulation models to understand the basic impact of measurement error on estimation and inference in profiling models. Further studies providing a more thorough statistical/theoretical treatment will contribute more general insights that may lead to additional guidance on the use of profiling models in the context of case-mix measurement error.