Ranking USRDS provider specific SMRs from 1998-2001

Abstract

Provider profiling (ranking/percentiling) is prevalent in health services research. Bayesian models coupled with optimizing a loss function provide an effective framework for computing non-standard inferences such as ranks. Inferences depend on the posterior distribution and should be guided by inferential goals. However, even optimal methods might not lead to definitive results and ranks should be accompanied by valid uncertainty assessments. We outline the Bayesian approach and use estimated Standardized Mortality Ratios (SMRs) in 1998-2001 from the United States Renal Data System (USRDS) as a platform to identify issues and demonstrate approaches. Our analyses extend Liu et al. (2004) by computing estimates developed by Lin et al. (2006) that minimize errors in classifying providers above or below a percentile cut-point, by combining evidence over multiple years via a first-order, autoregressive model on log(SMR), and by use of a nonparametric prior. Results show that ranks/percentiles based on maximum likelihood estimates of the SMRs and those based on testing whether an SMR = 1 substantially under-perform the optimal estimates. Combining evidence over the four years using the autoregressive model reduces uncertainty, improving performance over percentiles based on only one year. Furthermore, percentiles based on posterior probabilities of exceeding a properly chosen SMR threshold are essentially identical to those produced by minimizing classification loss. Uncertainty measures effectively calibrate performance, showing that considerable uncertainty remains even when using optimal methods. Findings highlight the importance of using loss function guided percentiles and the necessity of accompanying estimates with uncertainty assessments.

1 Introduction

Research on and application of performance evaluation steadily increases with applications to evaluating health service providers (Christiansen and Morris 1997; Goldstein and Spiegelhalter 1996; Landrum et al. 2000; Liu et al. 2004; McClellan and Staiger 1999; Grigg et al. 2003; Zhang et al. 2006; Normand and Shahian 2007; Ohlssen et al. 2007), prioritizing environmental assessments in small areas (Conlon and Louis 1999; Louis and Shen 1999; Shen and Louis 2000) and ranking teachers and schools (Lockwood et al. 2002). Inferential goals of these studies include evaluating the population performance, such as the average performance of all health providers and comparing performance among providers. Performance evaluations include comparing unit-specific, substantive measures such as death rates, identifying the group of poorest or best performing units and overall ranking of the units, e.g., profiling or league tables (Goldstein and Spiegelhalter 1996).

The Standardized Mortality Ratio (SMR), the ratio of observed to expected deaths, is an important service quality indicator (Zaslavsky 2001). The United States Renal Data System (USRDS) produces annual estimated SMRs for several thousand dialysis centers and uses these as a quality screen (Lacson et al. 2001; ESRD 2000; USRDS 2005). Invalid estimation or inappropriate interpretation can have serious consequences for these dialysis centers and for their patients. We present an analysis of the information from the United States Renal Data System (USRDS) for 1998-2001 as a platform for demonstrating and comparing approaches to ranking health service providers. From the USRDS we obtained observed and expected deaths for the K = 3173 dialysis centers that contributed information for all four years. The approach used by USRDS to produce these values can be found in USRDS (2005).

Though estimating SMRs is a standard statistical operation (produce provider-specific expected deaths based on a statistical model, and then compute the “observed/expected” ratio), it is important and challenging to deal with complications such as the need to specify a reference population (providers included, the time period covered, attribution of events), the need to validate the model used to adjust for important patient attributes (age, gender, diabetes, type of dialysis, severity of disease), and the need to adjust for potential biases induced when attributing deaths to providers and accounting for informative censoring.

The multi-level data structure and complicated inferential goals require the use of a hierarchical Bayesian model that accounts for nesting relations and specifies both population values and random effects. Correctly specified, the model properly accounts for the sample design, variance components and other uncertainties, producing valid and efficient estimates of population parameters, variance components and unit-specific random effects (provider-, clinician-, or region-specific latent attributes), all accompanied by valid uncertainty assessments. Importantly, the Bayesian approach provides the necessary structure for developing scientific and policy-relevant inferences based on the joint posterior distribution of all unknowns.

As Shen and Louis (1998) show and Gelman and Price (1999) present in detail, no single set of estimates or assessments can effectively address multiple goals and we provide a suite of assessments. Guided by a loss function, the Bayesian approach structures non-standard inferences such as ranking (including identification of extremely poor and good performers) and estimating the histogram of unit-specific random effects. For example, as Liu et al. (2004) show, when estimation uncertainty varies over dialysis centers, ranks produced by Z-scores that test whether a provider's SMR = 1 tend to identify providers with relatively low variance as extreme because these tests have the highest power; ranks produced from the provider-specific maximum likelihood estimates (MLEs) are more likely to identify dialysis centers with relatively high variance as extreme. Effective ranks depend on striking an effective tradeoff between signal and noise.

Lin et al. (2006) present estimates that minimize errors in classifying providers above or below a percentile cut-point. Our analyses build on Liu et al. (2004) by extending the application of Lin et al. (2006)'s estimates to combine evidence over multiple years via a first-order, autoregressive model on log(SMR), and by use of a nonparametric prior. For single-year analyses we compare the results from the log-normal prior to those based on the Non-Parametric, Maximum Likelihood (NPML) prior (Laird 1978).

In following, Sect. 2 presents our models; Sect. 3 outlines several ranking methods; Sect. 4 gives uncertainty measures; Sect. 5 presents results and Sect. 6 sums up and identifies additional research. Computing code for all routines is available at, http://people.umass.edu/rlin/jhuwebhost/usrds-ranking.htm.

2 Statistical models

We employ both single-year and longitudinal models for observed deaths and underlying parameters, with the former a sub-model of the latter. To this end, let (Y_kt, m_kt) be the observed and case-mix adjusted, expected deaths for provider k in year t, k = 1,... 3173, t = 0, 1, 2, 3 and ρ_kt be the SMR. The USRDS computes the expecteds under the assumption that all providers give the same quality of care for patients with identical covariates, see USRDS (2005) for details. We employ the conditional Poisson model,

$$\begin{array}{cc}\hfill Y_{kt}\mid m_{kt},{\rho }_{kt}& ~\text{Poisson}\left({\rho }_{kt}{m}_{kt}\right)\hfill \\ \hfill E(Y_{kt}\mid m_{kt},{\rho }_{kt})& =m_{kt}{\rho }_{kt}.\hfill \end{array}$$ (1)

If the provider has “average performance”, ρ_kt = 1. For both single-year and multiple-year analyses we model θ_kt = log(ρ_kt).

2.1 Single-year analyses

For single-year analyses, we assume that for year t; θ_kt $\stackrel{iid}{~}$ G_t, k = 1,…, 3173: We use a year-specific, normal prior (see the note after Eq. 2) and for the single-year analyses also use the non-parametric maximum likelihood (NPML) prior. See and Carlin and Louis (2008) and Paddock et al. (2006) for additional details and Appendix C for the estimation algorithm.

2.2 The longitudinal, AR(1) model

To model longitudinal correlation among (ρ_k0, ρ_k1, ρ_k2, ρ_k3), let ϕ = cor(θ_k,t, θ_k(t+1)), with -1 < ϕ < 1. Then, use a normal prior on the θ_kt and a normal prior on Z(ϕ) = 0.5 log {(1 + ϕ)/(1-ϕ)} in the hierarchical model,

$$\begin{array}{c}\hfill {\xi }_t\stackrel{iid}{~}N(0,V),{\lambda }_t={\tau }_t^{-2}\stackrel{iid}{~}\text{Gamma}(\alpha ,\mu ∕\alpha )\hfill \\ \hfill Z\left(\varphi \right)~N(0,V_{\varphi })\hfill \\ \hfill [{\theta }_{10},\dots ,{\theta }_{K0}\mid {\xi }_0,{\tau }_0]\stackrel{iid}{~}N({\xi }_0,{\tau }_0^2)\hfill \\ \hfill [{\theta }_{kt}\mid {\theta }_{k(t-1)},\dots ,{\theta }_{k0},\xi ,\tau ,\varphi ]\stackrel{ind}{~}N({\xi }_t+\varphi {\tau }_t{\tau }_{t-1}^{-1}\{{\theta }_{k(t-1)}-{\xi }_{t-1}\},\{1-{\varphi }^2\}{\tau }_t^2)\hfill \\ \hfill [Y_{kt}\mid m_{kt},{\rho }_{kt}]~\text{Poisson}\left(m_{kt}{\rho }_{kt}\right),{\rho }_{kt}=\text{exp}\left({\theta }_{kt}\right).\hfill \end{array}$$ (2)

The notation “iid” means independently and identically distributed and “ind” means independently distributed. The relation is first-order Markov, because though conditioning is on all prior θs, only ρ_k(t-1) appears on the right-hand side of Eq. 2.

Marginally, for year t, θ_kt iid N(ξ_t, ${\tau }_t^2$) and setting ϕ = 0 produces four, single-year analyses, each using the Liu et al. (2004) model with no borrowing of information over time. For ϕ > 0, we have a standard AR(1) model on the latent log(SMR)s and the posterior distribution combines evidence across dialysis centers within year and within dialysis center across years.

2.3 Posterior sampling implementation and hyper-prior parameters

We implement a Gibbs sampler for model (2) with WinBUGS via the R package R2WinBUGS, using the coda package to diagnose convergence (Spiegelhalter et al. 1999; Gelman et al. 2006; Plummer et al. 2006). We use V = 10, μ = 0.01, α = 0.05, values that stabilize the simulation while allowing sufficient adaptation to the data. With V = 10, the a priori, 95% probability interval for ξ_t is (-6.20, 6.20) [(0.002, 492.75) in the SMR scale]; the values for α and μ produce a distribution for τ² with center near 100, inducing large, a priori variation for the θ_kt. For the AR(1) model, reported results are based on the V_ϕ = 0.2. This produces an a priori 95% probability interval for ϕ of (-0.70, 0.70). In a sensitivity analysis, we also tried V_ϕ = 2, which produced the a priori interval (-0.99, 0.99) and yielded results virtually identical to those based on the V_ϕ = 0.2 hyper-prior. In both cases, the data likelihood dominated the priors. This can also be seen in the shrinkage of τ² towards zero, as reported in the Sect. 5.4. There is no strong posterior correlation observed between ϕ and the τ²s.

3 Loss function based ranking methods

Two general strategies for ranking are available. The preferred strategic approach first computes the joint posterior distribution of the ranks and then uses it to produce estimates and uncertainty assessments, generally guided by a loss function that is appropriate for analytic goals. This approach ensures that estimated ranks have desired properties such as not depending on a monotone transform of the target parameters. The other approach is based on ordering estimates of target parameters (MLEs or posterior means) or on ordering statistics testing the null hypothesis that SMR_k ≡ 1. If the posterior distributions of the target parameters are stochastically ordered, then for a broad class of loss functions (estimation criteria) optimally estimated ranks will not depend on the strategy. However, Lin et al. (2006) and others have shown that estimates not derived from the distribution of the ranks can perform very poorly and may not be invariant under monotone transformation of the target parameters. Producing the joint posterior distribution of the ranks is computationally intensive, but most estimates depend only on easily computable features.

We first define ranks and then specify candidate ranking methods. For clarity in defining ranks, we drop the index t and write $R_k\left(\rho \right)=\text{rank}\left({\rho }_k\right)={\sum }_{j=1}^K{I}_{\{{\rho }_k\ge {\rho }_j\}}$, with the smallest ρ_k having rank 1. Rank-based estimates are based on the joint posterior distribution of the R_k(ρ) and are invariant under monotone transform of the ρ_k.

3.1 Squared-error loss

Shen and Louis (1998) and Lockwood et al. (2002) study ranks that minimize the posterior risk induced by squared error loss (SEL): $\mathrm{E}\left[K^{-1}{\sum }_k{(R_k^{est}-R_k\left(\rho \right))}^2\right]$. It is minimized by the posterior expected ranks,

$${\stackrel{‒}{R}}_k\left(\mathbf{Y}\right)=E_{\rho \mid \mathbf{Y}}[R_k\left(\rho \right)\mid \mathbf{Y}]=\sum _{j=1}^K\mathrm{pr}({\rho }_k\ge {\rho }_j\mid \mathbf{Y}),$$ (3)

where pr(·) stands for probability. The optimal mean squared error (MSE) in estimating the ranks is equal to the average posterior variance of the ranks. Generally, the ${\stackrel{‒}{R}}_k$ are not integers; for optimal, distinct integer ranks, use ${\widehat{R}}_k\left(\mathbf{Y}\right)=\text{rank}\left({\stackrel{‒}{R}}_k\left(\mathbf{Y}\right)\right)$.

In the notation that follows, generally we drop dependency on ρ (equivalently, on θ) and omit conditioning on Y. For example, R_k stands for R_k(θ) and ${\widehat{R}}_k$ stands for ${\widehat{R}}_k(\theta \mid \mathbf{Y})$. We present either ranks (R_k) or, equivalently, percentiles [P_k = R_k/(K + 1)] with percentiles providing an effective normalization. For example, Lockwood et al. (2002) show that MSE for percentiles rapidly converges to a function of ranking estimator and posterior distributions of parameters that does not depend on K.

3.2 Optimizing (above γ)/(below γ) classification errors

The USRDS uses percentiles to identify the best and the worst performers. Let γ be the fraction of top performers among the total that we want to identify, 0 < γ < 1. A loss function designed to address this inferential goal was proposed by Lin et al. (2006). The loss function (Eq. 4) penalizes for misclassification and also imposes a distance penalty between estimated percentiles and the cutoff γ.

$$\tilde{L}\left(\gamma \right)=K^{-1}\sum _k{(\gamma -P_k^{est})}^2\{I_{\{P_k>\gamma ,P_k^{est}<\gamma \}}+I_{\{P_k<\gamma ,P_k^{est}>\gamma \}}\}$$ (4)

For ease of presentation, we have assumed that γK is an integer and so γ(K + 1) is not. It is not necessary to make the distinction between > and ≥. To minimize the posterior risk induced by (4), let $p_{k\ell }=\mathrm{pr}(R_k=\ell \mid \mathbf{Y})$ and ${\pi }_k\left(\gamma \right)=\mathrm{pr}(R_k>\gamma (K+1)\mid \mathbf{Y})={\sum }_{\ell =\left[\gamma K\right]+1}^K{p}_{k\ell }$.

$\tilde{L}\left(\gamma \right)$ is minimized by:

$$\begin{array}{cc}\hfill {\tilde{R}}_k\left(\gamma \right)& =\text{rank}\left({\pi }_k\left(\gamma \right)\right)\hfill \\ \hfill {\tilde{P}}_k\left(\gamma \right)& ={\tilde{R}}_k\left(\gamma \right)∕(K+1)\hfill \end{array}$$ (5)

Dominici et al. (1999) use this approach with γ = K/(K + 1), ordering by the probability of a unit having the largest latent attribute.

3.3 Equivalence of the ${\tilde{P}}_k\left(\gamma \right)$ and ordering posterior exceedance probabilities

Given an SMR threshold t, the ranks/percentiles induced by ordering the posterior probabilities that an SMR exceeds the threshold, pr(ρ_k > t|Y) allow us to make a connection between the ${\tilde{P}}_k\left(\gamma \right)$ and the substantive scale (in our application, SMR). Normand et al. (1997) rank providers based on these “exceedance probabilities” and Diggle et al. (2007) use them to identify the areas with elevated disease rates. Lin et al. (2006) shows that exceedance probability based percentiles are virtually identical to the ${\tilde{P}}_k\left(\gamma \right)$ by choosing the γth percentile of the average of posterior cumulative distribution function as the threshold t, i.e., $t={\stackrel{‒}{G}}_K^{-1}\left(\gamma \right)$, where ${\stackrel{‒}{G}}_k\left(t\right)=\frac{1}{K}{\sum }_k\mathrm{pr}({\rho }_k<t\mid \mathbf{Y})$. We denote the percentiles based on $\mathrm{pr}({\rho }_k>{\stackrel{‒}{G}}_K^{-1}\left(\gamma \right)\mid \mathbf{Y})$ as $P_k^{\ast }\left(\gamma \right)$. In addition to providing a connection to the SMR scale, the $P_k^{\ast }\left(\gamma \right)$ are far easier to compute than are the ${\tilde{P}}_k\left(\gamma \right)$. Note that the $P_k^{\ast }\left(\gamma \right)$ are invariant under the monotone transform of ρ_k.

4 Performance measures

As for all statistical procedures, estimated ranks/percentiles must be accompanied by uncertainty statement. A wide variety of univariate and multivariate performance measures are available and we propose three univariate measures of uncertainty.

4.1 Mean squared error

Using MCMC, the posterior mean squared error of percentiles produced by any method and 95% posterior intervals of dialysis center-specific percentiles can be computed. As a baseline, if the data are completely uninformative so that the percentiles ${\widehat{P}}_k$ (1/3174, 2/3174,…, 3173/3174) are randomly assigned to the 3173 dialysis centers, then $10000\times MSE=1666,100\times \sqrt{MSE}=41$.

4.2 Operating characteristic for (above γ)/(below γ) classification

The vector of $P_k^{est}$ from any ranking method can be used to classify units into (above γ)/(below γ) groups and the posterior classification performance (operating characteristic) can be computed. Following Lin et al. (2006), and suppressing dependence on P^est

$$OC(\gamma \mid \mathbf{Y})=ABR(\gamma \mid \mathbf{Y})+BAR(\gamma \mid \mathbf{y})=BAR(\gamma \mid \mathbf{Y})∕\gamma$$ (6)

where,ABR(γ|Y) = pr(percentile) > γ|percentile estimated < γ, Y) = pr(P > γ|P^est < γ, Y) BAR(γ|Y) = pr(percentile < γ|percentile estimated > γ, Y) = pr(P < γ|P^est > γ, Y).

The second equality in (6) results from the identity, γABR(γ|Y) = (1 - γ)BAR(γ|Y). If the goal is to identify units with the largest percentiles, then BAR(γ|Y) is similar to the False Discovery Rate (Benjamini and Hochberg 1995; Efron and Tibshirani 2002; Storey 2002; Storey 2003). ABR(γ|Y) is similar to the False non-Discovery Rate. When the data are completely uninformative, BAR(γ|Y)/γ ≐ 1 and so OC(γ|Y) produces a standardized comparison across γ values. Minimizing it produces the most informative cut point for a given P^est.

For any percentiling method, OC(γ|Y) provides a data analytic performance evaluation. The direct computation of it sums π_k(γ|Y) = pr(P_k > (γ|Y) over a P^est produced set of indices.

$$O{C}_{P^{est}}(\gamma \mid \mathbf{Y})=BA{R}_{P^{est}}(\gamma \mid \mathbf{Y})∕\gamma =\frac{\sum _{j=1}^K[1-{\pi }_k(\gamma \mid \mathbf{Y})]I_{\{P_k^{est}>\gamma \}}}{\gamma \{K-\left[\gamma K\right]\}}.$$

Plotting the π_k(γ|Y) versus the $P_k^{est}$ (see Fig. 1) displays percentile-specific, classification performance. For ideal fully informative data, the exceedance probability should be 1 for those classified as above γ and 0 for those classified as below γ. OC(γ) is the area between ${\pi }_{k_j}\left(\gamma \right)$ curve and 1 for j ≥ [γK] + 1 plus the area below ${\pi }_{k_j}\left(\gamma \right)$ curve for j ≤ [γK]. Using ${\tilde{P}}_k\left(\gamma \right)$ for the X-axis produces a monotone plot and $O{C}_{{\tilde{P}}_k\left(\gamma \right)}\left(\gamma \right)$ is the minimum attainable. This plot is similar to that proposed by Pepe et al. (2008)

Fig. 1.

π_k (0.8|Y) versus ${\tilde{P}}_k\left(0.8\right)$ for 1998. Optimal percentiles and posterior probabilities computed with the single year model (ϕ ≡ 0) and the AR(1) model (ϕ = 0.90) Two curves don't cross at γ = 0.8. The line for fully informative data, i.e., when there is no uncertainty associated with ranking results is given as reference

Computing the π_k(γ|Y) is numerically challenging. However, the virtual equivalence between ${\tilde{P}}_k\left(\gamma \right)$ and $P_k^{\ast }\left(\gamma \right)$ justifies replacing these posterior probabilities by the easily computed pr(ρ_k > t|Y) with $t={\stackrel{‒}{G}}_K^{-1}\left(\gamma \right)$.

4.3 Longitudinal variation

For most of dialysis centers, we expect that percentile estimates of different years are similar to each other. To measure variation in the ranks/percentile estimates within dialysis centers over the four years, we compute Longitudinal Variation:

$$L{V}_{P^{est}}=1000\times \frac{1}{3K}\sum _{k=1}^K\sum _{t=0}^3{(P_{kt}^{est}-P_{k\cdot }^{est})}^2,$$

where $P_{kt}^{est}$ is the estimated percentile for dialysis center k in year t and $P_{k\cdot }^{est}$ is the mean over the four years. A smaller LV value indicates better consistency in percentiles estimates of different years.

4.4 Subset dependency

Unlike estimating individual parameters (where there is individual shrinkage), ranks are highly correlated and so changing the posterior distribution of some target parameters or removing or adding units rearrange the order of individual parameters in a complicated manner. Ranks computed using the posterior distribution of the ranks are thus not subset invariant in that re-ranking the ranks for a subset of providers will not be the same as ranking only those providers. Section Appendix A gives a numeric illustrative example. However, if the prior distribution is known, ranks based on provider-specific summaries such as the MLEs, PMs, exceedance probabilities or single-provider hypothesis tests are subset invariant. Of course, in an empirical Bayes or fully Bayesian analysis with an unknown prior (thus, including a known hyper-prior), no method is subset invariant because the data are also used to estimate the prior or to update the hyper-prior. We investigate subset dependence by including/removing providers with small m_kt (high variance MLEs). These providers are generally small dialysis centers with very few patients. Ranking procedures excluding these centers imply that the centers are first categorized according to their sizes and rankings are then generated in different categories separately. We pursue our comparison under model (2).

5 Results

5.1 Simulated performance

We conducted simulation studies comparing ranking/percentiling methods for the Poisson sampling distribution similar to those reported in Lin et al. (2006) for the Gaussian sampling distribution. Conclusions were similar with ${\widehat{P}}_k$ performing well over a broad class of loss functions, with MLE-based ranks performing poorly and ranks of posterior mean performing reasonably well but by no means optimally (see Louis and Shen 1999; Gelman and Price 1999). Performances of all methods improved with increasing m_kt (reduced sampling variance), but generally the ranking results are quite indefinitive unless information in the sampling distribution (e.g., provided by the data) is very high relative to that in the prior.

5.2 Subset dependency and the effect of unstable SMR estimates

We studied the effect including or excluding providers with small m_kt (high-variance MLE estimates) by running both single-year and multiple-year analysis with and without the 68 providers with expected deaths <0.1 in 1998. Comparisons based on ${\widehat{P}}_k$ in a graph similar to Fig. 2 shows that, there is almost no change in percentiles for providers ranked either high or low, but noticeable re-ordering happens in the middle range. This is not surprising in that the ranks for high-variance providers are shrunken considerably towards the midrank (K + 1)/2 and are not ranked at the extremes. The high variance providers “mix up” with the ranks from more stably estimated, central region providers, but are not contenders for extreme ranks/percentiles. Also, there are more providers in a given interval length in the middle of the distribution of parameters than in the tails. The ranks of these large providers in the middle range will be more sensitive to the change of the joint posterior distribution caused by including small providers. Performance measures MSE and OC(γ) were very similar for the two datasets.

Fig. 2.

Comparison of 1998 $\tilde{P}\left(0.8\right)$ with NPML and Gaussian prior. Circles represent 40 dialysis centers evenly spread across percentiles estimated with NPML prior. The percentiles of the same center are connected

5.3 Comparisons using the 1998 data

We computed ranks (formula (7)) based on the MLE and hypothesis testing statistics Z-scores (testing the hypothesis H₀: ρ = 1 for 1998); we computed the Bayesian estimates ${\widehat{P}}_k,{\tilde{P}}_k\left(\gamma \right)$ and percentiles based on the posterior means (${\rho }_k^{pm}$) using model (2) with ϕ ≡ 0.

$${\rho }_k^{mle}=\frac{Y_k}{m_k},Z_k=\sqrt{m_k}\log \left(\frac{y_k}{m_k}\right),{\rho }_k^{pm}=E({\rho }_k\mid \mathbf{Y}).$$ (7)

Globally, if we regard a dialysis center with ${\rho }_k^{mle}$ greater than 1.5 as “flagged,” then 379 (12%) of dialysis centers will be identified; if we regard a dialysis center with Z-score greater than 1.645 as “flagged,” then 647 (20.4%) of dialysis centers are identified.

To compare methods, we select the 634 (20%) worst dialysis centers by ranking and selecting the largest MLEs and Z-scores and compare to those identified by $\tilde{P}$(0.8). The 80th percentiles of the ρ^mle and Z-score are 1.44 and 1.67, respectively, whereas the 80th percentile of the ρ^pm is 1.10 (these PMs are closer to 1 than their respective MLEs).

We calculate the kappa statistics between the (above γ)/(below γ) classifications based on ${\tilde{P}}_k\left(\gamma \right)$ and other estimators. The classifications based on ${\widehat{P}}_k$ and ρ^pm have high agreement with those based on ${\tilde{P}}_k\left(\gamma \right)$ with respective kappa statistics 0.90 and 0.94. The kappa statistics between the MLE and ${\tilde{P}}_k\left(\gamma \right)$ is 0.78, between the Z-score and ${\tilde{P}}_k\left(\gamma \right)$ is 0.83, and between MLE and Z-score is the lowest 0.68.

Figure 3 compares different methods based on their posterior probability of correct classification pr(P_k > 0.8|Y) The curve for ${\tilde{P}}_k\left(\gamma \right)$ is monotone and optimal because we construct ${\tilde{P}}_k\left(\gamma \right)$ by ranking these probabilities. The curves for percentiles based on ρ^pm and on ${\widehat{P}}_k$ are very close to that for ${\tilde{P}}_k\left(\gamma \right)$ (not plotted). The curves for MLE-based and Z-score-based percentiles are far from monotone.

Fig. 3.

π_k(0.8) versus estimated percentiles by three ranking methods using the 1998 data: ${\tilde{P}}_k\left(\gamma \right)$, MLE-based and Z-score-based. For small dialysis centers (fewer than 5 patients in 1998), the symbol “-” represents the MLE-based percentiles, the symbol “1” the Z-score-based percentiles and the symbol “ ^ ” the ${\tilde{P}}_k\left(\gamma \right)$

The MLE SMRs for centers with relatively small expected deaths have relatively large variances. To study the impact of large and small variances on estimated percentiles, Fig. 3 identifies those for the 147 dialysis centers that treated fewer than 5 patients in 1998. These constitute 4.5% of all centers. Generally, MLE-based percentiles for these centers are at the extremes whereas Z-score based percentiles tend to be near 0.5. However, because the posterior distribution of ρ_k for the high variance centers is concentrated around 1, the ${\tilde{P}}_k\left(\gamma \right)$ for these centers are near 0.5 and similarly for ${\widehat{P}}_k$ and percentiles based on ρ^pm. For dialysis centers with a large number of patients and thereby a small variance, the optimally estimated percentiles, ${\widehat{P}}_k$ and ${\tilde{P}}_k\left(\gamma \right)$ spread out to cover full range from 0 to 1. There is better agreement between MLE-based, Z-score-based and optimal percentiles when the small centers are removed from the dataset and estimates are recomputed.

Figure 4 displays estimates for the 40 providers at the 1/3174, 82/3174, 163/3174,…, 3173/3174 percentiles as determined by ${\widehat{P}}_k$. For each display, the Y-axis is 100 × ${\stackrel{‒}{P}}_k$ with its 95% posterior interval. The X-axis for the upper left panel is $\widehat{P}$, for the upper right is percentiles based on ρ^pm, for the lower left is percentiles based on ρ^mle, and for the lower right is percentiles based on Z-scores testing ρ_k = 1. To deal with cases where Y_kt = 0, for the hypothesis test statistic we use

$$\text{Z-score}=\log \left(\frac{y_k}{m_k}+0.25\right)\sqrt{m_k}.$$

Fig. 4.

SEL-based percentiles for 1998. For each display, the Y-axis is 100 × ${\stackrel{‒}{P}}_k$ with its 95% probability interval. The X-axis for the upper left panel is $\widehat{P}$, for the upper right is percentiles based on ρ^pm, for the lower left is percentiles based on the ρ^mle and for the lower right is percentiles based on Z-scores testing ρ_k = 1

See Conlon and Louis (1999) for a similar plot based on SMRs of disease rates in small areas.

Note that in the upper left display the ${\stackrel{‒}{P}}_k$ do not fill out the (0, 1.0) percentile range; they are shrunken toward 0.50 by an amount that reflects estimation uncertainty. Also, the posterior probability intervals are very wide, indicating considerable uncertainty in estimating ranks/percentiles. The plotted points in the upper left display are monotone because the X-axis is the percentile based on ranking of Y-axis values. Plotted points in the upper right display, which are based on posterior mean, are almost monotone and close to the best attainable. The lower left and lower right panels show considerable departure from monotonicity, indicating that MLE-based ranks and hypothesis test-based ranks are very far from optimal. Note also that the pattern of departures is quite different in the two panels, showing that these methods produce quite different ranks. Similar comparisons for SMRs estimated from the pooled 1998-2001 data would be qualitatively similar, but the departures from monotonicity would be less extreme.

We divide MSE for different ranking methods by the MSE of randomly assigned ranks (Sect. 4.1) for standardization. The methods based on posterior distributions, P^pm, ${\widehat{P}}_k$, $\tilde{P}\left(0.8\right)$ and P^☆ (0.8) perform pretty close to each other with standardized MSEs 44.5%, 44.4%, 46.2% and 46.2%, respectively. Rankings based on MLE and Z-score have less improvement (52.3% and 47.4%) over randomly assigned ranks. The differences in $\sqrt{MSE}$ are less substantial and the wide 95% intervals presented in Fig. 4 indicate that none of methods can give a conclusive ranking result.

5.4 Single year and multi-year analyses

Using model (2) we estimated single-year based and AR(1) model based percentiles. Table 1 reports that the ξ are near 0, as should be the case since we have used internal standardization (the typical log(SMR) = 0). The within year, between provider variation in 100 × log(SMR) is essentially constant at approximately 100 × τ = 24, producing a 95% a priori interval for the ρ_kt (0.62, 1.60). While we have a prior centering around 1000 for 100 × τ, the data likelihood dominates the prior information and the posterior 95% credible interval of 100 × τ_t for all 4 years is (22.8, 26.8). Use of the AR(1) model to combine evidence over years (with the posterior distribution for ϕ concentrated around 0.90) reduces 100 × $O{C}_{\tilde{p}}\left(0.8\right)$ from around 61 to around 48, a twenty percent decrease. Classification performance comparison using the ${\widehat{P}}_k$ is very close to that for the optimal 100 × ${\tilde{P}}_k\left(0.8\right)$.

Table 1.

Results for ${\widehat{P}}_k$ and $\tilde{P}\left(0.8\right)$. In the multi-year section, 100 × OC(0.8) is for the indicated year as estimated from the multi-year model and ₈₈90₉₂ is a notation for posterior median 90 and 95% credible interval (88, 92) (Louis and Zeger 2008)

	Single year: (ϕ ≡ 0)				Multi-year: 100 × ϕ ∼ 889092)
Parameter	1998	1999	2000	2001	1998	1999	2000	2001
100 × ζ	-2.8	-1.3	-2.3	-0.7	-3.1	-0.8	-1.7	-0.3
100 × τ	24.1	23.5	23.1	22.2	25.8	25.0	24.9	24.1
100 × $OC{\tilde{p}}_{\left(0.8\right)}\left(0.8\right)$	62	61	60	62	49	47	46	50
LV(${\widehat{P}}_k$)	62				4

Figure 1 displays the details behind the improvement of classification performance. In the upper range of ${\tilde{P}}_k\left(0.8\right)$, the curve for the AR(1) model lies above that for the single year, in the lower range it lies below. For the AR(1) model to dominate the single year at all values of ${\tilde{P}}_k\left(0.8\right)$, the curves would need to cross at ${\tilde{P}}_k\left(\gamma \right)$ = 0.8, but the curves cross at about 0.7. Appendix B provides some discussion on this phenomenon.

Longitudinal variation in ranks/percentiles (LV_Pest is dramatically reduced for the AR(1) model going from 62 for the year-by-year analysis to 4 for the multi-year. As a basis for comparison, if ϕ → 1, $L{V}_{\widehat{P}}$ → 0 and if the data provide no information on the SMRs (the τ → ∞), then $L{V}_{\widehat{P}}$ = 83.

We have not compared fit of the AR(1) model to other correlation structures such as compound symmetry (constant correlation rather than exponential damping). With only 4 years of data per center, the power to compare different correlation structures will be low. With ϕ's posterior mean 0.90 and 95% credible interval (0.88, 0.92), the AR(1) model is well supported by the data relative to independence. Note that the AR(1) model operates on the θ_kt = log(ρ_kt) and not on the observed estimates $\left({\theta }_{kt}^{mle}\right)$. The induced model for these is approximately ARMA(1, 1), a hidden Markov model.

5.5 Parametric and non-parametric priors

We compare percentiles based on posterior distributions under the parametric and NPML priors using 1998 data. Figure 5 displays Gaussian, posterior expected and smoothed NPML estimated priors for θ = log(ρ). The Gaussian is produced by plugging in the posterior medians for (μ₀, τ₀). The posterior expected is a mixture of Gaussians using the posterior distribution of (μ₀, τ₀). The posterior distribution of (μ₀, τ₀) has close to 0 variance, so the two parametric curves superimpose. The NPML is discrete and was smoothed using the “density” function in R with adjustment parameter 10 (i.e., the Gaussian kernel bandwidth is ten times of the default value, see Silverman (1986)). We smooth the NPML to graph a smooth curve, but use the NPML itself to produce ranks/percentiles. Note that the smoothed NPML has at least two modes with a considerable mass at approximately θ = 0.5; ρ = 1.65. However, this departure from the Gaussian distribution has little effect on classification performance. Using 1998 data, for the NPML 100 × OC(0.8) ≈ 67 while for the Gaussian prior the value is 62 (see Table 1). For performance evaluations of the NPML, see Paddock et al. (2006). Fig. 2 compares $\tilde{P}\left(0.8\right)$ under the two priors. The centers at the top or the bottom have less uncertainty in percentiles (strong signal), and their percentiles are generally same under two priors. For the dialysis centers with larger variance, the percentiles depend on the prior.

Fig. 5.

Estimated priors for θ = log(ρ) using the 1998 data. The solid curve is a smoothed NPML using the “density” function in R with adjustment parameter = 10. The dashed curve is Gaussian using posterior medians for (μ, τ); the dotted curve is a mixture of Gaussians with (μ, τ) sampled from their MCMC computed joint posterior distribution

5.6 Ranks based on exceedance probabilities

We compute P*(0.8) (see Sect. 3.3) using the Gaussian prior for θ and 1998 data. The θ-threshold, ${\stackrel{‒}{G}}_K^{-1}\left(0.8\right)=0.169$ (ρ-threshold = 1.184). Lin et al. (2006) prove the near equivalence of P*(γ) and ${\tilde{P}}_k\left(\gamma \right)$ and Fig. 1 displays this equivalence in that the curve based on P*(0.8) is nearly identical to that based on $\tilde{P}\left(0.8\right)$ for ϕ = 0.

6 Discussion

Ranks and percentiles are computed to address specific policy or management goals. It is important to use a procedure that performs well for the primary goals. A structured approach guided by a Bayesian hierarchical model and a loss function helps clarify goals and produces ranks/percentiles that outperform other contenders, such as those based on MLEs and Z-scores. When the uncertainties of the direct estimate vary considerably over providers, the estimates are very sensitive to the method used. In that situation, a structured approach is especially important.

Our data-analytic assessments support the Lin et al. (2006) finding that the ${\widehat{P}}_k$ (general purpose percentiles) perform well over inferential goals addressed by a range of loss functions, but that if a specific percentile cut-point, γ, is identified, ${\tilde{P}}_k\left(\gamma \right)$ (or P*(γ)) should be used. Unless the substantive application dictates otherwise, we recommend the use of these. Cost or other considerations can be incorporated to select γ.

Though the loss function guided estimates are the best possible, the ranking results might not be conclusive, partially indicated by the wide confidence interval as shown in the Fig. 4. Therefore, data-analytic performance evaluations are a necessary companion to estimated ranks. Uncertainty assessments include standard errors and tabulation or display of the probabilities of correct classification in a (above γ)/(below γ) assessment (our π_k(γ|Y)). These probabilities can be used to temper penalties or rewards. When available, data of multiple years can be combined to reduce the uncertainty in ranking results, as shown in Table 1 and Fig. 1.

Robustness of efficiency and validity are important attributes of any statistical procedure. For sufficiently large K, using a smoothed non-parametric prior is highly efficient relative to a correct, parametric approach and confers considerable robustness (see Paddock et al. 2006). Additional study of this strategy is needed.

Percentiles are prima facie relative comparisons in that it is possible that all providers are doing well or that all are doing poorly; percentiles will not pick this up. Indeed, the SMR is, itself, a relative measure and so percentiles produced from it are twice removed from a normative context! In situations where normative values are available (e.g., death rates), percentiles that have a normative interpretation are attractive and those based on posterior probabilities of exceeding some threshold (P*(γ)) are essentially identical to a loss function based approach $\left({\tilde{P}}_k\left(\gamma \right)\right)$ and so provide an excellent link to a substantively relevant scale. And, they confer a considerable computing advantage over using the posterior distribution of the ranks to find the ${\tilde{P}}_k\left(\gamma \right)$.

Finally, because percentiles can be very sensitive to the estimation methods and because there is considerable uncertainty associated with all percentiling methods, stakeholders need to be informed of the issues in producing percentiles, in interpreting them, in their role in science and policy, and in insisting on uncertainty assessments.

Appendices

Appendix A: An illustrative example of subset dependency

In general, ranks depend on the framework for comparison and the list of contenders; they are not necessarily subset invariant. For illustration, consider ranking 3 dialysis centers by ${\stackrel{‒}{R}}_k$ as defined in Eq. 3. We have,

$$\begin{array}{cc}\hfill {\stackrel{‒}{R}}_1& =\mathrm{pr}({\rho }_1>{\rho }_2)+\mathrm{pr}({\rho }_1>{\rho }_3)\hfill \\ \hfill {\stackrel{‒}{R}}_2& =\mathrm{pr}({\rho }_2>{\rho }_1)+\mathrm{pr}({\rho }_2>{\rho }_3)\hfill \\ \hfill {\stackrel{‒}{R}}_3& =\mathrm{pr}({\rho }_3>{\rho }_1)+\mathrm{pr}({\rho }_3>{\rho }_2).\hfill \end{array}$$

Let ${\theta }_i=\log \left({\rho }_i\right)~N({\mu }_i,{\sigma }_i^2)$, i = 1, 2, 3. Let μ₁ = 0, μ₂ = -0.15, μ₃ = 0.05, ${\sigma }_1^2={\sigma }_3^2={0.01}^2,{\sigma }_2^2={7.5}^2$, then pr(ρ₁ > ρ₂) = 0.51 > 0.49 = pr(ρ₂ > ρ₁); pr(ρ₁ > ρ₂) + pr(ρ₁ > ρ₃) = 0.51 < 0.98 = pr(ρ₂ > ρ₁) + pr(ρ₂ > ρ₃). Thus if we rank only dialysis centers 1 and 2, center 2 has better rank (smaller θ) than center 1; if we rank all 3 centers, center 2 has worse rank than center 1.

Appendix B: Crossing points of exceedance probability curves

We start from a simplified scenario where θ_k's share a common posterior variance. Assume θ_k ~ N(μ_k, ν) a posteriori, Φ(t; μ_k, ν) = pr(θ > t; μ_k, ν). Let ν₀ and ${\nu }_0^{\prime }$ be two possible values of ν, ν₀ > ${\nu }_0^{\prime }$. Without loss of generality, let μ₁ < μ₂ < …< μ_K.

For any given t,

• If $t<{\mu }_k,\frac{t-{\mu }_k}{\sqrt{{\nu }_0}}>\frac{t-{\mu }_k}{\sqrt{{\nu }_0^{\prime }}},\Phi (t;{\mu }_k,{\nu }_0)=\Phi (\frac{t-{\mu }_k}{\sqrt{{\nu }_0}};0,1)<\Phi (\frac{t-{\mu }_k}{\sqrt{{\nu }_0^{\prime }}};0,1)=\Phi (t;{\mu }_k,{\nu }_0^{\prime })$;

• If $t>{\mu }_k,\frac{t-{\mu }_k}{\sqrt{{\nu }_0}}<\frac{t-{\mu }_k}{\sqrt{{\nu }_0^{\prime }}},\Phi (t;{\mu }_k,{\nu }_0)=\Phi (\frac{t-{\mu }_k}{\sqrt{{\nu }_0}};0,1)>\Phi (\frac{t-{\mu }_k}{\sqrt{{\nu }_0^{\prime }}};0,1)=\Phi (t;{\mu }_k,{\nu }_0^{\prime })$;

By the common variance assumption and μ₁ < μ₂ <…μ_K, the posterior distributions of θ_k are stochastically ordered and the rank of θ_k is k. The curves $(\frac{k}{K+1},\Phi (t,{\mu }_k,{\nu }_0))$ and $(\frac{k}{K+1},\Phi (t,{\mu }_k,{\nu }_0^{\prime }))$ are both monotone increasing and cross each other between $\frac{i}{K+1}$ and $\frac{i+1}{K+1}$ if μ_i < t < μ_i+1. The value of y-coordinate of the crossing point is around 0.5 due to $\frac{t-{\mu }_i}{\nu }\approx 0$. We denote the x-coordinate of the crossing point as $\frac{i\left(t\right)}{K+1}$ ignoring at most 1/K difference.

If t₁ and t₂ satisfy

$$\frac{1}{K}\sum _{k=1}^K\Phi (t_1;{\mu }_k,{\nu }_0)=\frac{1}{K}\sum _{k=1}^K\Phi (t_2;{\mu }_k,{\nu }_0^{\prime })=1-\gamma$$

then the curves $(\frac{k}{K+1},\Phi (t_1,{\mu }_k,{\nu }_0))$ and $(\frac{k}{K+1},\Phi (t_1,{\mu }_k,{\nu }_0^{\prime }))$ cross each other at $\frac{i\left(t_1\right)}{K+1}$; the curves $(\frac{k}{K+1},\Phi (t_2,{\mu }_k,{\nu }_0))$ and $(\frac{k}{K+1},\Phi (t_2,{\mu }_k,{\nu }_0^{\prime }))$ cross each other at $\frac{i\left(t_2\right)}{K+1}$; And the crossing-over of curves $(\frac{k}{K+1},\Phi (t_1,{\mu }_k,{\nu }_0))$ and $(\frac{k}{K+1},\Phi (t_2,{\mu }_k,{\nu }_0^{\prime }))$ happens between $\frac{i\left(t_1\right)}{K+1}$ and $\frac{i\left(t_2\right)}{K+1}$. When γ is greater or smaller than both of $\frac{i\left(t_1\right)}{K+1}$ and $\frac{i\left(t_2\right)}{K+1}$, which depend on γ, ν₀, ${\nu }_0^{\prime }$ and vector (μ₁, μ₂,…, μK), the x-coordinate of the crossing point can not be at γ.

Denote the posterior distributions of θ_k as N(μ_k, ν_k) and N$({\mu }_k^{\prime },{\nu }_k^{\prime })$ in single year and multiple years analyses. If the dialysis centers perform consistently over years, inference uncertainty of θ_k should be reduced (assuming ${\nu }_k^{\prime }$ < ν_k) by accumulating data over years while the means do not change much (assuming ${\mu }_k^{\prime }$ = μ_k). Assuming N(μ_k, ν_k)'s are stochastically ordered, N$({\mu }_k^{\prime }={\mu }_k,{\nu }_k^{\prime })$'s are stochastically ordered, the above discussion applies to the crossing point of the curves $(\frac{k}{K+1},\Phi (t,{\mu }_k,{\nu }_k))$ and $(\frac{k}{K+1},\Phi (t,{\mu }_k,{\nu }_k^{\prime }))$.

In Fig. 1, two curves $(\frac{k}{K+1},\Phi (t,{\mu }_k,{\nu }_k))$ and $(\frac{k}{K+1},\Phi (t,{\mu }_k,{\nu }_k^{\prime }))$ are plotted without the stochastically ordered assumption and the location crossing point is more complicated. In general, it is not necessary that the x-coordinate of the crossing point will be at γ.

Appendix C: The NPML algorithm

Assume ρ_k ~ G, k = 1,…, K. G is discrete having at most J mass points u₁,…,u_J with probabilities p₁,…, p_J. We use EM algorithm (Dempster et al. 1977) to estimate the u's and p's. Start with $u_1^{\left(0\right)},\dots ,u_j^{\left(0\right)}$ and $p_1^{\left(0\right)},\dots ,p_J^{\left(0\right)}$, for each recursion,

$$\begin{array}{cc}\hfill w_{kj}^{(\nu +1)}& =\mathrm{pr}({\rho }_k=u_j^{\left(\nu \right)}\mid \text{data})\hfill \\ \hfill w_{kj}^{(\nu +1)}& =\frac{{\left(m_k{u}_j^{\left(\nu \right)}\right)}^{yk}{e}^{-m_k{u}_j^{\left(\nu \right)}}{p}_j^{\left(\nu \right)}}{\sum _l{\left(m_k{u}_l^{\left(\nu \right)}\right)}^{y_k}{e}^{-m_k{u}_l^{\left(\nu \right)}}{p}_j^{\left(\nu \right)}}\hfill \\ \hfill p_j^{(\nu +1)}& =\frac{w_{+j}^{(\nu +1)}}{w_{++}^{(\nu +1)}}\hfill \\ \hfill u_j^{(\nu +1)}& =\frac{\sum _k{w}_{kj}^{(\nu +1)}{y}_k}{\sum _k{w}_{kj}^{(\nu +1)}{m}_k}.\hfill \end{array}$$

This recursion converges to a fixed point $\widehat{G}$ and, if unique, to the NPML. The recursion is stopped when the maximum relative change in each step for both the $u_j^{\left(\nu \right)}$ and the $p_j^{\left(\nu \right)}$, j = 1, 2,…,K is smaller than 0.001. At convergence, $\widehat{G}$ is both prior and the Shen and Louis (1998) histogram estimate ${\widehat{G}}_K$.

Care is needed in programming the recursion. The w-recursion is:

$$w_{kj}^{(\nu +1)}=\frac{{\left(m_k{u}_j^{\left(\nu \right)}\right)}^{yk}{e}^{-m_k{u}_j^{\left(\nu \right)}}{p}_j^{\left(\nu \right)}}{\sum _l{\left(m_k{u}_l^{\left(\nu \right)}\right)}^{y_k}{e}^{-m_k{u}_l^{\left(\nu \right)}}{p}_j^{\left(\nu \right)}}.$$

Since $e^{-m_k{u}_j^{\left(\nu \right)}}$ can be extremely small ($m_k{u}_j^{\left(\nu \right)}$ > can be extremely large), to stabilize the computations we define,

$${\stackrel{‒}{\rho }}^{\left(\nu \right)}=\sum _j{p}_j\left(\nu \right)u_j^{\left(\nu \right)},$$

and write

$${\left(m_k{u}_j^{\left(\nu \right)}\right)}^{yk}=\text{exp}\left(y_k\log \left(m_k{u}_j^{\left(\nu \right)}\right)\right).$$

The w-recursion becomes:

$$\begin{array}{c}\hfill w_{kj}^{(\nu +1)}=\frac{{\left(u_j^{\left(\nu \right)}∕{\stackrel{‒}{\rho }}^{\left(\nu \right)}\right)}^{y_k}\text{exp}\left(-m_k\left(u_j^{\left(\nu \right)}-{\stackrel{‒}{\rho }}^{\left(\nu \right)}\right)\right)p_j^{\left(\nu \right)}}{\sum _{l=1}^J{\left(u_l^{\left(\nu \right)}∕{\stackrel{‒}{\rho }}^{\left(\nu \right)}\right)}^{y_k}\text{exp}\left(-m_k\left(u_l^{\left(\nu \right)}-{\stackrel{‒}{\rho }}^{\left(\nu \right)}\right)\right)p_l^{\left(\nu \right)}}\hfill \\ \hfill =\frac{p_j^{\left(\nu \right)}\text{exp}\left(y_k\log (u_j^{\left(\nu \right)}∕{\stackrel{‒}{\rho }}^{\left(\nu \right)})-m_k\left(u_j^{\left(\nu \right)}-{\stackrel{‒}{\rho }}^{\left(\nu \right)}\right)\right)}{\sum _{l=1}^J{p}_l^{\left(\nu \right)}\text{exp}\left(y_k\log (u_l^{\left(\nu \right)}∕{\stackrel{‒}{\rho }}^{\left(\nu \right)})-m_k\left(u_l^{\left(\nu \right)}-{\stackrel{‒}{\rho }}^{\left(\nu \right)}\right)\right)}\hfill \end{array}$$