Bayesian estimation of the accuracy of ICD-9-CM- and CPT-4-based algorithms to identify cholecystectomy procedures in administrative data without a reference standard

Abstract

Purpose

To estimate the accuracy of two algorithms to identify cholecystectomy procedures using International Classification of Diseases, 9th Edition, Clinical Modification (ICD-9-CM) and Current Procedural Terminology (CPT-4) codes in administrative data.

Methods

Private insurer medical claims for 30,853 patients 18–64 years with an inpatient hospitalization between 2006 and 2010, as indicated by providers/facilities place of service in addition to room and board charges, were cross-classified according to the presence of codes for cholecystectomy. The accuracy of ICD-9-CM- and CPT-4-based algorithms was estimated using a Bayesian latent class model.

Results

The sensitivity and specificity were 0.92 [probability interval (PI): 0.92, 0.92] and 0.99 (PI: 0.97, 0.99) for ICD-9-CM-, and 0.93 (PI: 0.92, 0.93) and 0.99 (PI: 0.97, 0.99) for CPT-4-based algorithms, respectively. The parallel-joint scheme, where positivity of either algorithm was considered a positive outcome, yielded a sensitivity and specificity of 0.99 (PI: 0.99, 0.99) and 0.97 (PI: 0.95, 0.99), respectively.

Conclusions

Both ICD-9-CM- and CPT-4-based algorithms had high sensitivity to identify cholecystectomy procedures in administrative data when used individually and especially in a parallel-joint approach.

INTRODUCTION

In the United States, approximately 700,000 cholecystectomy procedures are performed each year,¹ which makes it one of the most commonly performed surgical procedures. Administrative data have been used in several studies to identify cholecystectomy procedures.^2-9 These data are increasingly used for health services and outcomes research,^10-13 which is a concern due to uncertainty regarding the accuracy of diagnostic and procedural coding.^14-16 Administrative data are comprised of detailed health information including electronic insurance enrollment information, basic demographics, and claims submitted by providers and facilities for the purpose of billing and reimbursement. Given the distinct differences in data collected for billing versus research, it may be difficult to ascertain the true status of subjects with regard to an outcome or an exposure of interest when using administrative data.

To the best of our knowledge, there are no studies that assess the accuracy of different coding algorithms to identify cholecystectomy procedures in administrative data. The traditional method to validate a diagnostic algorithm is to compare it to a criterion with perfect accuracy, referred to as the reference or ‘gold’ standard. For administrative data, medical record (chart) review is often be used as the reference standard. However, a reference standard may not always be available for some administrative data studies due to the costs of obtaining another data source to use as a reference standard or lack of access to charts for record review. Without a reference standard, relying on a claims-based algorithm to identify subjects’ status in a study population could cause misclassification of the subjects’ outcome or exposure status, leading to biased estimates of prevalence or association between risk factors and an outcome.^17-19 In the absence of a reference standard, model-based methods, such as Bayesian latent class models, were developed to estimate the accuracy of diagnostic tests.^20-22 Bayesian methods allow incorporation of uncertainty as well as current knowledge regarding the parameters of interest into analysis through probability distributions, referred to as priors. Furthermore, Bayesian methods do not rely on large sample approximations for parameter estimation and provide direct probability interpretation of results from posterior distributions.²³

The purpose of this study was to estimate the accuracy of two different algorithms to identify cholecystectomy procedures based on the two commonly used coding systems in administrative data, the International Classification of Diseases, 9th Edition, Clinical Modification (ICD-9-CM) and Current Procedural Terminology (CPT-4) codes, using Bayesian latent class models. Moreover, we provide estimates of the accuracy of the coding algorithms when used jointly in a parallel scheme.

METHODS

Data

Medical claims data from 13 Blue Cross/Blue Shield health plans contained in the HealthCore Integrated Research Database (HIRD^SM) were used. The data included facility and professional adjudicated and paid claims for members ages 18–64 years enrolled in a non-capitated medical (hospital and physician) health plan from January 2006 through December 2010. We restricted this study to an inpatient population because ICD-9-CM procedure codes are reported by facilities for inpatient stays, but not routinely for outpatient or ambulatory visits. Each medical claim could include unlimited CPT-4 codes and up to five ICD-9-CM procedure codes.

The inpatient stay was established based on one or more claims with an inpatient place of service assigned by a facility or a provider; consecutive dates on claims with an inpatient place of service defined the duration of the surgical episode. A room and board charge, defined as Uniform billing revenue codes in the range 0100–0249, was required during the surgical episode to meet the definition of an inpatient surgical episode. Whenever there was more than one distinct surgical episode corresponding to cholecystectomy for a patient, only the first surgical episode was included.

Algorithms

Two algorithms were considered to identify cholecystectomy procedures. The first algorithm was based on ICD-9-CM procedure codes only. In this algorithm any patient with an inpatient stay with ICD-9-CM procedure codes 51.21–51.24 was considered positive for having had a cholecystectomy procedure performed.

The second algorithm was based on CPT-4 codes only. In the CPT-4 algorithm, any inpatient stay coded by a provider with 47562, 47563, 47564, 47600, 47605, 47610, 47612, or 47620 was considered positive for having had a cholecystectomy procedure performed.

Modeling

The accuracy of a binary diagnostic test is often presented by measures such as sensitivity, the probability of a positive result in the population with the condition of interest, and specificity, the probability of a negative result in the population without the condition of interest. Without a ‘perfect’ method of ascertainment or a reference standard, the accuracy of coding algorithms is ‘latent’,(i.e., the accuracy cannot be directly observed, but can be estimated from data).^24,25 We used a modified version of the Bayesian latent class model of ‘two tests, one population’, as discussed in Joseph et al 1995,²⁰ which does not rely on information from a reference standard to estimate the accuracy measures of interest, i.e. sensitivity and specificity. Hui and Walter 1980 provided methods to estimate the accuracy of two tests without a reference standard.²⁶ The problem in estimating the five parameter of interest, two sensitivities, two specificities and prevalence, is that there are only three independent cells (degrees of freedom) available in a cross-classified table of test results, which leads to statistical non-identifiability.²⁷ Therefore, the problem is unsolvable using frequentist methodology without additional assumptions. Hui and Walter required a second population with distinct prevalence in order to increase the degrees of freedom required to estimate the five parameters of interest in addition to the prevalence in the second population. Bayesian approach is able to tackle this problem and lack of identifiability (or weak identifiability) in the ‘two tests, one-population’ scenario by specifying probability distributions on the parameters of interest.²⁷

In a Bayesian approach, uncertainties regarding the accuracy of coding algorithms are represented by probability distributions. These probability distributions, commonly referred to priors, are combined with a likelihood function of data to obtain updated probability distributions, referred to as posterior distributions, for all parameters of interest. A detailed description of the model is in the Appendix. Briefly, the results for the two algorithms were cross-classified, where it was assumed that the observed cross-classified results have a multinomial sampling distribution. The multinomial probabilities associated with the cross-classified results can be described by the true-positive (sensitivity), false-positive (‘1 – sensitivity’), true-negative (specificity) and false-negative (‘1 – specificity’) probabilities, and the probability of the condition in the population.^20,22 It was assumed that the results of the two algorithms are independent of each other conditional on the true cholecystectomy status. The conditional independence assumption is reasonable since CPT-4 and ICD-9-CM procedure codes are coded independently by separate entities (i.e., facility medical coders for ICD-9-CM procedure codes and clinicians or their staff for CPT-4 codes). For situations where algorithms may be dependent, a model that incorporates conditional dependence could be used.^22,28

We then estimated the sensitivity and specificity of the two algorithms. A parallel-testing scheme is often implemented for screening or diagnostic algorithms in order to improve or gain increased sensitivity. In the parallel-testing scheme, having a positive result from either of the two algorithms would be considered as a positive outcome. Therefore, we also estimated the sensitivity and specificity of the two algorithms when they were used in parallel combination.

Priors

We used BetaBuster software (http://www.epi.ucdavis.edu/diagnostictests/betabuster.html) to obtain a unique beta prior corresponding to our prior belief. The BetaBuster requires two quantities to obtain a unique beta distribution. We first determined our best guess for the most likely value for a parameter, for example, sensitivity of the ICD-9-CM-based algorithm, and then our best guess that the most likely value for the parameter is more (or less) than a specific value. BetaBuster uses the most likely value as the mode and the other quantity as the 5th (or 95th) percentile of the corresponding beta distribution. For example, we assumed that the most likely value (mode) for the sensitivity of the ICD-9-CM-based algorithm is 0.95, and we are 95% sure that the mode is more than 0.80, which corresponds to the Beta(21.20, 2.06) distribution. The elicited priors used for the Bayesian model parameters are summarized in Table 1.

Table 1.

Elicited priors for the sensitivities and specificities of the ICD-9-CM- and CPT-4-based algorithms.

Algorithm	Parameter	Parameter description and corresponding prior	Alternative prior for sensitivity analysis 1	Alternative prior for sensitivity analysis 2
	Pr	Mode = 0.97, 95% sure that mode > 0.90; Beta(53.58, 2.63)	Mode = 0.97, 95% sure that mode > 0.90; Beta(53.58, 2.63)	Mode = 0.97, 95% sure that mode > 0.90; Beta(53.58, 2.63)
ICD-9-CM-based	Se	Mode = 0.95, 95% sure that mode > 0.80; Beta(21.20, 2.06)	Mode = 0.95, 95% sure that mode > 0.65; Beta(8.45, 1.39)	Mode = 0.80, 95% sure that mode > 0.60; Beta(14.8442, 4.4611)
	Sp	Mode = 0.99, 95% sure that mode > 0.97; Beta(212.12, 3.13)	Mode = 0.99, 95% sure that mode > 0.95; Beta(88.28, 1.88)	Mode = 0.99, 95% sure that mode > 0.95; Beta(88.28, 1.88)
CPT-4-based	Se	Mode = 0.95, 95% sure that mode > 0.80; Beta(21.20, 2.06)	Mode = 0.95, 95% sure that mode > 0.65; Beta(8.45, 1.39)	Mode = 0.80, 95% sure that mode > 0.60; Beta(14.8442, 4.4611)
	Sp	Mode = 0.99, 95% sure that mode > 0.97; Beta(212.21, 3.13)	Mode = 0.99, 95% sure that mode > 0.95; Beta(88.28, 1.88)	Mode = 0.99, 95% sure that mode > 0.95; Beta(88.28, 1.88)

Pr = probability of the condition in the population; Se = sensitivity; Sp = specificity.

The model described in the previous section is non-identifiable; however, achieving identifiability is not mandatory in Bayesian analysis with informative priors.²⁷ Therefore, priors were specified to be consistent with the data, where implausible (or extremely unlikely) and non-informative priors were not considered. Sensitivity analyses for our proposed model were performed using two sets of less informative priors (Table 1) to assess how posterior estimates would be affected by our choice of priors.

Computation

The Bayesian modeling was performed in JAGS version 3.4.0²⁹ through ‘rjags’³⁰ library in R³¹ software.³¹ All inferences were based on 100,000 iterations thinned from 100,000,000 after a burn-in of 10,000,000 iterations. Geweke’s statistic was used to assess lack of convergence for the Markov chains through ‘coda’³² library in R.

RESULTS

The study population included 30,853 patients with a cholecystectomy coded during an inpatient hospital stay. Table 2 presents the cross-classified results for the ICD-9-CM- and CPT-4-based algorithms for the study population.

Table 2.

Cross-classified results of the ICD-9-CM- and CPT-4-based algorithms for cholecystectomy procedures.

	ICD-9-CM-based	CPT-4-based	Frequency
	+	+	26 446 (85.7%)
	+	−	2 091 (6.8%)
	−	+	2 316 (7.5%)
Total			30 853

The posterior median and 95% probability interval for the sensitivities and specificities of the two algorithms when used separately and also when used in parallel are summarized in Table 3. Both algorithms had high accuracy to identify cholecystectomy procedures in this data. When used individually, it was estimated that the false negative proportions were about 8% and 7% for the ICD-9-CM- and CPT-4-based algorithms, respectively. The highest sensitivity of detection was obtained when the algorithms were used jointly in the parallel scheme (0.99, probability interval [PI]: 0.99, 0.99; Table 3). Sensitivity analysis with separate sets of less informative priors produced similar results (Table 3).

Table 3.

Posterior medians and 95% probability intervals for the accuracy of the ICD-9-CM- and CPT-4-based algorithms when used separately and jointly in parallel.

Algorithm	Parameter	Posterior median (95% PI)	Posterior median (95% PI) using priors in sensitivity analysis 1	Posterior median (95% PI) using priors in sensitivity analysis 2
ICD-9-CM-based	Se	0.92 (0.92, 0.92)	0.92 (0.92, 0.92)	0.92 (0.92, 0.92)
	Sp	0.99 (0.97, 0.99)	0.98 (0.94, 0.99)	0.98 (0.94, 0.99)
CPT-4-based	Se	0.93 (0.92, 0.93)	0.93 (0.92, 0.93)	0.93 (0.92, 0.93)
	Sp	0.99 (0.97, 0.99)	0.98 (0.94, 0.99)	0.98 (0.94, 0.99)
Parallel-joint	Se	0.99 (0.99, 0.99)	0.99 (0.99, 0.99)	0.99 (0.99, 0.99)
	Sp	0.97 (0.95, 0.99)	0.96 (0.91, 0.99)	0.96 (0.91, 0.99)

PI = probability interval; Se = sensitivity; Sp = specificity.

DISCUSSION

Our study suggests that both the ICD-9-CM- and CPT-4-based algorithms have high accuracy, especially when used in parallel combination, to identify cholecystectomy procedures in administrative data. Several studies using administrative data have relied on ICD-9-CM procedure codes alone to identify cholecystectomy procedures.^4,5,8 Some investigators have required additional ICD-9-CM diagnosis codes for conditions such as gallbladder disease along with ICD-9-CM procedure codes to identify cholecystectomy.^2,3,6,7,9 This approach, commonly referred to as serial-testing, could potentially lead to a higher specificity to detect cholecystectomy at the expense of a loss in sensitivity. In some studies, a reference standard was used to validate the detection algorithm only for subjects who had ICD-9-CM procedure codes for cholecystectomy;^4,8 consequently, unless adjusted for in the analysis or the design of the study, this could lead to partial verification bias, also referred to as workup bias.³³ Weinhandl and Gilbertson showed that the commonly used case-control design, where the validation study population includes a random sample of the subjects without any code in addition to all the subjects that are positive due to an algorithm, could potentially result in biased estimates of the accuracy and invalid findings because estimates for specificity and (consequently) predictive values vary considerably with the sampling rate.³⁴

In our study, we did not rely on a reference standard to verify the true classification status of subjects for each algorithm. Instead, we relied on a model-based approach to estimate the accuracy of the algorithms through Bayesian modeling, where current knowledge and uncertainty about the accuracy of the algorithms were represented by probability distributions. Model-based estimation of the accuracy of algorithms in administrative data has previously been used to identify conditions or procedures such as osteoarthritis,¹⁹ systemic lupus erythematosus,³⁵ autoimmune rheumatic disease³⁶ and screening colonoscopy³⁷ among others. An important feature of our study is that we used a national insurer database that represents multiple hospitals and physician groups of various sizes in both academic and private settings, which strengthens its generalizability.

In general, inaccuracy in administrative data can occur in several ways. For example, coding for some procedures may be complicated and subject to interpretation. Untimely documentation by clinicians could lead to inaccuracy in medical records if coders need to complete their coding documentation before operating notes are complete.³⁸ Some hospitals may use commercially available coding software rather than medical coders, likely resulting in some degree of coding error.

It is possible that we had missing records on CPT-4 codes from providers in our data potentially due to denial of provider claims by private insurers or incomplete enrollment information for patients with dual insurance; consequently, we may not have had complete information on CPT-4 codes from providers, which would have resulted in a lower sensitivity for the CPT-4-based algorithm. The number of ICD-9-CM procedure codes on each hospital claim in our data was limited to five. The primary purpose of administrative data is reimbursement of healthcare costs; thus, multiple diagnosis or procedure codes are likely to be listed in order to maximize remuneration. It is possible that ICD-9-CM procedure codes could have been dropped if there were other conditions associated with higher reimbursement occurring during the same inpatient admission.¹² Another option is that ICD-9-CM codes may be listed temporally, so some codes may be missed if a procedure was performed relatively late into the admission, especially for long hospitalizations. However, possible dropping of ICD-9-CM procedure codes did not appear to be a problem in our data. Only about 0.4% of patients, who were assigned a CPT-4 code for cholecystectomy by providers, had a facility claim that used all five fields allocated for ICD-9-CM procedure codes during their surgical admission without having an ICD-9-CM procedure code for cholecystectomy on their claims.

Our study population was based on inpatient hospitalizations; therefore, the parallel combination would not be applicable for ambulatory procedures since CPT-4 codes are assigned by both facilities and providers in ambulatory settings. The parallel combination would also not be applicable for databases containing only hospital billing data, as those data would be restricted to only ICD-9-CM procedure codes. Moreover, the parallel approach, for the purpose of maximizing sensitivity, may not be the most reasonable choice in all settings or for other procedures, for example when identifying true-positives does not outweigh consequences associated with including more false-positives in the study population.

Even though we have used administrative data from a large private insurer that cover regions in the Southeast, Mid-Atlantic, Eastern, Central, and Western regions of the United States, it should be noted that the assumption of constant sensitivity and specificity across populations may not hold unless the factors contributing to heterogeneity of populations could be measured and controlled. Therefore, sensitivity and specificity of our algorithms may not be the same when they are applied to populations and administrative datasets with distinctly different characteristics.³⁹ The two algorithms that we evaluated are coded independently by distinct entities (ICD-9-CM procedure codes by facilities and CPT-4 codes by providers) and are considered being inherently independent. There are uncommon instances where these could be coded by the same group in a hospital, for example by a hospital-employed surgeon. However, we avoided estimating two additional parameters for conditional covariances between the sensitivities and specificities of the algorithms because there is no extra information in data and the posterior estimates for conditional covariances would be the same as priors on these covariances.

The findings of this study suggest that both the ICD-9-CM- and CPT-4-based algorithms, especially in parallel combination, have high accuracy to identify cholecystectomy procedures in inpatient hospital populations using administrative data. Using multiple algorithms to increase the accuracy of detection is a common practice; however, the gain by parallel interpretation of these two algorithms is marginal since both algorithms are estimated to be highly accurate. In practice, whether an investigator uses multiple algorithms in parallel-, serial- or sequential-schemes or even individually depends on a specific setting, the accuracy of each individual algorithm, and the trade-off between sensitivity and specificity of the chosen scheme. Further, the study illustrates use of Bayesian latent class model to estimate accuracy of coding algorithms in administrative data.

KEY POINTS.

• The ICD-9-CM- and CPT-4 based algorithms have high accuracy to identify cholecystectomy procedures in administrative data. This has not been demonstrated before despite the use of administrative data to identify cholecystectomy in several past studies.

• Whenever possible, the use of ICD-9-CM- and CPT-4-based algorithms used in parallel combination is recommended to achieve the highest sensitivity to identify cholecystectomy procedures in administrative data.

APPENDIX

The resulting data after applying the ICD-9-CM- and CPT-4-based algorithms, denoted as algorithms 1 and 2, respectively, are cross-classified. It is assumed that the results of the two algorithms for a given patient are independent, conditional on the true status of each patient with regard to whether a cholecystectomy procedure was performed or not.

Let X = (X₁₁, X₁₀, X₀₁, X₀₀) be the vector of results for the two algorithm, where X₁₁ is the number of patients that are positive for both algorithm 1 and 2, X₁₀ (X₀₁) is the number of patients that are positive (negative) for algorithm 1 (algorithm 2), and X₀₀ is the number of patients that are negative for both algorithms 1 and 2. The observed cross-classified results are modeled as multinomial:

$$X~\text{multinomial}(N,(p_{11},p_{10},p_{01},p_{00}\left)\right),$$

where N is the sampled population and p’s are the multinomial probabilities corresponding to observed data, and are given by

$$\begin{array}{cc}\hfill p_{11}=& \pi {\mathrm{Se}}_1{\mathrm{Se}}_2+(1-\pi )(1-{\mathrm{Sp}}_1)(1-{\mathrm{Sp}}_2),\hfill \\ \hfill p_{10}=& \pi {\mathrm{Se}}_1(1-{\mathrm{Se}}_2)+(1-\pi )(1-{\mathrm{Sp}}_1){\mathrm{Sp}}_2,\hfill \\ \hfill p_{01}=& \pi (1-{\mathrm{Se}}_1){\mathrm{Se}}_2+(1-\pi ){\mathrm{Sp}}_1(1-{\mathrm{Sp}}_2),\hfill \\ \hfill p_{00}=& \pi (1-{\mathrm{Se}}_1)(1-{\mathrm{Se}}_2)+(1-\pi ){\mathrm{Sp}}_1{\mathrm{Sp}}_2,\hfill \end{array}$$

where π is the probability of the condition of interest in the population, and Se and Sp are the corresponding sensitivities and specificities of algorithms 1 and 2. For example, p₁₁ is calculated as p₁₁ = Pr(T1+,T2+) = Pr(T1+,T2+|D+)Pr(D+) + Pr(T1+,T2+|D−)Pr(D−) = πSe₁Se₂ + (1 − π)(1 − Sp₁)(1 − Sp₂), where T refers to an algorithm and D refers to true cholecystectomy status.

Furthermore, the sensitivity and specificity of the parallel-joint scheme are Se_Parallel = Se₁ + Se₂ − Se₁Se₂ and Sp_Parallel = Sp₁Sp₂, respectively. JAGS version 3.4.0 code to run the model:

    model{

    # likelihood

    X[1:4] ~ dmulti(p[1:4], N)

    p[1] <– pi * Se1 * Se2 + (1 − pi) * (1 − Sp1) * (1 − Sp2)

    p[2] <– pi * Se1 * (1 − Se2) + (1 − pi) * (1 − Sp1) * Sp2

    p[3] <– pi * (1 − Se1) * Se2 + (1 − pi) * Sp1 * (1 − Sp2)

    p[4] <– pi * (1 − Se1) * (1 − Se2) + (1 − pi) * Sp1 * Sp2

    Se.Parallel <– Se1 + Se2 − Se1 * Se2

    Sp.Parallel <– Sp1 * Sp2

    # priors

    pi ~ dbeta(, )

    Se1 ~ dbeta(, )

    Sp1 ~ dbeta(, )

    Se2 ~ dbeta(, )

    Sp2 ~ dbeta(, )

       }

If X₀₀ is not available, data X = (X₁₁, X₁₀, X₀₁) can be modeled as

$$X~\text{multinomial}(N,(p_{11},p_{10},p_{01}\left)\right)$$

where

$$\begin{array}{cc}\hfill p_{11}=& \pi {\mathrm{Se}}_1{\mathrm{Se}}_2+(1-\pi )(1-{\mathrm{Sp}}_1)(1-{\mathrm{Sp}}_2),\hfill \\ \hfill p_{10}=& \pi {\mathrm{Se}}_1(1-{\mathrm{Se}}_2)+(1-\pi )(1-{\mathrm{Sp}}_1){\mathrm{Sp}}_2,\hfill \\ \hfill p_{01}=& 1-(p_{11}+p_{10})\hfill \end{array}$$

The second scenario is more common in administrative data because often study data are extracted from complex and multi-layered administrative databases by querying for specific diagnosis or procedure codes; however, such data may not be accessible on all patients, especially patients who don’t have these specific codes.