Integrated statistical and decision models for multi-stage health care audit sampling

Abstract

Health care audits are crucial in managing the government insurance programs that are estimated to have losses amounting to billions of dollars every year. Statistical methods such as sampling have long been used to handle their size and complexity. Sampling from health care claims data can benefit from multi-stage approaches, especially when the evaluation of the tradeoffs between precision and cost is important. The use of decision models could facilitate health care auditors and policy makers make the best use of these sampling outputs. This paper proposes an integrated multi-stage sampling and decision-making framework that enables auditors address the tradeoffs between audit costs and expected overpayment recovery. We illustrate the framework and discuss insights utilizing a variety of overpayment scenarios for payment populations including U.S. Medicare Part B claims payment data.

1. Introduction

Increasing health care costs create financial concerns for policy makers in many countries. For instance, in the USA, annual health care expenditures reached $3.6 trillion or $11,172 per person in 2018, which accounted for $17.7\mathrm{\%}$ of the nation's Gross Domestic Product [7]. Up to 10% of this spending is estimated to be lost as overpayments in the form of fraud, waste and abuse [5]. These health care overpayments range from upcoding to sophisticated kickback networks; see Ekin [14] for an overview of examples. In addition to the adverse direct cost implications on both the government and tax-payers, these overpayments could also result in diminished quality of care and indirect negative health impacts on patients. In order to control and oversee health care spending, most health care insurance programs utilize audits. Traditionally, domain experts would manually investigate the billings and make informed decisions whether they are overpaid or not. Hence, health care audits are costly and time-consuming. The size and complexity of the systems deter health care officials from comprehensive auditing. Therefore, statistical sampling methods have become an integral part of health care audits.

Statistical sampling corresponds to choosing representative subsets of claims of interest to estimate the population parameters. Once health care audit sampling output is retrieved, the current practice mostly includes ad-hoc decision making. Despite the fact that a variety of sampling methods have been proposed, the statistical output has not been incorporated into decision models neither in practice nor the literature. This could decrease the efficiency of analytical methods used within health care fraud assessment. Health care audit decisions are complex and multi-faceted. They can involve consideration of audit costs, expected recovery amounts, accuracy of the extrapolations (overpayment estimation) and potential regret costs. In particular, the audit costs include the investigation time spent by the experts and the physical resources. The recovery costs correspond to the recovery due to audits as well as the projected recovery. Even a small improvement in these audit sampling decisions could be crucial in a multi-billion dollar domain that impacts most, if not all, of the population [13].

1.1. Statistical sampling for health care audits

The general objective of sampling is to make inference about overpayment through variables such as the proportion of overpaid claims and the overpayment amount. In the USA, current governmental sampling guidelines [3] recommend to use the lower limit of a one-sided $90\mathrm{\%}$ confidence interval for the total overpayments as the recovery (recoupment) amount from the audited provider. Using the lower bound protects the provider by preventing the recovery of an amount greater than the actual value of erroneous payments with higher confidence. Such overpayment estimation could be challenging for skewed populations, see [12,16,27] for alternative approaches. In practice, Rat-Stats [30] is among the statistical software that is used by the Office of Inspector General, Office of Audit Services to help with the statistical auditing. Woodard [37] presents the use of health care audit sampling practices by U.S. governmental programs.

The statistical sampling methods include but are not limited to simple random sampling, stratified and multi-stage sampling methods. Among these, simple random sampling is the most popular due to its ease of use and communication. Stratified sampling could be preferred when the auditor has information and/or interest about population subgroups. It separates the population into mutually exclusive, homogeneous segments (strata) by a stratification variable, and then draws samples from each segment (stratum). The weighted stratum overpayment estimates would yield a total overpayment estimate. Given a fixed pre-determined sample size, stratified sampling could result in a smaller margin of error compared to simple random sampling [8]. Having said that stratified sampling designs need to be constructed carefully, which include the choice of number of strata, the stratum boundaries, and the sample sizes of each stratum. There are a number of strategies that can be used to determine the stratum boundaries [20]. These include but not limited to Neyman allocation [29], cumulative square root rule [9] and the stratification approach [24]. For instance, Neyman allocation is based on considering the variance of the estimates within all strata.

The complexity of the health care claims data can benefit from multi-stage approaches, especially when the evaluation of the tradeoff between precision and cost is important. Rat-Stats allows the iterative use of sampling methods. Ignatova and Edwards [21] propose a multi-stage framework in that the total sample size is determined by evaluating the number of overpaid claims in a probe sample. Musal and Ekin [28] introduce an iterative stratified sampling method that uses Lindley's entropy measure to evaluate the expected information of the prospective samples. They use the 10th percentile of the posterior distribution of the total number of overpaid claims in the population to compute the recovery estimate. Their information-theoretic approach is shown to decrease estimation errors for heterogeneous medical claims data. However, their method is based on the estimation of a percentile of distribution and they employ it only for a known initial sample size.

1.2. Decision models for healthcare audits

Decision models have been used in health care services and administration extensively. The successful implementations include but are not limited to supporting management decisions in healthcare organizations, evaluation of health care providers and helping physicians to identify effective treatments [17]. However, the use of decision models to help subject domain experts remains to be an understudied step in the overall health care fraud assessment procedures [15]. The method of Iyengar et al. [22] is among the limited decision model-related work in health care fraud assessment. They propose a rule generation method for identifying and ranking candidate audit targets from a database of prescription drug claims. The modeling of the tradeoff between costs and recovery amounts in health care audits is discussed by Ekin [13], albeit without any numerical analysis. The need for measures to determine the effectiveness of the analytical methods has also been highlighted by the U.S. Governmental Accountability Office [18].

Health care fraud detection and the respective audit activities could benefit from decision models for better utilization of scarce resources. Such health care resource allocation methods have been applied in other domains of healthcare [1,38]. In the general fraud detection literature, a number of decision-theoretic approaches are considered within the financial and auto insurance industries. The evaluation of the tradeoffs among the cost of false positives and negatives is crucial and cost-based metrics such as ROC analysis are utilized [31]. Dionne et al. [11] propose a scoring based auditing approach in auto insurance fraud cases. Ulvila and Gaffney [36] present an integration of ROC and cost analysis to develop an expected cost metric for evaluating computer intrusion detection systems. Sahin et al. [32] propose a cost-sensitive decision tree approach for credit card fraud detection. Torgo and Lopes [35] address the prioritization of investigation leads for efficient allocation of audit resources. Their utility-based fraud detection model is based on using the likelihood of fraud, inspection costs and expected payoff.

1.3. Motivation, contribution and overview

This paper aims to fill the gap in health care audits literature and practice by proposing an integrated decision-making framework that utilizes the output of multi-stage sampling framework. This will facilitate health care audits and help policy makers. The proposed decision models consider initial and additional audit costs and expected recovery amount as well as a budget constraint. This framework enables auditors to analyze the resource allocation tradeoffs between audit costs and expected overpayment recovery while considering the impact of statistical learning within a predetermined budget.

To the best of our knowledge, other than a few simple rule-based frameworks, no decision models are utilized in the practice of health care audit sampling. In addition, none of the existing work in the literature has studied the impact of learning within the data-driven sampling decision model. This paper uses the entropy-based information quantification to determine the next stratum to sample from within a multi-stage sampling framework. It differs from the sampling method of [28] by utilizing Lindley's entropy for a set of decision alternatives and retrieving the overpayment recovery estimate with a different estimation method that is based on the confidence interval approach using the standard deviation and the mean of sample overpayments. This innovative application provides a semi-automated decision-making alternative within the health care fraud assessment systems that can help the auditors utilize the scarce audit resources efficiently. The proposed models are general in that it could be used within other audit sampling settings.

The paper is organized as follows. The following section presents the proposed multi-stage sampling method and the decision models. Section 3 presents the payment and overpayment data, while Section 4 illustrates the use of the integrated decision making and statistical framework with an analysis. The paper concludes with an overview and a discussion of future research directions. A brief Appendix presents complementary statistics of the data.

2. Methods

2.1. Multi-stage sampling framework

The information-theoretic multi-stage stratified sampling method used in this paper utilizes Neyman allocation for the initial allocation and Lindley's entropy measure for additional allocation. Neyman allocation is based on constructing homogeneous strata by the use of the mean and the variance of each stratum. Initial allocations are done so that the within variances among all strata are minimized. However, it does not consider the whole information content of the prospective samples. In order to quantify the available information, entropy has been long utilized in a number of different disciplines [33]. In this paper, we use Lindley's entropy interpretation [25]. In particular, entropy corresponds to the uncertainty summary about a probability distribution and is defined as the expected value of the log probability distribution [34]. We compute the information content of the current and prospective samples to determine the stratum that provides the highest expected information gain. This framework is based on quantifying the uncertainty of the number of overpaid samples, which is summarized by the derived Beta-binomial distribution. In order to satisfy the governmental guidelines, the left tail of a 90% confidence interval of the estimated mean overpayment amount is computed as the recovery amount. This framework enables us to compute the expected recovery amount estimates for all the decision alternatives of sample allocations to be used as inputs for the proposed decision models.

In the following, the details of the utilized sampling method are presented. First, we introduce the notation. $N_h$ is the number of claims in stratum h, which adds up to the total size of payment population as in $N=\sum _{h=1}^L{N}_h$. Similarly, $K_h$ and ${\rho }_h$ correspond to the number and proportion of overpaid claims in stratum h. The payment and overpayment amounts of claims in stratum h are represented by the vectors ${\mathbf{X}}_h=\{X_{(h,1)},\dots ,X_{(h,N_h)}\}$ and ${\mathbf{Y}}_h=\{Y_{(h,1)},\dots ,Y_{(h,N_h)}\}$, respectively. Once a sample of size n is retrieved and allocated, ${\mathbf{x}}_h=\{x_{(h,1)},\dots ,x_{(h,n_h)}\}$ and ${\mathbf{y}}_h=\{y_{(h,1)},\dots ,y_{(h,n_h)}\}$ denote the sample payment and overpayment amounts of claims in stratum while $k_h$ refers to the number of overpaid claims in stratum h with size $n_h$.

Next, we present how the initial and additional allocation of claims and the respective payment values are done. As part of the initial allocation, $n_{\mathrm{init}}$ number of claims are allocated to strata based on the mean and standard deviation of payment $\mu ({\mathbf{X}}_h)$ and ${\sigma }_{{\mathbf{X}}_h}$, for each stratum, $h=1,\dots ,L$. The initial sample size for stratum h is written as

$$n_{h,\mathrm{init}}=\frac{n_{\mathrm{init}}{N}_h{\sigma }_{{\mathbf{X}}_h}}{\sum _{h=1}^{h=L}{N}_h{\sigma }_{{\mathbf{X}}_h}}$$ (1)

where ${\sigma }_{{\mathbf{X}}_h}^2=\frac{\sum _{i=1}^{i=N_h}(X_{h,i}-\mu ({\mathbf{X}}_h){)}^2}{N_h}$. For allocation of additional samples, Lindley's entropy is used. In general, for stratum h, we would like to obtain $E_{k_{h,(\mathrm{add})},k_h}$, the expected information gain of observing an additional overpaid sample claim, $k_{h,(\mathrm{add})}$, from a pre-determined size of additional claims, $n_{h,(\mathrm{add})}$ given the number of overpaid claims in stratum h, $k_h$. In doing so, we will utilize the distribution of $k_h$ after already having observed $n_h$ claims with $k_{h,(n)}$ over-payments. For clarity, the stratum notation h is suppressed from the following equations:

$$\begin{array}{rl}& E_{k_{(\mathrm{add})},k}[\log \frac{p(k|k_{(n)},k_{(\mathrm{add})})}{p(k|k_{(n)})}]\\ & =\sum _{k_{(\mathrm{add})}}\sum _{k}p(k|k_{(n)},k_{(\mathrm{add})})p(k_{(\mathrm{add})}|k_{(n)})\log \frac{p(k|k_{(n)},k_{(\mathrm{add})})}{p(k|k_{(n)})}\end{array}$$

Since $p(k|k_{(n)},k_{(\mathrm{add})})=\frac{p(k,k_{(\mathrm{add})}|k_{(n)})}{p(k_{(\mathrm{add})}|k_{(n)})}$ ; this could be rewritten as

$$\begin{array}{rl}{E}_{k_{(\mathrm{add})},k}& =\sum _{k_{(\mathrm{add})}}\sum _{k}p(k,k_{(\mathrm{add})}|k_{(n)})[\log p(k,k_{(\mathrm{add})}|k_{(n)})\\ & -\log (p(k_{(\mathrm{add})}|k_{(n)}))-\log (p(k|k_{(n)}))]\end{array}$$ (2)

The probability distribution of k among n claims is Binomial with parameters $(n,\rho )$: $(k|n,\rho )\sim \mathrm{Bin}(n,\rho )$. We define the prior distribution of ρ with a Beta distribution with the parameter vector $\omega =(\alpha ,\beta )$. The parameter vector $\omega$ ; n, and $n_{(\mathrm{add})}$ are suppressed for notational clarity. The parameters, α and β are updated after observing data: $\alpha =a+k_{(n)}$ and $\beta =b+n-k_{(n)}$, respectively. The hyper-parameter values a and b are set to be 0.01 and 0.01, which corresponds to a weakly informative distribution. These are determined such that there exists a non zero probability of a claim being legitimate or fraudulent while having high sensitivity to the observed data. The posterior distribution of $p(k|k_{(n)})$ is therefore defined as Beta-binomial due to conjugacy. Similarly, the posterior distribution of $k_{(\mathrm{add})}$ is retrieved as a Beta-binomial distribution:

$$\begin{array}{rl}p(k_{(\mathrm{add})}|k_{(n)})& \sim (\genfrac{}{}{0ex}{}{n_{(\mathrm{add})}}{k_{(\mathrm{add})}})\frac{\mathrm{\Gamma }(k_{(\mathrm{add})}+\alpha )\mathrm{\Gamma }(n_{(\mathrm{add})}-k_{(\mathrm{add})}+\beta )\times \mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(n_{(\mathrm{add})}+\beta +\alpha )\times \mathrm{\Gamma }(\alpha )\times \mathrm{\Gamma }(\beta )}.\end{array}$$ (3)

These distributions are used within the following sampling framework to retrieve the overpayment recovery estimate as a function of decision alternatives of the initial and additional allocation sample sizes. The steps of the utilized sampling framework are:

1. For each stratum, use an initial Neyman allocation scheme and retrieve $k_{h,(n)}$ overpayments among n sampled claims.

2. The expected information gain of an additional sample, $n_{h,(\mathrm{add})}$ from stratum h, with the outcome $k_{h_{,}(\mathrm{add})}$, is computed by using Equation (2.1) for each stratum.

3. Determine $h^{\ast }$, the stratum with the highest expected information gain $h^{\ast }={\mathrm{argmax}}_h[E_{k_{(\mathrm{add})},k}]$ and sample from that stratum.

4. The expected total overpayment amount, $\hat{Y}$ is computed given the sample mean overpayment for each stratum, ${\overline{y}}_h$, $h=1,\dots ,L$ : $\hat{Y}=\sum _{h=1}^{h=L}{N}_h{\overline{y}}_h$.

5. The pooled standard deviation estimate for the total overpayment amount is computed as $s_{\hat{Y}}=\sqrt{\sum _h\frac{(N_h^2\ast (N_h-n_h)\ast s_h^2)}{(n_h\ast N_h)}}$. The overpayment standard deviation estimate of each stratum h is $s_h={\overline{x}}^2{s}_{\rho }^2+{\overline{\rho }}^2{\sigma }_{\mathbf{X}}^2+s_{\rho }^2{\sigma }_{\mathbf{X}}^2$. The sample mean of overpayment proportion is $\overline{\rho }=\frac{k_h}{n_h}$ whereas its sample variance is denoted as $s_{\rho }^2$, and ${\sigma }_{\mathbf{X}}$ is the payment standard deviation.

6. Finally, the overpayment recovery estimate, $Y_{\mathrm{rec}}$ is computed as

   $$Y_{\mathrm{rec}}=\hat{Y}-t_{0.1,n-1}\ast s_{\hat{Y}}.$$ (4)

As a result, we retrieve the total overpayment recovery estimate, $Y_{\mathrm{rec}}$, as a function of $n_{\mathrm{init}}$ and $n_{(\mathrm{add})}$ to be used within the proposed decision models.

2.2. Integrated decision models

We present two audit sampling decision models to be used depending on the state of the audit. The first model is proposed for cases where the auditor has only determined the provider of interest, and has access to the related payment data. This model enables auditors to consider the tradeoffs between audit costs and expected recovery while deciding how to allocate the sampling resources among the initial and potential additional investigations within the budget. The second model solves for the decision problem where the objective is to find the optimal additional sample size for a given initial sample.

Model 1 is written as

$$\begin{array}{rl}\underset{n_{\mathrm{init}},n_{(\mathrm{add})}}{max}& E[rY{r}_{\mathrm{rec}}+(n_{\mathrm{init}}+n_{(\mathrm{add})})\overline{y}]-c_1{n}_{\mathrm{init}}-c_2{n}_{(\mathrm{add})}\\ \mathrm{s}.\mathrm{t}.& c_1{n}_{\mathrm{init}}+c_2{n}_{(\mathrm{add})}\le B,(n_{\mathrm{init}},n_{(\mathrm{add})})\in A\end{array}$$ (5)

The objective of this simple optimization model is to minimize the total audit cost and maximize the expected recovery gain within a given budget. The objective function could be vaguely referred to as net gain. The decision variables are the initial and additional number of claims to sample; $n_{\mathrm{init}}$ and $n_{(\mathrm{add})}$.

Total audit costs correspond to the initial and additional sample resource allocation. The additional samples generally cost more than the initial samples, taking into account the extra investigation setup after the initial allocation. Hence, the unit audit cost for initial investigations is assumed to be less than the unit audit cost for additional investigations; $c_1<c_2$. The total audit cost, $c_1{n}_{\mathrm{init}}+c_2{n}_{(\mathrm{add})}$, cannot exceed the total audit budget, B.

The recovery consists of both the recovery due to audits and expected recovery from the population. The sample overpayment value is generally requested in full from the investigated provider as a result of the audits. The average recovery due to audits is estimated as the average sample overpayment, $\overline{y}$. The demanded refund from the provider under investigation is $(n_{\mathrm{init}}+n_{(\mathrm{add})})\overline{y}$. The expected recovery, $Y_{\mathrm{rec}}$, is computed according to the governmental guidelines as part of the sampling framework. It is a function of the decision alternatives from set A ; as in $n_{\mathrm{init}},n_{(\mathrm{add})}\in \mathit{A}$. In practice, the government is only able to recover a certain percentage of the inferred population overpayments. Thus, the recovery gain from the population is discounted by using a recoupment percentage, r; so the gain due to recovery can be written as $r{Y}_{\mathrm{rec}}$.

In this model, the objective function consists of $E[r{Y}_{\mathrm{rec}}+(n_{\mathrm{init}}+n_{(\mathrm{add})})\overline{y}]$. The expectation function is used to consider the randomness in the drawn samples and the resulting sample overpayment values. $Y_{\mathrm{rec}}$ and $\overline{y}$ are the random variables in our model, and differ in a given replication. While solving for the model, we use Monte Carlo simulation with a reasonably large number of replications to estimate the expectation function.

Model 2 could be considered as a subset of Model 1. It is based on maximizing expected utility given the already allocated initial sample size: $\underset{n_{(\mathrm{add})}}{max}E[n_{(\mathrm{add})}\overline{y}+r{Y}_{\mathrm{rec}}]-c_2{n}_{(\mathrm{add})}$. It focuses on the tradeoffs between the cost and expected recovery involved with choosing the additional sample size. The functional details are as same as Model 1, therefore excluded from the discussion for brevity.

3. Health care claims data

This paper utilizes two payment populations to demonstrate the versatility of the framework. The first payment population is compiled by Musal and Ekin [28], using the publicly available data from 2008 CMS Outpatient Procedures [4]. It includes the procedure codes with frequent overpaid billings in investigations such as the billings for ‘J9041’ (Injection of Bortezomib 0.1 mg). This data set consists of 8278 claims with payment values. The actual allocation of payments to L = 5 strata is done using the R package GA4Stratification [23] so that each stratum consists of increasing dollar amounts of payments. Table 1 presents the descriptive statistics of payment values.

Table 1.

Descriptive statistics of real world payment population.

Stratum	Mean	Std. dev.	Median	$N_h$
h = 1	69.40	46.92	40.00	3949
h = 2	915.69	180.87	70.00	1402
h = 3	2335.37	244.61	2500.00	588
h = 4	3076.60	206.76	3000.00	1675
h = 5	4012.65	246.71	4100.00	664
Overall	1298.47	1441.22	600.00	N = 8278

We simulate another payment population which has left skewness, see Figure 1 for the density plot. Left skewness is oft seen in health care claims payment data where providers may upcode the system. Our left skewed population has the same population size as the real life data. It has lower variance and correspond to relatively higher payment values compared to the real world payment population. Table 2 presents the descriptive statistics of the left skewed payment population.

Figure 1.

The density plot of the left skewed payment population.

Table 2.

Descriptive statistics of left skewed payment population.

Stratum	Mean	Std. dev.	Median	$N_h$
h = 1	7390.03	474.59	7523.10	527
h = 2	8274.94	195.62	8296.77	1351
h = 3	8844.99	145.28	8849.42	1970
h = 4	9286.63	114.38	9289.63	2329
h = 5	9679.78	127.71	9665.91	2101
Overall	8995.53	672.08	9125.18	N = 8278

3.1. Overpayment data

While using the proposed framework in an actual audit, the decision makers would have access to the audit results including overpayment values. However, such audit results and overpayment data are not publicly available due to privacy concerns. In addition, the proposed general decision making framework should be tested for different overpayment patterns in order to demonstrate its versatility. Therefore, we resort to simulation and construct a number of overpayment scenarios. The overpayment amounts of populations, ${\mathit{Y}}_h$ for each stratum, $h=1,\dots ,L$ are retrieved.

We assume that all claims are either fully legitimate or overpaid, similarly to the existing literature, i.e. [12]. For each scenario, the overpayment data is generated by using the overpayment proportion parameter ${\pi }_{i,h}$ for stratum h and ith scenario. In order to represent various scenarios, we utilize seven scenarios as presented in Table 3.

Table 3.

Overpayment proportions for each stratum and scenario.

Scenario	${\pi }_{i,h}$	h = 1	$h=2$	h = 3	h = 4	h = 5
i = 1	${\pi }_{1,h}$	0.00	0.25	0.50	0.75	1.00
i = 2	${\pi }_{2,h}$	1.00	0.75	0.50	0.25	0.00
i = 3	${\pi }_{3,h}$	0.05	0.25	0.50	0.75	0.95
i = 4	${\pi }_{4,h}$	0.95	0.75	0.50	0.25	0.05
i = 5	${\pi }_{5,h}$	0.50	0.50	0.50	0.50	0.50
i = 6	${\pi }_{6,h}$	0.25	0.25	0.25	0.25	0.25
i = 7	${\pi }_{7,h}$	0.75	0.75	0.75	0.75	0.75

The first two scenarios have cases that represent differing proportions of overpaid claims in each stratum. For instance, the first and fifth strata for the second overpayment scenario has ${\pi }_{2,1}=1$ and ${\pi }_{2,5}=0$. This means all claims in the first stratum are fraudulent indicating that low payment claims are fraudulent in that particular scenario. Whereas all high payment claims (which correspond to the fifth stratum) are fully legitimate. On the other hand, scenarios 5, 6 and 7 have the same overpayment proportions throughout all strata.

Using these choice of parameters, we run a simulation of size 5000, and record the averages of overpayment data for each payment population and overpayment scenario. This enables us to consider different settings, and demonstrate the versatility of the proposed models. The average descriptive statistics per claim of the total overpayment data for the real world and simulated payment populations are presented in the Appendix.

4. Application

This section mainly demonstrates the application using the simulated left skewed payment population. First, we go through the steps of the integrated statistical sampling and decision making process to illustrate the learning aspect. We discuss expected net gains based on decisions involving an auditing budget. Within our framework a decision involves an initial $(n_{\mathrm{init}})$ and an additional $(n_{\mathrm{add}})$ sample size whose overall auditing costs compose the budget. The alternatives for the initial sample size decision should be large enough for proper implementation of Central Limit Theorem and stratified sampling. It is difficult to recommend a general region for the decision search space of the sample size, as it depends on the characteristics of the population. These are generally determined by the auditors. One practical issue to consider is the search size of $n_{\mathrm{add}}$ at each iteration. In this particular paper, we consider increments of 1 which corresponds to investigating the impact of sampling a new claim for each decision alternative of initial sample size. This provides the most general and flexible application of the proposed decision framework. In practice, it might be preferred to sample a fixed amount of additional claims, say 5, all at once for convenience. This can be accommodated by the proposed iterative decision-making framework. Next, we present evidence of the statistical validity of the results. Finally, we conduct a sensitivity analysis with respect to varying parameters, and discuss the changes in outcomes for various overpayment scenarios and payment populations.

First, we present the step by step demonstration of the multi-stage sampling algorithm for a particular alternative of the initial sample size using a randomly chosen replication. For demonstration, we choose the initial sample size as 45, and increase additional sample size with increments of 1 up to 90. Table 4 presents the output of the iterative allocation of the additional samples for some iterations as well as the initial and final allocation among strata.

Table 4.

One replication of the multi-stage sampling framework evolution for left skewed payment population and overpayment scenario 2.

Iteration	h = 1	h = 2	h = 3	h = 4	h = 5
Initial allocation	8	9	10	9	9
t = 1, n = 46	0	0	0	1	0
t = 2, n = 47	0	0	0	1	0
t = 3, n = 48	0	1	0	0	0
t = 4, n = 49	0	1	0	0	0
⋮	⋮	⋮	⋮	⋮	⋮
t = 42, n = 87	0	1	0	0	0
t = 43, n = 88	0	0	0	1	0
t = 44, n = 89	0	0	1	0	0
t = 45, n = 90	0	0	1	0	0
Final allocation	8	14	34	25	9

Initially, at t = 0, Neyman allocation provides the initial allocation of samples to 5 strata. The remaining allocations are done with respect to the expected amount of information to be gained from the next sampling iteration. 8 claims are allocated to the first stratum at t = 0. In overpayment scenario 2, the probability of overpayments in the first stratum is 1. Therefore, the algorithm concludes that there is no more information to be gained by sampling from that stratum. That stratum is not allocated any other samples during the remaining iterations. Similarly, the fifth stratum has no overpayments in the population, hence no more samples are allocated. On the other hand, there is a larger number of samples allocated to strata 3 and 4 since there is more information to be gained by sampling from these strata. This indicates learning and results in a decreased standard error for these strata in the final recovery dollar amount.

The proposed statistical method facilitates iterative learning, which differentiates it from other multi-stage sampling frameworks such as Neyman allocation. Table 5 provides a comparison of sampling resource allocation between the Neyman allocation and the utilized entropy-based method. The impact of learning within the ultimate sampling allocation could be recognized from this output. While the entropy-based method does not allocate any more samples to the first and fifth strata after the very initial sample allocation, the Neyman allocation keeps allocating many samples to those strata due to variance.

Table 5.

The sample allocation of Neyman allocation vs information theoretic sampling for left skewed payment and overpayment scenario 2.

Algorithm	h = 1	h = 2	h = 3	h = 4	h = 5	Total
Information theoretic	8	14	34	25	9	90
Neyman allocation	20	17	16	17	20	90

The outcomes of the 5000 replications of the sampling method are used as inputs for the decision models. For the decision models, we choose the unit sampling costs as $1000 and $3500 for demonstration purposes. The sampling costs include but are not limited to the investigation setup, salary and expenses of the auditor. The recoupment percentage, r is assumed to be 0.1, using the information in the Annual Report to Congress on the Medicare and Medicaid integrity programs for fiscal years 2013 and 2014 [6]. In particular, 490 billion dollars were spent on Medicare in 2014. Assuming 10 % overpayment rate, out of 49 billion dollars overpayments, only 4.765 billion dollars were recovered. This approximately corresponds to a recovery rate of 10 cents for each overpaid dollar. The budget, B is set as 1,000,000. All parameters can be modified depending on the nature of the medical audit.

First, we utilize the proposed Model 1 to determine the tradeoffs between all decision alternatives when the analyst does not have any prior allocation information. For illustration, we have defined the discrete set of decision alternatives, A in the ranges of $[15,60]$ for $n_{\mathrm{init}}$ and $[1,60]$ for $n_{(\mathrm{add})}$ with increments of 1. For instance, for the left-skewed payment population and overpayment scenario 2, the optimal decision is found to be $\{60,10\}$. This provides the auditor with the best resource allocation within the budget when he/she does not have any information with regards to the initial sample size. On the other hand, Model 2 can be utilized for any given initial sample size. For instance, in the following, we share the optimal solutions of Model 2 given four alternatives of initial sample sizes for the left-skewed payment population and overpayment scenario 2. In particular, the initial sample sizes of 15, 30, 45 and 60 are already allocated to 5 strata. For this pre-determined initial sample size and cost assumptions involving the initial and added samples, we obtain the additional sample size for which the net gain is maximized.

Table 6 shows the total sample allocation decisions that provide the optimal net gain for each of the selected decision alternatives of initial sample sizes. These are 29 ( 15 + 14 ), 57 ( 30 + 27 ), 60 ( 45 + 15) and 70 ( 60 + 10), respectively. The higher number of additional samples do not necessarily result in higher net gains. For instance, in the case of an initial sample size of 60; the net gain becomes the highest when 10 additional samples are allocated. The cost of additional sampling units is not justified by the potential additional expected recovery that is provided by the learning from samples. It should be noted that in the unlikely case of negligible sampling costs; sampling more could be preferred since that would decrease the standard error and improve estimation; and recovery amount would be closer to the actual population overpayment. Figure 2 further illustrates the proposed model where every circle represents the average net gain for each of the decision alternatives of additional samples. The trends were smoothed out by using the Loess curves to help recognize concave behaviour in some cases.

Table 6.

Optimal additional sample allocation decisions for select decision alternatives of initial sample sizes.

$n_{\mathrm{init}}$	$15+$	$30+$	$45+$	$60+$
$n_{(\mathrm{add})}^{\ast }$	14	27	15	10

Figure 2.

Expected net gain (y-axis, in 1000 s) for additional allocation and total sample size alternatives (x-axis). In particular, from top left to bottom right, additional claims up to the sizes of $\{15,30,45,60\}$ are added with increments of one to samples with the initial claim sizes of $\{15,30,45,60\}$, to result with a total number of audited claims up to maximum values of $\{30,60,90,120\}$, respectively.

Next, we present a discussion based on the fitted Loess smoothing curve for the 45 additional samples taken on top of the initial sample of 45. For this scenario, the total population overpayment is computed as $26,398,584. The total overpayment recovery is estimated to be between $20,595,164 to $22,390,622 for total sample sizes ranging from 46 to 90, as standard error varies between $2,936,329 and $4,107,211.

Figure 3 provides the expected net gain for the decision alternatives of additional allocation. The expected net gain on the y-axis of this plot is the difference between extrapolated recovery amount and the sampling cost. The initial steep increase is followed by a flattening out and a steep decrease of the expected net gain. Two vertical lines are drawn to illustrate how the expected net gain differs with respect to the decision of additional sample size. If 50 claims (5 additional claims in addition to the initial 45) are used, it is quite clear it would lead to a sub-optimal solution. The preferred solutions seem to concentrate between 60 and 70 claims. Increasing sample size decreases the standard error of the recovery at a diminishing rate.

Figure 3.

Expected net gain (y-axis, in 1000 s) for select decision alternatives (x-axis). In particular, additional claims up to a size of 45 are added with increments of one to a sample with the initial claim size of 45 to result with a total number of audited claims up to maximum values of 90.

These results show the importance of performing cost analysis for the decision maker to determine the optimal sampling resource allocation. This is where the proposed decision-modeling framework becomes beneficial. The proposed decision model allows the decision maker to assess tradeoffs with respect to the cost and learning due to the additional samples and its impact on expected net gain. This is the main contribution of the proposed integrated decision making and sampling framework.

It is crucial that sampling output and overpayment estimates are statistically valid to be used in officially acceptable health care audits. Since we use the officially accepted lower limit of a one-sided $90\mathrm{\%}$ confidence interval for the total overpayments as the recovery (recoupment) amount from the audited provider, the coverage probabilities are expected to be around $90\mathrm{\%}$. In order to assess that, Figure 4 displays the average coverage probabilities for the aforementioned case. The probabilities for almost all additional sample sizes are around $90\mathrm{\%}$ line. If they get higher compared to that line, they are deemed as inefficient. Whereas if they are lower than 90 %; they have the risk of not having enough coverage which indicates statistical invalidity. For brevity, we do not provide the coverage probabilities for other cases. They are reasonable and available on request.

Figure 4.

Expected coverage probability (y-axis) for select decision alternatives (x-axis). In particular, additional claims up to a size of 45 are added with increments of one to a sample with the initial claim size of 45 to result with a total number of audited claims up to maximum values of 90.

4.1. Sensitivity analysis

So far, we have presented the results for a particular case and set of parameters. In the following, we will share results for different parameter values to assess the sensitivity. First, we run a sensitivity analysis for the unit cost of additional samples; $c_2$.

Table 7 gives the optimal allocation for different values of $c_2$ for initial allocation of 45. The inverse relationship between $c_2$ and $n_{(\mathrm{add})}$ can easily be recognized. When it becomes very costly to draw additional samples, i.e. $c_2=\mathrm{\$}6000$; the optimal additional sample size decreases to 11, whereas when there is no additional cost other than the regular sampling cost, i.e. $c_1=c_2=\mathrm{\$}1000$; the optimal additional sample size is found as 44.

Table 7.

Optimal additional sample sizes for initial allocation of 45 with varying $c_2$.

$c_2:$	$1000	$2250	$3500	$4750	$6000
$n_{(\mathrm{add})}^{\ast }$	44	44	15	15	11

Next, we analyze the sensitivity of the results for different overpayment scenarios. In doing so, we provide a comparison of the integrated decision and sampling framework for decision alternatives of initial sample sizes of 15, 30, 45 and 60 for all overpayment scenarios and left-skewed payment population.

Table 8 reveals the differences in the optimal allocation for various overpayment scenarios. For instance, the differences for the optimal decisions of Scenario 6 compared to other scenarios can be recognized. This could potentially be explained by the relatively low and equal probability of overpayments across all strata. Hence, the additional samples may not provide as much as value to justify their cost. The additional allocation also highly depends on the initial sample size. For the lower initial allocation alternatives such as 15, initial allocations result in large standard errors which in turn increases optimal additional allocation, even in scenario 6. Whereas when the initial sample allocation is 60, the optimal additional allocation values are not high compared to the initial allocation for any of the scenarios.

Table 8.

Optimum additional sample sizes for various overpayment scenarios and the left-skewed payment population.

$n_{\mathrm{init}}$	Scen 1	Scen 2	Scen 3	Scen 4	Scen 5	Scen 6	Scen 7
15	15	14	14	15	14	12	15
30	23	27	29	29	28	1	30
45	29	15	37	12	43	2	43
60	33	10	35	10	17	1	35

Lastly, we replicate the analysis for the real world payment population. Table 9 presents the final sampling resource allocations for all overpayment scenarios. Figure 5 presents the allocation and its impact on expected net gain for the four alternatives of initial sample allocation.

Table 9.

Optimum additional sample size for various overpayment scenarios for the real world payment population.

	Scen 1	Scen 2	Scen 3	Scen 4	Scen 5	Scen 6	Scen 7
$15+$	15	14	15	14	15	1	15
$30+$	30	19	29	9	29	1	30
$45+$	34	9	31	3	37	1	38
$60+$	21	3	27	1	14	1	42

Figure 5.

Expected net gain (y-axis, in 1000 s) for additional allocation and total sample size alternatives (x-axis) for the real world payment population. In particular, from top left to bottom right, additional claims up to the sizes of $\{15,30,45,60\}$ are added with increments of one to samples with the initial claim sizes of $\{15,30,45,60\}$, to result with a total number of audited claims up to maximum values of $\{30,60,90,120\}$, respectively.

The results for the first payment (real-world) population are slightly different, but are still in line with the earlier findings. The resource allocation differs with respect to the initial sample size and the overpayment scenario. The results of the tradeoff between investigation costs and expected recovery amounts becomes more evident for overpayment scenarios with a low probability of overpaid claims. Similar to the previous findings, this is clearly demonstrated for the 6th overpayment scenario, where the population overpayment probabilities are fixed at 25%. The additional sampling allocation is shown to have the least amount of benefit on expected net gain despite the improvements in expected recovery estimation.

Overall, the proposed frameworks enable auditors to assess the tradeoffs between costs and expected recovery amount while potentially considering the budget constraint. In particular, the tradeoff between the cost of drawing an additional sample and improvement in expected recovery due to learning and decreasing standard error is crucial. Additional sampling can be preferred despite its higher unit cost when the expected recovery becomes higher due to learning. The relative advantage of the information-theoretic method is smallest in the cases of small additional sample sizes especially for lower sizes of initial allocations. The monetary gain contribution is shown to decrease with increasing total sample size. It should be emphasized that since auditors generally utilize relatively small samples for medical audits, even small improvements in sampling designs are crucial.

5. Conclusion

The size and heterogeneity of medical claims data prompt the use of statistical and decision modelling tools to aid in medical audits. Although the statistical and data mining tools have been adopted widely in the last three decades, decision models have not been incorporated to the health care audit practice. This creates inefficiency in handling health care overpayments, which are estimated to be in billions of dollars. This paper fills that gap by presenting an integrated medical audit sampling decision analytics framework. Our decision-making framework builds on the output of information-theoretic multi-stage stratified sampling approach and ensures the overpayment amount estimates are statistically valid. In doing so, the impact of learning within the data-driven statistical model on the decision model is evaluated.

The proposed multi-stage sampling framework presents overpayment recovery estimates that are valid with respect to governmental guidelines. This output is utilized by the proposed decision models, and in turn, enable health care policy decision makers and auditors assess the tradeoffs between the audit costs and expected overpayment recovery for decision alternatives of initial and additional sample sizes. In cases where the auditor is in the beginning of an investigation, Model 1 could help with both sample resource allocation decisions at the initial and possibly additional phases. Model 2 assumes that the initial allocation is already done, and lets the decision maker choose the size of the additional sampling allocation. In cases where the auditor is not interested in additional samples, which is generally the practice of choice, the models help the auditors evaluate potential inefficiencies as a result of their decisions. Payment data from the U.S. Medicare Part B claims and simulated left-skewed data are used for various overpayment patterns to demonstrate the use of proposed decision models and to illustrate tradeoffs.

Recent advances in information technology allow wider adoption of statistical and decision models to deal with challenges in the health care industry. This could make semi-automated decision-making frameworks a viable option for health care audit sampling. This innovative application presents great potential to improve the efficiency of health care audit sampling decision making. Given the scarce nature of health care fraud assessment resources, even a small improvement would correspond to gain of millions of dollars as well as the improvement in public confidence to health care audit systems. The proposed framework is general in that it could be used within other audit sampling settings such as tax audits.

There are a number of limitations and potential future research directions. The extent of sensitivity analysis can be improved. For instance, the impact of changes for the unit recoupment percentage could be further analyzed. A more comprehensive decision-making framework could be designed over multi-stages in which the initial investigation decision at the first stage can impact the potential decisions at later stages. An alternative version could include using the entire distributions of the number of overpaid claims and recovery amount in a fully Bayesian framework.

In terms of the decision model, other decision criteria such as the potential regret cost due to overpayment estimation errors could be considered. The estimation errors decrease with larger sample sizes. Such incorporation of regret cost could be feasible for cases with known overpayment patterns. In addition, the increments in the decision alternatives of the additional sample size, $n_{\mathrm{add}}$, could be generalized by using any positive integer value instead of the value one as in our application. Our choice of decision alternative increments of 1 allows maximum flexibility in the decision-making framework realistically assuming the investigation of claims one at a time. However, for ease of implementation, the auditors may prefer to run the proposed framework for higher alternative values of the additional sample size.

Lastly, the proposed decision-making framework is compatible with alternative multi-stage sampling and estimation frameworks. Potential alternatives could include the proposal of a sequential interval estimation procedure, see [19,26] for overviews. For instance, Chattopadhyay and De [2] propose interval estimation of the Gini index with a specified confidence coefficient and a specified margin of error. They establish an iterative scheme where variance and confidence interval are recomputed at each iteration using the increased sample size, until the pre-specified confidence interval width is obtained. This could be modified to construct an alternative method for computing the required sample size. In cases where the claims data can be grouped by procedure codes to form clusters, sequential estimation frameworks as in [10] could also lead to more efficient sampling procedures.

Appendix.

Overpayment data

Tables A1 and A2 present the average descriptive statistics per claim of the total overpayment data for the real world and simulated payment populations, populations 1 and 2, respectively. The differences in the descriptive statistics between scenarios and the payment populations can be recognized easily. In order to provide a better understanding of the overpayment data, the descriptive statistics of a particular scenario are presented in Table  A3.

Table A1.

Overpayment population average descriptive statistics per claim generated for payment population 1 and each overpayment scenario.

Scenario	Mean	Std. dev.	2.5%	25%	Median	75%	97.5%
i = 1	910	1464	0	0	0	2469	4300
i = 2	388	800	0	0	50	150	3497
i = 3	896	1451	0	0	0	2383	4300
i = 4	403	833	0	0	50	150	4299
i = 5	649	1208	0	0	9	629	4300
i = 6	324	914	0	0	0	11	4300
i = 7	974	1369	0	8	90	2251	4300

Table A2.

Overpayment population average descriptive statistics per claim generated for payment population 2 and each overpayment scenario.

Scenario	Mean	Std. dev.	2.5%	25%	Median	75%	97.5%
i = 1	5667	4568	0	0	8951	9504	9992
i = 2	3344	4179	0	0	0	8410	9494
i = 3	5572	4577	0	0	8901	9484	9992
i = 4	3439	4220	0	0	0	8467	9966
i = 5	4505	4529	0	0	3433	9143	9990
i = 6	2254	3918	0	0	0	3569	9986
i = 7	6758	3943	0	3249	8849	9383	9991

Table A3.

Overpayment average descriptive statistics per claim for each stratum generated for payment population 1 and overpayment scenario 3.

Stratum	Mean	SD	2.5%	25%	Median	75%	97.5%
h = 1	52	51	0	6	40	78	347
h = 2	687	426	0	223	950	1000	1486
h = 3	1751	1033	0	860	2397	2500	2500
h = 4	2306	1345	0	1368	2900	3229	3500
h = 5	3010	1750	0	1857	3871	4200	4300

Funding Statement

This work was supported by the Texas State University 2019/2020 Faculty Development Leave and Presidential Research Leave Award. It was also partially supported by the National Science Foundation under Grant DMS-1638521 to the Statistical and Applied Mathematical Sciences Institute.

Disclosure statement

No potential conflict of interest was reported by the author(s).