Accuracy of prescription drug expenditure forecasts published in the American Journal of Health-System Pharmacy

Abstract

Purpose

To evaluate the accuracy of the forecast of drug expenditures in nonfederal hospitals and clinics published annually in the American Journal of Health-System Pharmacy (AJHP) compared to the drug expenditure forecasts produced annually by the Centers for Medicare and Medicaid Services (CMS).

Methods

The forecasted drug expenditure growth published in AJHP for nonfederal hospitals (for the years 2003 to 2013) and clinics (for the years 2004 to 2013) was compared to the actual growth each year. The actual and forecasted growth published by CMS was analyzed for the years 2003 to 2012. The mean absolute error (MAE) and directional accuracy for the AJHP forecasts for nonfederal hospitals and clinics, and for the CMS forecasts, were determined and compared.

Results

Actual growth was within the forecasted range in 2 of 11 years for nonfederal hospitals, and in 3 of 10 years for clinics. The forecasts were directionally accurate 27.3% and 60.0% of the time for nonfederal hospitals and clinics, respectively. The MAE for the nonfederal hospital and clinic drug expenditure forecasts were 2.0 and 4.7 percentage points, respectively. The CMS forecasts were directionally accurate 70% of the time, and the MAE was 2.2 percentage points and was not statistically different than the AJHP forecasts.

Conclusion

The forecasts published annually in AJHP have comparable accuracy that by CMS for predicting prescription expenditure growth. The forecast paper provides an overview of current trends, which must be combined with local information to accurately forecast institutional drug expenditures.

Introduction

Since 1992, the American Journal of Health-System Pharmacy (AJHP) has published an annual report describing trends and projections for national prescription drug expenditures.^1–23 Each report analyzed factors expected to impact drug expenditures in the upcoming year - such as recent or expected drug approvals, drugs expected to lose patent protection, policy changes, and historical data on drug spending - with the aim of providing information to health-system pharmacists to help understand national drug expenditure trends. The reports also sought to guide pharmacy executives in establishing budget projections for their own institutions, compare expenditure growth at their institution to the national average, and provide insights about factors influencing pharmaceutical expenditures.

The annual reports included forecasted growth in prescription drug expenditures, given as a percent increase or decrease from the previous year, and focused on non-federal hospitals and clinics. Such data are not available elsewhere and as a result the forecasts are frequently cited and widely used. However, the accuracy of the forecasts has not been systematically evaluated. Understanding how actual growth may deviate from that predicted each year is obviously important to users of the information. Further, while forecasting is inherently difficult, evaluating accuracy may help identify improvements that could be made in the methods of forecasting.

The purpose of this paper was 1) to examine the accuracy of the annual forecast of drug expenditures in non-federal hospitals, 2) and in clinics, and 3) to compare the accuracy of these forecasts to that produced annually by the Centers for Medicare and Medicaid Services (CMS).

Methods

This is a retrospective evaluation of the accuracy of the predicted percentage growth of drug expenditures (forecast) for nonfederal hospitals and clinics published annually in AJHP. The nonfederal hospital and clinic sectors were the focus of this analysis because these are of most interest to health-system pharmacists, were the most consistently forecasted over time, and are not forecasted elsewhere.

The data source for the predicted percentage growth of drug expenditures is the annual drug expenditure forecast papers published in AJHP from 1992–2014. For each article, the forecasted growth in expenditures for nonfederal hospital and clinics were abstracted. Papers not including these sectors were excluded from the analysis. In these articles, the forecasted growth for each sector each year was generally made as a 2% range in the anticipated increase or decrease in expenditures (except 2005 when a 3% range was forecasted). The forecasts were presented as ranges because the intent was to provide guidance to health system pharmacists for budgeting purposes, and ranges were thought to better allow for incorporating local market factors.

The data source for the actual drug expenditures is the IMS Health National Sales Perspective (NSP) database.²⁴ The NSP sample included data from more than 100 pharmaceutical manufacturers and more than 700 distribution centers, totaling more than 1.5 billion annual transactions. NSP is a statistically valid projected audit that describes 100% of the sales in every major distribution channel for prescription pharmaceuticals, nonprescription products, and select self-administered diagnostic products in the US, measuring both unit volume and invoice dollars.^{23, 25}

A time series approach was used to describe the predicted and actual growth for each year for hospitals and clinics separately. For each year we also determined if the actual growth in drug expenditures was within the predicted range, and if the forecast was correct in terms of the direction (increase or decrease) of growth (termed “directional accuracy”). We also determined the deviation (or “error”) of the forecast from the actual growth in drug expenditures, defined as the difference between the midpoint of the forecasted range and the actual percentage change in expenditures each year. The absolute values of this difference were used to determine the mean absolute error for both the non-federal hospitals drug expenditure forecast and that for clinics over the time period studied.

In order to have a benchmark to consider whether or not the accuracy of the AJHP predictions were reasonable, the annual health expenditure forecasts by the Centers for Medicare and Medicaid (CMS) were also analyzed. Each year, CMS publishes a forecast of health expenditures and a report on the actual growth of health expenditures, and within these papers are predicted and actual prescription drug growth in retail outlets.^26–41 An important point to note is that while the CMS forecasts were of growth in sales of prescription drugs in retail outlets and the AJHP forecasts were of drug spending by hospitals and clinics, our primary purpose was to compare the accuracy of these rather than the type of drug expenditures being forecasted.

The accuracy of CMS’s prescription drug growth forecasts was analyzed using the same methods described above. However, unlike the AJHP forecast, the CMS forecasted growth was given as an exact percentage increase or decrease. In order to make similar comparisons with the AJHP forecasts, we defined the CMS prediction as being in range if it was within +/- 1% than the predicted growth. This provided a 2% range for CMS predictions, consistent with the range used in the AJHP forecasts. A student’s t-test was used to determine if the mean absolute error of the AJHP forecasts for growth in drug expenditures in hospitals (and separately for clinics) differed from that of CMS forecasts for growth in overall prescription drug expenditures. An alpha of 0.05 was used to determine statistical significance.

Results

While the drug expenditures series of papers began in 1992, our analysis was limited to forecasts from 2003 to 2013 for non-federal hospitals and from 2004 to 2013 for clinics, which also generally corresponds to the time this author group prepared the publication each year. We excluded articles from 1) 1992 to 1999 because these only described historical trends without actually forecasting future spending,^1–8 and from 2) 2000 and 2001 because these did not include forecasts of expenditures in non-federal hospitals and clinics.^9–10 In 2002, two forecasts were made – the first was for a combination of hospitals and clinics and the second for ambulatory care settings. Since hospitals and clinics were not differentiated, 2002 was also excluded from this analysis.¹¹ In 2003, forecasts were made for the inpatient setting and ambulatory care settings. The inpatient setting was defined as non-federal hospitals, so 2003 was included in the non-federal hospital portion of this analysis, but was excluded from the analysis of clinic forecasts since ambulatory care referred to more than just clinics.¹² Finally, the 2014 forecast was excluded from this analysis because data on actual spending for that year were not available at the time of this analysis.²³

Annual forecasted and actual growth in drug expenditures in nonfederal hospitals are shown in Figure 1. Between 2003 and 2013 actual growth ranged from 6.4% to –0.6% (mean 2.9%, S.D. 2.3). As can be seen, forecasted and actual growth followed a similar pattern, and declined over the time period. Actual growth in expenditures was within the forecasted range in only 2 out of 11 years for non-federal hospitals (18.2%). However, the deviation of actual growth from that forecasted was minimal in most years – with the exception being in 2007 when an economic recession began. Actual growth in expenditures for non-federal hospitals was lower than predicted in 8 out of 9 years when not within the forecasted range. Actual growth was higher than predicted in 2013. The direction of the non-hospital forecast (increased or decreased growth compared to previous year) was correct in 3 of 11 years (27.3%). The mean absolute error of the forecasts of drug expenditures in non-federal hospitals between 2003 and 2013 was 2.0 (S.D. 1.2, range 0.6–4.8) – or, on average, the difference in the forecasted growth (expressed as a %) and actual growth (expressed as a %) was 2 percentage points.

Figure 1.

Annual forecasted and actual growth in drug expenditures in clinics are shown in Figure 2. Between 2004 and 2013 the growth in expenditures in clinics was more erratic that in hospitals – decreasing from a high of 20.9% in 2006 to 1.0% in 2008, then leveling out around 5.0–6.0% (mean 8.3%, S.D. 6.0) Actual growth in expenditures was within the forecasted range in 3 out of 10 years for clinics (30.0%). When not in range, actual expenditure growth was usually lower than predicted (4 out of 7 years, 57.1%). However, this was not the case in 2006, 2009, and 2013. The clinic forecast was directionally accurate in 6 of 10 years (60.0%). The mean absolute error of the forecasts of drug expenditures in clinics, 4.7 percentage points (S.D. 4.2, range 0.7–12.0), was greater than that of hospitals.

Figure 2.

As a comparison, the CMS annual forecasted and actual prescription drug growth are shown in Figure 3. The CMS data for actual growth were only available until 2012, one year less than the AJHP forecast. The CMS annual forecasts were in range 40.0% of the time (4 out of 10 years) and directionally accurate 70.0% of the time. The mean absolute error for the CMS forecast of overall drug expenditures was 2.2 percentage points (SD 1.7, range 0.4–2.6). This mean absolute error was not statistically different than that of the AJHP forecast for growth in drug expenditures in non-federal hospitals (2.0) or in clinics (4.7).

Figure 3.

Discussion

The forecasts of growth in prescription drug expenditures in non-federal hospitals and clinics published annually in AJHP are based on the best available data on drug expenditure trends and the opinions of individuals with insight on drug availability and pricing who serve as authors and reviewers of the papers, and anticipated changes in practice and health care that may impact drug spending. As demonstrated above, these predictions had varying degrees of accuracy. The forecast of growth in drug expenditures in non-federal hospital tracked closely with actual spending, though lagged by one year. This lag effect may suggest excessive influence on the forecast by spending in the previous year. Nevertheless, the mean absolute error for this forecast was better than that of the forecast of drug expenditures produced annually by CMS - though the CMS forecast had better directional accuracy.

On the other hand, the AJHP forecast of growth in drug expenditures in clinics had a mean absolute error just more than double that of the hospital and CMS forecasts, yet it had similar directional accuracy to CMS. The combination of high overall growth and variability over the past decade makes growth in drug expenditures in clinics difficult to predict. In addition, majority of drug expenditures in the clinics were driven by novel agents and oncology drugs where new evidence in any given year may change the patterns of use for these agents. The dynamic nature of health care and the many factors that work to increase or decrease future spending, make any prediction of expenditures such unlikely to be perfect. This reinforces the commonly stated caution that the annual forecast of drug expenditure growth published in AJHP should be just one of many resources that individual hospitals should draw upon when developing drug budgets. The annual forecast paper provides an overview of current trends, and health professionals must carefully consider those specific influences that will most significantly affect their own institution.

Figures 1 and 2 suggest that our ability to predict growth in drug spending in non-federal hospitals and clinics is better in some years than others. Where large deviations between the projected and actual growth were observed in a given year there were often unanticipated influences on spending that occurred. In fact many of these influences were identified in later editions of the forecast.^12–23 The most obvious of these is the unanticipated economic recession that began in 2007–2008. This clearly had a significant negative impact on hospital, clinic, and overall drug spending.

While unexpected macro-level influences on drug spending are important, so too are significant changes in patterns of use of individual high cost or high volume medications. In particular, we found that lower than expected growth in expenditures in non-federal hospitals and clinics may have resulted from newly identified safety concerns or the unexpected release of generic medications. For example, safety concerns regarding erythropoietin-stimulating agents (ESA’s) emerged in 2007, which accounted for a significant decrease in expenditures in both hospitals and clinics.¹⁷ Unexpected availability of generic agents also led to decreased non-federal hospital expenditures in certain years.²⁰ Higher than forecasted growth occurred rarely. This occurred in 2013 due to higher access to drugs as a result of healthcare reform. Other reasons for higher than forecasted growth could include changes in treatment guidelines to include expensive agents and drug shortages leading to higher prices.¹⁵ Changes such as these which occur during a budget period should lead to adjustment of budgets to account for changing utilization. Although an initial budget may be impacted, adjusting expectations to account for new and unanticipated information is valuable.

Our analysis of the accuracy of the annual drug expenditure forecast has several limitations, particularly with respect to the comparison to the CMS forecast. The CMS forecast that we used as a comparison differed in a number of ways from the AJHP forecast. Most importantly the CMS forecast is of growth in sales of prescription drugs in retail outlets in the US, whereas the AJHP forecasts are for growth in drug spending by hospitals and clinics. As such, there are intrinsic differences in nature of what is being forecasted. However, it is accuracy of the forecast that is being compared here, not the forecast itself. A recent evaluation of the accuracy of CMS forecasts finds that the actual growth between 2006–2012 was slower than projected due to a number of factors with the recession being a key factor overall, but lack of technological advances and generic availability of branded drugs being important for pharmaceutical spending.⁴² Some of these same factors may explain the differences between the projected and actual spending for the AJHP forecasts. In addition, in a few cases we found different growth rates reported by CMS for the same year, presumably because projections are updated. When this occurred the most recent reported growth was used in order to be most consistent with the AJHP forecast, which used the actual growth reported in the 2014 paper. Note that the calendar years we compared were also slightly different.

Last, our statistical comparison of the AJHP and CMS forecast was likely underpowered because of the limited number of observations (e.g., years in which forecasts were made) and this could explain the non-significant findings. This may be more important for the comparison to the clinic forecast where the mean absolute error values were larger. The mean absolute error of the non-federal hospital forecast was slightly lower (better) than that of CMS (2.0 v. 2.2 percentage points). Given the significant resources and technical expertise that goes into the CMS forecast, this finding was reassuring. Nevertheless, this analysis suggests that there may be opportunities to improve the annual AJHP forecast with better or more comprehensive consideration of factors that drive future drug spending.

Conclusion

Though not perfect, the annual forecast published in AJHP is reasonably accurate for predicting growth in prescription expenditures when compared to other available drug expenditure forecasts. Health-system administrators should use the forecast with an appropriate understanding of its limitations and assumptions, while applying their local context when developing their own drug budgets.