Categorizing Accountable Care Organizations: Moving Toward Patient-Centered Outcomes Research That Compares Health Care Delivery Systems

A reporter once asked Mahatma Gandhi, the key leader of the Indian rebellion against British colonial rule, “Mr. Gandhi, what is your opinion of Western Civilization?” Gandhi thought for a moment, and then replied: “I think that it would be a very good idea.”

Given the high cost and variable quality of U.S. medical care, it would also be a very good idea if there were organizations that take responsibility for the cost and quality of care for their patients and that work proactively and systematically to improve their patients' health. Accountable care organizations (ACOs) are intended to be just such organizations (Crosson 2011; Lewis et al. 2013). If ACOs succeed, they will be a critical and lasting legacy of the Affordable Care Act, which led to the creation of the Medicare Shared Savings, Pioneer, and Advanced Payment ACO programs (Center for Medicare and Medicaid Innovation 2014). Some health insurance companies and state Medicaid agencies have also been signing ACO-like contracts with provider organizations (Song et al. 2012). The article by Shortell et al. in the current issue of Health Services Research, as well as other recent publications, shows that the number of ACOs has grown very rapidly in just a few years (Petersen, Muhlestein, and Gardner 2013; Muhlestein et al. 2014). Details of the contracts vary, but the ACO contracts of both public and private payors give ACOs financial incentives to contain the overall cost of care for a defined group of patients whom the payor “attributes” to the ACO, as well as incentives to score well on measures of patient experience and of the quality of care.

Shortell et al. make four useful contributions to the study of ACOs and to health services research more generally, based on their first national survey of ACOs that is a milestone in itself:

First, they give a more precise sense of the number of operating ACOs and provide descriptive information about these ACOs. Very little beyond the anecdotal is known about ACOs, so this is helpful. The number of ACOs identified by Shortell et al. in 2012—at least 220—is probably a more accurate estimate than much higher estimates given by other authors (Petersen, Muhlestein, and Gardner 2013; Muhlestein et al. 2014), who appear to count as ACOs some organizations that merely have relatively minor pay-for-performance contracts with health insurers.

Second, they provide a taxonomy of ACOs. Creating clear categories is the first step to understanding a phenomenon and is necessary if one wants to make comparisons.

Third, Shortell et al. suggest that use of their taxonomy to compare the performance of ACOs both between categories and within categories will yield important results. That the performance of different organizational types should be compared may seem uncontroversial, but in fact such comparisons are not common in health services research (Casalino 2006). Comparative effectiveness research—now called “patient-centered outcomes research”—overwhelmingly focuses on comparing the effectiveness of specific therapies (e.g., different treatments for benign prostatic hyperplasia) or of specific processes of care (e.g., programs to reduce hospital readmissions) (Casalino 2014). There is not much research comparing types of organization. But it may be that the type of organization is at least as important for patient outcomes as the process of care. For example, good organizations will adopt good processes and will adapt them as needed to make them work for patients, whereas patients of less effective organizations may experience poor outcomes even when these organizations attempt to use processes shown to be effective by outcomes research. Moreover, many health care policies, either by design or as an unintended consequence, shape the types of organizations that are created and that succeed (Casalino 2004). It would be helpful if policy makers had some evidence on what types of organizations actually provide better care for their patients. But there isn't much of this evidence. The paper by Shortell et al. may be read in part as a call for “comparative effective research on organizations.” Creating a taxonomy is a necessary step to make such research possible for ACOs.

Fourth, Shortell et al. use explicit theory to guide their approach and the construction of their taxonomy. Use of theory is not very common in health services research, and not generally encouraged within the tight word counts permitted by most clinical journals.

The team of which Shortell et al. are members defines an ACO as “a group of providers that are collectively held accountable for the total cost and quality of care for a defined patient population” (Colla et al. 2014). Using a variety of methods, they identified and surveyed 220 ACOs in late 2012 and received reasonably complete survey responses from 162 of them. They identified an additional 42 organizations that seemed to be ACOs but did not respond to screening questions, so whether they are ACOs is uncertain.

Shortell et al. created their taxonomy based on eight organizational characteristics derived from their survey data (e.g., size, whether or not the ACO was an integrated delivery system, and whether the ACO was physician-led, jointly hospital-physician led, or had some other type of leadership arrangement). They used cluster analysis, pairwise comparisons, and discriminant analysis to create “clusters” of organizations that shared characteristics, and wound up with three clusters—that is, three types of ACO.

The first cluster, which they label as “larger integrated systems” and elsewhere in the paper call “integrated delivery system ACOs,” included 40 percent of ACOs. These ACOs were the largest (mean 566 physicians), included the broadest range of types of provider (e.g., 29 percent included a nursing home) and had the lowest percentage of primary care physicians (43 percent). Ninety-four percent of these ACOs self-identified as an “integrated delivery system” (IDS) and 40 percent stated that they were physician-led.

The second cluster, which they name “smaller, physician-led ACOs,” included 34 percent of ACOs. These were the smallest ACOs (mean 181 physicians), included fewer types of provider (3.6 percent included a nursing home) and had the highest percentage of primary care physicians (69 percent). Eleven percent of these smaller, physician-led ACOs self-identified as an integrated delivery system.

The third cluster, which they label “hybrid ACOs,” included 28 percent of ACOs. These ACOs were intermediate in size (mean 351 physicians) and percentage of primary care physicians (59 percent); 29 percent included a nursing home. Twenty-six percent reported being part of an IDS; 21 percent reported being physician-led (the others were led by hospitals or by “some other arrangement”).

These clusters have a fair amount of face validity, especially if the hybrid cluster is considered to be primarily a hospital-led-but-not-IDS cluster. They usefully show how certain sets of organizational characteristics tend to hang together. But the three clusters also illustrate problems with relying on a statistical approach to creating a taxonomy. Because there is overlap across the clusters—that is, certain key characteristics are common in each of the three clusters—the clusters may not be useful for answering important questions. For example, in each cluster/category there are substantial numbers of ACOs characterized as integrated delivery systems. Shortell et al. label the first cluster as “integrated delivery system ACOs.” From a statistical point of view, there is justification for this, but could one reasonably compare this category to the other two categories to determine whether integrated delivery systems perform better, given that there are many IDSs in each of the other two categories? Similarly, though the highest percentage of physician-led ACOs is in the second category (“smaller, physician-led ACOs”), there are substantial numbers of physician-led ACOs (21 and 40 percent) in the other two categories, and 31 percent of the ACOs in the “smaller, physician-led” category are not physician-led.

When categories/a taxonomy are going to be used for research and policy purposes, the categories created should ideally depend on the questions one wants to answer. Useful taxonomies help one to answer these questions and/or suggest new questions that could be asked. For example, one might want to know whether ACOs that are integrated delivery systems perform better than ACOs that are not. Or one might want to know whether ACOs that are physician-led perform better than those are not physician-led. These questions could not be answered using the taxonomy provided by Shortell et al. However, their categories do suggest questions that might be asked, as well as possible alternate sets of categories. For example, the complexity of the mixture of characteristics that ACOs may have suggests that questions about ACOs as defined by two characteristics of prime interest, rather than a single characteristic (or eight characteristics, as defined by Shortell et al.), might be important. For example, one might ask: “How do ACOs that are integrated delivery systems and are physician-led perform compared to ACOs that are integrated systems and are hospital-led? Or “how do physician-led IDS ACOs compare to all other ACOs?” And so on. In the first paper they published based on their ACO survey, the authors in fact created a simple two-category taxonomy of ACOs—physician-led versus non-physician-led (Colla et al. 2014). Other authors have provided alternative taxonomies of ACOs (Muhlestein et al. 2014).

More generally, the questions one tries to answer, and thus the categories that one creates, should ideally flow from theory. Many health services research papers begin by listing the questions they will address, but leave the theory behind these questions implicit. Admirably, Shortell et al. try to incorporate two prominent sociological theories of organizations—resource dependence theory and institutional theory—into their paper. However, the connections between these theories and the ACO characteristics that the authors use to create their taxonomy seem relatively loose, and the theories don't appear to yield any novel insight in this case.

A close reading of the paper by Shortell et al. leads to other interesting questions. For example, why are many hospitals creating ACOs, even though one of the main objectives of ACOs is to reduce the number of hospital admissions? Shortell et al. do not directly address this question, but institutional theory suggests at least a partial answer. As Shortell et al. point out, institutional theory “emphasizes that organizations are embedded within larger societal norms and cultures that over time reflect ‘appropriate’ or desired behavior.” In other words, the decision of hospital leaders to create or participate in an ACO may be influenced by emerging cultural norms (within groups of people important to hospital leaders) that creating an ACO is “the thing to do”—something that all progressive leaders are doing—rather than by strict considerations of economic rationality.

The category “larger integrated delivery system” raises several questions. The concepts of “integration” and of “integrated delivery systems” have always been difficult. What is meant by integration? Classical and recent work on this topic has been done by Shortell and colleagues, describing integration as having three components: clinical, physician-system, and functional/administrative (Gillies et al. 1993; Kreindler et al. 2012). How integrated are ACOs? Are certain types of ACOs more likely to be integrated than other types of ACOs? Do more highly integrated ACOs provide higher quality and/or lower cost care? Answering these questions will require a clear definition of integration, interview and/or survey data to determine the extent of integration of individual ACOs, and linking of this data to performance data (e.g., from Medicare claims) to determine the relationship among integration, quality, and costs.

What exactly is an integrated delivery system? Shortell et al. don't define an IDS in this article, but in earlier work Stephen Shortell defined an IDS as “A network of organizations that provides or arranges to provide a coordinated continuum of services to a defined population and is willing to be held clinically and fiscally accountable for the outcomes and health status of the population served” (Shortell et al. 1996). This is essentially the same definition that Shortell et al. use for an ACO, yet their taxonomy makes clear that not all ACOs are integrated delivery systems. If an IDS is defined as an organization that is highly integrated, then identifying IDSs by their structural components would not be enough, because the degree to which an organization is integrated is an empirical question that would take in-depth research to answer. The lowest common denominator structural definition of an IDS would be that it is an organization that includes one or more hospitals, plus medical groups, within a single ownership structure. However, defining a single hospital and the medical groups that it employs as an IDS would seem too broad. The term “integrated delivery system” is often used in a normative way—that is, to indicate that the organization is both integrated and provides excellent care, neither of which is necessarily true of a hospital and its employed physicians. Unless the concept of an IDS can be clearly defined in a way that distinguishes this type of organization from others, it may not be useful to use the term.

As Shortell et al. point out, their paper raises the question of whether their three categories of ACO differ in their performance. Their survey data alone cannot provide information about this, but they do contain information on such important things as the extent to which ACOs use care management processes for patients with chronic diseases and the extent to which ACOs use clinical health information technology. It will be interesting to learn whether their categories of ACO differ in use of these processes and technology. Shortell et al. briefly mention this and also address the issue to some extent in their paper on physician-led ACOs (Colla et al. 2014), but perhaps more extensive analysis will be forthcoming soon.

Like any thoughtful article, the article by Shortell et al. raises more questions than it answers. The article's statistically created taxonomy of ACOs is far from perfect, but the reality the authors are trying to describe is inherently complex, with a multiplicity of types of organization with different but overlapping characteristics. This article provides a useful starting point for categorizing ACOs and for beginning to think about the key questions that should be asked about them.