Augmenting Predictive Modeling Tools with Clinical Insights for Care Coordination Program Design and Implementation

Abstract

Context:

The Center for Medicare and Medicaid Innovation (CMMI) awarded Denver Health’s (DH) integrated, safety net health care system $19.8 million to implement a “population health” approach into the delivery of primary care. This major practice transformation builds on the Patient Centered Medical Home (PCMH) and Wagner’s Chronic Care Model (CCM) to achieve the “Triple Aim”: improved health for populations, care to individuals, and lower per capita costs.

Case description:

This paper presents a case study of how DH integrated published predictive models and front-line clinical judgment to implement a clinically actionable, risk stratification of patients. This population segmentation approach was used to deploy enhanced care team staff resources and to tailor care-management services to patient need, especially for patients at high risk of avoidable hospitalization. Developing, implementing, and gaining clinical acceptance of the Health Information Technology (HIT) solution for patient risk stratification was a major grant objective.

Findings:

In addition to describing the Information Technology (IT) solution itself, we focus on the leadership and organizational processes that facilitated its multidisciplinary development and ongoing iterative refinement, including the following: team composition, target population definition, algorithm rule development, performance assessment, and clinical-workflow optimization. We provide examples of how dynamic business intelligence tools facilitated clinical accessibility for program design decisions by enabling real-time data views from a population perspective down to patient-specific variables.

Conclusions:

We conclude that population segmentation approaches that integrate clinical perspectives with predictive modeling results can better identify high opportunity patients amenable to medical home-based, enhanced care team interventions.

Introduction

The Center for Medicare and Medicaid Innovation (CMMI) awarded Denver Health’s (DH) integrated, safety net health care system $19.8 million to integrate a “population health” approach into the delivery of primary care. This major practice transformation builds on the Patient Centered Medical Home (PCMH) and Wagner’s Chronic Care Model (CCM) to achieve the “Triple Aim”: improved health for populations, care to individuals, and lower per capita costs.1,2,3,4 Briefly, it leverages Health Information Technology (HIT) to risk stratify patients and employ enhanced care team staffing to tailor care management services to patient need, especially for patients at high risk of avoidable hospitalization. Developing, implementing, and gaining clinical acceptance of the HIT technology solution for patient risk stratification was a major grant objective.

Risk stratification for care coordination purposes often relies on predictive modeling and risk adjustment tools that were originally developed for payment and are often imperfectly suited to this expanded use. In Iezzoni’s Risk Adjustment for Measuring Health Care Outcomes, risk adjustment is defined as “accounting for patient-associated factors before comparing outcomes across different patients, treatments, providers, health plans, or populations.”5 For example, risk adjustment is used to ensure fair compensation to prepaid health plans or providers that serve large numbers of high-risk patients because such patients typically use more services than lower risk patients. However, financially oriented risk adjusters are designed to predict health care expenditures according to prevailing practice patterns, which may include both necessary and potentially avoidable service use. The emerging literature on risk-stratified approaches to care coordination has highlighted the need to home in on avoidable utilization—especially avoidable hospitalizations—not simply high utilization generally.6 The observation that alternative risk adjusters and predictive models were designed for, and therefore tend to identify, different populations takes on enhanced salience in this context, as some types of utilization may be more easily modified than others.7

To assess the technical performance of a predictive modeling tool for payment purposes, the major consideration is the tool’s “predictive value” or its relative accuracy in predicting future hospitalization and other high-cost utilization. A relatively robust literature exists that uses statistical models to evaluate the predictive value of individual risk adjustment tools, including a few head-to-head comparisons of the predictive performance of alternate methodologies.8,9 As noted, however, it is important to parse avoidable from unavoidable utilization, which generally requires clinical input. Therefore, assessing the technical performance of a predictive modeling tool for care coordination purposes must consider factors in addition to its predictive value.10 Furthermore, one must also consider the often substantial effort to integrate the IT solution with point-of-care program design considerations such as care coordination protocols for outreach and follow-up with identified patients and other workflow considerations.11

This paper presents a case study of how DH developed and implemented a clinically acceptable and actionable risk-tiering approach by pairing the use of published predictive models with front-line clinical judgment. In addition to describing the IT solution itself, we focus on the leadership and organizational processes that facilitated its multidisciplinary development and ongoing iterative refinement, including the following: team composition, target population definition, algorithm rule development, performance assessment, and optimization of clinical workflow. We provide examples of how dynamic business intelligence (BI) tools facilitated program design decisions by enabling real-time data views from a population perspective down to patient-specific variables. We conclude that population segmentation approaches that integrate clinical perspectives with predictive modeling results can better identify “high opportunity” patients amenable to medical home-based, enhanced care team interventions.

Context

DH is an integrated safety net system and the largest provider of Medicaid and uninsured services in Colorado, serving approximately 200,000 users annually. DH consists of eight federally qualified community health centers, 15 school-based health centers, outpatient specialty services, a 525-bed, acute care hospital with an academic Level One Trauma Center, a 100-bed nonmedical substance abuse and detoxification facility, Denver’s 911 emergency medical response system, the Denver Public Health department, and an HMO managed care plan serving commercial and public payers. This vertical and horizontal integration as well as a shared electronic health record translates into excellent data capture that provides detailed utilization information across the continuum of care.

The DH system provides health care services to nearly a third of residents in the city and county of Denver, Colorado. In 2013, the DH system recorded more than 550,000 outpatient visits, including 295,000 primary care visits. Approximately 65 percent of DH patients have incomes below 185 percent of the federal poverty level, 75 percent are members of racial and ethnic minority groups, and approximately 40 percent were uninsured in 2013 (prior to the implementation of the Affordable Care Act).

Purpose of the Risk Stratification Framework

DH’s 21^st Century Care practice transformation effort builds on the patient-centered medical home to provide individualized clinical and health information technology (HIT) services with the overall goal of advancing the Triple Aim: improved health and care at a lower cost.12 Specifically, DH seeks to implement a population health approach to primary care by explicitly defining and risk stratifying an accountable population for which DH is—or should be—providing comprehensive, patient-centered ambulatory care services. This new model of care provides a graduated set of enhanced clinical and HIT services that are matched to risk tiers and allocated according to individual needs within tiers, with more and higher intensity services reserved for higher tiered patients.

For example, all patients are offered text message reminders about appointments and recommended preventive services. For lower risk patients, this low-touch, panel management approach may be sufficient to support their good health. However, higher risk patients often need more frequent and more comprehensive follow-up care as well as substantial social and behavioral health support. Therefore, DH expanded its primary care staffing model to include new team members to optimize clinical visits, to provide behavioral health and disease management services, as well as to offer complex care coordination support. This enhanced care team includes patient navigators and clinical pharmacists for adults, pediatric nurse care coordinators, as well as social workers and behavioral health consultants (BHCs). For the highest risk patients, DH also funded three high intensity clinics with small patient panels that focus exclusively on high-risk populations: children with special health care needs, medically complex adults with recent multiple admissions, and adults with significant mental health diagnoses and recent multiple readmissions.

As illustrated in Figure 1, risk stratification is central to 21^st Century Care’s population health approach to the allocation of tiered care coordination services.

Figure 1.

Original Grant Risk Stratification Model: Tier 1 (Lowest Risk) Through Tier 4 (Highest Risk) with Associated Member Months (MM), Per Member Per Month (PMPM) Spending, Staffing Model, and Services

Notes: Baseline period is July 2010 through June 2011. This initial “proof of concept” tiering algorithm was implemented by Milliman using CDPS predictive modeling tool thresholds to define tiers. Tier sizes were pre-determined according to estimated resource capacity. The attributed managed care population was identified through membership files, whereas the fee-for-service population was selected at a single point in time at the beginning of the time period and fixed for the duration. All attributed individuals were tiered. MM: Member months, PMPMs: Per member per month, PN: Patient Navigator. RN CC: Nurse Care Coordinators, PharmD: Clinical Pharmacist, BHC: Behavioral Health Consultant, eTouch: Health Text Messages Programs. Grant tiers (Beta version).

The stratification process is meant to dynamically sort the population into four tiers of higher (Tier 4) and lower risk (Tier 1) patients. These tiers are a key factor in defining the target populations for specific clinical interventions, providing a higher level of care coordination resources to higher tier patients and vice versa. The dynamic nature of the tiering is required both to capture new patients and to detect when individual circumstances change (e.g., low-risk patients are reclassified as higher risk when their health status changes).

Figure 1 displays the original risk stratification that was submitted in our grant proposal as a proof of concept. For this initial tiering effort, DH did not perform its own risk stratification but outsourced it to its actuary, Milliman. Nearly 130,000 patients were placed into risk tiers using a financially oriented risk-adjustment tool known as the Chronic Illness and Disability Payment System (CDPS), which is used by the Colorado Medicaid agency to adjust capitated payments for health status.12 Using billing claims data, CDPS employs a regression-based methodology to predict future utilization according to an algorithm that considers mainly patient age, gender, and diagnoses. A numeric “risk score” is calculated that expresses individual risk in relation to the average risk of future spending (from a payer perspective). Risk-score threshold ranges defined tiers, and tier sizes were predetermined according to the estimated clinical resource capacity available to each tier.

As revealed in the average per member per month (PMPM) spending by tier in Figure 1, this initial proof of concept tiering resulted in excellent financial stratification for adults patients (Tier 1: $137 PMPM; Tier 2: $614 PMPM; Tier 3: $3,449 PMPM; Tier 4: $7,801 PMPM).

However, clinical team members reviewed patient diagnoses within tiers, conducted chart reviews, and judged the resulting population segmentation to lack sufficient clinical relevance for primary care-based, care coordination purposes. For example, the health services research team produced lists of patients sorted by tier, by risk score, and by diagnoses. These lists revealed that individuals with the same risk score were often clinically heterogeneous. Tier 3, for instance, contained individuals with pulmonary conditions (medium), eye conditions (low level), substance abuse (low level), psychiatric conditions (medium-, low level), and pregnancy (complete), among many other diagnostic profiles. Clinical team members felt that care coordination approaches would be different for these disparate groups. Furthermore, although this risk stratification model accurately predicted high cost patients, it was not designed to distinguish avoidable from less avoidable utilization. Chart reviews revealed that a subset of higher tier patients displayed utilization patterns that are relatively insensitive to change (at least through primary care-based interventions), such as patients undergoing chronic dialysis or active chemotherapy, as well as patients with recent, catastrophic events.

In keeping with this initiative’s focus on meeting the Triple Aim, subsequent tiering efforts sought to better identify high opportunity patients that were both potentially costly and amenable to medical home-based, enhanced care-team interventions. Note that care coordination programs directed at different aims might segment the population differently.

Case Study: Developing a Clinically Acceptable, Risk-Tiering Approach

We detail here the multidisciplinary and iterative refinement of the 21^st Century Care tiering algorithm, including three separate updates, known as versions 1.0, 2.0, and 3.0, respectively. As displayed graphically in Figure 2, key steps in the algorithm development included the following: assembling a team, agreeing on and defining an accountable population, creating risk stratification algorithm rules to define population segments, evaluating the financial and clinical performance, iteratively optimizing the algorithm through care model design and workflow development, and implementing performance monitoring. Although our population health model encompasses the entire primary care population—adults and children—this discussion focuses on efforts to refine risk stratification rules for higher tiered adults.

Figure 2.

Steps to Developing a Population Segmentation Strategy for Care Coordination

Inspired by Institute for Healthcare Improvement (IHI) BHLC Collaborative

Assembling a Team

Tiering algorithm development has been a multidisciplinary effort led by a health policy expert with doctoral-level public health training and an IT business analyst with experience in building large data models with companion BI tools. The algorithm rules development team has evolved and grown at each iteration and includes adult and pediatric clinical directors and senior management, a clinical pharmacy administrator, health services researchers, clinical operations staff, finance experts, and IT developers. Clinical input was also obtained from practicing primary care providers as well as those with quality improvement expertise. Collectively, this multidisciplinary group provided the broad content expertise necessary to conceptualize and implement a risk-stratified, population health approach to primary care service delivery. Explicit support for the effort by DH’s CEO and the director of ambulatory care services was also critical to the effort.

Defining the Target Population

Broadly speaking, the target population for the 21^st Century Care practice transformation is the approximately 130,000 publicly insured and uninsured patients (60,000 adults) who receive or could benefit from primary care at a DH primary care clinic. This target population includes current primary care patients, members of one of DH’s managed care plans, as well as certain frequent users of DH emergency, urgent care, and hospital services who do not currently use primary care services.

Arriving at this operational definition of the target population engendered much discussion and debate because, as an integrated delivery system, DH provides care to a wide variety of patients in numerous settings, including the following: primary care clinics, specialty clinics, hospital and trauma center, emergency department (ED), urgent care, sexually transmitted disease (STD) clinic, and travel clinic, among others. However, a population health orientation requires thinking about populations in need of primary care services—such as higher-risk patients and out-of-care patients—not solely those who present for care. For example, data analyses revealed that among the 3 percent of adult patients that accounted for 30 percent of DH facility charges, nearly half were not current primary care users.13 This self-reflection ultimately resulted in expanding the scope of who should be targeted for our primary care transformation effort. DH managed-care members were included under the assumption that their enrollment signals an expectation that DH serve as their medical home. Similarly, the utilization behavior of individuals with repeated ED visits, urgent care visits, and hospitalizations suggest that these patients think of DH as their delivery system, and therefore DH should make proactive efforts to engage them in primary care.

The 21^st Century Care population is dynamically defined through monthly runs of a population attribution and risk tiering algorithm. A portion of the algorithm includes a daily (not monthly) assessment of tier promotion events. The attributed population is not a fixed cohort. Individuals move in and out of the 21^st Century Care population and up and down in tiers as the attribution algorithm is reapplied.

Developing Risk Stratification Algorithm Rules

Adult Tiering Algorithm 1.0: (Implemented, November 2012–April 2013)

Model Development Summary

The task of bringing risk stratification in house and developing a more transparent and clinically relevant tiering algorithm was a major interdisciplinary effort that spanned six months of bimonthly team meetings. Although the Tiering 1.0 algorithm retained CDPS risk scores to assign patients to an initial tier, it also contained a set of rules for promoting adults to higher tiers, triggered by unusually frequent inpatient utilization or evidence of poor clinical control of chronic disease.

The process for developing these tier promotion rules illustrates the multidisciplinary nature of this work. DH clinical leaders identified high opportunity adult subpopulations thought to have potentially avoidable utilization, based on the literature and clinical experience. These clinicians then worked collaboratively with health services research and IT staff to develop detailed data specifications. The latter group then conducted iterativemanual simulations to quantify and describe each subgroup, including their inpatient and ED utilization trends over time. Final tier promotion rules considered the subpopulation size, the existence of aligned, evidence-based interventions, and their utilization risk profile. While the bulk of the clinical and analytical work related to patient tiering was completed within the six-month planning period, an additional three months were required for final algorithm testing and implementation by the IT team.

Financial Assessment

To assess each algorithm’s predictive performance, Milliman actuaries calculated the PMPM payer spending by tier for those defined under the original grant tiering algorithm as compared to those defined by the Tiering 1.0 algorithm (Table 1). As revealed in the average PMPM spending by tier in Table 1, this revised tiering resulted in a less distinct financial stratification, particularly among the lower tiers: Tier 1: $386 PMPM; Tier 2: $621 PMPM; Tier 3: $1,064 PMPM; Tier 4: $8,829 PMPM.

Table 1.

Tiering Algorithm 1.0 Versus Original Grant Tiers: Member Months (MM) and Per Member Per Month Spending (PMPM) by Tier

	TIERING 1.0 ADULT MEMBER MONTHS (MM) (NOV. 1, 2011–OCT. 31, 2012)	TIERING 1.0 ADULT PMPM (NOV. 1, 2011–OCT. 31, 2012)	GRANT TIERS ADULT MEMBER MONTHS (MM) (JULY 1, 2010–JUNE 30, 2011)	GRANT TIERS ADULT PMPM (JULY 1, 2010–JUNE 30, 2011)
Tier 4	11,578	$8,829	7,348	$7,801
Tier 3	35,915	$1,064	25,007	$3,449
Tier 2	263,634	$621	326,636	$614
Tier 1	116,470	$386	175,023	$137
Tier 0 (Not yet tiered)	114,684	$429	0	N/A
TOTAL	542,281		534,015

Note: Patients are attributed to the DH population on a monthly basis to account for patients joining and leaving the DH system during the observation period. As a result, not all patients are part of the attributed population for all observed months. Patient counts are therefore expressed as member months (MM), which is the sum of months in the attributed population for all attributed patients. Member months is also the denominator for patient-level spending that is expressed in dollars on a per member per month (PMPM) basis.

Clinical Assessment: Clinical Coherence and Care Model Implications

Clinical performance was subjectively assessed by team clinicians who considered the degree to which the population segments were clinically coherent, facilitating aligned interventions that could be implemented for populations identified as high risk. Because tier assignments were made visible through an automated process at the point of care on the patient “face sheet,” the algorithm team was able to obtain feedback from front-line clinical teams, as well as rendering their own opinions. Assessment of the algorithm’s clinical performance differed by tier, with greater clinical satisfaction with the Tier 4 redefinition than with the lower tier rule changes.

Whereas the original grant tiering algorithm classified adults as Tier 4 based on their CDPS risk score, the Tiering 1.0 algorithm identified Tier 4 adults according to multiple recent hospitalizations. These two methods identified different but overlapping populations, and both versions identified individuals with high per capita spending ($7,801 PMPM versus $8,829 PMPM). Because frequent readmissions have been shown to be a potential marker for unmet behavioral health or social needs in safety net populations,14,15 our clinical experts judged that Tiering 1.0 rules better aligned Tier 4 status with patients with potentially avoidable utilization. This “super-utilizing” Tier 4 adult population would eventually be targeted for a high intensity clinic organized under the ambulatory intensive caring unit (A-ICU ) model.16

A second set of Tiering 1.0 rules assigned individuals with poorly controlled chronic conditions to either Tier 2 or Tier 3 (depending on clinical control level) in order to position them to receive pharmacotherapy support and other enhanced care team services. As a result, a large number of adults were promoted to Tier 3 based on lab results indicating uncontrolled diabetes or hypertension. Because the rules did not consider severity of disease or persistence of uncontrolled status, this clinical relevance came at the direct expense of a flattened financial stratification, with much less tier differentiation in PMPM spending for Tiers 1–3, as compared to the original stratification. (See Table 1.)

The Tiering 1.0 algorithm also produced clinically unacceptable, unstable tier assignments at the individual patient level. Frequent “tier jumping” triggered by small changes in clinical indicators (e.g., blood sugar control) were judged to be problematic in a program designed to allocate resources according to tier. At the clinic level, this would mean that many patients qualify, disqualify, and requalify for clinical pharmacist services multiple times in short succession, which is inconsistent with a longitudinal approach to care coordination.

A lesson we learned from this experience was the need to distinguish the concept of a “tier” or population segment from that of a clinical “trigger.” In the DH tiering model, a tier is intended to identify who (which patients, populations) we should monitor more closely on a regular basis given their ongoing, elevated risk status, whereas a trigger indicates when we should take specific clinical action. The algorithm team determined that chronic disease clinical control status works better to prompt clinical activities within tiers (population segments) rather than to define tiers themselves.

Tiering Algorithm 2.0: (Implemented, May 2013–April 2014)

Model Development Summary

Tiering 1.0 succeeded in engaging key clinical partners in the risk stratification effort who were now motivated to consider predictive performance as well as clinical coherence and intervention alignment. After another extended multidisciplinary planning process, the Tiering 2.0 algorithm was implemented in May 2013.

A major change in 2.0 was to replace the CDPS risk-scoring tool with an alternative, predictive modeling tool and diagnosis grouper: a 3M product called Clinical Risk Groups (CRGs) for the initial sorting of patients into four risk tiers. Used for a variety of analytical purposes by Colorado’s state Medicaid program, CRG was recommended by a clinical team member and selected primarily for its clinical accessibility.

CRG differs from regression-based risk-adjustment models (such as CDPS) in that it categorizes patients into 9 mutually exclusive and clinically coherent groups (and thousands of subgroups) that are ranked according to financial risk. Briefly, CRG status 1 corresponds to healthy individuals, CRG status 2 includes those with acute issues only, CRG statuses 3–7 are reserved for those with an increasing number and severity of chronic diseases, CRG status 8 corresponds to metastatic cancers, and CRG status 9 is for catastrophic events, such as trauma.17

The CRG model development team included physicians, and the process for model refinement was similar to that employed by the DH algorithm development team. As detailed by Hughes et al., “after creating an initial set of hypothesized risk groups, the research staff calculated mean expenditures for each risk group, beginning a highly iterative process in which the hypothesized risk groups and their interactions with other chronic and acute conditions were tested, modified, and tested against through multiple cycles. Whenever there was a conflict between statistical results and a plausible clinical rationale, the final decision always favored the clinical rationale.”18 CRG was developed using Medicare, commercial and (limited) Medicaid data, and has been validated against United States Medicare and Canadian data sets, performing comparably to other available risk-adjusters at the time.19,20, 21 It has also been clinically validated as accurately identifying, via chart review, children with lifelong chronic conditions.22

DH conducted simple validation exercises using three cohorts of adults, as defined on a point-in-time basis on January 1 of the years 2012, 2013, and 2014. (See Table 2.) Specifically, it examined mean facility charges for each cohort during the year postidentification to assess the degree to which CRG status is associated with next year’s charges. For at least two of the three cohorts, CRG produced a consistent financial gradient in which prospective year charges increased directly with CRG status level, with the exception of CRG statuses 2 and 3, which were reversed in two of the three cohorts (2012 and 2014).

Table 2.

Average Adult Charges by CRG, 2012–2014

CLINICAL RISK GROUP (CRG)	JANUARY 1, 2012 COHORT	JANUARY 1, 2013 COHORT	JANUARY 1, 2014 COHORT
CRG STATUS	MEAN FACILITY CHARGES 2012	MEAN FACILITY CHARGES 2013	MEAN FACILITY CHARGES 2014 (FIRST 6 MONTHS)
1: Healthy	$2,859	$3,058	$1,940
2: History of Significant Acute Disease	$5,686	$5,820	$3,450
3: Single Minor Chronic Condition	$5,243	$5,843	$3,213
4: Minor Chronic Disease in Multiple Organ Systems	$6,572	$7,055	$4,346
5: Single Dominant or Moderate Chronic Condition	$7,474	$7,571	$4,084
6: Significant Chronic Conditions in Multiple Organ Systems	$17,413	$18,437	$9,909
7: Dominant Chronic Disease in 3 or More Organ Systems	$45,277	$42,380	$29,353
8: Dominant, Metastatic and Complicated Malignancies	$39,243	$48,771	$34,689
9: Catastrophic Conditions	$81,538	$87,993	$48,372

Tiering 2.0 algorithm development proceeded similarly to the 1.0 process, with clinical teams taking the lead in developing rules to assign CRGs to tiers. BI tools greatly facilitated the CRG-to-Tier mapping process by creating an interface that could switch in real time between population- and patient-level views, with key summary data in each view. These BI tools allowed the clinical teams to drill down to the individual patient level to make mapping decisions as well as efficiently evaluate the effect individual CRG assignments had on tier size. (See Table 3, for an example of a data view used in CRG-to-Tier mapping.)

Table 3.

Top 25 Base CRGs, Tier Assignments, and Adult Patient Counts

BASE CRG	CRG DESCRIPTION	TIER	# OF PATIENTS	% OF PATIENTS
1000	Healthy	1	26,657	38%
5192	Hypertension	2	3,069	4%
6144	Diabetes and Hypertension	2	2,507	4%
5424	Diabetes	2	2,055	3%
6270	Two Other Moderate Chronic Diseases	2	1,996	3%
6143	Diabetes and Other Moderate Chronic Disease	2	1,827	3%
2030	One Significant Acute Illness Excluding ENT	1	1,779	3%
4000	Multiple Minor Chronic PCDs	2	1,435	2%
6201	Psychiatric Disease (Except Schizophrenia) and Other Moderate Chronic Disease	2	1,052	2%
5138	Asthma	2	1,008	1%
5441	Obesity	2	1,007	1%
9030	HIV Disease	2	879	1%
5784	Chronic Alcohol Abuse	2	798	1%
2020	One Significant Acute Illness – Span 90 Excluding ENT	1	730	1%
6293	One Other Moderate Chronic Disease and Other Chronic Disease Level 2	3	718	1%
5442	Chronic Endocrine, Nutritional, Fluid, Electrolyte and Immune Diagnoses – Moderate	2	706	1%
6171	Schizophrenia and Other Moderate Chronic Disease	3	701	1%
6241	Asthma and Other Moderate Chronic Disease	3	699	1%
6251	Moderate Chronic Substance Abuse and Other Moderate Chronic Disease	2	680	1%
5743	Schizophrenia	2	674	1%
3446	Chronic Thyroid Disease	2	633	1%
6260	One Other Dominant Chronic Disease and One or More Moderate Chronic Disease	2	620	1%
3755	Depression	2	619	1%
7071	Diabetes, Hypertension, Other Dominant Chronic Disease	4	583	1%

Although DH clinical partners generally assigned higher status CRGs to higher tiers, they also considered “clinical actionability” in making tier assignments. Examples of patients deemed as less amenable to intervention through enhanced care team services included patients hospitalized for catastrophic events or those undergoing active chemotherapy. As a result, certain subsets of patients within CRG 8 and CRG 9 were assigned to lower tiers.

In the Tiering 2.0 algorithm, the vast majority of attributed patients (97 percent) are assigned to tiers based on their CRG. In addition, several enhancements to tier promotion rules were also made.

New or retained tiering promotion rules included the following:

• Tier 1 women with a history of adverse birth outcomes were promoted to Tier 2 (new);

• Tier 2 adults with specific CRG profiles and recent utilization were promoted to Tier 3 (new); and

• Tiers 1–3 adults with frequent inpatient utilization were promoted to Tier 4 (retained).

In contrast to the 1.0 version, under Tiering algorithm 2.0, the comprehensive adult Tier 4 population definition was no longer restricted to high-utilizing “tier promoters” but now also included CRG-assigned individuals. Dropped from the Tiering 2.0 algorithm were Tiering 1.0 tier promotion rules that used clinical lab values to determine tier assignment.

Conceptually, both new and retained tier promotion rules sought to leverage patient information that is known to the provider but would be unlikely to influence the CRG status. Therefore, CRG alone would likely understate the risk profile of the patient. It was acknowledged that the use of utilization criteria for tier promotion purposes would contribute to instability in tier assignment, but the perceived clinical intervention opportunity outweighed this concern.

Financial Assessment

As Table 4 illustrates, financial stratification on a PMPM basis for Tiers 1–3 improved in Tiering 2.0 relative to Tiering 1.0. The Tier 4 PMPM is reduced in Tiering 2.0 because it now includes individuals assigned via their CRG status of 7 (Dominant Chronic Disease in 3 or More Organ Systems) as well as super-utilizers. As a predictive modeling tool, CRG aims to predict which patients are likely to experience outlier utilization before it has occurred, whereas super-utilizer status identifies individuals with elevated utilization after it has happened.

Table 4.

Tiering 2.0 Algorithm: Member Months (MM) and Per Member Per Month Spending (PMPM) by Tier

	TIERING 1.0 ADULT MM NOV. 1, 2011–OCT. 31, 2012	TIERING 1.0 ADULT PMPM NOV. 1, 2011–OCT. 31, 2012	TIERING 2.0 ADULT MM NOV. 1, 2011–OCT. 31, 2012	TIERING 2.0 ADULT PMPM NOV. 1, 2011–OCT. 31, 2012
Tier 4	11,578	$8,829	38,351	$4,350
Tier 3	35,915	$1,064	53,200	$1,499
Tier 2	263,634	$621	281,718	$451
Tier 1	116,470	$386	191,576	$271
Tier 0 (Not yet tiered)	114,684	$429	0	$0
TOTAL	542,281		564,845

Note: Patients are attributed to the DH population on a monthly basis to account for patients joining and leaving the DH system during the observation period. As a result, not all patients are part of the attributed population for all observed months. Patient counts are therefore expressed as member months (MM), which is the sum of months in the attributed population for all attributed patients. Member months is also the denominator for patient-level spending which is expressed in dollars on a per member per month (PMPM) basis. Although the actuarial reports consider the same period, total member months differ slightly in the assessments of Tiering 1.0 and Tiering 2.0 due to late claims.

Figure 3 depicts how Algorithm 2.0 compares to Algorithm 1.0 in terms per member spending and member months attributed to each tier.

Figure 3.

Tiering 1.0 Versus 2.0 Algorithm: Member Months (MM) and Per Member Per Month Spending (PMPM) by Tier

Clinical Assessment: Clinical Coherence and Care Model Implications

In addition to this improved financial performance, the use of CRG as the primary building blocks for assigning patient to tiers under Tiering 2.0 was also better accepted by clinical teams. CRGs were viewed as more transparent and clinically relevant than numeric risk scores. CRG could also accommodate meaningful changes in health status without being too sensitive to small fluctuations (e.g., in lab values). Although the addition of patients with CRG status 7 to Tier 4 reduced the tier’s PMPM, it had the clinical advantage of potentially identifying high-risk individuals before they experienced a super-utilizing episode.

Tiering Algorithm 3.0: (Implemented, May 2014–Present)

Model Development Summary

The Tiering 3.0 algorithm was implemented in May 2014. The key goal of this round of risk-stratification redesign was to continue to improve the identification of high opportunity individuals and to better align tier assignments with clinical interventions. By way of example, we describe how the algorithm was optimized for clinical pharmacist and High Risk Care Coordination (HRCC) interventions.

Optimizing Clinical Pharmacist Intervention Alignment

Ambulatory clinical pharmacy services are available for high-risk (Tiers 3 and 4) patients to help manage chronic diseases that require ongoing medication management. While Tiering 1.0 rules were highly aligned with evidence-based clinical pharmacist interventions focused on patients with uncontrolled diabetes and hypertension, it resulted in a population definition that was too broad. There was no mechanism to effectively prioritize which patients most needed care, such as those with worsening disease or those with high or increasing utilization. While Tiering algorithm 2.0 improved the ability to prioritize patients based on severity of disease, comorbidities, and recent utilization, feedback from our clinical pharmacists suggested that the patient populations identified as Tier 3–4 now too narrowly specified the population most likely to benefit from clinical pharmacist interventions. Specifically, clinical pharmacists believed that a substantial number of Tier 2 patients were inaccurately classified as lower risk. In an effort to further optimize the model and improve its clinical acceptance, the Tiering 3.0 algorithm development team worked on a hybrid model that incorporated strengths from the previous two tiering approaches.

The first step of this process involved reviewing the CRG assignments for Tier 2 patients (under Tiering algorithm 2.0) for whom clinical pharmacists had provided services in order to identify specific CRGs that should be considered for mapping to Tier 3 in the updated algorithm. In a parallel process, the IT team collaborated with the health services research staff to identify additional subpopulations that might be aligned with clinical pharmacist interventions, including for example, individuals with elevated blood sugar (A1C) levels and Tier 1–2 individuals with significant chronic disease (CRG 5 or 6).

For each of these subpopulations, a cohort of patients was defined using historical data. Each population was identified during August 2011, and a data view was created that summarized the cohort’s utilization and costs in the year prior to identification (year 0) as well as two subsequent years (years 1 and 2). This approach to reviewing utilization and cost data over a three-year period allowed the team to identify which CRGs had stable or increasing utilization and costs, and which CRGs demonstrated regression to the mean. As illustrated in Figure 4 the data views had drill down capabilities that enabled clinical team members to review the clinical conditions of patients among the CRG profiles exhibiting a high level or increasing utilization. Clinicians could drill down to the individual patient level, if desired. This analysis informed the final selection CRGs to map into Tier 3 in the new Tiering 3.0 algorithm.

Figure 4.

Example of Data Views with Drill Down Capacity Facilitated Tiering 3.0 Algorithm Development for Patients with an A1C>9.0%

Examining actual utilization trends on actual DH patients enabled the clinical members of the team, in particular, to gain comfort with the predictive modeling aspect of CRG. Figure 4 displays the results of this exercise for the subpopulation defined as patients with A1Cs > 9 in August 2011.

Developing Associated Workflows and Performance Monitoring

In several instances, the Tiering 3.0 algorithm and associated clinical workflows were developed in parallel. For example, much of the clinical design work in 2013–2014 focused on the development of a multidisciplinary, adult HRCC intervention that aimed to identify patients with complex chronic disease that might benefit from enhanced care team services. A target “adult high-risk” group—defined by CRG and recent utilization—was identified and assigned to Tiers 3–4. In a concurrent process, clinical work flows were piloted for the HRCC intervention. The data simulations related to the CRG selection for adult high-risk tier promotion, and the clinical pilots of the HRCC intervention mutually informed each other such that the HRCC target population is a subset of the tier promotion group.

While the tiering algorithm development team remained the same as previous iterations, the HRCC pilot involved a broader array of clinic staff, including clinical operations, nurse clinic managers, social workers, patient navigators, behavioral health consultants, primary care physicians, and clinical pharmacists. This resulted in a rich discussion around which patients had avoidable utilization and would be most responsive to the HRCC intervention. In a manner analogous to the above clinical pharmacist discussion, the HRCC pilot team reviewed clinical utilization and cost patterns by CRG over time to arrive at a target population defined by a combination of CRG status and recent utilization.

The HRCC patient identification logic was revised many times in response to clinical feedback. Specifically, DH’s Quality Improvement (QI) coaches worked with the clinic teams to operationalize the HRCC process for each clinic according to each clinic’s patients’ needs and staffing capacity. The pilot clinic sites reviewed lists of adult high-risk patients (subsets of Tiers 3 and 4 patients) that were generated on a weekly basis and distributed to patient navigators and providers for care coordination services. These patient lists included information about patient CRG status, recent utilization, upcoming and recent clinic visits, and the pattern of charges in the past 18, 12 and 6 months. Our HRCC teams met on a regular basis, and as providers gained experience with the patients identified through this automated process, the work list logic was refined to optimize clinical actionability. For example, requested refinements included excluding patients from HRCC lists if they had recently experienced a onetime catastrophic event or had a recurrent history of substance abuse resulting in multiple visits to DH’s detoxification unit. The content and format of the care coordination outreach worklists were also modified based on team input.

As a result of these refinements, the HRCC target-patient population definition remains largely in alignment with tier promotion rules, but the latter (tier promotion) group is more broadly defined than the former (HRCC target population). This again gets to the distinction between tier (population segment) definitions and clinical triggers as well as the emerging clinical consensus that there is a broad group of potentially clinically actionable, high opportunity adult patients who need tailored interventions of which HRCC is just one approach.

In addition to the development of aligned clinical workflows, additional BI tools for performance monitoring were also developed at this stage. For example, the reach of care coordination program services were measured at various levels of population: adult Tier 4, adult Tier 3, CRG subgroups, and super-utilizers. Inpatient and ED utilization was also trended for these various groups. Finally, DH clinical coaches continue to elicit front-line feedback to continue to refine this and other interventions targeting high-risk patients.

Financial Assessment

At the time of this paper’s submission, actuarial information on the Tiering 3.0 algorithm’s predictive performance is not available.

Clinical Assessment: Clinical Coherence and Care Model Implications

The above-described engagement with front-line clinical teams improved clinical acceptance of tiering. As discussed, this approach leveraged the predictive modeling strength of the CRG tool and combined it with clinical expertise to identify actionable patient populations for clinical pharmacy services and prioritize those patients in a meaningful way to direct work. Given this success, future optimization efforts are expected to follow this same approach. For example, front-line clinicians have identified the need for improved modeling of social determinants of health. Although the Tiering 3.0 optimization work was intensive, overall, the resulting rule changes impacted only a small proportion of patients.

In addition to remapping certain CRGs to Tiers, the Tiering 3.0 algorithm team also made minor refinements to tier promotion rules, the most significant of which further broadened the definition of adult Tier 4 super-utilizers to consider outlier ED utilization as well as outlier inpatient utilization, although different thresholds apply. Future iterations of the algorithm will likely narrow the definition of Tier 4 adult super-utilizers as clinical teams currently devote significant effort to screen out individuals unlikely to benefit from their services. Corroborating this front-line feedback, the health services research team has noted significant cycling between Tier 4 and the lower tiers, triggered by the super-utilizer tier promotion rules. This finding speaks to the need to distinguish individuals who are episodically high utilizing from individuals who are persistently high utilizing.

Figure 5 illustrates the Tiering 3.0 algorithm, summarizing the combined effect of the revised CRG mapping assignments and tier promotion rules in determining who gets into which tier and why. The front-facing triangle view captures individuals assigned to tiers based on their CRG status, whereas the side slope pictures those who have been promoted to a higher tier based on one of three tier promotion rules.

Figure 5.

Tiering 3.0 Algorithm: CRG-Assigned Versus Tier Promoters

Notes: Adverse Birth Outcomes: Empaneled. Tier 1 CRG-assigned mothers are promoted to Tier 2 when there is record of a previous high-risk delivery, including the following factors: teen pregnancy (patient <=18 years @ delivery); low birth weight (patient delivered an infant weighing less than 2.5 kg); NICU (patient delivered an infant requiring transfer to the NICU); and C5HCN (patient has a child followed on DH’s CSHCN registry).

Tier 4 Super-Utilizers: Empaneled or unempaneled patients of any Tier (1–3) are promoted to Tier 4 when they match the following: They have 3+ of Inpatient/Boarder/Observations (any combination) in a 12-month rolling window and are not otherwise CRG-assigned to Tier 4. Or. they have 2+ Inpatient/Boarder/Observations in a rolling 12-month period and a serious mental health diagnosis. Or, they have 10+ ED visits in a rolling 12-month period.

Adult High Risk: Empaneled patients who are otherwise CRG-assigned to Tier 2 are promoted to Tier 3 when the following conditions are met: They have a CRG Status of 6, 7, or 9 and have a CRG Severity Level > 3 and (have one or more Denver Health Inpatient stays or have two or more Denver Health ED visits in the previous six months.)

Conclusions and Next Steps

Table 5 summarizes the major features of each iteration of algorithm development, the clinical and technical lessons learned, and insights for future innovation.

Table 5.

Major Tiering Strategy, Lessons Learned, and Future Innovation Directions

	MAJOR FEATURES OF TIERING STRATEGY	LESSONS LEARNED	INSIGHTS AND FUTURE INNOVATION DIRECTIONS
“Proof of Concept” Grant Submission	100% sorted by CDPS (predictive model) risk score	Excellent financial stratification Poor clinical stratification	Need to align tier assignments with clinical interventions
Tiering Algorithm 1.0	78% sorted by CDPS risk score 19% sorted by clinical lab values 3% sorted by recent utilization	Defining risk status by clinical and utilization criteria improved clinical acceptability. Unstable tier assignments complicated aligned clinical interventions. Use of lab values unacceptably reduced financial discrimination (Tiers 1–3).	Lab values are better used to trigger interventions within risk tiers than to define risk tiers.
Tiering Algorithm 2.0	97% sorted by Clinical Risk Group (CRG) 3% sorted by “tier promotion” (patient-specific utilization) criteria	The relative clinical transparency of predictive modeling tools affects their clinical acceptance. Super-utilizer tier promotion rules have clinical salience but result in unstable tier assignments. Clinical alignment and financial discrimination can be simultaneously met.	Longer-term clinical interventions require a reasonably stable definition of risk status.
Tiering Algorithm 3.0	97% sorted by Clinical Risk Group (CRG) 3% sorted by “tier promotion” (patient-specific utilization) criteria	BI tools can help make predictive modeling tools more transparent and improve clinical acceptance. Front-line clinical feedback improves clinical acceptance through improved tier alignment with clinical interventions.	Need to distinguish episodically high utilizing from persistently high utilizing Need to improve modeling of social determinants of health Need additional clinic-level feedback

From a process perspective, there were additional lessons learned. The algorithm development team provided a mechanism to structure the process of coming together to define common purpose. We found that interdisciplinary population health work requires time and leadership to develop a common language for mutual understanding before it is possible to arrive at a shared framing of the problem, much less a solution. Adopting an intentionally iterative process from the outset reduced the pressure of getting it right the first time. Through trial and error, we learned which strategies work best to identify population tiers and which work best to trigger specific clinic action.

We found that predictive modeling tools are necessary but not sufficient to identify populations and individual patients appropriate for enhanced care team services. Clinical expertise is equally critical. Dynamic BI tools greatly facilitated this interdisciplinary communication by organizing and displaying data in ways that were familiar to the different disciplines engaged in the process. Particularly invaluable were the abilities to analyze the target population in real time and, especially, to toggle between data views organized at the population, subpopulation, and patient levels.

Although initially controversial, over a two-year period, tiering language has been incorporated into the DH vernacular. Particularly at the clinical leadership level, focus has largely shifted from a question of “why” to a question of “how.” As chronicled here, our evolution moved us from a solely statistically driven approach to a blended focus that incorporates additional dimensions of clinical acceptability and actionability. We conclude that population segmentation approaches that integrate clinical judgment with predictive modeling tools can better identify high opportunity patients amenable to medical home-based, enhanced care team interventions.