Assessing the Quality and Value of Psychological Health Care in Civilian Health Plans

Abstract

The Military Health System (MHS) strives to provide high-quality care and improve outcomes for individuals with psychological health conditions. Over the last decade, the MHS has provided care to a growing number of individuals with psychological health conditions, such as post-traumatic stress disorder (PTSD) and major depressive disorder (MDD). However, little is known about the extent to which the MHS delivers care that is consistent with evidence-based clinical practice guidelines or if it is achieving positive outcomes for its service members. To better understand these issues, the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury (DCoE) asked the RAND Corporation to describe civilian health plansâ; approaches to systematically measuring the quality of psychological health care delivered by providers in their networks. This work was part of a larger effort by RAND to develop a framework and identify a set of measures for monitoring the quality of care provided by the MHS for psychological health conditions.

Key Findings

• Psychological health quality measurement is common in health plans.

• Health plans primarily focus on collecting and reporting measures of the process of care.

• Health plans use quality measures to improve the quality of psychological health care.

• Measuring the quality of psychological health care is challenging.

• Measuring the value of psychological health care is more difficult and less common.

Summary

The Military Health System (MHS) strives to provide high-quality care and improve outcomes for individuals with psychological health conditions. Over the past decade, the MHS has provided care to a growing number of individuals with psychological health conditions, such as post-traumatic stress disorder (PTSD) and major depressive disorder. However, little is known about the extent to which the MHS delivers care that is consistent with evidence-based clinical practice guidelines or whether it is achieving positive outcomes for its service members. To better understand these issues, the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury (DCoE) asked the RAND Corporation to conduct case studies to identify the ways in which civilian health plans measure the quality of psychological health care delivered by providers in their networks. This work was part of a larger RAND effort to develop a framework and identify a set of measures for monitoring the quality of care the MHS provides for psychological health conditions.

Measuring the Quality and Value of Psychological Health Care

Health care quality is a concept that has been variously defined in the literature. For example, the Institute of Medicine (2001) has described quality as the “degree to which health care systems, services and supplies for individuals and populations increase the likelihood for positive health outcomes and are consistent with current professional knowledge,” while the World Health Organization states that “quality of care is the level of attainment of health systems' intrinsic goals for health improvement and responsiveness to legitimate expectations of the population” (Murray, Lauer, and Evans, 2000). Although these definitions of quality vary, they highlight the fact that quality refers to characteristics of and actions taken by providers that contribute to patient health. This concept of quality is most commonly measured across three domains (Donabedian, 2005):

• Structures are provider, practice, or system characteristics that are associated with high quality of care and may be considered a necessary but insufficient condition for quality. These structures might include the presence of key technologies (e.g., electronic medical record system) or characteristics of providers and staff (e.g., mix of registered nurses and licensed practical nurses on a nursing unit). Structural characteristics are also relatively concrete and more easily assessed compared to processes of care. The decision about which structural characteristics to measure should be guided by the extent to which those structures are associated with processes of care and outcomes of interest.

• Processes of care refer to interventions performed on and for patients and are often operationalized as compliance with evidence-based care guidelines. Instead of measuring the outcomes of medical care, processes of care can be used to assess the extent to which providers deliver “good” or “appropriate” medical care. Processes of care variables reflect the actual actions of a health care provider or system and, therefore, may be more appropriate variables in comparing providers. To measure processes of care, however, health care systems must carefully consider which processes of care are most appropriate to measure and the extent to which those processes lead to the outcomes of interest.

• Outcomes are heath states of individuals that result from the delivery of health care. Using outcomes to measure the quality of medical care can be especially useful as most stakeholders agree that health states, such as recovery, restoration, and survival from illness, are the end goal of medical care, and outcomes tend to be concrete and validly measured (e.g., death). Outcomes, however, are not always the direct result of medical care, and many conditions outside of the control of health care providers (e.g., patients' socioeconomic status) affect outcomes, rendering the comparison of health care providers difficult. Furthermore, it is not always clear which outcomes should be used, and many outcomes can be challenging to measure.

More recently, measurement experts have identified other domains across which quality of care might be measured. Particularly, the National Quality Measure Clearinghouse, a publicly available repository of quality measures established by the Agency for Healthcare Research and Quality, includes access to care and patient experience as key quality measure domains (Agency for Healthcare Research and Quality, 2014):

• Access to care refers to the extent to which patients can attain timely and appropriate care from health care providers. Access to care can be measured in a variety of ways, such as accessing data from an appointment management system, calling a provider practice to assess the availability of appointments, or estimating the number of providers of a certain type per 1,000 individuals.

• Patient experience refers to patient reports of their observations of and participation in the health care system. A patient can be asked to report on structure (e.g., timeliness of care), process of care (e.g., provider listened and respected patient, provider discussed treatment options), and outcomes of care (e.g., patient's perception of the helpfulness of treatment).

Measurement of health care quality is difficult regardless of the clinical conditions of interest. However, the systematic measurement of psychological health care quality is particularly complex relative to the measurement of physical health care quality (Addington et al., 2005; Hermann et al., 2004; Institute of Medicine, 2006; Kilbourne, Keyser, and Pincus, 2010; Williams, Cerese, and Cuny, 2007). Many of these challenges have been described in detail in the Institute of Medicine's report Improving the Quality of Mental Health Care and Substance Abuse (2006). Key barriers described in the report are:

• Diverse workforce: Psychological health care is delivered by a variety of provider types working in a number of different settings, such as primary care practices, therapist offices, prisons, schools, and community organizations. Providers working in these various settings have very different objectives, clinical approaches, and skills, making systematic measurement across groups challenging.

• Unclear locus of accountability: Because many patients with psychological health conditions see multiple providers in a variety of settings, it is challenging to determine which providers are accountable for providing various processes of care for such patients.

• Different types of evidence: Compared to many physical health conditions, a significant amount of the evidence for psychological health treatments is drawn from quasi-experimental designs, as opposed to randomized controlled trials. Therefore, wider stakeholder groups might view many psychological health quality measures as lacking the requisite evidence to be considered valid.

The concept of “value” incorporates both quality and cost into the definition. Value has been defined in a number of different ways and, importantly, holds different meanings for various stakeholders. However, a single, unified definition clearly and succinctly defines value as “outcomes relative to costs” (Porter, 2010). Because value incorporates measures of quality, measuring the value of psychological health care is susceptible to all the challenges listed above. Furthermore, because value combines both outcomes and cost data, measuring value requires a theoretical and computational burden that is much greater than that for developing either type of measure alone.

Despite these challenges, health plans are assessing the quality of psychological health care delivered by the providers in their networks. In some cases, health plans may be using quality and value measures to systematically monitor care delivered by the providers in their networks and to inform and monitor quality improvement and cost reduction efforts. In other cases, health plans may forgo use of standardized measures in favor of other means to assess quality, such as credentialing processes, peer review, or tracking patient complaints. Health plans often consider efforts to assess the quality of care to be proprietary (because demonstrating increased capacity for quality measurement and improvement is considered a strategic advantage), and health plans' “best practices” in this domain are generally not publicly reported in academic journals or gray literature (e.g., trade publications). Therefore, we conducted a small set of case studies of health plans to describe their approaches to assessing the quality of psychological health care. These case studies had three primary aims:

1. to describe health plans' efforts to assess the quality of psychological health care and the extent to which quality measures were used to monitor health plan performance and guide quality improvement strategies

2. to describe challenges and opportunities related to health plans' efforts to assess the quality of psychological health care

3. to describe health plans' efforts to actively assess the value of psychological health care and the extent to which value measures were used to monitor health plan performance and guide cost reduction efforts.

The overall goal of the case studies was to provide the Department of Defense (DoD) with information related to how psychological health care quality and value are being assessed in other health care systems to inform the development of a comprehensive measurement system for psychological health care within the MHS.

Case Study Methodology

The sampling frame for this study included national managed care organizations (MCO), integrated delivery systems (IDS, i.e., a network of organizations that provide a full continuum of care, such as physician services, hospital care, and insurance coverage), and behavioral health managed care plans (sometimes referred to as “behavioral health carve-outs” because they are contracted to provide only behavioral health benefits). RAND chose these three organization types because they represent the three primary types of civilian-sector payers for psychological health services. The case study participants were chosen based on convenience sampling, relying on RAND's familiarity with organizations that pay for and provide psychological health services, as well as the key individuals within those organizations. RAND experts nominated organizations believed to be leaders in quality measurement and improvement within the civilian health care sector and could be exemplar organizations. The selection of the organizations was based on personal communication and experience with the organizations, as well as a review of literature in which these organizations' work was highlighted. Arguably, the U.S. Department of Veterans Affairs (VA) would be the most analogous organization to the MHS. However, there are several joint DoD and VA efforts that facilitate cross-agency communication (e.g., the VA/DoD Integrated Mental Health Strategy). Therefore, we believed that the MHS would already be more familiar with initiatives within the VA. In consultation with DCoE, they agreed with our strategy to sample civilian health plans.

We then contacted executives at each of these organizations to assess their interest in participating in an interview. Of the three organizations we initially selected, all participated. For each organization, we conducted two interviews with one to two representatives. Our goal was to interview mid- to high-level executives responsible for overseeing performance management, clinical programs, and/or analytics.

The first plan was a national MCO, where we conducted interviews with a manager for quality improvement for behavioral health and a senior clinical executive. The second health plan was a large regional IDS, which was organized as a health maintenance organization comprising physicians, hospitals, and insurance coverage. We chose to include an IDS because this was the organization in the civilian sector most similar to the MHS. At the IDS, we interviewed a senior clinical director for mental health and substance abuse services and a senior executive for quality improvement for behavioral health. The third health plan was a large, national, managed behavioral health organization (MBHO), where we interviewed a senior executive for quality improvement and a senior executive for research.

We conducted semistructured interviews with each of the health plan representatives. RAND developed an interview protocol based on the content areas of most interest to DCoE (i.e., how civilian health plans were measuring quality and value). The protocol covered three main topics. First, we asked participants to describe their health plan and the types of products offered. Second, we asked about quality measures and how they were used to monitor performance and inform quality improvement efforts (e.g., How would you define “quality” as related to psychological health care? How do you measure the quality of psychological health services? Which measures have been the easiest or hardest to implement? Do you use these measures for quality improvement?). Third, we asked about value measures and effects of these measures on cost reduction efforts (e.g., How do you measure the cost/efficiency of psychological health care? Which measures have been the easiest or hardest to implement? Do you use these measures to monitor cost and efficiency?).

Two RAND researchers conducted each interview, with one leading the interview and the second taking extensive notes. All interviews were conducted by phone. All participants provided oral informed consent. Participants were told that the information gathered in the interview would be used for research purposes only and that their responses would be kept confidential but that readers might be able to identify organizations or interviewees by inference. After the completion of the interviews, the two researchers used a conventional content analysis (Hsieh and Shannon, 2005) to extract themes directly from typed notes. The researchers independently identified (1) how health plans were measuring quality and value, and (2) any themes that were similar across the interviews. The researchers then met to discuss their findings, identify common themes across the interviews, and reach a consensus on any thematic discrepancies.

Psychological Health Quality Measurement in Health Plans Is Common

Participants reported that they regularly measured and tracked psychological health quality measures using four key data sources:

1. administrative data

2. medical record review

3. member feedback

4. patient-reported outcomes.

Select Types of Quality Measures Used by Health Plans, by Data Source, as Cited During Interviews

Administrative Data

• Process measures assessing the receipt of care for treatment of alcohol and drug dependence and depression

• Process measures assessing the receipt of care following mental health hospitalizations

• Utilization (e.g., medication possession ratio and hospitalizations)

• Readmissions

• Inpatient length of stay

Medical Record Reviews

• Appropriate assessments

• Treatment plans

• Interdisciplinary communication

Member Feedback

• Consumer complaints

• Patient experience

Patient-Reported Outcomes

• Changes in general health

• Changes in psychological health symptoms

• Changes in drug and alcohol use

Administrative Data

Administrative data are records of services provided to patients, which are generally collected for the purposes of billing or registration. The data can include records of what service was delivered, any diagnoses related to the service, and service dates.

All the health plans reported measuring and tracking quality using administrative data-based measures. In this context, administrative data were generally used to construct Healthcare Effectiveness Data Information Set (HEDIS) measures and HEDIS-like measures (e.g., measures similar to HEDIS but using different measure specifications). HEDIS measures are constructed using health insurance claims data and used to measure the performance of health plans (National Committee for Quality Assurance [NCQA], 2013). In particular, the civilian health plans that we interviewed routinely implemented HEDIS measures required for NCQA accreditation. NCQA is a nationally recognized private 501(c)(3) not-for-profit organization that offers various levels of health plan, physician, and other organizational designations that include accreditation, certification, and recognition (NCQA, 2012). For accreditation, health plans are required to track and report the use of evidence-based care, particularly antidepressant medication management, follow-up after hospitalization for mental illness, identification of alcohol and other drug services, initiation and engagement of alcohol and other drug dependence treatment, follow-up care for children prescribed attention deficit hyperactivity disorder (ADHD) medication, and mental health utilization (among other nonpsychological health measures). Because its performance was reported through NCQA, the IDS monitored its own performance on a regular basis and strove to be at the 90th percentile or above on each measure. However, other plans did not report having a regular schedule to monitor their performance on HEDIS measures or to set specific performance goals.

The IDS was also required to report similar measures related to access to psychological health care services to a state regulatory agency. These measures included the extent to which members have access to:

• a nonurgent psychological health provider within ten days of a request for an appointment

• nonurgent specialist care within 15 business days of a request for an appointment

• urgent care within 48 hours of a request for an appointment.

Other administrative data-based measures tracked by the health plans included readmissions, length of stay, duration between admissions, and medication possession ratio. The MCO and MBHO stated that they preferred developing administrative data-based measures, such as medication possession ratio, because the requisite data are available in claims and do not require collecting primary data from patients or providers.

Medical Record Reviews

Two of the health plans reported performing regular reviews of patient medical records to assess both the accuracy and sufficiency of provider documentation. These audits were part of the process for credentialing providers and were often performed in an ad hoc manner. While these efforts did not yield data to construct quality measures per se, the health plans reported that such efforts were an important part of assessments of the quality of care. The MCO conducted medical record reviews with a random selection of 60–80 psychological health providers (including primary and specialty care) per year (out of 120,000 network providers) and examined these records for documentation of appropriate assessments (e.g., assessing for homicide and/or suicide, measuring patient-reported outcomes), follow-up after a specific diagnosis (e.g., substance abuse), discharge plans, and other types of clinical care. In all, about 40 items were examined, and every provider received feedback about these audits (described in more detail below). The IDS also conducted medical record reviews and specifically mentioned monitoring progress notes and specific quality measures tied to quality improvement initiatives. The IDS randomly selected two patient intake visits and three follow-up visits for each provider for review to ensure that patient outcomes were routinely measured, treatment plans were present in the progress notes, and reminders were sent to patients about how they could communicate with their provider if crisis services were needed. An important aspect of the IDS's medical record review was to examine coordination of care and whether interdisciplinary communication was occurring within the IDS (e.g., whether primary care physicians and psychiatrists were communicating about a patient). At least once a year, chairs from each department across the country discussed these reviews and other quality improvement efforts as part of their quality improvement committee. The MBHO did not discuss the use of medical record reviews as a key aspect of its quality assessment efforts.

Respondents cited the medical record review as an important aspect of their efforts to assess and improve the quality of care delivered by providers in their network. However, these efforts did not yield quality measures that could be systematically tracked, and none of the plans conducted record reviews to routinely assess patient outcomes.

Member Feedback

All the health plans reported collecting patient feedback in the form of consumer complaints and/or patient experience surveys. For example, the MCO and IDS reported that they regularly collected, tracked, and followed up on complaints about individual providers (e.g., if patients reported mistreatment, privacy breaches, or that an office was not conducive to therapy). In addition, the IDS and MBHO reported administering a patient or member survey to track service satisfaction. The IDS reported using a member survey, which included 15–20 items related to satisfaction (topics included ratings of treatment results and interactions with providers). Surveys were designed to be administered by providers and their administrative staffs when patients arrived for appointments. As part of their health plans' general NCQA accreditation, the MCO and IDS also were required to measure and report patient experience using the Consumer Assessment of Healthcare Providers and Systems instrument (Agency for Healthcare Research and Quality, 2012). However, these efforts were not specifically mentioned during the interviews as a key part of their efforts to measure and monitor the quality of psychological health care.

Patient-Reported Outcomes

Two of the health plans reported collecting patient-reported outcomes and that these measurements helped with several quality initiatives. The IDS administered a patient survey that assessed symptom improvement (e.g., the nine-item Patient Health Questionnaire (PHQ-9) with four additional items that assess anxiety and general functioning). This health plan reported asking providers to administer an assessment within the first two sessions and, four months later, to assess depression symptoms (PHQ-9; Spitzer, Kroenke, and Williams, 1999), mental and physical functioning (Medical Outcomes Survey Short Form) (Ware, Kosinski, and Keller, 1996), substance use (CAGE-AID) (Brown and Rounds, 1995), and other clinical symptoms. The MBHO asked providers to administer a patient assessment that included items from the Symptom Checklist-90 (Derogatis, 1977), CAGE-AID Health Performance Questionnaire (Brown and Rounds, 1995), Short Form (SF)-36 (Ware, 2003), and other questions regarding comorbid conditions and utilization of medical services in the previous six months. The MBHO asked providers to administer this assessment within the first two visits to the provider. The provider was then expected to enter this information into an online portal or fax it to the MBHO. After four months, the MBHO mailed a follow-up assessment tool, containing the same items, to the patient to fill out and return. Both the IDS and the MBHO reported that these data were designed to inform providers about how their patients were progressing and to identify at-risk patients. For example, providers from the MBHO could log onto a portal to obtain feedback about how their clients were improving or worsening and also had high-risk clients flagged. However, it is not clear whether providers actually use this information. Both health plans also reported that data collection from patients was challenging because of poor provider and patient response rates.

Health Plans Primarily Focused on Collecting and Reporting Measures of the Process of Care

The health plans that we interviewed were actively assessing the quality of care delivered by providers in their networks. However, the quality measurement efforts were relatively limited in scope and scale and may not have provided a comprehensive view of quality. The measures that were most consistently reported assessed processes of care, such as documentation of appropriate symptom assessments or rates of follow-up after hospitalizations. These process measures were generally limited in scope, however, focusing on drug and alcohol dependence, depression, and posthospitalization care, and were often assessed on a rather small scale (e.g., medical record audits of a small number of providers). Health plans rarely reported assessing the structure of psychological health care across their networks. Health plans reported the use of access measures to inform whether provider networks were compliant with state regulatory requirements (e.g., availability of appointments).

While health plans were enthusiastic about the use of patient-reported outcome measures, their efforts were still in the nascent stages as they continue to examine opportunities to engage patients and providers in the data collection process by helping them see value in having such data available. Health plans routinely tracked such measures as consumer complaints and patient satisfaction, which may be proxies for factors associated with the quality of care. For example, consumer complaints likely reflect the fact that providers do not deliver processes of care appropriately for their patients or that there are problems with some aspect of the structure of and access to care (e.g., availability of appointments, cleanliness of office space).

Health Plans Use Quality Measures to Improve the Quality of Psychological Health Care

Health plans used quality measures in a variety of ways. First, all health plans reported using administrative data-based measures to plan and monitor quality improvement efforts. For example, the health plans often used their own care management services to help ensure that their beneficiaries received recommended care. The IDS undertook a number of initiatives to improve HEDIS scores, such as having in-house care managers schedule outpatient follow-up visits while patients were still in the hospital to improve aftercare rates. They also monitored “high-risk” patients on discharge and reviewed their cases to ensure that they had a follow-up appointment within seven days of discharge (and called the patient and provider if not). The MBHO used claims data (e.g., utilization, comorbidities, hospitalizations) to flag high-risk patients. The plan then sent associated alerts to providers and offered additional outreach to members when these types of flags were identified. In addition, the plan tracked gaps in care in certain process measures, such as medication prescribing, as a sign that patients may be at high risk for rehospitalization.

Health plans also reported that an important part of their quality improvement strategies was to provide feedback to providers and facilities on their performance. For example, the MCO reported using medical record reviews to provide feedback to providers. If any of the 60–80 providers whose patients' medical records were reviewed received an overall score below 80 percent, the plan instituted a corrective action plan. Providers generally improved when reaudits were conducted because, if they did not comply, the health plan removed them from the network. The MCO also monitored patient complaints; provider recredentialing could be affected if there were three or more complaints. The MCO has considered sending providers reports on their performance compared to peer physicians but has not implemented this strategy yet because initial efforts were unsuccessful. Representatives from the MCO noted that their previous attempts to do this were impeded by difficulties communicating with some physicians who were not connected through electronic records and that physicians with “low performance” assumed the data were wrong and did not trust the data. Finally, the MCO is also examining ways to provide feedback to facilities regarding patients' length of stay compared to those of competitors.

The IDS reported using patient-reported outcomes to understand how well each location and provider managed care. This information was then used to either target resources to providers that needed more support or to identify providers that were top performers. The MBHO enabled providers to view their patient-reported outcomes compared to organizational benchmarks or other clinical thresholds for certain psychological health conditions. This online portal was tied to the billing portal system to encourage providers to use the platform. Patient outcomes were also connected to patient claims data to help providers identify and monitor high-risk patients.

Finally, health plans were starting to use performance data to implement “value-based insurance designs,” wherein health plans provide information and incentives to encourage patients to choose “high-performing” providers (Chernew, Rosen, and Fendrick, 2007). High-performing providers are selected using either cost or quality metrics or a combination of the two depending on the goals of the insurance design. Such strategies are often referred to as “steering” (Scanlon and Martsolf, 2009). The MCO was beginning to use quality measures to steer clients toward preferred drug and alcohol programs that report better outcomes. The strategy included ranking each facility by key outcomes, such as readmission rates and cost of care. The health plan would then provide members with incentives (i.e., reduced copay) to visit those providers. Such steering programs do face significant challenges. Particularly, many providers remain unconvinced that methodological approaches used to categorize providers are valid and reliable, leading to concerns that some providers would be unfairly penalized in such a scheme. In fact, there have been a number of high-profile court cases attempting to block efforts by insurance companies to implement such steering programs (Scanlon and Martsolf, 2009).

Measuring the Quality of Psychological Health Care Is Challenging

Health plans reported a number of challenges to measuring the quality of care delivered by their network providers. All the health plans stated that collection of patient-reported outcome data was arguably the most difficult aspect of their quality measurement initiatives. The health plans all stated that collection of these data was especially hampered by provider resistance. Many of these providers have argued that psychological health is subjective, individual-based, and not “one size fits all”; therefore, having objective outcome standards was not reasonable. The health plan representatives also stated that providers were not typically trained to monitor the quality of their services and were resistant to measuring, collecting, and reporting these data. Furthermore, for many providers, only a small fraction of their patients was covered by any given health plan. Thus, providers did not see the value of completing additional paperwork to document quality and value measurement for a single health plan. Also, some providers felt that collecting and entering patient-level data interfered with patient care. Echoing these reported challenges for providers, the literature suggests that collecting and using patient-reported data is difficult. For example, while one study found that 74 percent of psychological health systems collect patient-reported data (Schoenwald et al., 2008), another study found that few providers actually use this information in their clinical practice (Garland, Kruse, and Aarons, 2003).

To overcome provider reluctance related to collecting patient-reported outcome data, the MCO was piloting a program with three to four provider groups in different parts of the country to pay providers for using evidence-based assessments at the beginning of treatment and then again three months later. The goal of this effort was to incentivize providers to conduct these assessments and monitor clinical outcomes over time. The providers asked four questions about the clinical status of the patient at the beginning of the session and four questions about the process of therapy. Providers' performance was then compared to a national database.

Health plans stated that claims records, such as receipt of certain lab tests, were much easier to collect but often lacked substantive information. For example, the IDS reported that administrative-based measures (e.g., HEDIS measures) were easier to implement because the data were available in claims, did not need to be collected separately, and did not rely on provider compliance with data collection requests. However, these measures are limited because they focus on what services were delivered by providers, as opposed to patient outcomes (e.g., it was easy to know whether a prescription was filled but not whether the patient got better). Furthermore, the health plans reported that complete claims data were often unavailable for three months or more after the date of service.

Health plans noted several recommendations that could help contribute to successful quality measurement, including using measures already established in the literature, creating an organizational climate supportive of quality measurement (e.g., a video of members and providers sharing their experiences and emphasizing the importance of measurement), and utilizing provider champions to help increase provider buy-in. Finally, two health plans reported that, when measures were integrated into providers' clinical operations (e.g., electronic health record system), the data were easier to collect.

Measuring the Value of Psychological Health Care Is More Difficult and Less Common

The participants in our study commented significantly less on value measurement compared to quality measurement. This was driven both by the fact that many of the participants worked in areas directly related to quality of care and by the lesser development of their health plans' efforts to systemically measure value relative to quality. Nonetheless, health plans reported that they did monitor and track utilization patterns among their beneficiaries. For example, the MCO reported monitoring such measures as admissions per 1,000 members, hospital days per 1,000 members, and length of stay, while the MBHO tracked access rates, diagnosis rates, and changes in utilization by facility.

There was noticeable variation in the extent to which the plans measured the actual costs of care among members with psychological health diagnoses. The MCO reported regularly tracking actual costs using a “per member per month (PMPM)” measure. The plan produced reports on PMPM trends on a monthly basis, which were broken out by a number of categories, such as chemical dependency versus mental health, residential versus outpatient, pharmacy types, and patient age. The MBHO also tracked PMPM costs but did not provide specific information on how these costs were used or disseminated across the institution. Instead, it reported more regularly tracking utilization rates. The IDS did not produce any regular reports of its cost of care. This is largely explained by the fact the IDS was fully integrated, so it functioned primarily on a fixed-cost basis and did not pay fees for services. Therefore, marginal service costs were more difficult to track. The director of mental health of that plan reported that he could know the fixed costs associated with the psychological health delivery system but could not estimate marginal costs or efficiency. IDS did track utilization and noted that a long-term goal was to be able to use this utilization data to produce estimates of efficiency and answer such questions as “How many visits does it take to achieve a certain outcome?” He believed that once IDS was able to obtain enough patient-reported outcome data, it might be able to do some of this analysis.

The health plans that did track and monitor utilization and costs used these metrics for a number of different purposes. For example, the MCO monitored costs and utilization closely and, when there were significant changes in these metrics, deployed staff for greater oversight in that area. Similarly, the MBHO analyzed utilization and cost data to better understand trends and patterns to determine whether these changes were occurring at specific facilities or among specific demographic groups. Then, the staff would work with facilities to understand these spikes in utilization (e.g., whether patients are being referred to lower levels of care) and to create initiatives to reduce costs. However, the interviewees acknowledged that measuring value and reducing cost based on these measurements was challenging and something at which they were less skilled.

Implications for Psychological Health Quality and Value Measures Within the MHS

In this section, we consider how the findings from these case studies can inform future efforts to measure and improve quality and value of psychological health care in the MHS.

Continue and Expand Tracking of Access and Process of Care Measures

The health plans focused first on generating measures based on readily available data. Specifically, plans were generating administrative data-based measures of access to care (i.e., appointment availability) and care processes (i.e., the clinical procedures provided). Like the civilian health plans that we interviewed, the MHS routinely collects similar data as well. For example, the MHS regularly tracks its performance on 18 HEDIS measures within the direct care system and 12 measures within the purchased care system (Department of Defense, 2014a; Department of Defense, 2014b). Four of these measures are related to psychological health.

The civilian health plans, however, continue to expand their collection of access and process measures, and MHS might do likewise. Specifically, MHS might continue to expand measurement related to psychological health. MHS could improve on its measurement of quality by ensuring that such measures are collected systematically across all military treatment facilities (MTFs) and services and that measures are regularly reevaluated to include additional measures that may be of relevance to the MHS. While some MTFs may collect several psychological health quality measures, very few measures are collected and assessed consistently across all MTFs, services, and for direct and purchased care (Department of Defense, 2013, Department of Defense, 2014c). Collecting different measures of quality makes it challenging to compare across MTFs and care systems. A recent RAND report provides an extensive list of potential quality measures that could be considered. This report presents a list of 58 process, structure, and outcome measures that are candidates to be used to assess the quality of PTSD and MDD treatment (Hepner et al., 2015). An important step in expanding the scope of the quality measurement is to update and expand DoD policies related to the collection and assessment of psychological health quality measures. If the MHS were to expand its performance assessment capabilities, it would collect more quality information than many civilian health plans currently track and assess.

Engage Providers in the Process

Health plans reported that clinician autonomy often acted as a large barrier to quality improvement efforts. Interviewees and other literature report that provider buy-in is critical for successful implementation of any quality measurement program (Addington et al., 2005; Baars et al., 2010; Baker, 1998; Baker, 1999; Bremer et al., 2008; Gaebel et al., 2012; Kilbourne, Keyser, and Pincus, 2010; Stein et al., 2010; Reich et al., 2003). Particularly, while the health plans we interviewed were very enthusiastic about measuring patient-reported outcomes, they reported that providers were reticent to collect such information. To ensure adequate implementation of such measurement efforts, health plans must provide a compelling case to providers that the use of such data is personally or clinically beneficial. For example, the MCO that we interviewed has provided bonus payments to providers for using the patient-reported outcome measure tool. However, the initiative is in its early stages, and the plan has been unable to assess the impact of this incentive on the use of the measure. In other settings, such pay-for-performance mechanisms have been used in an attempt to improve process of care and outcome measures (Damberg et al., 2014). Health plans have also considered providing incentives to patients for filling out the tools, sending the information to the health plan, and discussing them with the providers. Likewise, in order to expand beyond the measurement of administrative data-based access and processes of care measures, MHS would need to more effectively engage MTF providers and have the flexibility to require purchased care contractors to collect and report quality data.

While financially incentivizing providers within the MHS may not be feasible, the MHS may use other nonfinancial incentives, such as mandates, to encourage provider compliance. For example, through an explicit policy memo, the military is encouraging the use of the Behavioral Health Data Portal (BDHP) to monitor symptoms for patients with PTSD, depression, and anxiety disorders (Department of Defense, 2014a; Brewin, 2013). These data could populate outcome measures in the future. Further, the MHS might work with the DoD to mandate clinics to collect quality of care data. Alternatively, the MHS might formally recognize high-performing MTFs and providers or publicly report MTF performance. This increase in transparency was one of the key recommendations in a recent report on MHS care (Department of Defense, 2014a).

The MHS might also experiment with financial and other incentives for purchased care providers. The MHS might consider incentivizing purchased care providers to collect quality data and participate in data collection efforts, consistent with recommendations provided in the Military Health System Review Report to the Secretary of Defense (Department of Defense, 2014a). Incentives could be incorporated into contracts with purchased care providers or through preferential referrals for providers who participate in data collection and sharing. For providers at MTFs, MHS could use other nonfinancial means to encourage provider participation in the collection of quality data not available in administrative data, especially patient-reported outcomes. Specifically, the MHS could work with providers to demonstrate that the use of quality measurement systems can help them deliver better care to their patients. A number of studies have demonstrated that patient-reported outcome tools can be used to engage patients and families in the care process, which can, in turn, improve outcomes (Azocar et al., 2003; Brodey et al., 2005; Duffy et al., 2008).

Implement Measures of Costs and Utilization, Which Provide an Essential Foundation for Assessing Value

Although the health plans we interviewed expressed interest in measuring the value of psychological health care, a number of significant barriers to implementation exist. Value measures remain underdeveloped for all health conditions but especially for psychological health. Some have suggested calculating the “value” of depression care by estimating the cost required to achieve a given outcome (e.g., quality-adjusted life years or depression-free days). However, these measures require the collection of valid and reliable patient-reported outcomes over time. As discussed earlier, efforts to collect such outcome data within health plans have been extremely time consuming and require substantial engagement by providers. Given the struggles of civilian health plans to collect outcomes data, the MHS might be a leader in developing value measures, given they are beginning to see success systematically collecting patient outcome data through such programs as the BDHP. However, to move toward developing measures of value, the MHS might consider regularly measuring cost and utilization data. Such data could be linked to the growing outcome measures to create measures of value. The first step toward measuring cost would be to consistently track utilization data, readily available in claims, which include hospitalizations, readmissions, and length of stay, as well as utilization of prescription drugs and ambulatory care visits. Although these are not explicitly measures of cost, they provide a useful starting point for the MHS to move toward measuring the cost and value of care it provides.

Limitations

The case studies have at least three limitations. First, we relied on RAND experts to nominate health plans known for their quality measurement efforts. This process for selecting organizations was neither formal nor unbiased, but we were purposive in this regard because we wanted to better understand what leading health plans were doing and how their efforts could inform DoD. Second, the comments provided by the two representatives from each organization may not represent the larger views and efforts of the entire health plan. Finally, because we interviewed only three health plans, the results gathered in this study are not necessarily generalizable to all civilian health plans.

Conclusions

In summary, we used a case study approach to better understand how civilian health plans use quality and value measures to monitor their performance and to plan and evaluate quality improvement and cost reduction efforts. We found that health plans were regularly collecting and acting on quality measures but less often on value measures. Some measures, particularly patient-reported outcomes, have proven to be difficult to collect and use. These findings may be useful for DoD in the further development of its own psychological health care quality and value measurement programs.