Skip to main content
World Psychiatry logoLink to World Psychiatry
. 2018 Jan 19;17(1):30–38. doi: 10.1002/wps.20482

Measuring and improving the quality of mental health care: a global perspective

Amy M Kilbourne 1,2, Kathryn Beck 2, Brigitta Spaeth‐Rublee 3, Parashar Ramanuj 4,5, Robert W O'Brien 1, Naomi Tomoyasu 1, Harold Alan Pincus 6
PMCID: PMC5775149  PMID: 29352529

Abstract

Mental disorders are common worldwide, yet the quality of care for these disorders has not increased to the same extent as that for physical conditions. In this paper, we present a framework for promoting quality measurement as a tool for improving quality of mental health care. We identify key barriers to this effort, including lack of standardized information technology‐based data sources, limited scientific evidence for mental health quality measures, lack of provider training and support, and cultural barriers to integrating mental health care within general health environments. We describe several innovations that are underway worldwide which can mitigate these barriers. Based on these experiences, we offer several recommendations for improving quality of mental health care. Health care payers and providers will need a portfolio of validated measures of patient‐centered outcomes across a spectrum of conditions. Common data elements will have to be developed and embedded within existing electronic health records and other information technology tools. Mental health outcomes will need to be assessed more routinely, and measurement‐based care should become part of the overall culture of the mental health care system. Health care systems will need a valid way to stratify quality measures, in order to address potential gaps among subpopulations and identify groups in most need of quality improvement. Much more attention should be devoted to workforce training in and capacity for quality improvement. The field of mental health quality improvement is a team sport, requiring coordination across different providers, involvement of consumer advocates, and leveraging of resources and incentives from health care payers and systems.

Keywords: Mental disorders, quality of care, quality measurement, health informatics, electronic health records, patient‐centered outcomes, health care systems, health policy


Mental disorders are responsible worldwide for 32% of years of disability and 13% of disability adjusted life years1. In addition, persons with these disorders face increased rates of morbidity from general medical conditions2, 3, 4 and a higher risk of premature mortality5. Among persons with mental disorders, disparities in quality and outcomes of care are more pronounced for racial/ethnic minorities6, 7, 8, and those from lower socio‐economic status groups9. Severe mental illness (e.g., schizophrenia and bipolar disorder) is emerging as a prominent health disparity category, given estimates that persons in this group die 8‐25 years younger than the general population10, 11. Despite the contribution of mental disorders to the global burden of disease, the quality of care for these disorders remains suboptimal, and there are persistent gaps in access to and receipt of mental health services worldwide12, 13, 14, 15, 16, 17, 18.

Quality of care, as described by the Donabedian framework, includes structure, or organization of care, the influence of structure on clinical processes of care as delivered by providers, and ultimately, patient‐level health care outcomes19, 20, 21 (see Table 1). This system‐level perspective of health care quality (structure, process, outcomes) became the foundation for two US Institute of Medicine's reports: Crossing the Quality Chasm22 in 2001 and Improving the Quality of Health Care for Mental and Substance‐Use Conditions23 in 2006.

Table 1.

Mental health quality measures: key examples

Description Examples
Structure Are adequate personnel, training, facilities, quality improvement infrastructure, information technologies, and policies available for providing care? Adequate number of components available in assertive community treatment program
Availability of mental health specialists in primary care practices
Presence of a mental health care manager
Process Are evidence‐based processes of care delivered? Percent of patients in mental health program who have documented substance use screening
Receipt of adequate dose of psychotherapy
Outpatient follow‐up within 7 days after mental health hospitalization discharge
Outcome Does care improve clinical outcomes? Functioning (e.g., assessed by WHO‐DAS)
Employment (% patients returning to work)
Symptoms (e.g., depressive, assessed by PHQ‐9)
Recovery

WHO‐DAS – World Health Organization Disability Assessment Scale, PHQ‐9 – Patient Health Questionnaire‐9

The Crossing the Quality Chasm report highlighted six aims towards quality improvement – safe, effective, patient‐centered, timely, efficient, and equitable care – and stated that “quality problems occur typically not because of failure of goodwill, knowledge, effort or resources devoted to health care, but because of fundamental shortcomings in the ways care is organized”22. The 2006 report further noted the persistent gaps in quality of mental health care and called for systematic efforts to improve quality in this area23.

Nonetheless, the overall quality of mental health care has hardly improved since publication of these reports and, in some cases, has worsened over time24. In the US, only a third of those in need receive adequate mental health care25. The level of mental health quality of care is poor and the rate of improvement is slow compared to general medical conditions26. For example, recent data indicate that less than half of patients with publically funded insurance get adequate follow‐up after mental health hospitalization27. This persistent gap in quality of mental health care is due in part to lack of systematic methods for measuring quality. We cannot improve what we cannot measure.

As health care costs continue to rise and mental disorders become more prevalent worldwide, health care leaders and providers will need valid information on quality of care, in order to: a) identify population needs and make decisions on how to provide the best services, and b) apply effective strategies to improve quality and reduce disparities. This paper describes the current state of quality measurement of mental health care and the challenges it poses to health care systems internationally, and suggests next steps for health care systems around the world to better implement quality measurement and ultimately improve quality of mental health care.

CURRENT STATE OF MEASURING MENTAL HEALTH CARE QUALITY

Worldwide, efforts to standardize mental health care quality measurement are slowly evolving. Measuring and reporting quality of care on a routine basis enables the application of quality improvement at provider, clinic and health system levels, as well as accountability mechanisms that include public reporting and financial penalties and rewards. However, measuring quality of mental health care is challenging worldwide, as it can vary based on the organization of services by country. In general, structure, process and outcome measures have all been employed for accreditation, standard setting, quality improvement and accountability in health care generally and in mental health care. Each have strengths and weaknesses and, ultimately, a balanced portfolio across these categories is needed.

Health care structural components, such as resources (personnel, training, facilities) and policies that support measurement‐based care, are fundamental to achieving high quality care. However, while adequate structure measures create the necessary infrastructure for reporting on processes and outcomes and conducting improvement activities, they do not provide sufficient detail as to whether quality services are actually being delivered as intended (fidelity) nor if the outcomes obtained are acceptable.

Ideally, process measures can fill this gap and assess whether evidence‐based practices are in fact being implemented. These measures generally involve operationalizing clinical guidelines into specifically defined denominators and numerators, using data that can be reliably obtained from feasibly accessed data sources. However, many widely used mental health process measures lack evidence to be used in mental health quality and outcome improvement. Only a few studies have linked quality of care process measures to improvements in patient functioning and clinical outcomes, calling into question the clinical validity of these measures. Some notable exceptions that have been reported recently show that measures for improved processes of care (e.g., appropriate pharmacotherapy, continuity of care, and psychotherapy use) are associated with reduced mortality28, 29, 30, 31 and reduced symptom severity32. Still, even among existing mental health process measures that could be reported, not all have been validated25, 26, 33, 34, 35, 36, 37, 38, 39.

Outcome measures assess whether the care that a patient receives actually improves his/her symptoms – e.g., improvement or remission in Patient Health Questionnaire‐9 (PHQ‐9) scores – or functioning. These measures can also assist providers in planning, monitoring and adjusting treatment options (e.g., change in medication, multi‐component treatment collaboration). However, in order to address the complexity of mental disorders, mental health outcome measures should not only focus on symptoms and functioning, but also on issues such as quality of life, recovery, and community tenure.

Furthermore, the use of outcome measures for the purpose of evaluating the quality of mental health care requires sophisticated risk adjustment approaches to control for underlying patient risk factors beyond providers’ control, such as severity of illness, medical history/health status, socio‐demographic factors, in order to minimize “cherry‐picking” of the healthiest patients. This, however, may be challenging, due to typically limited available data on psychiatric symptoms, social context and other patient characteristics. Increasingly, there are calls to add patients’ experiences to a balanced portfolio of measures, to get their view about a system's structures, the care they have received, as well as self‐reported outcomes.

In addition, the mental health service field lacks consistent outcome measures and tools that are embedded in current information systems and other rapidly changing technologies. Lack of ability for system‐wide routine data collection within existing electronic health care systems can ultimately impede continuous quality improvement for patients. To mitigate this challenge, mental health experts are embracing measurement‐based care to promote the use of outcome measures on a routine basis.

Measurement‐based care is a core component of the chronic care model40, 41, 42, which uses proactive data collection to provide patient‐centered care plans. These are delivered by a care manager who also coordinates care between different providers so that it is tailored to the patient's current disposition and self‐management preferences. The chronic care model has been shown in multiple randomized trials to improve physical and mental health outcomes across different mental disorders, with little to no added cost42. Measurement‐based care relies on clinical measures (e.g., PHQ‐9, mental health vital signs) as well as systematic, longitudinal and action‐oriented care to track, assess and respond to changes in individualized outcomes, such as symptom severity and goal attainment, frequently and over the long term.

Key international examples of measurement‐based care include the Improving Access to Psychological Therapies (IAPT) program within the UK National Health Service43, 44, the Dutch Depression Initiative primary mental health collaborative care model45, and the Australian TrueBlue model46. Notably, after initial pilot testing and successful evaluation, the IAPT was expanded in the UK for at least 1.5 million adults to access care each year by 2020/2147, and the Depression Initiative primary mental health collaborative care model was included in the Netherlands into the list of national essential benefits as part of the Health Insurance Act45. However, these programs do not reach all patients with mental disorders, and a majority of health care providers do not routinely apply measurement‐based care48, 49.

In the US, there are a few notable examples of public and private measurement‐based care programs in primary and specialty mental health care settings that are adopted as clinical tools, but to date not widely used for quality measurement. For example, the Sequenced Treatment Alternatives to Relieve Depression (STAR*D)50, the US Department of Veterans Affairs Behavioral Health Laboratory model51, 52 and the US Department of Defense Behavioral Health Data Portal53 are all examples of measurement‐based care applied to patient populations. In the State of Minnesota, the Depression Improvement Across Minnesota, Offering a New Direction (DIAMOND) initiative implemented measurement‐based care to help benchmark quality improvement efforts as part of a bundled payment initiative for depression care management54.

UNIQUE CHALLENGES TO MENTAL HEALTH CARE QUALITY MEASUREMENT

In the US and worldwide, mental health care quality measurement and measurement‐based care have a weak infrastructure in health care systems. This is due to a multitude of barriers specifically related to mental health, that involve limitations in policy and technology as well as limited scientific evidence for mental health quality measures, lack of provider training and support, and cultural barriers to integrating mental health care within general health environments.

The development and application of mental health care quality measures has lagged behind other areas of medicine, in part to lagging policy and technological initiatives. For example, in the US, quality measures are used for chronic medical conditions to set reimbursement through Medicare, the government's public insurance program for elderly individuals (e.g., Value‐Based Purchasing Modifier55), Medicaid56 and State Medicaid Reporting Programs57, and to benchmark care quality in the private sector (e.g., PhysicianCompare.Gov58, HospitalCompare.Gov59). Yet, despite the mental health parity laws passed ten years ago, which stipulate equal coverage for mental health conditions, and the availability of over 500 measures for monitoring quality of mental health care, only 5% of these measures are actually used in the above major quality reporting programs, and only 10% of the measures have been endorsed by the US National Quality Forum60 (e.g., Value Based Inpatient Psychiatry Quality Reporting Program61). Of these available measures, the majority (72%) focus on processes quite distal to outcomes (e.g., screening/assessment)60 rather than on process measures that indicate treatment adequacy or intensity for mental health care.

On the other hand, there are many important gaps in the evidence base to support mental health quality measurement, especially for outcomes that are most meaningful to consumers, as well as for specific populations such as children. Measures are also lacking for mental health conditions commonly experienced in populations, such as anxiety disorders, and lacking in depth for evidence‐based treatments such as psychotherapy. While there is well‐established evidence for mental health interventions such as pharmacotherapies, specific manualized psychotherapies (e.g., cognitive behavioral therapy), and team‐based interventions (e.g., assertive community treatment), the evidence base for many other psychosocial interventions needs to be strengthened62. For evidence‐based psychotherapies, quality measures may not fully capture whether they were delivered adequately. Moreover, many providers are able to codify psychosocial interventions in administrative data, but not whether the intervention was delivered with fidelity23, 63.

There is also insufficient attention to the development and implementation of performance measures that reflect patients’ views and treatment choices. As a result, few endorsed mental health quality measures assess patient‐centered care, notably mental health recovery. The US Substance Abuse and Mental Health Services Administration defines mental health recovery as “a process of change through which individuals improve their health and wellness, live a self‐directed life, and strive to reach their full potential”64. Yet, identifying valid recovery measures has been hampered by a lack of consensus about an operational and measurable definition of recovery among providers, the research community and, most importantly, consumers of mental health services. While this is partially inherent to the subjective process of recovery, it has resulted in a large variation in reliability and validity of recovery measures and tools. Beyond the needs for further evidence to support clinical guideline development and a broader array of valid and useful patient reported outcomes, there has been little investment in the development and testing of mental health care quality and recovery measures to assure their validity, utility and comprehensiveness.

Furthermore, the mental health field is far behind other areas of medicine with regard to the implementation of technologies, notably health information technology to capture relevant health information that could support reporting on mental health care quality measures. Despite some incentives to implement electronic health records (e.g., the HITECH Act in the US), there is no specific requirement worldwide to include mental health data in electronic records. Currently, many mental health care quality measures are not linked to existing data sources, which mostly rely on claims data rather than data derived from electronic health records or electronically‐reported patient outcomes26, 65. As a result, these measures cannot be automated to generate meaningful data60, which in return could support quality measurement and inform routine medical practices and procedures. In addition, mental health providers often use separate electronic medical record systems from their general medical provider counterparts, or do not have access to these systems at all, creating big challenges to engage the mental health field as a whole in quality measurement and improvement of care for patients who often require coordinated services across different sectors.

In some countries with common claims datasets or electronic medical records, mental health care measures have been variably adopted66, 67. For example, the UK National Health Service has a long tradition of using electronic medical records in primary care for routine quality measurement, most notably through the Quality Outcomes Framework, the largest payment‐by‐results program in the world. Over the past ten years, the National Health Service has tried to implement a similar outcome‐based reimbursement program in mental health care68. This would have made routine measurement mandatory for funding. However, the administrative burden involved and the risk of gaming (i.e., biased reporting to improve apparent performance) has led to resistance from the profession68, 69. The program has now been indefinitely postponed in implementation in favor of smaller areas of work70. One of these areas is the above‐mentioned IAPT initiative, which embedded routine outcome measurement – using validated tools such as the PHQ‐9 and the Clinical Outcomes in Routine Evaluation (CORE) – and could demonstrate good outcomes that have led to further funding into the initiative71. In Canada, there has been the adoption of mental health care quality measures in electronic medical records67. Still, due to long‐standing stigmatization and functional challenges, consumers of mental health services may feel burdened by the data gathering. Overall, integrating health information technology into routine mental health treatment practices is paramount to support measurement‐based care for mental health72, 73.

In addition, heterogeneity of provider training and certification requirements within mental health care can also hinder quality measurement implementation. For example, in spite of their extensive involvement in mental health care, less than one third of US social workers receive training in quality measurement and effective clinical practices74. Moreover, many of the challenges that providers address with their patients include service needs beyond health care (employment, housing, education, criminal justice and welfare), and quality of care for these services is rarely measured to ensure improved mental health outcomes and recovery. These services often require coordination across different providers, settings, agencies and even sectors, but there is little incentive to improve quality when there are no measures to assess accountability for these services. A notable exception to this has been the US cross‐agency priority goal of ending Veteran homelessness, where the US Department of Veterans Affairs began working with other federal, state and local agencies to provide housing vouchers and track outcomes over time75.

Finally, cultural and administrative differences between physical and mental health providers hinder quality measurement. “Physical” and “mental health” services, in many if not most countries, are often administratively separated at clinical, organizational, policy and financial levels. Mental health care also requires more of a team effort between psychiatrists, social workers, psychologists and case managers, and mental health visits are typically longer, due to the nature of the illnesses.

INNOVATIONS IN MENTAL HEALTH CARE QUALITY MEASUREMENT AND IMPROVEMENT

Several innovations are underway worldwide for measuring and improving quality of mental health care. These initiatives combine advances in technology or measurement‐based care with concerted efforts to obtain patient and provider buy‐in towards continuous quality measurement and improvement.

International innovations in quality measurement include the World Health Organization (WHO)'s Assessment Instrument for Mental Health Systems76, and the International Initiative for Mental Health Leadership77, which provides data on reporting, ability to report, and ascertainment of data across countries.

In the Netherlands, routine outcome monitoring has been incorporated into health insurance reimbursement mechanisms. This evaluates three aspects of quality – effectiveness of treatment, safety and client satisfaction – through ten measures that are repeated at the start and end of treatment78. The initiative stipulates that the indicators are collected centrally and published transparently to stimulate continuous quality improvement.

In Australia, the use of standard outcome measures for all mental health service users was mandated in 2000, and all Australian states have signed agreements to submit routinely collected outcomes and case mix data. The principal outcome measures are the Health of the Nation Outcome Scales (HoNOS) and a quality of life instrument. To be able to implement this initiative on such a large scale required considerable investment in mental health providers, ongoing training and a broad program of engagement79.

In New Zealand, mental health providers focus on monitoring of key indicators, such as seclusion and restraint minimization, and suicide reduction80. In the UK, the National Health Service Benchmarking Network81 is a collaboration between all mental health provider organizations, which supply data to benchmark their own practice against others. The Benchmarking Network was developed because of the perceived inadequacy of the national data collection system and the lack of feedback on the large amount of data collected. As a ground‐up initiative, the Benchmarking Network required a large degree of engagement and dynamic leadership.

In the US, national efforts are underway to identify cross‐cutting mental health care quality measures and to determine who “owns” responsibility for improving quality. In the Department of Veterans Affairs, quality measures are set by central leadership for implementation in over 160 medical centers. While quality of mental health care in the Department has been widely documented, regional variations in processes and outcomes of care are common82, 83, 84, 85, 86. Hence, while regional service directors are ultimately responsible for improving quality, the Department has launched national initiatives to improve quality of care and reduce disparities in mental health care, notably through the implementation of the Uniform Mental Health Services Handbook87 and the deployment of mental health care managers in primary care settings to promote integrated care. The Department has also sponsored the national implementation of evidence‐based psychotherapy for post‐traumatic stress disorder88.

Pay‐for‐performance (now more often termed “value‐based payment”89) models are also increasingly being advocated in the US and internationally. These initiatives reward providers for outcomes improvement and are also increasingly becoming used in mental health care90, 91. Other innovations involve care beyond the clinic walls, including the measurement of recovery‐oriented services92 and incorporation of mobile health to capture outcome data65, 93. The US Centers for Medicare & Medicaid Services is also deploying initiatives that seek to improve provider use/engagement in evidence‐based practices as well as delivery system changes to sustain them. The main focus has been to integrate mental health treatment into primary care, where most patients with mental health symptoms initially present. The Institute for Healthcare Improvement Breakthrough series used business practices to integrate chronic illness care management for depression in primary care settings94. There also exist other pockets of innovations in integrating mental health into primary care (e.g., the Health Care System Research Network, the Community Mental Health ‐ Cherokee Health System95), but few frameworks to scale up and spread.

In the UK, the Commission for Quality and Innovation is implementing pay‐for‐performance for mental health, in which payments are based on meeting national quality improvement targets96. The targets are set locally, but with centrally agreed goals. Nonetheless, inevitable variations in care delivery make the development of quality measures a more difficult process in the mental health field.

Finally, there are emerging efforts to engage multi‐stakeholder groups to solicit feedback throughout the entire process of quality measurement development and implementation. While frontline clinicians are often able to provide input for quality measures development, garnering feedback from consumers and their caregivers is also considered essential for buy‐in97. Byron et al98 describe a process of engaging stakeholders at all levels of measure development and implementation for Children's Health Insurance Program Reauthorization Act (CHIPRA) quality measures. The Measure Development Plan outlines the planned process, including engaging stakeholders99. The National Quality Forum uses a consensus process for review and endorsement of measures, including periods for public comment100. Moreover, the Centers for Medicare & Medicaid Services recently convened technical expert panels to help develop, select and maintain measures including clinicians, statisticians, quality improvement experts and methodologists101.

RECOMMENDATIONS

We offer several recommendations for implementing quality measurement as an ultimate tool for improving quality of mental health care. First, health care payers and providers will need a portfolio of validated measures of patient‐centered outcomes across a spectrum of conditions commonly experienced, as well as for special populations, including children/youth102. Moreover, valid measures that assess mental health care access are also needed, in order to more comprehensively determine quality of care beyond what happens within the clinical encounter. Measures need to be validated across the Donabedian spectrum (structure, process, outcome).

Second, common data elements should be developed and implemented for diagnoses, clinical measures and mental health “vital signs” and embedded within existing electronic health records and other information technology tools such as smartphones. Other elements that need to be standardized include coding in both electronic health records and administrative datasets for interventions such as medications, psychotherapies (including fidelity measures) and other treatments or care processes. Innovations such as natural language processing, or the automated capture of information from electronic medical records, are already being used to facilitate data capture for information (e.g., homelessness or suicide risk) not readily apparent from claims data.

Third, mental health outcomes will need to be assessed more routinely, and measurement‐based care not only needs to be embedded within existing technologies, but should become part of the overall culture of the treatment setting and health care system. Regular outcome assessments have been linked to improvements in service delivery and lower readmission rates103, whereas infrequent outcome measurement did little to improve quality104. Moreover, routine outcome measurement that was fed back to the clinician and used to make joint treatment decisions with the patient did lead to better quality of life105. Quality measures need to be used in health systems that can generate near‐real time data on quality in order to promote continuous quality improvement, and need to be monitored for unintentional consequences such as gaming.

Fourth, health systems need to provide investment, leadership and coordination to improve and link data sources in order to measure quality across settings. Systems will need to involve frontline providers and consumers in quality measurement endorsement and design measures that fit the needs of these providers and consumers rather than those of the administrators. Too often systematic quality outcome measurement is driven by a desire to inform policy or reduce expenditure rather than improve treatment decisions for individuals, which may have an adverse effect if staff (who are meant to be collecting the data) perceive it as a distraction with little value. Efforts like the UK Benchmarking Network are a good way of incorporating these perspectives106.

Finally, health care systems need a valid way to stratify quality measures, in order to address potential gaps among subpopulations and identify groups in most need of quality improvement. A much greater expectancy for workforce training in and capacity for quality improvement is essential. Strategies for quality improvement and accountability need to be adapted, developed, and applied routinely in mental health settings.

In Table 2, we propose a broad multilevel process that outlines barriers to quality measurement and potential facilitators leading to quality improvement107. This process, based on the US National Academy of Medicine Learning Health Care System framework, is updated to include “levers” that address organizational barriers experienced in mental health care108. Learning health care systems leverage existing data (e.g., electronic health records) to deploy and evaluate innovations and best practices across health care organizations with the goal of improving population health.

Table 2.

Learning health care system framework for mental health care quality improvement

Barriers Leverage opportunities in learning health care systems
Patients Medical and behavioral health conditions co‐occur
The majority of patients are still seen in small primary care practices
Adopt mental health measurement‐based care (continuous use of validated outcome assessments that inform changes in treatment decisions)
Consumer organizations link patients to recovery‐oriented services in the region
Providers The majority of providers lack training in quality improvement and evidence‐based practice implementation
Lack of incentives for non‐mental health providers to incorporate mental health services where patients are more likely seen (e.g., primary care), and lack of integration with social services
Professional organizations mandating training in quality measurement and improvement methods
Same‐day billing for mental health and physical health care
Mental health professional organizations adopt common quality measures, guidelines, and improvement strategies
Practices/Organizations Limited electronic medical record use in the majority of mental health sites
Lack of effective strategies to scale up and spread evidence‐based mental health treatments and models of care
Standard health information exchanges need to include mental health services
Embed quality improvement experts to help identify, test and scale up treatment models to promote measurement‐based care
Purchasers/ National health systems Primarily fee for service, few bundled payment models Plan‐level mental health care coordination
Instability in health insurance markets Value‐based reimbursement payment models benchmark on improved quality rather than volume
Population Stigma Public reporting of quality measures

CONCLUSIONS

Improving quality of mental health care is a team sport, requiring coordination across different providers, involvement of consumer advocates, and leveraging of resources and incentives from health care payers and systems. Figure 1 offers a roadmap for measuring and improving quality of mental health care. First, patients, providers and health care systems need to provide input on the choice of measures and their implementation. The steps to be taken include establishing an evidence base for quality measures through practice guidelines, operationalizing guidelines into quality measures that have a numerator and denominator based on data easily captured from health care settings, testing quality measures for their reliability and validity (ensuring that they also do not lead to gaming or manipulation), finalizing measures based on endorsement from patients, providers and system leaders as well as professional organizations, adopting the measures for use in routine practice, aligning measures across multiple settings (e.g., primary care, social services), and finally, identifying a group to “own” the measures that will continually monitor and provide strategies to incorporate quality improvement where necessary.

Figure 1.

Figure 1

Multi‐stakeholder roadmap for measuring and improving quality of mental health care

The recommendations for improving quality of mental health services presented here can apply to health care in general. Indeed, mental health has led the way in other health care innovations, including moving care into the community, use of innovative models of integrated care, as well as measures of patient‐centered recovery. Moreover, there are lessons learned from mental health services that will inform the rest of health care to adopt a learning health care system. For years, mental health consumers and their family members have advocated for “patient‐centered” care and greater focus on the personal goals of the patient, above and beyond receipt of medical services.

The diverse nature of mental health providers also challenges the health care system to take into consideration the perspectives of frontline staff including nurses, social workers, and increasingly peer specialists in owning quality improvement. It is not surprising that many of the quality improvement methods used in mental health care have influenced the growing field of implementation science109, which is the study of provider behavioral change within the context of organizational constraints. Finally, the growth of value‐based payment models that reward health systems and providers on achieving outcomes rather than on volume of services holds great promise for improving the quality of mental health care.

ACKNOWLEDGEMENTS

This paper was supported by the Department of Veterans Affairs, Veterans Health Administration, the Irving Institute, and the National Institutes of Health (R01 MH 099898). Additional funding support was provided by the Commonwealth Fund (grant no. 20141104) and by the National Center for Advancing Translational Sciences, National Institutes of Health (grant no. UL1 TR000040). The views expressed in the paper are those of the authors and do not necessarily represent those of the Department of Veterans Affairs, the National Institutes of Health, or other public entities.

REFERENCES

  • 1. Vigo D, Thornicroft G, Atun R. Estimating the true global burden of mental illness. Lancet Psychiatry 2016;3:171‐8. [DOI] [PubMed] [Google Scholar]
  • 2. De Hert M, Correll CU, Bobes J et al. Physical illness in patients with severe mental disorders. I. Prevalence, impact of medications and disparities in health care. World Psychiatry 2011;10:52‐77. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3. Goldberg D. The detection and treatment of depression in the physically ill. World Psychiatry 2010;9:16‐20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Lin EH, Rutter CM, Katon W et al. Depression and advanced complications of diabetes: a prospective cohort study. Diabetes Care 2010;33:264‐9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Walker ER, McGee RE, Druss BG. Mortality in mental disorders and global disease burden implications: a systematic review and meta‐analysis. JAMA Psychiatry 2015;72:334‐41. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Carson N, Vesper A, Chen C et al. Quality of follow‐up after hospitalization for mental illness among patients from racial‐ethnic minority groups. Psychiatr Serv 2014;65:888‐6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Cook BL, Zuvekas SH, Carson N et al. Assessing racial/ethnic disparities in treatment across episodes of mental health care. Health Serv Res 2014;49:206‐29. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. Coleman KJ, Stewart C, Waitzfelder BE et al. Racial‐ethnic differences in psychiatric diagnoses and treatment across 11 health care systems in the Mental Health Research Network. Psychiatr Serv 2016;67:749‐57. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Case A, Deaton A. Rising morbidity and mortality in midlife among white non‐Hispanic Americans in the 21st century. Proc Natl Acad Sci USA 2015;112:15078‐83. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. Parks J, Svendsen D, Singer P et al. Morbidity and mortality in people with serious mental illness. Alexandria: National Association of State Mental Health Program Directors, 2006.
  • 11. Liu NH, Daumit GL, Dua T et al. Excess mortality in persons with severe mental disorders: a multilevel intervention framework and priorities for clinical practice, policy and research agendas. World Psychiatry 2017;16:30‐40. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12. Demyttenaere K, Bruffaerts R, Posada‐Villa J et al. Prevalence, severity, and unmet need for treatment of mental disorders in the World Health Organization World Mental Health Surveys. JAMA 2004;291:2581‐90. [DOI] [PubMed] [Google Scholar]
  • 13. Esposito E, Wang JL, Adair CE et al. Frequency and adequacy of depression treatment in a Canadian population sample. Can J Psychiatry 2007;52:780‐9. [DOI] [PubMed] [Google Scholar]
  • 14. Harris MG, Hobbs MJ, Burgess PM et al. Frequency and quality of mental health treatment for affective and anxiety disorders among Australian adults. Med J Aust 2015;202:185‐9. [DOI] [PubMed] [Google Scholar]
  • 15. Lopes CS, Hellwig N, de Azevedo e Silva G et al. Inequities in access to depression treatment: results of the Brazilian National Health Survey – PNS. Int J Equity Health 2016;15:154. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Lu CY, Roughead E. New users of antidepressant medications: first episode duration and predictors of discontinuation. Eur J Clin Pharmacol 2012;68:65‐71. [DOI] [PubMed] [Google Scholar]
  • 17. Wang PS, Aguilar‐Gaxiola S, Alonso J et al. Use of mental health services for anxiety, mood, and substance disorders in 17 countries in the WHO World Mental Health Surveys. Lancet 2007;370:841‐50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Whiteford HA, Degenhardt L, Rehm J et al. Global burden of disease attributable to mental and substance use disorders: findings from the Global Burden of Disease Study 2010. Lancet 2013;382:1575‐86. [DOI] [PubMed] [Google Scholar]
  • 19. Donabedian A. Evaluating the quality of medical care. Milbank Q 2005;83:691‐729. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Donabedian A. The quality of care: how can it be assessed? JAMA 1988;260:1743‐8. [DOI] [PubMed] [Google Scholar]
  • 21. Kilbourne AM, Fullerton C, Dausey D et al. A framework for measuring quality and promoting accountability across silos: the case of mental disorders and co‐occurring conditions. Qual Saf Health Care 2010;19:113‐6. [DOI] [PubMed] [Google Scholar]
  • 22. US Institute of Medicine . Crossing the quality chasm: a new health system for the 21st century. Washington: National Academies Press, 2001. [PubMed] [Google Scholar]
  • 23. US Institute of Medicine . Improving the quality of health care for mental and substance‐use conditions. Washington: National Academies Press, 2006. [PubMed] [Google Scholar]
  • 24. Hayes JF, Marston L, Walters K et al. Mortality gap for people with bipolar disorder and schizophrenia: UK‐based cohort study 2000–2014. Br J Psychiatry 2017;211:175‐81. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25. Wang PS, Demler O, Kessler RC. Adequacy of treatment for serious mental illness in the United States. Am J Publ Health 2002;92:92‐8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26. Pincus HA, Scholle SH, Spaeth‐Rublee B et al. Quality measures for mental health and substance use: gaps, opportunities, and challenges. Health Aff 2016;35:1000‐8. [DOI] [PubMed] [Google Scholar]
  • 27. US National Committee for Quality Assurance . The state of health care quality, 2016. Washington: National Committee for Quality Assurance, 2016.
  • 28. Harris AHS, Gupta S, Bowe T et al. Predictive validity of two process‐of‐care quality measures for residential substance use disorder treatment. Addict Sci Clin Pract 2015;10:22. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29. Schmidt EM, Gupta S, Bowe T et al. Predictive validity of outpatient follow‐up after detoxification as a quality measure. J Addict Med 2017;11:205‐10. [DOI] [PubMed] [Google Scholar]
  • 30. Watkins KE, Paddock SM, Hudson TJ et al. Association between process measures and mortality in individuals with opioid use disorders. Drug Alcohol Depend 2017;177:307‐14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31. Watkins KE, Paddock SM, Hudson TJ et al. Association between quality measures and mortality in individuals with co‐occurring mental health and substance use disorders. J Subst Abuse Treat 2016;69:1‐8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32. Schmidt EM, Gupta S, Bowe T et al. Predictive validity of a quality measure for intensive substance use disorder treatment. Subst Abus 2017;38:317‐23. [DOI] [PubMed] [Google Scholar]
  • 33. Bremer RW, Scholle SH, Keyser D et al. Pay for performance in behavioral health. Psychiatr Serv 2008;59:1419‐29. [DOI] [PubMed] [Google Scholar]
  • 34. Dausey DJ, Pincus HA, Herrell JM. Performance measurement for co‐occurring mental health and substance use disorders. Subst Abuse Treat Prev Policy 2009;4:18. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35. Hepner KA, Watkins KE, Farmer CM et al. Quality of care measures for the management of unhealthy alcohol use. J Subst Abuse Treat 2017;76:11‐7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36. Kilbourne AM, Keyser D, Pincus HA. Challenges and opportunities in measuring the quality of mental health care. Can J Psychiatry 2010;55:549‐57. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37. Martsolf GR, Osilla KC, Mandel D et al. Assessing the quality and value of psychological health care in civilian health plans: lessons and implications for the Military Health System. Rand Health Q 2016;5:16. [PMC free article] [PubMed] [Google Scholar]
  • 38. Watkins K, Horvitz‐Lennon M, Caldarone LB, et al. Developing medical record‐based performance indicators to measure the quality of mental healthcare. J Health Qual 2011;33:49‐66. [DOI] [PubMed] [Google Scholar]
  • 39. Watkins KE, Keyser DJ, Smith B, et al. Transforming mental healthcare in the Veterans Health Administration: a model for measuring performance to improve access, quality, and outcomes. J Health Qual 2010;32:33‐42. [DOI] [PubMed] [Google Scholar]
  • 40. Wagner EH, Austin BT, Von Korff M. Organizing care for patients with chronic illness. Milbank Q 1996;74:511‐44. [PubMed] [Google Scholar]
  • 41. Bodenheimer T, Wagner EH, Grumbach K. Improving primary care for patients with chronic illness. JAMA 2002;288:1775‐9. [DOI] [PubMed] [Google Scholar]
  • 42. Woltmann E, Grogan‐Kaylor A, Perron B et al. Comparative effectiveness of collaborative chronic care models for mental health conditions across primary, specialty, and behavioral health care settings: systematic review and meta‐analysis. Am J Psychiatry 2012;169:790‐804. [DOI] [PubMed] [Google Scholar]
  • 43. Pincus HA, Jun M, Franx G et al. How can we link general medical and behavioral health care? International models for practice and policy. Psychiatr Serv 2015;66:775‐7. [DOI] [PubMed] [Google Scholar]
  • 44. Richards DA, Bower P, Pagel C et al. Delivering stepped care: an analysis of implementation in routine practice. Implement Sci 2012;7:3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45. Goorden M, Huijbregts KML, van Marwijk HWJ et al. Cost‐utility of collaborative care for major depressive disorder in primary care in the Netherlands. J Psychosom Res 2015;79:316‐23. [DOI] [PubMed] [Google Scholar]
  • 46. Morgan MAJ, Coates MJ, Dunbar JA et al. The TrueBlue model of collaborative care using practice nurses as case managers for depression alongside diabetes or heart disease: a randomised trial. BMJ Open 2013;3:e002171. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47. England NHS. Adult mental health: common mental health problems. Implementing the five year forward view for mental health. London: NHS England, 2016. [Google Scholar]
  • 48. Hatfield D, McCullough L, Frantz SH et al. Do we know when our clients get worse? An investigation of therapists' ability to detect negative client change. Clin Psychol Psychother 2010;17:25‐32. [DOI] [PubMed] [Google Scholar]
  • 49. Zimmerman M, McGlinchey JB. Why don't psychiatrists use scales to measure outcome when treating depressed patients? J Clin Psychiatry 2008;69:1916‐9. [DOI] [PubMed] [Google Scholar]
  • 50. Trivedi MH, Rush AJ, Wisniewski SR et al. Evaluation of outcomes with citalopram for depression using measurement‐based care in STAR*D: implications for clinical practice. Am J Psychiatry 2006;163:28‐40. [DOI] [PubMed] [Google Scholar]
  • 51. Oslin DW, Ross J, Sayers S et al. Screening, assessment, and management of depression in VA primary care clinics: the Behavioral Health Laboratory. J Gen Intern Med 2006;21:46‐50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52. Tew J, Klaus J, Oslin DW. The Behavioral Health Laboratory: building a stronger foundation for the patient‐centered medical home. Fam Syst Health 2010;28:130‐45. [DOI] [PubMed] [Google Scholar]
  • 53. US Department of Defense . Plan for development of procedure to measure data on mental health care provided by the Department of Defense. Washington: US Department of Defense, 2016. [Google Scholar]
  • 54. US Institute for Clinical Systems Improvement . The DIAMOND program: treatment for patients with depression in primary care. Bloomington: US Institute for Clinical Systems Improvement, 2016. [Google Scholar]
  • 55. US Centers for Medicare & Medicaid Services . Value‐based payment modifier. Baltimore: US Centers for Medicare & Medicaid Services, 2017. [Google Scholar]
  • 56. Zivin K, O'Malley A, Bigby J et al. Behavioral health integration in primary care: a review and implications for payment reform. Mathematica Policy Research, 2016. [Google Scholar]
  • 57. Seibert J, Fields S, Fullerton CA et al. Use of quality measures for Medicaid behavioral health services by state agencies: implications for health care reform. Psychiatr Serv 2015;66:585‐91. [DOI] [PubMed] [Google Scholar]
  • 58. US Centers for Medicare & Medicaid Services . Physician compare. Baltimore: US Centers for Medicare & Medicaid Services, 2017. [Google Scholar]
  • 59. US Centers for Medicare & Medicaid Services . Hospital compare. Baltimore: US Centers for Medicare & Medicaid Services, 2017. [Google Scholar]
  • 60. Patel MM, Brown JD, Croake S, et al. The current state of behavioral health quality measures: where are the gaps? Psychiatr Serv 2015;66:865‐71. [DOI] [PubMed] [Google Scholar]
  • 61. US Centers for Medicare & Medicaid Services . Inpatient Psychiatric Facilities Quality Reporting (IPFQR) Program. Baltimore: US Centers for Medicare & Medicaid Services, 2017. [Google Scholar]
  • 62. Institute of Medicine US. Psychosocial interventions for mental and substance use disorders: a framework for establishing evidence‐based standards. Washington: National Academies Press, 2015. [PubMed] [Google Scholar]
  • 63. Gaynes B, Brown C, Lux LJ et al. Relationship between use of quality measures and improved outcomes in serious mental illness. Rockville: US Agency for Healthcare Research and Quality, 2015. [PubMed] [Google Scholar]
  • 64. US Substance Abuse and Mental Health Services Administration . Recovery and recovery support. Rockville: US Substance Abuse and Mental Health Services Administration, 2015. [Google Scholar]
  • 65. Ranallo PA, Kilbourne AM, Whatley AS et al. Behavioral health information technology: from chaos to clarity. Health Aff 2016;35:1106‐13. [DOI] [PubMed] [Google Scholar]
  • 66. Druss BG, Dimitropoulos L. Advancing the adoption, integration and testing of technological advancements within existing care systems. Gen Hosp Psychiatry 2013;35:345‐8. [DOI] [PubMed] [Google Scholar]
  • 67. Riahi S, Fischler I, Stuckey MI et al. The value of electronic medical record implementation in mental health care: a case study. JMIR Med Inform 2017;5:e1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 68. Yeomans D. Clustering in mental health payment by results: a critical summary for the clinician. Adv Psychiatr Treat 2014;20:227‐34. [Google Scholar]
  • 69. Royal College of Psychiatrists UK. Royal College of Psychiatrists' statement on mental health Payment Systems (formerly Payment by Results). London: UK Royal College of Psychiatrists, 2014. [Google Scholar]
  • 70. Wang R, Shaw I, Middleton H. Delaying the implementation of Payment by Results in mental health: the application of standardisation. Ment Health Rev 2015;20:156‐65. [Google Scholar]
  • 71. McShane M, Mitchell E. Person centred coordinated care: where does the QOF point us? BMJ 2015;350:h2540. [DOI] [PubMed] [Google Scholar]
  • 72. Harding KJ, Rush AJ, Arbuckle M et al. Measurement‐based care in psychiatric practice: a policy framework for implementation. J Clin Psychiatry 2011;72:1136‐43. [DOI] [PubMed] [Google Scholar]
  • 73. Fortney JC, Unützer J, Wrenn G et al. A tipping point for measurement‐based care. Psychiatr Serv 2017;68:179‐88. [DOI] [PubMed] [Google Scholar]
  • 74. Insel TR. Translating scientific opportunity into public health impact: a strategic plan for research on mental illness. Arch Gen Psychiatry 2009;66:128‐33. [DOI] [PubMed] [Google Scholar]
  • 75. US Department of Veterans Affairs . VA is working to end homelessness among veterans. Washington: US Department of Veterans Affairs, 2017. [Google Scholar]
  • 76. World Health Organization . WHO‐AIMS mental health systems in selected low‐income and middle‐income countries: a WHO‐AIMS cross‐national analysis. Geneva: World Health Organization, 2009. [Google Scholar]
  • 77. International Initiative for Mental Health Leadership . IIMHL and IIDL Annual report. Lambton Quay: International Initiative for Mental Health Leadership, 2016. [Google Scholar]
  • 78. Dutch Association of Mental Health and Addiction Care . Performance indicators mental health care in the Netherlands. www.ggznederland.nl.
  • 79. Burgess P, Coombs T, Clarke A et al. Achievements in mental health outcome measurement in Australia: reflections on progress made by the Australian Mental Health Outcomes and Classification Network (AMHOCN). Int J Ment Health Syst 2012;6:4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 80. New Zealand Mental Health and Addictions KPI Programme . KPI Dashboard for financial years 2013/14 to 2016/17 YTD (Jul‐Dec). www.mhakpi.health.nz.
  • 81. UK National Health Service Benchmarking Network . Work programme report 2016/17. Manchester: UK National Health Service, 2017.
  • 82. Hussey PS, Ringel JS, Ahluwalia S, et al. Resources and capabilities of the Department of Veterans Affairs to provide timely and accessible care to veterans. Rand Health Q 2016;5:14. [PMC free article] [PubMed] [Google Scholar]
  • 83. Shields M, Rosenthal M. Quality of inpatient psychiatric care at VA, other government, nonprofit, and for‐profit hospitals: a comparison. Psychiatr Serv 2017;68:225‐30. [DOI] [PubMed] [Google Scholar]
  • 84. O'Hanlon C, Huang C, Sloss E et al. Comparing VA and non‐VA quality of care: a systematic review. J Gen Intern Med 2017;32:105‐21. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 85. Watkins KE, Smith B, Akincigil A et al. The quality of medication treatment for mental disorders in the Department of Veterans Affairs and in private‐sector plans. Psychiatr Serv 2016;67:391‐6. [DOI] [PubMed] [Google Scholar]
  • 86. Hepner KA, Paddock SM, Watkins KE et al. Veterans' perceptions of behavioral health care in the Veterans Health Administration: a national survey. Psychiatr Serv 2014;65:988‐96. [DOI] [PubMed] [Google Scholar]
  • 87. Department of Veterans Affairs US. Uniform Mental Health Services in VA Medical Centers and Clinics. Washington: US Department of Veterans Affairs, 2008. [Google Scholar]
  • 88. Sayer NA, Rosen CS, Bernardy NC et al. Context matters: team and organizational factors associated with reach of evidence‐based psychotherapies for PTSD in the Veterans Health Administration. Adm Policy Ment Health 2017;44:904‐18. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 89. Roland M, Guthrie B. Quality and outcomes framework: what have we learnt? BMJ 2016;354:i4060. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 90. Bao Y, McGuire TG, Chan YF et al. Value‐based payment in implementing evidence‐based care: the Mental Health Integration Program in Washington state. Am J Manag Care 2017;23:48‐53. [PMC free article] [PubMed] [Google Scholar]
  • 91. Unützer J, Chan Y‐F, Hafer E et al. Quality improvement with pay‐for‐performance incentives in integrated behavioral health care. Am J Publ Health 2012;102:e41‐5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 92. Pincus HA, Spaeth‐Rublee B, Sara G et al. A review of mental health recovery programs in selected industrialized countries. Int J Ment Health Syst 2016;10:73. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 93. Price M, Yuen EK, Goetter EM et al. mHealth: a mechanism to deliver more accessible, more effective mental health care. Clin Psychol Psychother 2014;21:427‐36. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 94. US Institute for Health Care Improvement . The Breakthrough Series: IHI's collaborative model for achieving breakthrough improvement. Cambridge: US Institute for Healthcare Improvement, 2003. [Google Scholar]
  • 95. Agency for Healthcare Research and Quality . Cherokee health systems. www.integrationacademy.ahrq.gov. [DOI] [PubMed]
  • 96. Ramanuj PP, Breslau J, Strathdee G et al. Carrots and sticks on opposite sides of the Atlantic: integration incentives for people with serious mental illness in England. Psychiatr Serv 2016;68:430‐2. [DOI] [PubMed] [Google Scholar]
  • 97. US Institute of Medicine . Quality measurement. Psychosocial interventions for mental and substance use disorders: a framework for establishing evidence‐based standards. Washington: National Academies Press, 2015. [PubMed] [Google Scholar]
  • 98. Byron SC, Gardner W, Kleinman LC et al. Developing measures for pediatric quality: methods and experiences of the CHIPRA Pediatric Quality Measures Program grantees. Acad Pediatr 2014;14(Suppl. 5):S27‐32. [DOI] [PubMed] [Google Scholar]
  • 99. US Centers for Medicare & Medicaid Services . CMS quality measure development plan: supporting the transition to the Merit‐based Incentive Payment System (MIPS) and Alternative Payment Models (APMs). Baltimore: Centers for Medicare & Medicaid Services, 2016. [Google Scholar]
  • 100. US National Quality Forum . Consensus development process. www.qualityforum.org.
  • 101. US Centers for Medicare & Medicaid Services . Technical expert panels. Baltimore: US Centers for Medicare & Medicaid Services, 2017. [Google Scholar]
  • 102. Hall CL, Moldavsky M, Taylor J et al. Implementation of routine outcome measurement in child and adolescent mental health services in the United Kingdom: a critical perspective. Eur Child Adolesc Psychiatry 2014;23:239‐42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 103. Slade M, McCrone P, Kuipers E et al. Use of standardised outcome measures in adult mental health services: randomised controlled trial. Br J Psychiatry 2006;189:330‐6. [DOI] [PubMed] [Google Scholar]
  • 104. Ashaye OA, Livingston G, Orrell MW. Does standardized needs assessment improve the outcome of psychiatric day hospital care for older people? A randomized controlled trial. Aging Ment Health 2003;7:195‐9. [DOI] [PubMed] [Google Scholar]
  • 105. Priebe S, McCabe R, Bullenkamp J et al. The impact of routine outcome measurement on treatment processes in community mental health care: approach and methods of the MECCA study. Epidemiol Psychiatr Soc 2002;11:198‐205. [DOI] [PubMed] [Google Scholar]
  • 106. NHS Scotland . Mental health project final report. Edinburgh: Scottish Government, 2008. [Google Scholar]
  • 107. Pincus HA, Hough L, Houtsinger JK et al. Emerging models of depression care: multi‐level (‘6 P’) strategies. Int J Methods Psychiatr Res 2003;12:54‐63. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 108. US Institute of Medicine . Best care at lower cost: the path to continuously learning health care in America. Washington: National Academies Press, 2013. [PubMed] [Google Scholar]
  • 109. Chambers DA, Feero WG, Khoury MJ. Convergence of implementation science, precision medicine, and the learning health care system: a new model for biomedical research. JAMA 2016;315:1941‐2. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from World Psychiatry are provided here courtesy of The World Psychiatric Association

RESOURCES