Skip to main content
NCBI home page
Translational Behavioral Medicine logoLink to Translational Behavioral Medicine
. 2019 Jul 26;9(6):1157–1162. doi: 10.1093/tbm/ibz121

From obligation to opportunity: future of patient-reported outcome measures at the Veterans Health Administration

Theresa M Coles 1,✉, Sarah M Wilson 2,3,4, Bo Kim 5,6, Jean C Beckham 3,4,7, Warren A Kinghorn 3,4,8
PMCID: PMC6875646  PMID: 31348511

When implemented systematically and longitudinally outside of the clinical encounter, patient-reported outcome measures may bolster the therapeutic alliance, enable new opportunities for clinician-patient communication, and improve patient satisfaction and clinical outcomes.

Keywords: Patient-reported outcome, Routine outcome measure, Veterans Health Administration, Mental health

Abstract

Patient-reported outcome (PRO) measures are particularly important in mental health services because patients are the central and essential source of information about their mental health status. PRO measures have the potential to engage patients in meaningful and focused conversations during clinical encounters, but unfortunately they often do not serve this purpose in mental health care. Administration of routine outcome measures has often been viewed by clinicians as an obligatory quality improvement process that takes time away from the clinical encounter. This commentary describes current practical barriers to using PRO measures in practice. Then, focusing specifically on the Veterans Health Administration, a unit of the U.S. Department of Veterans Affairs (VA), we propose processes within which PRO measures in mental health services could support the clinical encounter and enhance patient-centered mental health care. With the increasing number of Accountable Care Organizations and other integrated health-care systems that focus on mental health-care delivery, VA has an opportunity to leverage its long-standing electronic medical record technologies and integrated health system to serve as a model for incorporating PRO measures into mental health-care practices. This commentary provides a vision for the future of mental health delivery by incorporating PRO measures at the VA and in other health-care systems.

Implications.

Practice: Well-designed and carefully chosen patient-reported outcome measures and measure administration can enhance mental health care by strengthening the clinician–patient relationship, fostering a culture of systematic patient-centered decision-making and team-based care, and enabling large-scale evidence synthesis across patient populations.

Policy: Policymakers should consider patient-reported outcome (PRO) measures not simply as measures of quality of care but also tools to foster patient agency and to enable clinic encounters to focus on building therapeutic alliance.

Research: Future research should examine how PRO measures might be implemented and disseminated in a way that strengthens patient autonomy, collaborative decision-making, and clinician–patient rapport.

INTRODUCTION

The Veterans Health Administration, a unit of the U.S. Department of Veterans Affairs (VAs), serves over 9 million veteran patients [1]. The largest integrated health-care system in the USA, VA was an early leader of integrated care, using electronic medical records to enhance measurement-based approaches to care since the 1980s [2,3]. One such approach is the use of patient-reported outcome (PRO) measures, defined by the U.S. Food and Drug Association as, “... any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else” [4]. PRO measures are questionnaires that are often developed using qualitative and quantitative (psychometric) methods.

There is much evidence supporting the benefits of using PRO measures in clinical practice. PRO measures can facilitate patient engagement in meaningful and focused conversations during the clinical encounter [5]. Mental health clinicians agree that tracking patient progress is very important [3], and a number of researchers and clinicians have called for the integration of PRO measures in the clinical workflow to track progress and support mental health care [6,7]. Integration of PRO measures in clinical workflow has been shown to improve patient outcomes [8–11], to reduce time to treatment response [12], and to improve patient–clinician communication and patient satisfaction [13]. Recent highly cited literature reviews show positive or mixed evidence for the benefits of integrating PRO measures in mental health workflows [14,15]. Patient outcomes are more likely to improve when clinicians review patient responses to PRO measures [16].

PRO measures are administered either on paper or electronically, and studies have shown the feasibility and acceptability of remote symptom monitoring [17]. More recently, experts have noted the capacity for mobile health technologies (mHealth) to potentially enhance patient treatment engagement, especially for patients who face barriers to attending face-to-face appointments with providers [18–20]. However, much of the extant literature focuses on either stand-alone mHealth interventions or automated texting interventions, rather than specifically on strategies to use technology to incorporate PRO measures between provider encounters. Hollis et al. advised that potential for digital innovations has not yet been fully realized [21]. PRO measures are uniquely important within mental health services, which rely on the patient as the central and essential information source [15]. In an era of substantial attention to veteran mental health, it is therefore vital for VA to leverage its sophisticated EMR technologies to incorporate electronic patient-centered routine outcome measurement into its mental health-care processes, and to leverage its integrated system to support mental health services for the 1.6 million veterans per year receiving mental health services at the VA [22].

As clinicians, health services or PRO researchers, and/or leaders within the VA mental health-care system, we describe current utilization of mental health PRO measures in VA and review potential untapped benefits and possible methods of incorporating PRO measures more widely, systematically, and longitudinally to enhance patient care and outcomes.

HISTORICAL AND CURRENT USE OF PRO MEASURES IN VA

In 1997, a VA foundational primer described applications for using health measurement outcomes to improve health-care decision-making. The primer emphasized leveraging measurement to accomplish tasks such as the following: develop and set benchmarks for performance indicators, develop risk adjustment tools, conduct performance monitoring, and develop report card measures [23]. Patient-reported measurement was missing from this list.

Since the primer was published in the 1990s, the most widely implemented PRO measures at the VA are automated one-time screening tools programmed into the VA’s EMR. For example, screening alcohol use disorder was implemented across VA using computerized clinical reminders in the EMR. Using the Alcohol Use Disorders Identification Test-Consumption (AUDIT-C [24]), more than 90% of VA patients were screened for alcohol use disorder [25]. Metrics were used to tabulate the number of times the AUDIT-C was administered, rather than evaluate how the metric informed patient–clinician communication or patient-related outcomes. This example emphasizes how PRO measurement could be viewed as an obligation rather than an opportunity to support the mental health encounter.

More recently, researchers have been more focused on patient outcomes. Landes et al. developed an integrated informatics system to collect mental health PRO measures at the VA [3]. Results from this pilot study showed that clinicians were in favor of the new system both from an administrative and clinical perspective. In 2018, Wray et al. published a protocol to evaluate the effectiveness of a measurement-based mental health implementation via VA’s primary care clinics [26]. Although Landes and Wray’s publications represent progress, systems and tools enabling the consistent collection of mental health PRO measures are yet to be robustly implemented.

To support dissemination of PRO measure initiatives at the VA, it is important to describe the current mental health-care workflow to reveal barriers and limitations. Figure 1 shows a typical VA workflow in mental health care. We will consider a hypothetical patient for this example: a 58-year-old woman veteran who is receiving primary care at a rural VA community-based outpatient clinic. The patient is referred for a psychiatry visit from her local primary care provider (PCP). She is scheduled for an initial visit with a psychiatrist, and then she attends a return visit during which her new psychiatrist opens VA’s current system for accessing mental health PRO measures. This nationally available VA-specific system, called Mental Health Assistant [27], is not often used in clinical encounters. Specifically, preliminary evidence suggests that less than 4% of patients who had been seen for three or more mental health encounters had more than 1 documented PRO score in Mental Health Assistant, indicating that patients and clinicians were not using this system to document PRO measures to longitudinally monitor patients [3]. Another study of VA PCPs treating veterans diagnosed with depression, PTSD or anxiety disorder found that mental health screens and brief symptom measures were used in less than 10% of all encounters, and health measures were infrequently linked to treatment decisions (<10%) [28]. There has been an increase in the proportion of veterans who were administered at least one PRO measure as part of their mental health care as the VA’s measurement-based care initiative was announced [29]. Still, less than 16% of a sample of veterans from 58 sites were administered PRO measures in 2018, and psychiatrists were the least likely to use PRO measures in practice compared to other providers [29].

Fig. 1.

Fig. 1

Open in a new tab

Typical Veteran Affairs (VA) patient-reported outcome (PRO) measure workflow. Boxes indicate encounters with the health system and arrows indicate the flow of the encounters. From left to right, patients matriculate into VA’s mental health system via a referral. After the referral, patients are scheduled for a visit and then they visit their mental health provider. The cycle continues until the visits are deemed unnecessary by the patient and provider. The circle indicates administration of PRO measures.

Although Mental Health Assistant offers over 35 PRO measures for use by VA clinicians, the PRO measures are not easily identifiable (only acronyms are listed in the system without descriptions or purposes of the measures). The psychiatrist chooses and prints the PRO measures, then administers PRO measures to the patient on paper to assess depression, posttraumatic stress disorder (PTSD), and generalized anxiety symptoms. The psychiatrist must take valuable therapy time to administer the paper-based PRO measures, calculate total scores, and enter scores into the veteran’s medical record in progress notes. Because the emphasis is performance measurement and because time spent administering PRO measures is often perceived as an obstacle to building rapport, the psychiatrist has little incentive to prioritize or to review PRO scores during or after the clinic visit. If veterans miss appointments or participate via telehealth, PRO measures may not be assessed at all. Finally, the current VA EMR does not easily allow providers (or patients) to review historical PRO scores or to graphically trend PRO scores across time.

REIMAGINING PRO OPPORTUNITIES

Figure 2 illustrates opportunities for incorporating PRO measures in the clinical workflow leveraging modifications to the current VA EMR. The primary difference between Fig. 1 and Fig. 2 is that in Fig. 2, PRO measures are not administered during the hypothetical patient’s psychiatry visit (yellow Xs). Available technologies could support the administration of PRO measures via computer, smartphone, or tablet, or in waiting rooms on tablets (indicated by the yellow X immediately prior to the office visit box). The example veteran would complete valuable PRO measures on symptoms of depression, PTSD, and anxiety, before her time with her new psychiatrist, allowing her time with the psychiatrist to be exclusively focused on addressing specific areas of concern, building rapport, and determining optimal treatment.

Fig. 2.

Fig. 2

Open in a new tab

Opportunities for incorporating patient-reported outcome (PRO) measures in Veteran Affairs (VA) Mental Health Workflow. Boxes indicate encounters with the health system and arrows indicate the flow of the encounters. Yellow Xs illustrate opportunities for PRO measure administration, scoring, and provider PRO score review. Patients are referred into the system through a referral, which could be initiated by a conversation with another provider or using PRO scores administered by another provider. After the referral, patients are scheduled for a visit and then they visit their mental health provider in person or via telehealth communications. Opportunities to incorporate PRO measures (yellow Xs) are available between visits and during the clinical encounter. The cycle continues until the visits are deemed unnecessary by the patient and provider.

The workflow presented in Fig. 2 offers an option for providers to longitudinally monitor patients’ progress inside and outside of clinic or telehealth visits. Monitoring of PRO scores over time would identify changes in symptoms or functioning that require clinical attention. In the example of the 58-year-old woman veteran seeking psychiatry care, the psychiatrist could “prescribe” in vivo PRO monitoring (e.g., quality-of-life measures, symptom measures) between visits to better understand the veteran’s functioning over time. PRO measures would be administered at a frequency that makes sense for the patient and condition, and scores could be tabulated electronically before clinician review, reducing clinician burden. Ideally, the PRO scores could be viewed via easy-to-understand figures [30] that convey the severity, important changes over time, and norms.

Reimagining a system like the one illustrated in Fig. 2 allows us to consider how well-designed and carefully chosen PRO measure administration provides opportunities to enhance care. We envision five valuable ways in which a process like the one illustrated in Fig. 2 could enhance care:

  1. Integrating PRO measures in mental health care provides additional points of connection outside of clinic time, enhancing the clinician–patient relationship. By monitoring patients’ outcomes outside the visit, providers can use PRO scores to inform important content in the therapy session or support patient care as necessary outside of clinic visits [21,31]. Electronic PRO measures facilitate engagement among patients and providers for patients who are seen less often to still be engaged by being informed of drivers of the direction in which their health care should evolve. Consistent with Fig. 2, clinicians spend less time administering, scoring, and entering PRO scores, allowing more time during the clinical encounter to evaluate the meaning of symptoms rather than simply ascertaining the presence of symptoms.

  2. PRO measures have been developed to measure a wide range of outcomes and are not limited to symptom measures [32]. For example, knowledge of veteran’s social functioning via PRO scores could provide systematic whole-person insight into the ability of a patient to participate in social roles and activities, companionship, emotional support, and informational support, all of which provide context and information to the veteran’s mental health status.

  3. Electronically collected PRO scores collected within integrated EHR systems designed to share results across clinicians may reduce burden [5] on patients and clinicians because PRO scores collected by one provider can be seen by all providers (no readministration of the same PRO measure). Burden may be further reduced with prompts to complete or review PRO measures, and the number and timing of those prompts should be carefully assessed. With electronically collected PRO measures, clinicians do not have to spend time scoring patient responses because scoring occurs within the system [33]. The record of electronically captured PRO scores over time provides clinicians with a history that can be shared with other providers (including providers in other VA facilities when patients relocate), or be used as a “baseline” for comparison in the future. It should be noted that the most effective PRO measure integrations include frequent PRO measure administrations including administrations timed close to the clinical encounter [15].

  4. Administration of PRO measures and the routine discussion of scores instills a culture of systematic patient-centered decision-making and team-based care in patients, providers, and the system. Routine discussion and review of scores also communicates patients’ status and progress back to the patient and clinician, revealing perceived or unperceived progress. For example, a mental health clinician providing treatment for PTSD would be able to view the intensity of their patient’s PTSD symptoms together with the patient over time to assess the efficacy of a treatment strategy. Electronically captured scores would be accessible by other providers at the VA for continuity of care or by patients themselves to foster engagement, transparency, and patient autonomy.

  5. Integrating PRO score collection at the individual level also contributes to large-scale evidence synthesis informing research and VA’s integrated health system. Using aggregated PRO scores, the VA could evaluate treatment effectiveness, implementation of a new intervention, or reduce variation in patient care [34]. Clinicians and researchers can compare patient outcomes across patients, or patient populations (e.g., by treatment, time since diagnosis, demographic characteristics), possibly revealing patterns or disparities that should be addressed. For example, data from a particular clinic could demonstrate that veterans in one group for depression treatment are less likely to experience a decrease in depression symptoms than those in other groups. This could then more easily facilitate audit and feedback of care, thus helping the clinic explore the reasons for this disparity—individual provider differences, group psychotherapy protocol, etc.

CONSIDERATIONS AND POTENTIAL UNINTENDED CONSEQUENCES

Two along with PRO measures, VA’s Mental Health Assistant offers important patient- or caregiver-reported questionnaires focusing on behavioral (e.g., alcohol use), personality, or burden (e.g., stigma, caregiver burden) that provide context for patient-centered mental health care. Together, these questionnaires and PRO measures support patient–clinician communication and enhance the clinician’s ability to effectively treat the patient.

As with any system-level intervention, changes must be pursued carefully to limit unintended consequences and misuse of PRO scores. First, patients may be concerned that PRO scores will be used for disability benefit decisions [29], particularly for longitudinally administered measures. The VA should carefully develop policies to protect patients from losing important benefits due to PRO scores only. Although PRO scores provide important support to providers and patients, PRO scores alone cannot ultimately determine patients’ needs: this must be discerned within the clinician–patient relationship.

Second, when clinicians have real-time longitudinal access to patients’ PRO scores, it will be important to determine when it is their responsibility to intervene [35]. Protocols should set standards and expectations for clinician response because of potential legal and moral implications. These standards and expectations should be shared with patients so that they are aware of the strengths and limitations of real-time longitudinal PRO score access. For example, responsibility of clinicians to respond to indications of risk for suicide or harm to others given real-time longitudinal access to patients’ PRO scores should be clearly addressed in the protocols and discussions with patients.

Third, not all veterans have access to and fluency with technology; PRO measures must be implemented in a variety of formats [35].

Fourth, recognizing that electronic real-time PRO measurement will influence how veterans interpret their daily lived experience, it is important that implemented PRO measures focus on meaningful functions and outcomes and not contribute to medicalization by focusing inordinately on symptoms. Symptom-focused PRO measures may actually cause unintended harm by focusing on veterans’ attention on the presence or absence of symptoms rather than on the attainment of meaningful life goals. PRO measures can become occasions for symptom amplification or, at worst, for the malingered reporting of symptoms in the context of secondary gain.

Finally, workflow changes may not need to be made across all VA centers. Use of Mental Health Assistant and PRO measures may vary by location [29], thus metrics for Mental Health Assistant and PRO measure usage should be collected and published so that administrators know where to focus intervention efforts.

An important consideration for the integration of any PRO measure in clinical practice to support care is the development or use of a conceptual model. Conceptual models provide the theory and context for how and why integrating a PRO measure supports care, and how success is measured. A conceptual model will describe the relationship between the workflow change, intermediate outcomes (such as confidence or behavior change), and ultimate outcomes of care (e.g., health status).

GROWING EMPHASIS ON MEASUREMENT-BASED MENTAL HEALTH CARE

The VA is making strides toward integrating PRO measures in mental health care. In 2016, VA launched a large-scale multiphase initiative to implement measurement-based care as the standard of care across its mental health clinics [36]. The VA also recently published a literature synthesis on the use of PRO measures in mental health decision-making [37].

This review provides VA administration with important evidentiary information that may influence future policies for routine outcome assessment within the VA.

There is opportunity and tremendous promise to fully integrate PRO measures in VA’s mental health services workflow. PRO measures are of critical importance in mental health because patients are the primary source of information about their mental health, thus it is encouraging that VA has recently begun to increase its focus on patient-reported and patient-centered measures [3,26]. As other health systems plan to integrate PRO measures into routine care, the forthcoming VA model could serve as an open door for VA and other systems to learn from each other.

Acknowledgments

This article emerged from conversations at the conference, “Prescribing Wisely: Psychiatric Medications and the Whole Person,” held at Duke University in June 2018 with the support of the McDonald Agape Foundation. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs, the United States government, or any of the institutions with which the authors are affiliated. The authors would like to acknowledge Ashley Wilson for enumerating mental health questionnaires available at the VA, and Drs. Dave Oslin and Laura Wray for providing their expert insight into VA’s current usage of PRO measures.

Funding: The preparation of this work was supported in part by a VA Career Development Award from VA Health Services Research and Development (IK2HX002398), a VA Research Scientist Award from the VA Office of Research and Development (ORD) Clinical Sciences Research and Development Service (IK6BX003777), and the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis; through an award from the National Institute of Mental Health (5R25MH08091607), National Institute on Drug Abuse, and VA HSR&D QUERI.

Compliance with Ethical Standards

Conflicts of interest: T.C. declares that she has no conflicts of interest. S.M.W., B.K., J.B. and W.K. are employees of the VA but otherwise declare that they have no conflicts of interest.

Human rights: This article does not contain any studies with human participants performed by any of the authors.

Informed Consent: This article does not involve human participants and informed consent was therefore not required.

Welfare of Animals: This article does not contain any studies with animals performed by any of the authors.

References

  • 1. US Department of Veterans Affairs. VA Health Administration Available at https://www.va.gov/health/. Accessibility verified October 19, 2018.
  • 2. Fihn SD, Francis J, Clancy C, et al. Insights from advanced analytics at the Veterans Health Administration. Health Aff (Millwood). 2014;33(7):1203–1211. [DOI] [PubMed] [Google Scholar]
  • 3. Landes SJ, Carlson EB, Ruzek JI, et al. (2015). Provider-driven development of a measurement feedback system to enhance measurement-based care in VA mental health. Cogn Behav Pract. 22(1):87–100. [Google Scholar]
  • 4. U. S. Department of Health & Human Services, F. D. A. (2016). Guidance for Industry & Food & Drug Administration Staff Regarding Factors to Consider When Making Benefit-Risk Determinations in Medical Device Premarket Approval and De Novo Classifications Available at https://go.usa.gov/xUPw6. Accessibility verified August 1, 2018.
  • 5. Philpot LM, Barnes SA, Brown RM, et al. Barriers and benefits to the use of patient-reported outcome measures in routine clinical care: a qualitative study. Am J Med Qual. 2018;33(4):359–364. [DOI] [PubMed] [Google Scholar]
  • 6. Bickman L. A measurement feedback system (MFS) is necessary to improve mental health outcomes. J Am Acad Child Adolesc Psychiatry. 2008;47(10):1114–1119. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Wolpert M. Uses and abuses of patient reported outcome measures (PROMs): potential iatrogenic impact of PROMs implementation and how it can be mitigated. Adm Policy Ment Health. 2014;41(2):141–145. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. Basch E, Deal AM, Dueck AC, et al. (2017). Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA. 318(2):197–198. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Guo T, Xiang YT, Xiao L, et al. Measurement-based care versus standard care for major depression: a randomized controlled trial with blind raters. Am J Psychiatry. 2015;172(10):1004–1013. [DOI] [PubMed] [Google Scholar]
  • 10. Hansson H, Rundberg J, Österling A, Öjehagen A, Berglund M. Intervention with feedback using Outcome Questionnaire 45 (OQ-45) in a Swedish psychiatric outpatient population. A randomized controlled trial. Nord J Psychiatry. 2013;67(4):274–281. [DOI] [PubMed] [Google Scholar]
  • 11. Dailey YM, Humphris GM, Lennon MA. Reducing patients’ state anxiety in general dental practice: a randomized controlled trial. J Dent Res. 2002;81(5):319–322. [DOI] [PubMed] [Google Scholar]
  • 12. Unützer J, Chan YF, Hafer E, et al. Quality improvement with pay-for-performance incentives in integrated behavioral health care. Am J Public Health. 2012;102(6):e41–e45. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Chen J, Ou L, Hollis SJ. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res. 2013;13:211. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Kendrick T, El-Gohary M, Stuart B, et al. Routine use of patient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. Cochrane Database Syst Rev. 2016;(7):CD011119. doi:10.1002/14651858.CD011119.pub2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Fortney JC, Unützer J, Wrenn G, et al. A tipping point for measurement-based care. Psychiatr Serv. 2017;68(2):179–188. [DOI] [PubMed] [Google Scholar]
  • 16. Krägeloh CU, Czuba KJ, Billington DR, Kersten P, Siegert RJ. Using feedback from patient-reported outcome measures in mental health services: a scoping study and typology. Psychiatr Serv. 2015;66(3):224–241. [DOI] [PubMed] [Google Scholar]
  • 17. Naslund JA, Marsch LA, McHugo GJ, Bartels SJ. Emerging mHealth and eHealth interventions for serious mental illness: a review of the literature. J Ment Health. 2015;24(5):321–332. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Martin T. Assessing mHealth: opportunities and barriers to patient engagement. J Health Care Poor Underserved. 2012;23(3):935–941. [DOI] [PubMed] [Google Scholar]
  • 19. Germack HD. Using mHealth as a lever to promote mental health care quality through patient engagement. J Am Psychiatr Nurses Assoc. 2016;22(4):320–322. [DOI] [PubMed] [Google Scholar]
  • 20. Sahin C. (2018). Rules of engagement in mobile health: what does mobile health bring to research and theory? Contemporary Nurse. 54(4–5):374–387. [DOI] [PubMed] [Google Scholar]
  • 21. Hollis C, Morriss R, Martin J, et al. Technological innovations in mental healthcare: harnessing the digital revolution. Br J Psychiatry. 2015;206(4):263–265. [DOI] [PubMed] [Google Scholar]
  • 22. Department of Veterans Affairs, O. O. P. A., Department of Veterans Affairs. (2016). Fact Sheet: VA Mental Health Care Available at https://www.va.gov/opa/publications/factsheets/April-2016-Mental-Health-Fact-Sheet.pdf. Accessibility verified October 30, 2018.
  • 23. Zimmerman DL, Daley J (1997). Using Outcomes to Improve Health Care Decision Making Available at https://www.hsrd.research.va.gov/publications/internal/outcomes.pdf. Accessibility verified October 19, 2018.
  • 24. Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA. The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol Use Disorders Identification Test. Arch Intern Med. 1998;158(16):1789–1795. [DOI] [PubMed] [Google Scholar]
  • 25. Bradley KA, Williams EC, Achtmeyer CE, Volpp B, Collins BJ, Kivlahan DR. Implementation of evidence-based alcohol screening in the Veterans Health Administration. Am J Manag Care. 2006;12(10):597–606. [PubMed] [Google Scholar]
  • 26. Wray LO, Ritchie MJ, Oslin DW, Beehler GP. Enhancing implementation of measurement-based mental health care in primary care: a mixed-methods randomized effectiveness evaluation of implementation facilitation. BMC Health Serv Res. 2018;18(1):753. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27. Department of Veterans Affairs Office of Information and Technology (OI&T). (2019). Product Development. Mental Health Assistant User Manual Version 3. Available at https://www.va.gov/vdl/documents/Clinical/Mental_Health/YS_MHA_UM.pdf. Accessibility verified May 15, 2019. [Google Scholar]
  • 28. King PR, Beehler GP, Vair CL, Gass J, Funderburk JS, Lilienthal KR, Novi JH (2017). Identifying measurement-based care practices of VHA co-located collaborative care providers. Prof Psychol: Res Pract. 48(4):236–242. [Google Scholar]
  • 29. Resnick SG, Hoff RA (2019). Observations from the national implementation of measurement based care in mental health in the department of veterans affairs. Psychol Serv. doi: 10.1037/ser0000351 [DOI] [PubMed] [Google Scholar]
  • 30. Snyder CF, Smith KC, Bantug ET, Tolbert EE, Blackford AL, Brundage MD; PRO Data Presentation Stakeholder Advisory Board What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability. Cancer. 2017;123(10):1848–1859. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31. Locklear T, DeBar LL, Willig J, et al. Case studies from the clinic: initiating and implementing patient-reported outcome measures. EGEMS (Wash DC). 2017;5(1):7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32. HealthMeasures: Transforming how health is measured. (2018). PROMIS List of Adult Measures Available at http://www.healthmeasures.net/explore-measurement-systems/promis/intro-to-promis/list-of-adult-measures. Accessibility verified October 19, 2018.
  • 33. Snyder CF, Aaronson NK, Choucair AK, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 2012;21(8):1305–1314. [DOI] [PubMed] [Google Scholar]
  • 34. Baumhauer JF. Patient-reported outcomes—are they living up to their potential? N Engl J Med. 2017;377(1):6–9. [DOI] [PubMed] [Google Scholar]
  • 35. Snyder C, Wu A.. Guide to Integrating Patient-Reported Outcomes in Electronic Health Records. Baltimore, MD: Johns Hopkins University; 2017. [Google Scholar]
  • 36. Dollar K, Kirchner J, McGee-Vincent P, DePhilippis D, Resnick D (2018). Implementation of measurement-based care: Creation of an implementation planning guide. Paper presented at the 11th Annual Conference on the Science of Dissemination and Implementation in Health, Washington, DC. https://academyhealth.confex.com/academyhealth/2018di/meetingapp.cgi/Paper/29231. Accessibility verified May 15, 2019. [Google Scholar]
  • 37. Peterson K, Bourne AJ, Bourne D (2018). Evidence Brief: Use of Patient Reported Outcome Measures for Measurement Based Care in Mental Health Shared Decision-Making Available at https://www.hsrd.research.va.gov/publications/esp/Measurement-based-Care.cfm. Accessibility verified April 22, 2019.

Articles from Translational Behavioral Medicine are provided here courtesy of Oxford University Press

RESOURCES