Collecting, Integrating, and Disseminating Patient-Reported Outcomes for Research in a Learning Healthcare System

Abstract

Introduction:

Advances in health policy, research, and information technology have converged to increase the electronic collection and use of patient-reported outcomes (PROs). Therefore, it is important to share lessons learned in implementing PROs in research information systems.

Case Description:

The purpose of this case study is to describe a novel information system for electronic PROs and lessons learned in implementing that system to support research in an academic health center. The system incorporates freely available and commercial software and involves clinical and research workflows that support the collection, transformation, and research use of PRO data. The software and processes that comprise the system serve three main functions, (i) collecting electronic PROs in clinical care, (ii) integrating PRO data with non-patient generated clinical data, and (iii) disseminating data to researchers through the institution’s research informatics infrastructure, including the i2b2 (Informatics for Integrating Biology and the Bedside) system.

Strategies:

Our successful design and implementation was driven by three overarching strategies. First, we selected and implemented multiple interfaced technologies to support PRO collection, management, and research use. Second, we aimed to use standardized approaches to measuring PROs, sending PROs between systems, and disseminating PROs. Finally, we focused on using technologies and processes that aligned with existing clinical research information management strategies within our organization.

Conclusion:

These experiences and lessons may help future implementers and researchers enhance the scale and sustainable use of systems for research use of PROs.

Introduction

Advances in health policy, research, and information technology have converged to increase the electronic collection and use of patient-reported outcomes (PROs) to support clinical care and research. Through the Affordable Care Act and other legislation, the United States government has invested in research and care delivery that focus on evidence-based treatments and reflect patients’ preferences and values.1,2 To support these initiatives, and concomitant with increasing electronic health record (EHR) adoption,3–5 researchers and clinicians are demanding information technology and data infrastructures that support routine collection and use of PROs.6–8 PROs allow patients to directly report on their health status and health care, such as quality of life, function, symptoms, or care experiences.9 Furthermore, systematic PRO collection and use in research supports the learning health care system goal of increasing the efficiency and scope of scientific discovery related to health care delivery.10

One approach to meeting the clinical and research demand for PROs is to incorporate electronic data collection into everyday clinical processes.11–13 This approach is appealing because it utilizes existing infrastructure and interactions with patients, which may allow data to be captured more efficiently and to be acquired for larger, more representative populations. Also, by collecting data during care delivery, PROs may be more easily integrated with complementary data, such as demographics, diagnoses, laboratory data, and medications. Indeed, systems for collecting and sharing PROs with clinicians have been shown to be usable,14–16 to improve patient-clinician communication,17–19 and to be supportive of both clinical care and research,20,21 such as the feasibility of linking PROs with a cancer registry.22 Furthermore, sharing PRO data with clinicians at the point of care may improve patients’ health-related quality of life.23 However, broader analyses of the literature fail to reveal a consistent positive impact of PRO collection and use on care processes, clinical decision-making, and health outcomes.24–28 This lack of consistent impact may relate to inconsistencies in how electronic PRO systems are implemented, including how burden on clinical workflows is managed and how PROs are communicated to clinicians.29–32 Therefore, to more effectively meet the demand for PRO data, there is a need to describe lessons learned in implementing PRO systems.

Describing lessons learned with PRO systems may help future implementers and researchers enhance the scale and sustainable use of such systems. The purpose of this case study is to describe a novel information system for electronic PROs and our lessons learned in implementing the system, which supports clinical care and research, in an academic health center. Uniquely, our lessons learned stem from a system that starts with point-of-care electronic PRO collection and ends with institutionwide access to PROs linked to other clinical data for research. We describe how PROs are collected in primary care practices, integrated with other clinical data in an EHR, and then loaded into an institutional data warehouse. The system utilizes interfaces between freely available and commercial software and involves clinical and research workflows that support the collection, transformation, and research use of PRO data. Researchers in the institution can use web-based software33 to query for patients of interest based on PROs and other clinical data before requesting detailed data for in-depth analysis.

Our evaluation of the system focused on assessing the feasibility of collecting, integrating, and then reusing PRO data for research. Therefore, we used qualitative data gleaned from informal and formal interactions with stakeholders in varied roles, including practice staff, clinicians, patients, system designers, researchers, and administrative leadership. These data were collected during the system’s design, implementation, and the first six months after implementation. To supplement these qualitative data, we also collected quantitative data that describes the frequency with which PROs were collected in clinical settings and reused by researchers.

Context

The system was implemented in collaboration with the University of Florida health system. The technology and lessons learned described here are based on our initial implementation of PRO data being collected in two family medicine practices affiliated with the university’s Family Medicine department. The system was implemented to achieve two goals. The first goal was to improve care for chronic pain by incorporating patient-reported data into clinical processes. This goal was set by pain medicine physicians and family medicine physicians who were interested in improving chronic pain care. The second goal was to create a process through which PROs collected during routine clinical care could also be reused by many researchers across the institution. This second goal was set by researchers and administrators in the institution who wanted to build infrastructure that supports the regular reuse of PROs by many researchers. Therefore, the family medicine practices began collecting PRO data related to pain, physical function, sleep, and mental health. At the same time, the first and third authors of this case study led a related pragmatic clinical trial to evaluate the effect of integrating PROs in an EHR on clinician and patient satisfaction with care for chronic pain.34 The system’s design and implementation were led by a team that included two physician researchers, an information science researcher, system developers, and the health system’s chief data officer. The team received support from administrative leadership, including the Community Health and Family Medicine department chairperson, the College of Medicine’s assistant dean for clinical informatics, and the academic health center information technology organization. While this article primarily describes the electronic data collection and workflows for research use of PROs, details of clinician and patient use of the PRO system are described elsewhere.30

Case Description

The software and processes that comprise the system serve three main functions: (1) collecting electronic PROs in clinical care, (2) integrating PRO data with other clinical data not reported by the patient, and (3) disseminating data to researchers (Figures 1 and 2). To accomplish these tasks, the system encompasses multiple software tools and interfaces between them.

Figure 1.

Process of Collecting Electronic PROs, Integrating PRO Data with Other Clinical Data, and Disseminating Research Data to Researchers

Figure 2.

Screen Shot of Data Elements Available for Querying in i2b2

Electronic PRO Data Collection in Clinical Care

To collect PROs at the point of care, we adopted an existing web-based system that administers PRO assessments called the “Collaborative Health Outcomes Information Registry” (CHOIR). CHOIR was first developed and implemented at Stanford University to support specialty pain care and clinical research.35,36 Our university received a no-cost license to use and modify the software within our health system. Under a similar license, CHOIR is also in various stages of design and implementation at other academic health centers.35 CHOIR allows patients to complete PRO assessments via web-enabled devices. CHOIR also contains administrative functions that allow clinicians or practice staff to register patients, assign surveys, and review PRO results at a point in time or longitudinally. Through a computer-adaptive testing engine, CHOIR administers PRO measures, including Patient-Reported Outcome Measurement Information System (PROMIS) measures.6,37 Computer-adaptive and static instruments can be combined to create different patient assessments. New instruments can also be added to those already available in CHOIR.

In the current system, patients complete an assessment that includes nine computer-adaptive PROMIS measures (pain interference, pain behavior, fatigue, physical function, sleep disturbance, sleep-related impairment, anger, depression, and anxiety)38,39 as well as static measures of pain location, pain intensity (0–10 scale), pain catastrophizing,40 and risk of opioid-related aberrant behavior.41 While these measures have specific relevance to patients with chronic pain, the domains are also generally relevant to primary care patients and clinical management of other chronic diseases. Patients complete assessments using a tablet computer prior to seeing their provider, either in a waiting room or exam room. Because the data are initially collected for regular clinical care, patients do not consent to research reuse when reporting their outcomes. However, as we describe later in the case, researchers who wish to reuse the PROs must obtain Institutional Review Board (IRB) approval.

Once an assessment is complete, CHOIR computes standardized numeric scores that summarize each PRO result. For the PROMIS measures, the standardized scores include a t-score (mean 50, standard deviation 10) and a percentage score (0–100 percent), which are based on calibration samples that are representative of the United States population.42 Also, CHOIR produces a comprehensive PRO result set in portable document format (PDF) that can be viewed, saved, or printed by clinicians. This PDF includes patients’ responses to individual items and summarized scores. All data are stored as discrete entities in an Oracle database within the health system’s secure clinical data environment.

Integrating PRO Data with Other Clinical Data

To integrate PRO results with other clinical data in support of clinical care the CHOIR system sends the standardized PRO scores to our health system’s Epic EHR. CHOIR creates HL7 Version 2.1 messages and sends discrete PRO results through Epic’s laboratory and results interface. Each message contains unique patient and encounter identifiers that link the PRO results to patients’ other health record information. EHR developers built discrete results structures to store each PRO result. Currently, results are uniquely identified in the EHR using local identification numbers.

We are in the process of mapping the PRO measures to Logical Observation Identifiers Names and Codes (LOINC). LOINC is a universal system for coding laboratory results and clinical observations, including patient assessments.43 Also, within United States government initiatives to increase EHR use and health data exchange, LOINC is the preferred vocabulary standard for results.44 Therefore, by mapping the PRO results to LOINC, we expect to increase the ability to share results both internally and with external organizations over time. Finally, in addition to the discrete numeric results, CHOIR sends the comprehensive results set PDF document in a separate HL7 Version 2.6 message. This PDF document contains the same PRO results scores that are sent as discrete numeric results through the EHR’s laboratory and results interface. However, the PDF also contains patients’ responses to the individual PRO assessment items in case clinicians are interested in obtaining the full details of their patients’ responses.

Both the discrete PRO scores and the PDF, which includes individual item responses, are available to clinicians in the EHR. Using Synopsis, which provides graphical trending functionality, the PRO scores can be plotted against other clinical data, such as medications that have been prescribed. The PRO scores can also be accessed through Results Review, which is the function through which laboratory results are typically viewed. The PRO scores can also be accessed using shortcut phrases, which allow them to be pasted directly into clinicians’ notes. Finally, the PDF, which contains PRO scores and individual item responses, is available to clinicians through the EHR’s document storage functionality called “Media Tab.”

Disseminating Data to Researchers

The health system’s research enterprise makes PRO data, which is integrated with other clinical data as described above, available to researchers through two mechanisms. First, university-affiliated researchers can obtain count-based information through the institution’s i2b2 (Informatics for Integrating Biology and the Bedside) system by simply requesting access to the i2b2 web client.33,45 i2b2 provides count-based information such as the number of unique patients seen within the health system who meet certain clinical criteria and their breakdown by age, race, and sex. For example, a researcher could query for the number and age ranges of women with diabetes who have documented PROMIS depression and anxiety scores in the 75^th percentile or above. Such information may be useful for determining feasibility of recruitment for prospective studies or availability of existing data for retrospective studies.

Second, researchers can obtain detailed data through the institution’s integrated data repository (IDR).45 The IDR encompasses an enterprise clinical data warehouse, supporting personnel, and information management and governance processes. The IDR provides a single source of administrative and clinical information that supports research, clinical care, and operations. The IDR is funded by the health system and the university’s Clinical and Translational Science Institute (CTSI). The IDR is continuously growing in terms of the amount and types of clinical data that it contains. With IRB approval, university-affiliated researchers can obtain detailed PRO and other clinical data from the IDR using a standardized data request process. Often, these researchers’ plan only to analyze the secondary clinical data. Thus, their studies may be deemed nonhuman by the IRB (if using de-identified data) or may be approved with a waiver of patient informed consent. Furthermore, the university, through the CTSI, proactively promotes and educates researchers on i2b2 and IDR data access. This occurs through regular tutorials to students and faculty groups, an “IDR studio” in which researchers bring specific research questions for review and consultation, and monthly email blasts that describe new data elements when they become available for research use.

Two export, transform, and load (ETL) processes move data from the EHR to the IDR and i2b2, as described below. First, on a daily basis, select clinical data elements that were chosen to support operational and research needs are copied from the EHR’s database to the IDR’s clinical data warehouse.

Both databases use SQL Server 2012 databases and reside within the health system’s secure clinical data environment. The ETL is managed by scripts written in SAP Data Services software. Second, on a monthly basis, the i2b2 database is refreshed with data from the clinical data warehouse. The i2b2 database, which is also implemented in SQL Server 2012, contains a Health Insurance Portability and Accountability Act (HIPAA) limited data set of patient information and resides outside of the clinical enterprise’s privacy wall so that researchers can query it using i2b2. A separate mapping table that contains unique patient identifiers is maintained inside the enterprise’s privacy wall and supports fulfillment of research data requests that are based on the results of previously executed i2b2 queries.

Findings

Each of the three system functions—PRO data collection, integration with other clinical data, and dissemination to researchers—is currently operating in the institution. For each, we qualitatively summarize barriers and facilitators to clinic operation, which we identified through regular check-ins with practice staff; our own workflow observations; and discussions with patients, clinicians, system developers, and administrators. When possible, we also quantitatively describe the extent to which each function was used in the first six months following implementation (Table 1).

Table 1.

Summary of Outcomes Assessed During System Design, Implementation, and Six Months Postimplementation

SYSTEM FUNCTION	OUTCOMES
Collecting electronic PROs in clinical care	Two practices adopted and using system. Automated identification of patients who qualified to complete assessments reduced concerns about workflow disruption. Patients typically completed questionnaires in less than 10 minutes. Eighty-four percent of patients said they would be comfortable reporting outcomes using a tablet computer. Patients completed 309 total PRO assessments.
Integrating PRO data with other clinical data not reported by the patient	PRO results consistently loaded into the EHR after patients reported them. PROs results are a standard part of regular data loads from clinical information systems to the research information systems (i.e., IDR).
Disseminating data to researchers	Two research studies actively acquiring PRO data via i2b2 and data request process. Average of 9.5 i2b2 queries per month involving PRO data. Average of 519 i2b2 queries per month involving any clinical data.

In terms of PRO data collection, clinicians in two family-medicine practice locations have been using the CHOIR system to collect electronic PROs from select patients on a daily basis. Also, two other departments are currently reviewing the system for adoption in their clinics. In the two family-medicine practice locations, initial barriers to data collection included clinician uncertainty about the clinical benefit of the data and concerns about slowing down workflow. To help overcome the barriers, we developed a process that leveraged EHR data to pre-identify patients with chronic pain who were then targeted to complete the PRO assessments. In collaboration with practice staff and providers, we also found existing patient wait times within workflows during which patients could reliably complete assessments before seeing their provider. In our testing, patients typically completed the questionnaires in less than 10 minutes. Also, patients typically expressed willingness to use the tablet computers to report outcomes. While we did not precisely measure the number of patients who refused the PRO assessment in practices, in our related study on the same types of patients with chronic pain,34 84 percent of patients responded “yes” when asked if they would “feel comfortable and able to use a tablet computer to answer questions about your health at your doctor’s office.” Finally, after implementation, clinicians and staff did not express concerns that the data collection slowed down patient throughput. Therefore, the practice processes tolerated regular data collection without significant changes to existing workflows. We further discuss elsewhere barriers to PRO data collection in clinics and how we overcame them.30

While a majority of patients with chronic pain who were approached were willing and able to complete the PRO assessment prior to seeing a provider, patients with known chronic pain conditions represented a minority of all patients in the practices. Still, in the first six months, a total of 309 complete PRO assessments, which include 13 distinct PRO measures, were recorded by 203 unique patients.

These PRO data were integrated with clinical data in the EHR and became part of the regular data loads in the clinical data warehouse and i2b2 (Figure 2). This represents a moderate-size and growing data set on which future research analyses can be conducted. We discovered no problems in integrating the PRO data with patients’ electronic records once the system went live in the practices. PRO results consistently loaded into the EHR after patients reported them. Furthermore, because the integration of PROs into the clinical data warehouse and i2b2 followed established institutional processes for curating and integrating data, the PROs quickly became a standard part of regular data loads from the clinical-information system infrastructure to the research-information system infrastructure.

With respect to facilitating research use of PROs, we have two examples of research studies that were actively supported by the system in the first six months after implementation. Our previously mentioned study on the effect of introducing PROs into the EHR during patient visits has been supported by all three systems.34 That study obtains regular data extracts, including PROs and other clinical data, through the IDR’s data request process. Second, a future study that aims to qualitatively examine patient reporting of PROs during office visits recently used i2b2 to identify a cohort of patients with high pain interference who can be contacted about potential research participation. Overall, in the first six months after implementation, researchers ran 39 i2b2 queries in support of these 2 exemplar studies. With that said, one of the goals of the system is to support PRO data reuse by a larger and more diverse group of researchers across the institution. As an early understanding of success in that regard, other researchers in the institution ran i2b2 queries involving PRO data 18 times, or an average of 3 queries per month. Over the same period, i2b2 received an average of 519 total queries per month, which indicates overall system feasibility. Therefore, while it is still in the early stages of use, we expect the system will increasingly be used for point-of-care PRO data collection and subsequent reuse of PRO data for research.

Strategies

Our successful design and implementation of an information system that supports PROs in research was driven by three overarching strategies (Table 2). First, we selected and implemented multiple interfaced technologies to support PRO collection, management, and research use. In other words, rather than using one system only (e.g., Epic) to collect, integrate, and disseminate PROs to researchers, we chose to leverage different systems with unique strengths in these functions. Second, we aimed to use standardized approaches to measuring PROs, sending PROs between systems, and disseminating PROs. Finally, we focused on using technologies and processes that aligned with existing clinical-research information-management strategies within our organization. Together, these strategies helped drive a successful system that is currently collecting PROs from patients on a daily basis and feeding those data into the research enterprise for widespread use. Furthermore, we believe these strategies will increase the long-term growth and sustainability of the system locally and extensibility of our general approach to other institutions. Below, we expand on these strategies and describe specific challenges and lessons learned from our experiences.

Table 2.

Strategies and Lessons Learned in Collecting, Integrating, and Disseminating Patient-Reported Outcomes

STRATEGIES	LESSONS LEARNED
Strategy 1: Use of multiple interfaced technologies rather than a single system, such as the EHR, to support PROs	The system benefitted from functionality not available in the EHR, including computer-adaptive PRO assessments and an easy-to-use interface for patients. More resources were required during start-up and for ongoing maintenance due to using multiple and less mature technologies.
Strategy 2: Use of standardized approaches to measuring, exchanging, and disseminating PROs	Using standardized and validated PROMIS measures likely increased the data’s usefulness for internal researchers and future inter-institutional research partnerships. PROMIS measures are not fully mapped to data standards for reporting results (e.g., LOINC). PROMIS measure scores are not well understood by clinicians for use in patient care. Using accepted HL7 messaging standards to communicate between systems will reduce the complexity of future system maintenance, upgrades, and other required changes.
Strategy 3: Use of technologies and processes that aligned with existing clinical-research information management strategies	The implementation benefitted from broad institutional support for collecting electronic PROs and from the institution’s interest in increasing the use of electronic clinical data in research generally. Institutional strategies and processes evolve. Thus, to be successful, the PRO system must adapt to new organizational information technology strategies and requirements.

Strategic Selection and Implementation of Multiple Technologies to Support PROs

We implemented technologies from multiple sources, including an open-source tool in i2b2 and another freely licensable tool in CHOIR. We chose this approach so that we could take advantage of innovative software and functionality that were unavailable in the EHR. Using Epic to collect PROs would have prevented us from using computer-adaptive assessments. This would have increased patient response burden and required patients to be registered with Epic’s personal health record tool. Instead, CHOIR provides an EHR-independent platform into which we can easily incorporate and modify static and computer adaptive assessments.

Our technology choices also increased the availability of PRO data for researchers in our institution and the potential for our process to be replicated by and allow data sharing with other health systems. Rather than keeping PRO data in CHOIR, we made it available through our institution’s enterprisewide IDR and i2b2. This process makes PRO data collected by one clinician or collected for a particular research study readily available to other researchers. Also, because CHOIR and i2b2 software are freely available and EHR independent, they offer low-cost options for other institutions to acquire. i2b2 has already been adopted by more than 100 institutions worldwide and provides a platform for increasing cross-institutional sharing of PRO and other clinical data.46 At the same time, because they are not mature vendor-based products, CHOIR and i2b2 require additional overhead in terms of development and maintenance.

Another lesson learned and ongoing challenge relates to the ongoing cost of maintaining multiple systems from different sources, including maintaining the interfaces between these systems. Currently, the interfaces require minimal maintenance to remain operational because they use standard approaches to data exchange that are also used to share data between other systems within our institution. However, as systems are updated or organizational policies change, the interfaces for the PRO system may also need updating. For example, if organizational policies were changed to no longer support sending PRO data through the EHR’s laboratory and results interface, an alternate approach—such as using Epic’s device interface—would be needed. This would require continued administrative support for the PRO system so that relevant development, testing, and implementation resources could be allocated. Furthermore, we are hopeful that the increasing informatics and policy emphasis on systems and data interoperability will help minimize the cost of ongoing interface maintenance between our EHR and other systems.47,48

Our success thus far has required champions in key clinical and research leadership positions and dedicated funding to support system development. On one hand, the CHOIR software implementation was supported by a single-study research grant. On the other hand, i2b2 and related IDR resources are supported by the health system and the university’s research enterprise as part of general investments in research informatics infrastructure. Ongoing maintenance, expansion, and improvements to the system will require ongoing funding. Such funding will likely need to come from research infrastructure funds and clinical departments interested in collecting PROs to support their clinical services.

Using Standardized Approaches to Measure, Exchange, and Disseminate PROs

In designing our system, we also aimed for processes that use widely available data, technology, and messaging standards for measuring PROs, communicating PROs between systems, and disseminating PROs to researchers. For measuring PROs, we focused on PROMIS measures, whose development, validation, and use has grown dramatically in recent years. PROMIS provides an increasingly popular approach to measuring PROs, computer-adaptive assessments, and measures that span a broad range of domains, which we can add to our system in the future. For sending PROs from CHOIR to the EHR, we used HL7 messages. HL7 messaging standards are widely understood and would allow our interface process to be replicated by other institutions and in other EHR systems. In disseminating PROs to researchers, our system reports PRO results using standardized quantitative scores, including t-scores and population percentiles. We also use i2b2, which has been widely adopted in academic settings. Each of these choices increases the potential scalability of our processes, in part or in whole, within and beyond our institution.

Our choices of standards have also introduced some challenges. In particular, the relative novelty of PROMIS has made it difficult to interpret the clinical significance of PROMIS results. Similarly, the newness of PROMIS has prevented us from linking each PROMIS result to a standard coding system, such as LOINC. While LOINC codes exist for some PROMIS measures, current codes are not comprehensive. Therefore, we are currently in the process of mapping our PRO measures to existing LOINC codes, and we will likely request the creation of new LOINC codes for some measures.

Aligning Technologies and Processes with Existing Clinical Research Information Management Strategies

Finally, our design and implementation strategy required us to align our decisions and technologies with institutional strategies for collecting, managing, and disseminating clinical data for research. This alignment between individual systems and overarching strategies was necessary for both short- and long-term support for the growth and sustainability of the system. However, we learned that alignment with organizational strategies can be a moving target. For example, the goal of mapping PROMIS results to LOINC was prompted by not only the general value of standardization for communicating PRO data but also by our institution’s shift to using LOINC, generally, for results. Similarly, our institution has increasingly fielded requests from researchers and clinicians to interface its EHR system with other systems. Because of this, administrators have reconsidered how many different interfaces and interface processes should be supported. Therefore, in the future, maintaining the PRO system may require reworking to adapt to changing institutionwide strategies.

Overall, as we prospectively devised and retrospectively reflected on our approach to collecting, integrating, and disseminating PROs, we identified many specific challenges and lessons learned. Beyond those above, we also continue to be challenged by the need to make PRO data collection as easy as possible and clinically relevant in practice settings. While collecting PROs at the point of care is critical because it allows data capture from essentially all patients, the value of PRO data for research is not generally a compelling rationale for busy clinicians and office staff with a full schedule of patients. Therefore, the cost of collecting PROs must be negligible and the benefits must be clearly articulated.

Discussion and Conclusion

This case study described the context and lessons learned in implementing an information system that supports collecting, integrating, and using PROs for research in a learning health care system. Our description and lessons learned can help others as they think about how to implement useful research information systems while adhering to principles that allow their systems to scale and persist over time.

With this case study, we also hope to spur dialogue in the research community on others’ successes (and failures) in designing, implementing, and managing systems for collecting and managing electronic PROs in support of research. As other academic health centers grapple with the challenge of managing electronic clinical data and technology for research, the community would benefit from a more robust and diverse body of literature describing experiences. These descriptions may come from experiences of consortiums, such as those funded by the Patient-Centered Outcomes Research Institute49 and others that manage cross-institutional data research networks. Also, these descriptions could come from individual institutions that developed effective systems to support individual clinical studies that grew to be part of broader organizational solutions and can thus provide more general informatics-oriented contributions. Initially, growth in published literature on such systems may offer qualitative syntheses of lessons learned across organizations. As the knowledge base matures, it may be possible to conduct cross-organizational quantitative studies that examine relationships between PRO and other data management strategies and outcomes such as research productivity and quality.

As with other case studies, our experiences and lessons learned are not completely transferrable to other organizations. We describe a system that is still in the early stages of maturity. The complete set of processes has been in place for only six months and PRO data collection is not yet in operation throughout the health system. Thus, it remains to be seen whether the strategies we employed will scale up within our institution or be adopted by others. Furthermore, the system’s effectiveness should continue to be evaluated. In addition to more qualitative evaluations via stakeholder feedback, quantitative evaluation should include ongoing tracking of the volume of electronic PRO data collection as it expands to other care settings and new types of patients. Also, the institution logs i2b2 query volume and detailed research data requests. Therefore, to assess ongoing research use of PROs, we should track the volume and types of queries and data requests involving PRO data.

An academic health center that affiliates with multiple health systems may face additional challenges in using a system like ours. In such a setting, system designers may face challenges in developing consensus on a common PRO data set and challenges in developing standards for integrating and disseminating data that come from multiple organizations. However, our general approach of using multiple systems that interface using standard data and messaging approaches may also be beneficial in settings with multiple health systems. Our approach does not require each health system to use the same EHR or even to use CHOIR for collecting PROs. This flexibility may also be helpful if a health system decided to withdraw support for a given software or vendor and switch to another. Also, using CHOIR required significant start-up and moderate ongoing maintenance effort given that it is not a mature, vendor-based product. Therefore, another institution that aimed to adopt the same technologies would need in-house software development expertise.

We also note that the PRO data are initially generated in a clinical setting without a researcher present. Thus, the quality of the data may not be as high as data generated in a controlled setting where a researcher directly oversees the process. This concern is mitigated by the fact that the PRO assessments were designed to be completed independently by patients. Therefore, we believe the PRO data are of sufficient quality for most observational studies and for cohort discovery purposes. We also believe that the PRO data are likely of similar or better quality as other clinical data commonly used for these research purposes.

New challenges will certainly arise as the corpus of PRO data in the institution grows and researcher demand for the data grows with it. That said, we are actively improving our system and working to scale up the implementation in our health system. We are also partnering with other institutions that are planning to adopt our approaches to regularly capturing and using electronic PROs in research. In conclusion, if learning health care systems across the United States and worldwide are to incorporate efficient solutions for generating PRO data for research on large, diverse populations, it is important to continue sharing and learning from these experiences and advancing research on the management of these systems.