PatientViewpoint: A Website for Patient-Reported Outcomes Assessment

Abstract

Purpose

To develop a prototype website to collect patient-reported outcomes (PROs) in outpatient clinical oncology and link the data with the electronic medical record (EMR).

Methods

A multidisciplinary Research Network, including experts in outcomes research, clinical oncology, nursing, social work, information technology, EMRs, behavioral science, decision science, clinical trials, law, and a cancer survivor, was formed to design the prototype website. The Research Network developed the initial website specifications, elicited feedback from patients (n=20) and clinicians (n=7), constructed the website, and conducted usability testing (n=10).

Results

Clinicians reported that the website could improve clinical practice if it was not burdensome and were most interested in tracking change over time. Patients were interested in using the website because of the potential to facilitate communication with their clinicians. Patients emphasized the importance of short and simple surveys and a user-friendly interface. The PatientViewpoint website was designed to meet these specifications. Usability testing suggested that patients had few problems accessing and using the site.

Conclusions

Preliminary reports from clinicians and patients suggest that a website to collect PROs and link them with the EMR could help improve the quality of cancer care. Further pilot-testing will evaluate the use, usefulness, and acceptability of PatientViewpoint.

BACKGROUND

Measures of health-related quality of life (HRQOL) and other patient-reported outcomes (PROs) are used commonly in clinical trials and observational studies. Research has recently focused on the potential use of PRO instruments to improve the interactions between individual patients and their clinicians. In contrast to use of PRO questionnaires in clinical trials, when PROs are used in clinical practice, the data are provided to the clinicians (and sometimes the patients) with the goal of helping the clinicians manage the patients’ care. The use of PROs in clinical practice is being investigated in a variety of disease areas, with oncology being one of the most common in recent years [1,2].

Early studies of PROs in clinical practice have had mixed results. Having patients complete these assessments and providing summaries of the patient scores to their physicians has been shown to facilitate communication about HRQOL between doctors and patients [3–9]. There is less evidence that these assessments alter management, affect patient outcome, or improve HRQOL and satisfaction [10]. Still, when asked, clinicians have generally found PRO data useful and not disruptive to their practices [3–4, 8–9].

Use of a web-based system may overcome some of the barriers to using PROs in clinical practice. Previous assessments of PROs in clinical practice have asked patients to complete questionnaires in the clinic prior to their visit. However, patients arriving late, getting interrupted while completing the questionnaire, or not having enough time to finish led to missing data [11]. Berry et al. [12] recommend interventions that ensure adequate time for patients to complete questionnaires without interruption. Additionally, questionnaires completed in the waiting room may be affected by patient anxiety about their visit, so asking patients to complete questionnaires at home may provide information about how they are doing separate from their visit. Having patients complete questionnaires via the Internet before visits could obviate these problems. For a more complete discussion of how a web-based system could also address other obstacles to use of PRO assessment in routine outpatient oncology practice, see Jones et al. [13]. However, as noted by Basch et al. [14], health information privacy, data security, and patient safety must also be addressed in developing web-based systems.

Another innovation would be to link the patient’s PRO data to the health care organization’s electronic medical record (EMR) to facilitate the use of PRO data during the clinical encounter. It is widely believed that EMR systems, which include current and historical patient information, clinical decision support, and a data repository, will provide better and more useful information when it is needed, reduce medical errors, increase efficiency, and improve health. Having information on patients’ functioning and well-being alongside their lab results, imaging studies, and treatment notes allows clinicians to see a fuller picture of the patient’s status. Currently, PRO data are rarely collected as part of routine clinical practice, and linking to the EMR is even more uncommon.

This paper reports the process we undertook to develop a web-based system to collect PROs and link with the EMR. As a first step, we designed the system for use in outpatient oncology practice, but the technology could be easily adapted for other disease areas and other care settings. These preliminary steps were designed to develop the website to the point of readiness for formal pilot-testing. The planned pilot-test will rigorously evaluate the use, usefulness, and acceptability of the website to patients and clinicians.

DEVELOPING A PROTOTYPE WEBSITE

Establishing a Multidisciplinary Research Network

This project was conducted in response to a request for proposals from the U.S. National Institutes of Health that called for “applications for meetings or networks to develop methodologies that will facilitate interdisciplinary health research among behavioral or social scientists and investigators in the biomedical, mathematical /computational, physical sciences, and/or engineering” [15]. The “Website for Outpatient QOL Assessment Research Network” (The Network) was formed using the organizing committee of the Quality of Life in Clinical Practice Special Interest Group of the International Society for Quality of Life Research as its basis and adding other areas of expertise. The Network includes experts in cancer outcomes and effectiveness research, clinical oncology, nursing, social work, information technology, electronic medical records, behavioral science, decision science, clinical trials, law, and a cancer survivor.

The Network undertook a multistep process to design the prototype website: (1) a literature review to identify similar projects and issues for the Network’s considerations, (2) development of initial website specifications, (3) interviews with patients and clinicians to elicit their feedback on the website design, (4) website construction, and (5) usability testing with patients to obtain their feedback on website operations. The results of the literature review have been reported [13]; this paper summarizes the remaining steps of the website development process. The protocol for interviewing patients and clinicians was reviewed and approved by the Institutional Review Board of the Johns Hopkins School of Medicine (NA_00006060), and all subjects provided written informed consent.

Initial Website Specifications

Based on the literature review, the Network outlined specifications for the website. The purpose of the website is to improve patients’ experience of care in real-time by facilitating doctor-patient communication, which can then improve patient care and outcomes. The target audience includes both patients and clinicians, and the website is designed to allow both patients and clinicians to track changes in status. To facilitate interpretability of the data, the website includes alerts for scores or changes in scores that exceed pre-set thresholds. Further, so that the results of the PRO data can be evaluated in the context of the patient’s other clinical information, the website links to an organization’s EMR. To facilitate both individual patient management and evaluating group trends, the website is designed so that physicians can view data both on an individual and group basis. Finally, implementation is planned to be multi-faceted, including training, education, in-servicing, and recruiting thought leaders.

We opted to develop our own system rather than use an existing commercially available survey product because we needed the website to be flexible and customizable. The website is more than a survey tool and multiple dimensions of functionality were required. The website design included developing an interface for clinicians to assign surveys to specific patients and track results over multiple survey responses. Also, an interface was built so that patients could track their results over time and leave comments for the clinicians. Finally, the website was designed specifically to link with the electronic medical record, a task with specialized security requirements.

To make the initial development manageable, we decided to first implement the website as a demonstration project at the outpatient clinic of the Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins. However, generalizability was considered during the design phase. Though we focused the initial website on a limited number of cancer types (i.e., breast and prostate), we designed it to easily expand to other cancer types and diseases. Any generic or disease-specific PRO could be programmed into the website, enabling its use in a broad range of patient populations.

Another issue addressed with the initial website specifications was the selection of PRO questionnaires to be administered. We determined that valid measures of PROs/HRQOL (broadly defined) will be available on the website. Clinicians would be able to review and evaluate the available questionnaires and select which ones to use with his or her patients. The website would provide explanations of the questionnaires to aid in selection.

Eliciting Feedback on Initial Specifications from Patients and Clinicians

Based on the above specifications, a preliminary website was constructed and presented to patients and clinicians to elicit their feedback. The purpose of these initial qualitative interviews was not to rigorously evaluate the website’s performance but to obtain preliminary information that could be used to develop the website and prepare it for formal pilot-testing. Using a semi-structured interview guide, we interviewed a convenience sample of 10 breast and 10 prostate cancer patients and 7 clinicians (3 oncologists and 4 research nurses) involved in their care. Patients were recruited in an outpatient clinic setting. As the website was targeted at all patients in the practice, no attempt was made to restrict eligibility. From the interviews we know that some patients had early and some later stages of disease. We know that one breast and one prostate patient were not computer users. Because the initial screening for interest in participation was done by clinic staff, our qualitative interviewers had no interaction with patients who refused to participate and therefore could collect no information from them. While statistics were not kept on the number who declined participation, it occurred very infrequently. Clinicians were recruited by the researchers based on their involvement in the treatment of breast or prostate cancer.

The preliminary website presented the screen shots and demonstrated the functionality of the website without having the actual data collection or storage system built in. During the interview, the interviewer presented a mock-up of the web application and described its proposed functions to each participant. Participants were asked to comment on the proposed features and give their impressions of the website. Questions were asked about participants' experiences using alternative questionnaire and report formats, access to computers and e-mail, concerns about maintaining the security of their data, and reactions to the possibility of using questionnaire responses for research purposes. Each interview was audio recorded, and relevant sections were transcribed. We analyzed the data qualitatively and identified themes. Specifically, as this was a relatively small dataset, the interviewer listened to all of the interviews several times and developed a thematic coding scheme. The themes were grouped by context, and example quotes were abstracted to illustrate the findings.

In general, clinicians were optimistic about the potential of the website to improve clinical practice but wary of the time and effort required to use the site (Table 1). One clinician commented, “I would want someone to print the information out to be reviewed with the chart, or linked directly in the EPR [electronic patient record]. I need someone to hit me over the head to indicate that there’s something I need to look at.” As with any lab test, clinicians must be able to interpret the meaning of the PRO data and, in particular, whether a change in an outcome is clinically significant. Thus, clinicians recommended that we present the results in a way that clearly depicts change over time for each patient. Beyond the outcomes measured by the web survey, one clinician suggested that patients identify the single most important issue they would like addressed at their next clinic visit.

TABLE 1.

Key findings from clinicians on website specifications

FINDING	SAMPLE QUOTE
Clinicians need guidance for interpreting the meaning of values.	“Of course I have no idea if this is a good score or a bad score.”
Clinicians are most interested in meaningful individual change over time.	“The individual patient values would be more useful, I want to know about this individual patient, some patients will always score low, this doesn’t represent change over time, you can’t always make a miserable person better, is there something that I can fix?”
Need ways to emphasize the key findings to focus on during the visit.	“We’re all taught to fix the one thing that the patient needs today, is there one parameter that is tipping the scale, or are they generally having trouble.”
The website could be useful in clinical practice, but it needs to be easy to integrate into current routines with little administrative burden.	“I would want someone to print the information out to be reviewed with the chart, or linked directly in the EPR [electronic patient record]. I need something to hit me over the head to indicate that there’s something I need to look at.”

Patients were generally enthusiastic and interested in using the website (Table 2). This interest was based on the premise that it would allow them to provide meaningful information to their clinicians. A few patients were concerned that their clinicians might not have time to review their information before their next appointment. Some patients mentioned the importance of the system not being burdensome. Most indicated that they would prefer short and simple surveys, though the meaning of “short” ranged from 5 to 15 minutes, with a few patients indicating they would be willing to spend more time on each survey. In terms of user interface, several had experience using websites that were confusing and difficult to navigate, and they suggested that using the website should be as simple and efficient as possible. When asked about their thoughts on different methods of presenting the results at the end of each survey, some patients were interested in seeing their scores and tracking changes over time, while others simply wanted to provide useful information to their clinicians.

TABLE 2.

Key findings from patients on website specifications

FINDING	SAMPLE QUOTE(S)
Keep surveys short and simple.	“I always have questions for him about my well-being, but if it becomes too long or too complicated, I’ll give up.”
Present multiple questions per survey page, but avoid long scrolling.	“I would [prefer multiple questions], as long as there’s not too much.”
Some patients are interested in their survey results, others just want to comply with the request of their care provider.	“Somebody is going to find it interesting, but I’m not, you know personally, once I’ve done the survey and made my additional feedback, I’m basically done.”
Patients interested more in changes over time than survey values.	“It will be more meaningful just to see the individual change. I want to be able to look back six months and say, where was I six months ago.”
Considering the option to add free text, patients like the direct communication, like their comments to inform the next visit, worried it will not be read.	“You need to go to your doctor with questions in writing, and it seems that this would be a vehicle to get those questions there.” “I am concerned where you say that you won’t read it until the next appointment, you’re not going to read it until I get there, what am I chopped liver or something?”
Patients perceive the site’s purpose as valuable, concerned it will become burdensome.	“It would be useful from their point of view, so I’d be willing to take five minutes to fill out the thing.”

Based on the feedback we received from the patients and clinicians, we refined the website specifications and developed the prototype. For example, in response to the clinicians’ request, we added a question that asks patients the single issue they most want their doctor to address at the next visit. Also, clinicians did not feel comfortable selecting surveys from a menu and recommended that, working with them, the PRO researchers should identify the appropriate questionnaire(s) to assign to patients with a given tumor type. In the future, as clinicians gain more familiarity with the PRO surveys, they will have the option of selecting which ones they want their patients to use.

Based on clinicians’ and patients’ feedback, we developed score reports that focus on changes over time and that highlight problem scores (either poor scores in absolute terms or a large change from the previous assessment). The clinicians’ report is more detailed, including both tabular and graphical presentation of the results along with the patients’ comments (Figure 1). Viewing the results is optional for patients and includes only the graphical display and comments. However, if further research suggests that patients want more detail, it would not be difficult to provide the same reports to patients as clinicians.

FIGURE 1.

Sample clinician report presenting results

Developing PatientViewpoint

Having revised the website specifications based on the feedback from patients and clinicians, we constructed the prototype website. The name, “PatientViewpoint,” was selected based on the website’s purpose of bringing the patient’s perspective on their functioning and feelings into the clinical picture.

Specifically, PatientViewpoint is a web-based system with a database back-end for management and storage of patient responses. The system has been developed with standard browser interaction to reduce the need for training of the patients. A standard, popular browser is all that is needed to complete the surveys and view the results. High-speed connectivity is preferred, but the survey is separated into multiple smaller forms, reducing the need for high-speed connectivity. We expect that most patients will complete the questionnaires using their own computers prior to their visits, but for patients without computer or Internet access, we will have a computer in the clinic for them to use.

There are three main functions provided by the system: (1) assigning questionnaires to patients, (2) completing questionnaires by patients, and (3) reporting results for clinicians and patients. The first function allows clinicians to assign questionnaires to their patients and schedule how often the patient should complete the questionnaires. The system has a built-in calendar function, which will automatically generate e-mails to patients to alert them when it is time for them to complete a questionnaire.

The second is the ability for the patients to enter responses to the questionnaires, submit comments, and view the results of their responses. As the patients complete questionnaires over time, they can view graphical representations of their responses and see their scores and explanations of their scores. Specifically, after logging in, the patients are presented with the survey(s) that they have been requested to complete and the dates by which they should be completed. By clicking on a specific survey link, the patient is presented with a series of forms, using checkboxes, drop-down selections, and other standard form elements including two open comment fields. One comment field asks the patient to specify the one thing s/he is most interested in the clinician addressing at the next visit, and the second text field is for any additional comments. The third function is giving clinicians a text and graphical view of their patients’ responses and scores over time (Figure 1). Specifically, the system generates reports of the patient’s questionnaire responses that are made available in the electronic patient record so that physicians can view patient clinical and PRO information in one place. Clinicians can search by survey name or by patient name to view results from previously completed surveys. The results are presented in table and graphical views. Domains with problem scores (i.e., poor scores in absolute terms or scores that are significantly worse than the previous assessment) are highlighted. A printable version of the report is available for the clinicians. To access the patients’ PRO results, the clinician simply needs to go to the “Documents” tab within the patient’s electronic medical record and click on “PRO report.”

The system was developed using Microsoft ASP.Net and SQL database. There are multiple firewalls in place to prevent unauthorized access to the servers that house the systems. The system runs on two servers, a web application server and a database server so that the data are physically separated from the application code. A user of the system cannot directly access the database. Access is restricted by roles, and patients are restricted to view their own data. There is a timeout after 20 minutes of inactivity. The servers are backed up on a regular cycle and backups are stored at a remote location. Secure Sockets Layer is implemented to secure the site as patients enter their responses and physicians view the results. Access to the system is controlled by requiring authentication involving username and password. The system was formally reviewed and approved by our institution’s Clinical Systems Review Committee.

Usability Testing

We conducted usability testing of the prototype website. A convenience sample of 10 patients (5 breast and 5 prostate) were recruited and interviewed in an outpatient cancer clinic setting. This sample was considered adequate to evaluate whether the website was ready for a planned, more rigorous pilot-test. Eligible patients, identified by clinic staff, were approached for the study prior to their scheduled visit. While we did not collect socio-demographic data from the patients as part of this initial usability testing, no patients refused to participate, minimizing the chances of selection bias.

During the usability testing, patients accessed the website and tested its features, including site login, survey completion, and review of results. Patients completed a semi-structured, in-depth interview about the site and its features, and their willingness to use it regularly prior to visits. These interviews showed that participants were interested in adding this type of online data collection to their care and that they had few problems accessing and using the site (Table 3). Many were excited about a new way to communicate with their doctors between visits, and liked the ability to track and review scores. Even participants who specifically stated they lacked computer skills had no difficulty using the website, commenting on how easy it was to use. Some differences were noticed between breast cancer and prostate cancer patients. Breast cancer patients, for example, were more interested in using the site frequently as a tool to monitor their health. Prostate cancer patients, on the other hand, were not as interested in viewing their scores and reports. However, regardless of how participants reported their own use of the website, the majority of all participants reported a minimal response burden and a willingness to log on and use the site regularly.

TABLE 3.

Key findings from patient usability testing

FINDING	SAMPLE QUOTE(S)
Overall, patients found the site simple and easy to use.	“I've filled out job applications that are harder than that…I'm limited in what I can do, and I found it very easy."
All patients expressed interest in being able to complete the questionnaire online because of time constraints during patient visits, the travel distance necessary for visits, and increased privacy.	“A lot of these questions…I could sit at home and answer, and think about it a little….So I think this is very good.”
Patients expressed concerns about the security of the website. Others asked which staff members would be able to access this information.	“Just the doctor and me. No, I wouldn't want everyone seeing my information.” “One of my biggest concerns is that [the site] is secure. People hack-in and get people's information and all that kind of stuff.”
Patients liked that questions asked about issues they might not have thought of, or would not normally talk about with their doctor.	“It would stop a lot of [people from] trying to hide or forget…This would bring it to the doctor’s attention to talk about. It may not be a big issue, but hey.”
Patients liked that the system automatically highlighted large changes in scores.	“You may not go [to the doctor] and say, ‘Hey, I haven't gone out of the house for a week, but the doctor can ask you about your report.’"

OUTSTANDING ISSUES AND NEXT STEPS

The initial usability tests confirmed the website’s readiness for formal pilot-testing, and we expect funding for a two-year, two-phase evaluation of the Patient Viewpoint website to assess its practicality and its value to patients and clinicians. Specifically, we aim to assess the website’s use (i.e., how often patients complete questionnaires and clinicians refer to results), usefulness (i.e., whether the website improves the patient encounter from the patient and clinician perspective), and acceptability (i.e., whether patients and clinicians find the website easy to use and not burdensome). These initial pilot tests will be conducted at a single institution (Johns Hopkins) and in only two cancer types (breast and prostate). While the initial testing conducted during the developmental phase did not collect detailed information on our participants and we, therefore, cannot comment on its performance based on various socio-demographic characteristics, the formal pilot-test will explore whether differences in participation and perspectives occur by age, sex, race, education, and prior computer/e-mail experience. Based on the findings of the pilot test, the website will be refined and then its use expanded to more institutions and cancer types.

While the website in its current form has the potential to meet our goal of improving the quality of interactions between patients and clinicians, a variety of issues remain in the website’s design. First, at this point, the website is primarily a web-based application for collecting survey data and summarizing the results. Its primary innovation is the ability to link the data collected via the website with the other data in the patient’s medical record. There is the potential to develop the website to do more. For example, the website could serve as an important patient education and self-management intervention, which some patients expressed an interest in. Links could be added so that patients could investigate potential solutions to problems identified through the questionnaires. Similarly, on the clinician side, the website could be designed not only to identify problem scores on the questionnaires but also to recommend actions to take in response to those problems.

Second, our initial plans involve having patients complete questionnaires via the website just prior to their visits, with the goal of having data from the questionnaires available to the clinician to aid in his/her assessment of how the patient is doing. However, one of the strengths and possibilities of the website is to facilitate care for patients between office visits. While improving the quality of the clinician-patient encounter during visits is valuable in and of itself, this greater potential of improving care between visits is also worthy of further exploration. Issues regarding reimbursement for clinicians’ use of the website will need to be addressed to facilitate use of the website more broadly.

Related to the second issue is the question of how to ensure that critical problems reported by patients between visits are addressed in a timely manner. The current website prototype warns patients clearly and repeatedly that they should not report urgent situations using the website because their results may not be viewed until their next office visit. Patients are instructed that they should call their doctor with urgent issues. However, safeguards need to be in place such that if patients do report a critical problem, the issue is addressed quickly and appropriately. One potential safeguard is to have automatic alerts to the patient’s clinician built-in for responses on the questionnaires that the clinicians feel merit an immediate response. The alerts would be sent as text to both the clinician’s e-mail and pager. For clinicians who are out-of-the-office, pager messages are forwarded to covering clinicians. The key is to build in such alerts so that they are not burdensome to clinicians and directing them to the appropriate personnel, perhaps nurses. However, patients’ free text entries are not amenable to automatic alerts.

A final issue involves the questionnaires that will be available via the website. For the most part, HRQOL and other PRO questionnaires were developed for group comparisons and the reliability thresholds required for individual patients are rarely met [16]. Further, clinicians emphasized the importance of making the data interpretable. However, few of the currently existing questionnaires have guidelines regarding what score represents a problem, either in absolute terms or changes in an individual patient’s score over time. We can provide initial guidelines based on score distributions for normative populations, but this area requires significantly more research. Additionally, clinicians suggested having patients report one issue they most want addressed at their next visit. While this may be useful from the clinician perspective, it remains to be seen whether patients, who might have multiple problems, have difficulties identifying one most important issue.

These issues and others will need to be explored and addressed as development of PatientViewpoint proceeds. Accommodations will need to be made for users with disabilities. Additionally, as with any technology, innovations will occur in hardware, software, and knowledge. Moving ahead, we will need to implement the prototype in a flexible and agile manner so that it can adapt to evolving demands and improve with technological innovations.

ABBREVIATIONS

EMR

electronic medical record

HRQOL

health-related quality of life

PRO

patient-reported outcome