Design and Evaluation of the Electronic Patient Generated Index (ePGI)

Abstract

Background.

The Patient Generated Index (PGI) is an individualized measure of health related quality of life. Previous work testing the PGI in the oncology setting identified threats to content validity due to navigational and computational completion errors using the paper format.

Objective.

The purpose of this pilot study was to refine and evaluate the usability and acceptability of an electronic PGI (ePGI) prototype in the outpatient radiation oncology setting.

Methods.

This pilot study used adaptive agile web design, cognitive interview and survey methods.

Results.

Three iterations of testing and refining the ePGI were required. Fifteen patients completed the refined ePGI using touchscreen tablets with little or no coaching required. Nearly all participants rated the ePGI as “easy” or “very” easy to use, understand and navigate. Up to one-half stated they did not share this type of information with their clinician but felt the information on the ePGI would be useful to discuss when making decisions about their care. Eight clinicians participated, all of whom felt the ePGI was a useful tool to initiate dialogue about quality of life issues, reveal infrequent or unusual effects of treatment and assist with symptom management.

Discussion.

The pilot study indicates the ePGI may be useful for use at the point of care. Larger studies are needed to explore the influence it may have in decision-making and restructuring patient/provider communication.

The concept of patient-centered care, in which patients are the “final arbiters in deciding what treatment and care they receive,” is core to efforts to transform health care in the U.S.(Crossing the Quality Chasm: A New Health System for the 21st Century, 2001). Within patient-centered care, the needs and values of the patient provide the basis for individualized care and decisions yet all too often, patients are unable to articulate or learn how treatment may affect what is most important to them (IOM). Communication and decision making becomes more meaningful when the impact of illness and its treatment on the individual is assessed from the patient perspective (Bowling, 1995; Lindblad, Ring, Glimelius, & Hansson, 2002; Patel, Veenstra, & Patrick, 2003).

Patients and clinicians are more likely to discuss health related quality of life (HRQL) issues when data are available to them. There is a need for patient relevant outcome measures that promote the patient perspective in health care discussions and decisions occurring during clinical interactions (Aburub, Gagnon, Rodríguez, & Mayo, 2016; Atkinson & Rubinelli, 2012; Hall, Kunz, Davis, Dawson, & Powers, 2015; McCleary et al., 2013).

The three-step patient-generated index (PGI; Ruta, Garratt, Leng, Russell, & MacDonald, 1994) provides a novel approach to the measurement of HRQL that accounts for individual values and preferences. First, patients identify areas most important to them that are effected by cancer and its treatment. Second, they score each item for severity. Third, they prioritize the importance of the items. The PGI, historically available only in paper format, has been studied in cancer (Camilleri-Brennan, Ruta, & Steele, 2002; Martin, Camfield, Rodham, Kliempt, & Ruta, 2007; Tang, Oh, Scheer, & Parsa, 2014; Tavernier, Totten, & Beck, 2011; Tavernier, Beck, Clayton, & Pett, 2011). A computerized version, however, has the capability to address documented navigational and computation errors (Tavernier, Totten, & Beck, 2011), threatening the content validity of the PGI. The purpose of this study was to refine and evaluate the usability and acceptability of an electronic version of the PGI (ePGI) prototype in the outpatient radiation oncology setting.

ePGI Prototype

Investigators developed a prototype based upon the three step PGI (Ruta, 1994). The initial prototype consisted of five screens with navigation and completion processes incorporated to address documented navigation and computation errors of the paper and pencil version (previous study).

Methods

Investigators obtained Institutional Review Board approval prior to beginning the study. Clinic liaisons assisted with identifying and inviting eligible participants; written consent was obtained by the investigators. The study occurred at a large outpatient radiation department with data collected data during routine appointments.

All full-time oncology physicians and registered nurses working in the radiation outpatient clinic were eligible and invited to participate. Investigators used convenience sampling methods to accrue adult patients receiving radiation treatment at the study site for at least two or more weeks at time of consent and who were able to speak and read English and being seen by a consenting clinician.

The study had two phases (Figure 1). In Phase 1 the investigators implemented an end-user adaptive agile design approach (Gustafson, 2011; Nielsen, 2000; Wolpin & Stewart, 2011) using cognitive interviews (Willis, 2005) and direct observation to examine and improve the usability (ease of use, understandability, navigational elements) and acceptability (interface layout and visual design) of the ePGI prototype (Health Literacy Online: A Guide for Simplifying the User Experience, 2016). The agile design specifically focused the development of the ePGI for the end-user, testing each iteration until there were no further changes suggested. In the second phase, investigators used a survey and structured interview to evaluate patient, nurse and physician perceptions of the usability and acceptability of the ePGI at the point of care.

Figure 1.

Two-phase study using adaptive agile design

Measures

Phase 1 cognitive interviews (Table 1) focused on the comprehension, navigation, layout and difficulties experienced or observed during completion of the ePGI. Phase 2 interviews addressed using the ePGI at the point of care (Table 2). For Phase 2, the investigators developed a 19-item survey of previous computer use and acceptability and usability of the ePGI, loosely adapted from other related survey questions (Basch et al., 2005; Carlson, Speca, Hagen, & Taenzer, 2001; Clark, Bardwell, Arsenault, DeTeresa, & Loscalzo, 2009). ePGI and surveys were completed using tablet computers and data automatically stored on a secure server using Research Electronic Data Capture (REDCap).

Table 1.

Cognitive Interview Questions

1. Tell me what you like about the ePGI survey.  a. What didn’t you like?

2. For each screen:  a. Tell me what this screen is asking  b. Was it easy or difficult to complete this screen?   i. What made it easy/difficult?   ii. What could make it easier?   iii. Probe for the specific elements of the screen    1. Entering words    2. Clicking/touching radio buttons for ratings    3. Selecting points  c. Tell me how you came up with your answer.   i. Is there anything that would have helped to make your answer easier?  d. Do you like the way this screen looks? Why or why not? 3. Was it easy to move between screens in the survey?  a. What made it easy/difficult?  b. What could make it easier? 4. Did you look at your answers from the previous week?  a. Why or why not?  b. If so, was it helpful?   i. Why or why not?  c. Did you understand how to look at your answers from the previous week?   i. What might make it easier to do/understand?  d. Did you make any changes to your answers this time? 5. Is there anything else you thing should be changed to the survey to make it easier to complete?

Table 2.

Phase 2 Patient and Clinician Interview Questions

Patient Questions: We are going to talk about how you would use the information from this survey when you meet with your nurse or doctor.  1. Describe a typical weekly visit with your oncology doctor and nurse.  2. Do you usually talk about these ____ areas when you meet with your nurse/doctor?   a. If so, how is it part of the conversation?   b. If so, are they part of the planning of your care?   c. Tell me why not.    i. Do you wish it were part of your conversation? Why or why not?  3. Do you think about these areas when you make decisions about your health?  4. Do you think about these areas when you make decisions about the treatment you get for your cancer?  5. Would you want your doctor or nurse to see your survey?  6. How do you think it would help your doctor or nurse to know what is on your survey?  7. Is there anything you would need to know to be able to use your survey when you meet with your doctor or nurse?  8. IS there anything else you would like to add about how this survey may or may not be used while you get treatment for cancer?  Clinician Questions:  1. Describe a typical weekly “on treatment” visit.   a. What factors do you consider when making planning care?  2. Tell me about your previous experiences using patient reported measures (provide examples if necessary)   a. Did you find them useful? Why or why not?  3. What are the benefits of using patient-reported outcomes in your practice?  4. What are the barriers to using patient-reported outcomes in your practice?  5. How do you define quality of life for your patient?   a. How do you routinely assess for patients quality of life in your practice now?   b. How do you usually document your assessment?  At this point, review ePGI and how it is completed using responses randomly selected from patient ePGI responses.  1. If you had this information available to you before you meet with this patient, would you review it?   a. Why or why not?  2. Reflecting back on the typical weekly visit you described, is the type of information in the ePGI ever discussed?   a. If so, describe an example and how/why you use the information.  3. How do you think using the ePGI would affect or change the interaction you have with your patients?  4. Do you think this information is useful to you?   a. Why or why not?   b. If so, how?   c. How would you use this in your practice?  Is there anything else you would like to add about using the ePGI or other patient reported outcomes in your clinical practice?

Procedure

Patients in Phase 1 completed the ePGI prototype using a touch screen tablet provided by the principal investigator. Participants had only the instructions within the prototype provided. The investigator observed the patient and provided directions or answered questions only if there was an inability to proceed further, taking notes of hesitations, completion errors and verbalized difficulties for reference during the subsequent interview. Cognitive interviews were recorded and assessed the ease of use, understandability, navigational elements, interface layout, visual design and potential usefulness of the ePGI. Patient recommendations were incorporated and tested with the next iteration, tested on the same and additional patients. The process was repeated until there were no observed difficulty with the completion of the ePGI nor patient suggestions to improve the ePGI.

In phase two, patients completed the ePGI on a computer tablet once a week for two consecutive weeks before seeing the clinician; the patient and treating clinician had the opportunity to share the ePGI results on the tablet during the visit. In the first week, patients answered survey questions about computer use and completed the ePGI. In the second week, the patient completed the ePGI, usability and acceptability survey questions and interview. Clinicians were interviewed individually at an agreed upon time after all patients had completed the study.

Analysis

Notes and recordings of Phase 1 interviews were reviewed by the first author for content related to suggestions for improving the ePGI. All suggestions were discussed with a bio-informatics professional to resolve any conflicting feedback and then incorporated into the ePGI for testing in the next iteration.Recorded interview data from Phase 2 were coded and analyzed directly from the recordings using content analysis. Quantitative data was downloaded from REDCap and analyzed using Excel for Windows. Because of the pilot study nature and thus small sample size, only descriptive analysis is reported.

Results

During Phase 1, three iterations of testing and refining the ePGI, involving seven patients were required to obtain a usable ePGI; one in which patients could easily complete, provide a description of what to do on each screen congruent with investigators’ intent, and not have additional questions about completion or suggestions for improvement. Cognitive interview results led to prototype changes in screen content, spacing and language. A sixth screen providing a summary of patient responses and overall HRQL index score was added based upon clinician feedback.

In phase 2, 15 patients completed the ePGI and interview; 12 (80%) were women and 12 (80%) were older than 60 years of age. Fourteen patients completed the survey questions (Table 3); six (42%) had little or no computer touch screen experience, three of whom had never used any type of computer prior to the study. Patients required less than 30 seconds or no coaching when completing the ePGI. Nearly all participants rated the ePGI as “easy” or “very easy” to use, navigate, understand, and follow instructions. Less than half stated they frequently shared the type of information they entered on the ePGI with their physician or nurse however most (n=8, 67%) felt the information on the ePGI would be useful in making decisions regarding their disease or treatment.

Table 3.

Frequency of Phase 2 ePGI Usability and Acceptability and Computer Use responses (n =14)

Question	Easy/Very Easy	In-between	Difficult/Very Difficult	No response*
Usability and Acceptability Questions	n (%)	n(%)	n(%)	n
How easy or difficult was the survey to complete?	12(86)	2(14)
How easy or difficult was it to use the touch sensitive screen?	13(93)		1(7)
How understandable were the questions?	12(86)	2(14)
How easy or difficult was it to read the instructions on the screen?	13(93)	1(7)
How easy or difficult was it to follow the instructions on the screen?	13(93)	1(7)
How easy or difficult was it to find your way around the pages of the survey?	14(100)
	Not at all/ very little	So-so or a little	Quite a bit/ very much
How helpful was it to complete the survey?	3(25)	4(33)	5(42)	2
Was the amount of time it took to complete the survey acceptable?			14(100)
How useful is the information you entered in the survey to the decisions you make about your health?	3(25)	2(17)	8(67)	2
Do you share this information with your nurse?	6(50)	1(7)	5(42)	2
Do you share this information with your doctor?	4(31)	3(23)	6(46)	1
How much did you enjoy using the survey?		2(18)	9(69)	3
Computer Use Questions
How often do you use computers?	3(21)	3(21)	8(57)
How much do you like computers?	1(8)	4(33)	7(58)	2
How much do you like to complete computer surveys?	5(42)	4(33)	3(25)	2
How much do you like to complete paper surveys?	7(50)	4(29)	3(21)
How much do you like to complete face-to-face surveys?		8(57)	6(43)

^*Non-responders not included in percentage calculations

Patient interview responses provided additional detail to the survey findings. Those who shared the ePGI results verbally or from the tablet with their doctor or nurse (n= 6 and 9, respectively) during the visit initiated the discussion, not the clinician. All patients stated they consider the areas they listed on the ePGI when making treatment decisions but are often unsure how the treatment will affect the areas identified. All patient participants also stated they would like to share the ePGI survey with clinicians even if uncertain if or how it might help.

Eight radiation oncology clinicians (four nurses and four physicians) participated in the study. They all described the typical visit as clinician driven, focused on assessing for common side effects of radiation and those related to the anatomical features within the radiation field. Patient quality of life was defined predominantly (n= 6) as minimal symptomatology and the ability to provide self-care. Two clinicians defined it as enjoying ones’ life, patient well being and lack of bother. Five clinicians stated they were familiar with patient reported outcome measures but had not seen them widely used in practice. All eight clinicians articulated the lack of time and possible disruption of patient flow as significant barriers to using patient-reported outcomes in practice.

The predominant theme voiced by clinicians after a demonstration of the ePGI was the potential value of ePGI use in starting a dialogue about quality of life issues, revealing infrequent or unusual effects of treatment and assist with symptom management. All clinicians felt the ePGI could be used in practice, provided the concerns of patient flow disruption, linking the ePGI to the patient’s electronic health record (EHR) and having adequate resources to respond to patient needs were addressed. All but one clinician stated patients did not typically share ePGI type of information with them during visits. Four clinicians were perplexed by some shared responses on the ePGI (Table 4), not understanding how areas listed were related to the patient’s treatment. For example, a patient listed a pet as most important and affected by cancer and its treatment. The clinician expressed “total befuddlement on how to explore the answer or if I would be able to do anything about it” and did not pursue it further. In the patient interview, the patient described their inability to walk or train the dog due to the patient feeling too tired and out of breath when walking, describing the impact of fatigue in their life.

Table 4.

Frequency of Categories of Areas Effected by Cancer and Its Treatment Listed by Patients in Step 1 of ePGI

Area	Frequency
Relationships (family, friends, pets)	12
Psychosocial (Stress, depression, embarrassment, worry)	12
External activities (work, civic, church, travel)	11
Body Image (appearance, embarrassment, ugly skin and incision)	6
Physical Symptoms (sleep, pain, drug effects, energy, sexual)	6
Everyday activities	2
Health	2
Longevity of life	1
Concentration	1

Discussion

This study is the first known use of an electronic platform for the Patient Generated Index, allowing ePGI completion from any internet capable device. Responses support the acceptability and usability of the final prototype at the point of care by patients and clinicians. While clinician concerns about integrating patient reported outcomes into the EHR are understandable, examples of strategies to do so are described in the literature (Bennett, Jensen, & Basch, 2012; Berry et al., 2011; Chung & Basch, 2015; Lobach et al., 2016; McCleary et al., 2013; Wagner et al., 2015). The study precludes generalization of findings, the perplexity felt by clinicians over patient responses demonstrates a lack of correspondence between how patients and clinicians view quality of life. Clinicians can explore the perplexities during the patient visit and may reveal previously unassessed effects of treatment..

The ePGI seems a simple and practical approach to restructure patient-provider interactions and enhance meaningful dialogue at the point of care. Moreover, using a tool such as the ePGI allows for the expression of patient voice and personalized care. Further study evaluating the effects of the ePGI on communication and decision-making are needed.