Usability Evaluation of the Community Use Interactive Personal Patient Profile -Prostate (P4) Decision Support System with African-American Men: A Think – Aloud Study

Abstract

Background

The Personal Patient Profile-Prostate (P4) program is an interactive web-based decision support system that provides men with localized prostate cancer customized education and coaching with which to make the best personal treatment decision. This study assessed functionality and usability of the P4 program and identified problems in user-computer interaction in a sample of African-American men.

Methods

Usability testing was conducted with twelve community dwelling African-American adult men. The health status of participants was not known or collected by the research team. Each participant worked with the P4 program and provided simultaneous feedback using the “think aloud” technique. Handwritten field notes were collated and assigned to three standard coded categories. Aspects of P4 program usability was made based on common issues in the assigned categories. Summary statistics were derived for types and frequency of usability issues noted in the coded data.

Results

Twelve participants reported a total of 122 usability comments, with a mean of nine usability comments. The most common usability issue by participant was completeness of information content, which comprised 53 (43%) of the total issues. Comprehensibility of text and graphics was second, comprising 51 (42%) percent of the total issues.

Conclusion

This study provided initial inventory of usability issues for community African-American men that may potentially interfere with application of the P4 system in the community setting and overall system usability, confirming the need for usability testing of culturally appropriate Internet-based decision support system before community application.

INTRODUCTION

Localized prostate cancer (LPC) is a common malignancy in the United States. Each year, approximately 218,000 men are diagnosed with LPC and are faced with the task of making a decision among several treatment options.^{1, 2} Extant treatment options include cyrotherapy, hormonal therapy, radical prostatectomy, brachytherapy, external beam radiation therapy, and watchful waiting. With the exception of watchful waiting, the treatment options often have detrimental effects on urinary, bowel and sexual functions. Because little research supports any one treatment as conferring better survival advantage over another, arriving at a decision can be particularly difficult. ^2–5

With the uncertainties surrounding treatment modalities, how do men with early stage prostate cancer make treatment decisions? Typically, physicians present their patients with a specific set of treatment options based on medical considerations and patients’ general health status. Patients invariably consider the impact treatment will have on their quality of life. Consequently, they want to integrate unique personal preferences and biases vis-à-vis specific treatment options into the decision process.^{6, 7}

The Personal Patient Profile-Prostate (P4) program - an innovative application of health informatics technology provides a fresh, web-based approach to customized patient education and support.⁸ The P4 assesses preferences for decisional control, integrates personal factors, and current symptoms and interactively guides men with LPC through a series of questionnaires and delivers multi-media interaction modeling and decision support that are customized for the patient’s preferences. A randomized clinical trial is ongoing to test the efficacy of P4 with regard to reducing decisional conflict and enhancing preparedness of a man with LOC to make an informed and personal treatment choice. A version of P4 program tailored for race/ethnicity and community access is now available.

African-American men bear disproportionate burden of prostate cancer morbidity and mortality. They have a greater likelihood to develop prostate cancer and are twice as likely to die from the disease compared to Caucasian men.^9,10 African-American men are less likely to be screened for prostate cancer; more likely to be diagnosed at an advanced stage; and less likely to receive aggressive treatment when diagnosed.^11–15 Research exploring issues of ethnicity and knowledge about prostate cancer suggest that African American men may have lower levels of prostate cancer knowledge than men from other ethnic groups, possibly attributable to low overall levels of education.^16,17

Given the notable disparities in prostate health among African-American men, it is imperative that access to interventional resources are available to enable individuals diagnosed with LPC to not only understand the potential risks, benefit and limitations of treatment modalities, but also articulate their own priorities and personal factors relevant to the treatment decision. The community-use P4 program is designed to promote informed decision-making about prostate cancer treatment and also prepares men for including personal factors in shared decision-making with their healthcare team.

Usability is a key in the acceptance of a new technology by its users. Usability testing is regarded as a major technique to be used by developers of novel information systems.¹⁸ Usability testing involves subjects who are representative of a target population to evaluate the degree to which the technology satisfies basic usability criteria as the subjects perform representative tasks.^{19, 20} The primary objectives of our study were assessment of P4 program functionality and usability, and identification of problems in user-computer interaction. In this paper, we report the results of an evaluative study of community-use P4 usability with African-American men.

STUDY DESIGN

This cross-sectional, descriptive evaluation focused on three features of the P4 program - the navigational architecture, information content, and text and graphics inform the design of this usability study.

Setting and Participants

Our study received exempt status approval from the Fred Hutchinson Cancer Research Center, Cancer Consortium Institutional Review Board. The evaluation team included the principal investigator (a Caucasian woman) and two male graduate research associates of Hispanic and African heritage respectively. We conducted the usability testing sessions in three Pacific Northwest community settings. Two sessions were held at fraternity group meetings and the third session was conducted at a church brotherhood meeting. Group leaders were contacted by one of the authors (CJ) and subsequently arranged a mutual date and time for the P4 usability testing. Potential participants were aware of the evaluation prior to the group meeting. All group members over the age of 18 were eligible to participate. Knowledge of prostate cancer or personal and familial experiences with the disease were not inclusion or exclusion criteria for this study. Twelve men volunteered to participate in the usability study sessions: eight men in groups of four and two respectively at the fraternity group meetings, and four men at the church brotherhood meeting.

Usability Testing

The testing sessions were divided into two phases. First, the participants were oriented to the purpose of the P4 program. Instructions as to how to use the P4 program on touch screen notebook computers were articulated clearly to the participants. Second, we used the think-aloud method to capture participants thoughts as they work through the P4 program. This method is particularly useful for evaluating a system’s design on usability flaws and is frequently used to gather information about a system’s usability in testing computer systems with potential end-users.^21–23 For instance, the think-aloud method has been used widely in nursing research to discern clinical decision-making processes, investigate clinical reasoning and strategies that nurses use when planning nursing care.²⁴ Think-aloud method has also been used in cognitive psychology as a method for understanding individual development of thought, and to address the cognitive processes encountered during problem-solving respectively.^25–27 The advantage of think-aloud is that it makes it easy for the study participants to use their own language and own words in the usability testing sessions.²⁸

During the usability testing sessions, participants were encouraged to verbalize freely their thoughts as they performed computer prompted tasks. When a participant fell silent for a long period (i.e., 20 to 30 seconds), a variety of simple prompt questions were made to encourage the subject to carry on thinking aloud. Examples of prompt questions include the following: What are you thinking? Was that question easy? Was there something that was hard to understand in that screen? What did you think when you saw that screen? Did that make sense? How were the instructions on the screen? Did the instructions presented on the screen seemed useful? More general queries included: “Do you think other men in the community will be able to use the program? What would you recommend changing about the program? Participants’ verbal responses were handwritten by the study researchers. The verbal data were coded and analyzed to discern usability issues.

Measures and Analysis

The verbal data from the think-aloud process informed the usability study. Each screen on the P4 program was assigned a code number. Participants’ verbalizations were noted for each screen. When two or more issues were verbalized for a particular screen by participants, or verbalized more than once by a participant, the researchers acknowledged a note of potential usability consideration. Think-aloud data were transcribed by two researchers (CJ, JP-A). The coding of the transcribed data to usability issues was done by one investigator (CJ) using standard coding scheme adapted from previous studies.^{28, 29} The coding categories were: “problems in navigation,” “completeness of information content,” and “comprehensibility of text and graphics.” Extrapolations about the usefulness and ease of use the P4 program were based on emergent issues in the assigned categories. Summary statistics were also derived for types and frequency of usability issues.

RESULTS

The P4 program contains forty-six interactive screens. There are eight screen types in P4 program. Figures 1 and 2 are representative screen shots of screen types. There are thirteen video clips with associated screens. Table 1 shows summary descriptive statistics for the data from the coding scheme by screen types. Fifty percent of usability issues were associated with the “influential factors” screens, with participants suggesting eight textual changes to question stems in the usability categories. Completeness of information and comprehensibility of text and graphic both constitute over eighty percent of the usability issues per screen type. Table 2 presents descriptive statistics of usability issues. The twelve participants identified 122 usability issues with an average of nine usability issues per participant. The most common usability issue by participant is completeness of information content, which comprised 53 of 122 or forty-three percent of the total issues. Comprehensibility of text and graphics was second, comprising 51 of 122 or 42 percent of the total issues. Table 3 depicts several sensitive issues that might potentially impact overall usability of the P4 program. Specific usability issues are elaborated upon below.

Figure 1.

Influential factors screen shot

Figure 2.

Treatment decision screen shot

Table 1.

Usability issues by survey screen type

Screen Shot Type	Number of screens	Number of suggested textual changes	Problem with navigation	Completeness of information Content	Comprehensibility of text & graphics	Total (No) & (%)
Program instruction screen	5	-	2	1	1	4 (9%)
Transition Screen	5	-	-	2	-	2 (4.3%)
Racial & demographic screen	5	2	2	2	2	6 (13%)
Influential factors screen	17	8	3	13	7	23 (50%)
Treatment Decision screen	6	1	1	1	4	6 (13%)
Program information screen	2	1	-	-	1	1 (2.1%)
PCA information screen	3	1	-	-	2	2 (4.3%
Current symptoms screen	3	1	-	2	-	2 (4.3%)
Total	46	14	8 (17%)	21(46%)	17 (37%)

Table 2.

Usability issue type by participants

Type of Usability Issue	P1	P2	P3	P4	P5	P6	P7	P8	P9	P10	P11	P12	Total
Problem with navigation	3	-	1	-	1	2	1	2	3	2	1	2	18(15%)
Completeness of information content	5	8	5	8	4	7	8	-	-	2	5	1	53(43%)
Comprehensibility of text & graphics	9	7	3	3	5	6	8	2	1	3	2	2	51(42%)
Total usability issues	17	15	9	11	10	15	17	4	4	7	8	5	122

Table 3.

Emergent issues from usability testing

Characterization of Issue	Think–aloud data	Usability problem
Inadequate knowledge of LPC treatment modalities and side effects	The problem I am having with this question is that it makes the assumption that I know what these treatment options are about (P2) “Is this for someone already diagnoses?” (P11). “Helpful question if I am knowledgeable about the treatment” (P4). “Does not tell me which effect of treatment, gives treatment instead” (P6)	Socio-technical
Inadequate knowledge of LPC symptoms	“Something to inform you about these symptoms. I don’t know if I have bladder dysfunction” (P6) “Why is this being brought up?” (P1). “Is bladder and urinary function a familial problem?” (P12). “Are these things that would bother me if I leave it alone?” (P1)	Socio-technical
Sensitivity about age - related issues	“For African-American men we don’t live beyond the 70s” (P1) “My father had prostate cancer at eighty-five years of age …so important question (P6). “I am still considering. Nervous about my age. I am still considering?” (P2). “Age? Explain what you mean” (P11). “Ridiculous question. I could be hit by a bus. What’s the importance of the question?” (P7). “This question answer the previous age question” (P4)	Socio-cultural
Issue of literacy	“I have been working toward my degree…now I would have to answer high school graduate because I don’t have a college degree yet; but it does not describe who I am.? (P1). “I am computer literate” (P7) “Every word needs to be evaluated and the question to ask is ‘ is there a simple way to say this’ [it looks higher than 6th grade]” (P1)	Socio-cultural
Issue of privacy	“Are you gaining data on a number of individuals you are talking to?” (P6). “Taking information about me that I am worried about” (P4).	Socio-cultural

Problems in Navigation

Navigation issues stem from imprecise navigational cues and technical issues. Participants experienced navigational problems with the program instruction, demographic, treatment decision and influential factors screens. Even though the second instruction screen provides succinct instruction with regards to forward and backward movement by means of directional arrow icons, a participant asked on the third screen: “Just touch here?” (P1). On the subsequent screen shot, a participant asked: “How do I change option?”(P3). On the eighth sequenced screen, one participant required prompting by an investigator to make a selection, while another participant asked: “So I go ahead and hit next?” (P1). Seven of twelve participants experienced slight confusion about how to play a video clip linked to one of the influential factors screen shot. One participant asked: “Does it play automatically?”(P10). Another participant hit the “next” icon before realizing that he had to play the video clip (P5). One of the PCA information screen with links to prostate cancer Internet sites was also a sources of navigation problem. Two participants (P6, P7) seemed not to understand the instruction to click on the links. Another participant appeared confused as to what to do with the information on the linked pages (P10).

The major technical issue during the testing session was with accessing the video clips. Touch screen laptops with wireless air-cards used for the testing session provided adequate connectivity with no difficulty accessing the online P4 program. However, during some of the sessions, the video streams played at exceedingly slow speed or played without sound. Participants expressed the need for faster access and run-time. However, participants who played video clips liked the customized questions to bring to the physician. One participant said that: “Overall it is very good preparation beforehand with pros and cons of treatment and who you would talk to” (P2).

Completeness of Information Content

Completeness of information content refers to whether the program presents information that enables participants to adequately perform tasks or respond unambiguously to questions. A usability issue is flagged if a participant draws erroneous conclusion or expresses disquiet about the question’s connotation. Several participants did not quickly respond to all P4 questions as evidenced by the disquiet expressed over demographic screens soliciting information about computer usage and educational levels. Examples of the most common verbal utterances were as follows: “Why is the question important?”(P3), “Why do you want to know?”(P4), “Why do I need to complete this?”(P3), and “What is the question trying to get at?” (P11). Participants (P5, P10) suggested textual changes to two questions as follows: adding “some college” to a question about level of education; changing the stem of another question to read “how much influence do the following people have in your decision for treatment choice?” instead of “How much influence might the following people have as you consider treatment choices?” One participant stated that the program used “…too many words to ask a simple question” (P9). Half of the twelve participants (P2, P3, P4, P5, P7, and P11) expressed ambivalence with regards to how knowledge of a famous individual afflicted with prostate cancer would influence their treatment decision: “Not a good question…is it someone I respect?” (P4). “I really don’t care about those kind of people…they wear pants the same way I do?” (P7).

The significance of age in informing treatment decision making was an issue for several participants. Background knowledge of prostate cancer treatment modalities was cited by participants as necessary to respond to queries about treatment choices adequately: “Is this for someone already diagnosed?” (P11), “I don’t know how to answer that?” (P1), and “Helpful question if I am knowledgeable about treatment” (P4). Several participants also stated their lack of knowledge about the role that prostate cancer associated symptoms and treatment side effects (i.e., sexual dysfunction, bowel movement, urinary incontinence, etc.,) would have on treatment decisions. Typical responses were as follows: “Something to inform you about these options…I don’t know if I have bladder dysfunction?” (P6), “These words ‘bladder’ and ‘urinary’ are they synonymous?” (P10), “This is one I have never hard about [bowel function] … ” (P1), “I would like to know what you mean by bowel function?”(P2), and “Are these things that would bother me if I leave it alone?” (P1). A vexing issue for some of the participants was the inability to make an informed treatment decision. As one participant puts it: “Up to this point nothing has prepared me to make a decision” (P2). Another participant states: “Now that the program will tell me what to do…what is it suppose to do for me? Is it going to tell me?” (P4).

Comprehensibility of Text and Graphics

Comprehensibility of text and graphics refers to whether the screens display of text and graphics are easily understood or confusing. Overall, participants found it easy to understand the textual content on many of the screens. However, there were some questions with the treatment options and influential factors screens for which the participants did not grasp all the content and terminology: “Is this the same question?”(P4, P11) two participants asked; and “A lot like the previous screen …I don’t know what has changed” (P8). Participants also suggested textual changes questions about treatment decision and work activities: Use “‘understand’ instead of ‘know’ which treatment” (P7).

All the participants flagged one “fill-in-box” screen with usability concern. Sample comments were as follows: “So I put a year in the box?” (P1); “What do I put in here?” (P11); “Make me pause” (P3), and “Type age…instruction is not clear” (P6). Also, participants did not find comment boxes particularly useful. Several participants suggested avoiding the use of screen shots that would require the participant to type in personal data, comment or feedback. One participant suggested keeping comments at the very end of the program: “Feedback at the end…otherwise just forget it” (P7). Another participant suggested a textual change for program feedback as follows: Do you have any recommendation or concerns? If yes, give brief sentences of what you’re concerned about” (P12). Participants were unimpressed with how prostate cancer mortality and morbidity statistics was graphically present: “Graphics does not have much impact…will push me toward watchful waiting” (P1); “Is it adjusted for race? Since the program knows my race, is it tailoring these statistics accordingly?” (P2); “Feel like reading science. You lost me on this one” (P6); “Difficulty trying to figure out. Don’t know what it is. I will say forget it” (P7), and “Took time to read” (P10). Several participants found the presentation of patient personal influential factors awkward. They preferred presentation of personal influential factors in a format indicating priority level for the participant.

DISCUSSION

Recent evidence indicates that African-American men have lower levels of knowledge and awareness about prostate cancer.^12–14 African-American men may possibly be disadvantaged in terms of integrating medical knowledge with personal values and preferences to promote informed decision-making about treatment modalities, or to participate in shared decision-making with their physicians. In light of the high-risk of prostate cancer among this group, healthcare interventions, based appropriately on the needs and concerns of African-American men are needed to prepare these individual for shared decision with their healthcare providers. The P4 program may be the first prostate cancer treatment decision support system designed for community use in this high-risk population. A Cochrane Collaboration review of 200 decision aids acknowledged their usefulness in reducing decisional conflicts, and synching patients’ values and with therapeutics decision-making.³⁰ However, we are unaware of any current health-related decision aids designed specifically for African-American diagnosed with localized prostate cancer. Thus, our aims were focused on usability issues that might be unique to this population and that might facilitate the (re)designing of a culturally appropriate decision support system.

Our study adopted the think-aloud method, a qualitative method used to collect verbal data about the cognitive and linguistic processes in a subject’s mind during performance of designated task. Our analysis of the think-aloud data on navigation architecture, completeness of information content, and comprehensibility of text and graphics of the P4 program revealed a total of one hundred and twenty-two usability concerns. Navigation problems however, were minimal, and were related to misunderstanding of directional icons or imprecise instruction cues. The participants generally expressed favorable comments about the comprehensibility of the text, but at times were confused with some terminologies or failed to discern difference in the content of sequenced questions. The presentation of graphics and race appropriateness of prostate cancer statistics, cumbersome presentation of participant influential factors, and clustered screens were related usability concerns.

The think-aloud method also revealed socio-technical issues such as the inability of participants to fully appreciate the P4 program interaction modeling and decision support matrix because of lack of medical knowledge of prostate cancer symptoms, treatment options, and associated side-effects. Perhaps this concern could be attributed to the fact the participants in our study were not prostate cancer patients and therefore may not have considered any of these terms or issues. Certain participants, particularly the younger men, may not have been familiar with terms related to a prostate cancer diagnosis as they may have never encountered or even considered such a diagnosis. While these men may once have represented a group unlikely to face a prostate cancer diagnosis, given the increase in screening rates, the younger African-American man may be exactly the type of man who will require such customized treatment decision support in the future. A basic premise of the P4 program is that the end-user would have some opportunity to acquire relevant medical terminology from their health care providers. Rethinking of this assumption is imperative. And so is rethinking of appropriate health literacy level of the end-user of the P4 program. This may be necessary because participants highlighted fourteen questions for textual revision to improve comprehensibility. African-Americans mistrust of research because of historical antecedents may be at the bottom of the privacy of information concern expressed by several participants.

Invariably, the essence of usability testing is to make appropriate modifications to the technology that is the object of study. Usability of a novel technology is determined by the effectiveness, efficiency and satisfaction with which users achieve specific sets of tasks in a particular environment.^{18, 31} Effectiveness refers to how accurately and completely users of a system can achieve specified goals; efficiency refers to the resources utilized in relation to the accuracy and completeness of goals achieved, and user satisfaction refers to the ease and acceptability of the system to its users. The user-interface issues identified in our study are not intractable. The issues with program navigation will be resolved with minor technical changes. For instance, a navigation video demo at the beginning of the program would help with the navigation issues as would activation of the voice-over feature as a default option on the program. More visibly presented video clip buttons, with unambiguous instruction cues; aesthetically presented graphics and statistics; and less clustered screen shots would improve the representation structure of displayed information and greatly increase ease of functionality of the P4 program.

Several study limitations must be acknowledged. The study recruitment method potentially may have biased the evaluation results. Twelve participants where recruited from fraternity and brotherhood groups. It is possible that these participants possess or lack characteristics present or lacking (for instance, computer savvy, high literacy level, etc.,) in other African-American male groups not recruited for this study. Our observations led us to conclude that the computer skill levels in these groups were varied. For usability studies, a small number of participants are deemed sufficient for determining the major usability issues.^{19, 20} Our study participants identified 122 usability issues. However, because of significant redundancy in the issues identified by the participants, the usability study data indicate an appreciable range of possible feedback for our decision–support system.

CONCLUSION

This evaluative study is a step in an ongoing effort to develop a community-based, culturally appropriate decision support system for men newly diagnosed with prostate cancer. The merit of this study is that it has provided an initial inventory of usability issues and identified socio–technical and cultural issues for African American men. Robust iterative usability studies would validate changes made to the system design on the basis of the issues highlighted in our study. Furthermore, such studies must be conducted in larger communities to evaluate potential issues missed by this study and to also assess the relative salience of the socio-technical and socio-cultural identified in our study on system usability. Carefully designed iterative evaluation studies would foster the ultimate development of a culturally appropriate decision support system that African-American men could embrace. It is anticipated that our study is a step in that direction.