Usability of a CKD Educational Website Targeted to Patients and Their Family Members

Summary

Background and objectives

Web-based technology is critical to the future of healthcare. As part of the Safe Kidney Care cohort study evaluating patient safety in CKD, this study determined how effectively a representative sample of patients with CKD or family members could interpret and use the Safe Kidney Care website (www.safekidneycare.org), an informational website on safety in CKD.

Design, setting, participants, & measurements

Between November of 2011 and January of 2012, persons with CKD or their family members underwent formal usability testing administered by a single interviewer with a second recording observer. Each participant was independently provided a list of 21 tasks to complete, with each task rated as either easily completed/noncritical error or critical error (user cannot complete the task without significant interviewer intervention).

Results

Twelve participants completed formal usability testing. Median completion time for all tasks was 17.5 minutes (range=10–44 minutes). In total, 10 participants had greater than or equal to one critical error. There were 55 critical errors in 252 tasks (22%), with the highest proportion of critical errors occurring when participants were asked to find information on treatments that may damage kidneys, find the website on the internet, increase font size, and scroll to the bottom of the webpage. Participants were generally satisfied with the content and usability of the website.

Conclusions

Web-based educational materials for patients with CKD should target a wide range of computer literacy levels and anticipate variability in competency in use of the computer and internet.

Introduction

A goal of care in chronic disease management is enhanced communication with patients regarding the complexity of their diseases, importance of healthy lifestyles, and adherence to recommended treatments. Health information technology (IT) has offered new tools to facilitate communication about chronic diseases. Predialysis CKD is one chronic disease with multiple opportunities for use of health IT tools. Although numerous health IT applications have been developed and shown to be efficacious in various chronic diseases (1–6), the ultimate use of most health IT tools depends on the fluency of the target population with the internet through e-portals like the computer or mobile devices. The CKD population has a high preponderance of individuals who are underserved by the medical system, with lower socioeconomic status and educational levels. Although the use of the computer and mobile devices has increased in this portion of the population (7), there are few formal evaluations of how effectively CKD patients or their family members can use the internet to obtain relevant health information.

Ensuring patient safety in CKD requires effective communication with patients to make sure that they are aware of interventions with the potential for harm, because the interventions have not been tailored for use in this disease. In an attempt to increase patients’ awareness of the unique needs of CKD patients and the necessary modifications required in their medical care to avoid harm, the Safe Kidney Care (SKC) project conceived of and created a website designed to educate patients with CKD on disease-specific patient safety concerns. This study reports on usability testing of this patient safety website by patients with CKD and their family members.

Materials and Methods

Website Design

The SKC website (www.safekidneycare.org) was conceived and developed as an educational tool, with the intention of informing patients with predialysis CKD and their family members of common patient safety issues related to kidney disease. Disease-specific safety concerns were formulated based on the results of a national electronic (SurveyMonkey) survey to a broad range of US nephrologists (responders=142), who were asked to prioritize candidate patient safety topics and identify additional safety topics not considered. The final survey results were used to formulate a set of safety modules, which was built into www.safekidneycare.org. An expert panel assembled for the SKC study reviewed the safety modules for final approval, and these modules are shown in Figure 1.

Figure 1.

Safe Kidney Care website (www.safekidneycare.org). Main page of Safe Kidney Care website.

The website was designed using guidelines for color and content layout incorporated from the 508 Compliance and Disabilities Act and heuristic design (8,9). Emphasis was placed on ease of readability, with text written to target a seventh grade reading level. The primary placement of the “Safety Concerns” was in a circular distribution to avoid the implication of prioritization. The website was designed to limit the need to scroll up or down; however, on pages with a requirement for scrolling, the most important content was placed above the fold to ensure primary focus on the safety content. The ability to increase text size on every page and recognizable meaningful icons were also included (e.g., picture of a pill bottle to represent “Pills to Avoid”). A quick-launch side panel entitled “Learn More” included links to pages on the SKC website that provided users with basic information about the function of the kidneys, the meaning of kidney disease, the definition of GFR, and commonly used medication terms (e.g., BP, dialysis, and nephrologist). The website markup and content were constructed using HTML, with embedded functions programmed using PHP (version 5.3.5). Of note, dynamic images were not included because of an anticipated limited access to multimedia applications (e.g., Adobe Flash). The design was conceived with the goal of optimal use on all internet portals.

Study Participants and Testing

Between November of 2011 and January of 2012, a total of 12 participants underwent usability testing of the SKC website. A convenience sample of 10 volunteers with CKD (defined as an estimated GFR<60 ml/min per 1.73 m²) was recruited from the Early Renal Insufficiency clinics at the University of Maryland and Baltimore Veterans Affairs Medical Center in Baltimore, MD, along with two family members of enrolled participants, for a total of 12 participants. Feedback from the first eight participants was used to make website improvements, such as bulleting of text materials for ease of reading and removal of scroll bars; these modifications were used during testing of the final four participants. All materials and templates used for usability testing, included sample size, were adapted from guidelines obtained from www.usability.gov (10), and they were personalized for use in this study. The study was approved by the Institutional Review Board of the University of Maryland at Baltimore as an exempt protocol (Title 45, Code of Federal Regulations part 46) (11); therefore, written informed consent was not obtained, and protected health information was not collected.

Individual interview sessions were conducted in a quiet, private office setting on the University of Maryland School of Medicine campus, and they included the participant, a single interviewer, and a note taker to record participant actions and comments. Testing was conducted on a desktop personal computer with a 19-in, 1280×1024-pixel liquid crystal display monitor, keyboard, and mouse; the Internet Explorer web browser was used to access the internet. The interviewer directed the participant through a series of tasks, including (1) entering the URL into the address bar, (2) selecting the patient/family portal, (3) navigating through the website pages, (4) returning to the homepage, (5) navigating offsite to other websites, (6) returning to the SKC site, and (7) using an automated estimated GFR calculator (Appendix). A user satisfaction questionnaire was conducted at the end of each testing session. Each interview lasted up to 2 hours. All participants received a volunteer honorarium after completion of the testing.

Usability Testing Process

Usability is a systematic process that incorporates feedback into the iterative development of a program or technology (12). Two major types of usability assessments exist: usability evaluation and usability testing (10). The former is generally performed by expert evaluators (usability specialists), who apply a set of broad usability guidelines using inspection methods (such as heuristic evaluation) (13) to a program or website. By contrast, usability testing generally involves representative users (such as study participants), whose performance speed, success in completing tasks, and satisfaction with the program or website are recorded and incorporated into future modifications of the program or website (13,14). The usability process in this study was based on performance usability testing by representative users to assess potential concerns that could be addressed to improve website user satisfaction (10,12,14). Task completion success rates, adherence to dialogue scripts, error rates, and subjective evaluations were used to measure participant performance. A score was calculated for each task based on the following rating system. A task-completed score indicated that the participant completed the task independently and without difficulty. A noncritical error score (recoverable error) indicated that the participant used a different pathway to complete a task, causing minor problems to occur, and thus, that the participant had difficulty completing the task independently. A critical error score (fatal error) indicated that the participant was unable to complete the task independently.

Statistical Methods

No formal hypothesis testing was conducted in this analysis, and descriptive analyses were consistent with the applied qualitative study methods of usability testing. Continuous variables were presented with mean and SEM (SD). Binomial and categorical variables were presented as n (%). Frequencies of both critical and noncritical errors were presented per subject and then, per group designated by prior experience with computer use.

Results

Participant demographics are shown in Table 1. Over one-half of participants were older than age 50 years, were male, were non-Hispanic black, completed at least some college, and used the computer daily. Only one participant reported an annual household income of more than $100,000. Table 2 illustrates the task completion categories for all usability tasks by frequency of computer use. Of 252 total usability study tasks (21 tasks per participant), there were 55 (21.8%) critical errors. As a group, participants who used the computer daily had a lower rate of critical errors than those participants who used the computer less than daily or never (9.5% versus 46.4%). Median completion time of the usability testing protocol was 17.5 minutes for all participants (range=10–44 minutes), 13 minutes for daily internet users (range=10–22 minutes), and 35 minutes for nondaily internet users (range=19–44 minutes). Median completion time before website modification (participants 1–8) was 17.5 minutes (range=10–39 minutes); this time decreased to 16.5 minutes (range=12–44 minutes) for last four participants.

Table 1.

Participant demographics (n=12)

Participanta	Ageb	Sexc	Race/Ethnicityd	Incomee	Educationf	Computer Use
1	65+	F	AA/non-Hispanic	>$100K	7th to 12th	Never
2g	50–64	F	AA/non-Hispanic	$20K–$50K	SC	Daily
3	30–49	M	Cauc/Hispanic	$50K–$100K	CG	Daily
4	30–49	F	Cauc/non-Hispanic	$50K–$100K	TV	Daily
5	50–64	M	AA/non-Hispanic	$50K–$100K	CG	Daily
6	50–64	F	AA/non-Hispanic	DWA	7th to 12th	Occasionallyh
7	65+	M	AA/non-Hispanic	$20K–$50K	SC	Daily
8	50–64	M	AA/non-Hispanic	$20K–$50K	SC	Daily
9	65+	F	AA/non-Hispanic	$20K–$50K	SC	Occasionally
10g	30–49	M	AA/non-Hispanic	$50K–$100K	HS	Daily
11	50–64	M	AA/non-Hispanic	$50K–$100K	HS	Daily
12	65+	M	AA/non-Hispanic	<$20K	7th to 12th	Never

^aWebsite modification occurred after testing completion by the first eight participants.

^bAge categories: 30–49, 50–64, and ≥65 (65+) years.

^cSex categories: male (M) and female (F).

^dRace categories: African American (AA), Caucasian (Cauc), and other. Ethnicity categories: non-Hispanic and Hispanic.

^eAnnual household income categories: <$20,000 (<20K), $20,001–$50,000 ($20K–$50K), $50,001–$100,000 ($50K–100K), >$100,000 (>$100K), and did not wish to answer (DWA).

^fEducation categories: 6th grade or less, 7th to 12th grade, no high school diploma (7th to 12th), high school graduate/general equivalency degree (HS), technical or vocational school degree (TV), some college (SC), and college graduate (CG).

^gFamily member.

^hOccasionally is defined as computer usage≤6 days per week.

Table 2.

Task completion categories

Type of Error	All Tasksa (n=12)	Frequency of Computer Use
		Daily (n=8)	Less Than Daily or Never (n=4)
Total tasks	252	168	84
Critical errors	55 (21.8%)	16 (9.5%)	39 (46.4%)
Noncritical errors	16 (6.3%)	10 (6.0%)	6 (7.1%)
Easily completed	181 (71.8%)	142 (84.5%)	39 (46.4%)

^aTwenty-one tasks per participant.

Figure 2 shows the individual task performance results by each participant. Two participants (participants 5 and 10) completed all 21 tasks without a critical error, and only one participant (participant 5) completed all 21 tasks without any type of error. There was no difference in error rates after website modification (participants 9–12) compared with before website changes (participants 1–8). The tasks most commonly resulting in critical errors before and after website modification are shown in Table 3. Over one-half of all participants could not find information on “where to find information on tests that hurt my kidneys,” whereas six participants could not increase the font size of the website or find the website on the internet. The next most frequently occurring critical errors were when participants were asked to “scroll to the bottom of the page” or “point to a calculator of kidney function” (both n=5; 41.7%). A listing of all 21 tasks can be seen in the Appendix.

Figure 2.

Task classifications by participant. *Occasional/never computer users.

Table 3.

Most common critical errors

Critical Error Description	All Participants (%)	Participants 1–8a (%)	Participants 9–12 (%)
Where to find information on tests that may hurt kidneys	7 (58.3)	4 (50.0)	3 (75.0)
Increase the font size	6 (50.0)	4 (50.0)	2 (50.0)
Find this website on the internet	6 (50.0)	4 (50.0)	2 (50.0)
Scroll to the bottom of the page	5 (41.7)	4 (50.0)	1 (25.0)
Point to the calculator of kidney function	5 (41.7)	3 (37.5)	2 (50.0)
Where to learn more information about the kidneys	4 (33.3)	2 (25.0)	2 (50.0)
Go to the internet	3 (25.0)	2 (25.0)	1 (25.0)
Estimate kidney function using the calculator on website	3 (25.0)	2 (25.0)	1 (25.0)

^aWebsite modification was performed after testing completion by the first eight participants.

Participant responses to satisfaction questions after usability testing are shown in Table 4. Comments by the first eight participants related to dislike of the website were incorporated into website revisions, which were used for the last four usability participants as iterative testing. Median time to complete the satisfaction survey was 6 minutes (range=1–14 minutes). Overall satisfaction with the website was high; the most frequent comments offered were on the level of detail in website content and ease of website use.

Table 4.

Testimonials

Question	Comments
Overall impressiona	“Well laid out. Info[rmation] on other websites is very broad; this can give generalized answers to questions about CKD.”
	“Unique [website], because it opens a window of opportunity to information about the kidneys. All information is right there … and it is nice to view this at home and research information further. You can also share with family to explain kidney disease.”
Liked bestb	“Ease of information … topics about kidney disease, and I liked that the font size can be changed.”
	“I liked the scope of information provided.”
Liked leastc	“The left side panel was hard to read because of [text] size.”
	“Size of text should be larger.”
How to improve websited	“Enlarge left side panel font.”
	“Add a ‘Click Here’ for ‘Home Page’ button or link at the bottom of the pages.”
Anything missinge	“[There are] many Hispanics; they could use a Spanish version with CKD.”
	“Anyone relatively computer literate could navigate through the site. But, if someone is not used to using computers, they may need help.”
Describe to a family memberf	“A place where you can learn more about kidney disease … has links of where you can go to find more information about the nature of your condition or things that concern you.”
	“You need to go on a kidney website that will tell you everything you need to know about the function of your kidneys and how you can prevent problems.”

^aWhat is your overall impression of the Safe Kidney Care website?

^bWhat did you like best about the site?

^cWhat did you like least about the site?

^dIf you were the website developer, what would be the first thing that you would do to improve the website?

^eIs there anything that you feel is missing on this site?

^fIf you were to describe this site to a family member or friend in a sentence or two, what would you say?

Discussion

Health IT use in the delivery of healthcare is growing rapidly, both in the adoption of electronic health records in healthcare systems and in the use of the internet and mobile phones for delivery of information between providers and patients (6,15–22). Applications vary in their design based on the expected interactive relationships ranging from provider to core knowledge reference source, provider with medical record, provider to provider, and provider with patient. Providers using health IT to connect with their patients may improve overall patient–provider communication and also help to promote patient autonomy and empower patients’ with management of their own health. Providers using health IT tools to communicate with patients have reported improved medication adherence and appointment compliance for various ailments (4,23–26). Whether health IT applications are preferred by patients and whether a broader disease population can effectively use and benefit from such devices remain unclear. Demonstration of empirical use in a target population with a broad range in dexterity and cognitive abilities is necessary, and longitudinal studies are needed to track the effectiveness of such interventions on mitigating healthcare outcomes long term (27).

Nonetheless, although providers see most patients only several times per year, the internet is an escalating presence in the daily lives of most Americans. In 2009, the US Census Bureau estimated that 69% of households use the internet, and 77% of households have at least one family member that accesses the internet from some location (28). It is estimated that over 55% of Americans 55 years and older have internet capability at home (28). Until recently, however, the use of the internet for distribution of health information was underused, despite growing prevalence of several chronic diseases, including CKD (29). Although healthcare providers routinely use internet-based applications and databases in the treatment of patients, use of the internet to communicate with these patients is much more infrequent (30). Importantly, patients in the United States are becoming less likely to seek health information from sources other than their doctor than in past years (31), which is compounded by racial differences in health information access, because blacks are estimated to use the internet for health information 21% less frequently than whites (32–34). Although unequal access to information certainly contributes to the digital divide (for example, ethnic and racial minorities trail whites in broadband internet access) (32), the cause of these disparities seems to be because of more than just access to technology, and it is likely a complicated interface between socioeconomic factors and cultural practices (35). Fortunately, the divide may be partially narrowed with access to the internet on mobile devices, because ethnic and racial minorities are more likely to own a cell phone than whites (87% versus 80%) and also, the most active users of the mobile web (7). Therefore, versatility of tool design across various platforms will be instrumental in broad dissemination of educational materials to a diverse patient population. This finding highlights the need for development of effective and efficient communication tools between a provider and the patient, particularly in the field of CKD, which is facing a substantial predicted nephrology workforce shortage, despite an increasing prevalence of individuals with kidney dysfunction (36).

To our knowledge, usability testing of a website designed to promote patient safety among individuals with CKD has not been previously well-studied. Although it is proposed that individuals with CKD have a variable degree of health literacy, defined as the ability to understand basic health information, very little is known about this population’s knowledge and ability to use computers and technology effectively (e-literacy) (37). Our study samples from a socioeconomically diverse population with an equally diverse ability to perform required computer tasks in our usability testing. These results show the importance of addressing wide ranges of both health literacy and e-literacy when designing web-based educational tools for individuals with CKD (37,38). The findings suggest that there is a broad range of degree of success in use of the site and its intentions, which was largely expected given the heterogeneity of the study population. Key problem areas identified were mainly heuristical in nature (relating to website design and function), and they are applicable to any website development strategy seeking to optimize users’ ability to navigate a site, regardless of content. This study provides a good framework from which to base future efforts to construct websites with educational information directed at patients.

Limitations relate to the qualitative and descriptive nature of the study and difficulty in hypothesis testing. The sample size, although limiting hypothesis testing, is consistent with the range in the number of participants recommended for usability testing, with no more than eight individuals, and a smaller second iterative round of testing evaluated the efficacy of the changes made as a result of initial feedback (39,40). The website tested was specific in its theme, which may limit the generalizability of the findings to other websites designed for kidney disease patients. Nevertheless, usability testing is a valuable tool to evaluate the real world use of the growing number of health IT applications that have become available to individuals with kidney disease. Furthermore, we did not inquire if participants were able to access the internet in a place where they would feel comfortable using it for CKD patient safety inquiries, which is an important component of information-seeking practices. Finally, despite modification of the website to address frequent critical errors, some participants remained unable to complete the requested tasks. However, we anticipate that critical errors will persist regardless of the degree of website modification, in part because of variability in e-literacy of participants. Future studies will need to evaluate efficacy of such a website in its ability to improve outcomes.

Websites intended for specific disease populations must be dynamic to both incorporate new and changing information relevant to patients and adapt to the specific profile of the disease’s demographics. Some might view a report such as this study, which evaluates web usage, as anachronistic given the rapid progress in mobile technologies and device applications. However, it is important to keep in mind the age, socioeconomic status, and behavior patterns of patients for whom the technologies are intended. Use of traditional websites may become less prominent with the growth of social media and personal device applications, which allow instantaneous receipt of information and in many cases, interaction and dialogue. However, the population with CKD is also dynamic, and as the disease population adds younger members, a cohort effect with regard to technological aptitude will likely transform the reception of this group to newer technologies. With this rapidly changing health IT environment, assessment of the target population’s ability to access and successfully use these tools will become increasingly important.

Disclosures

J.C.F. received prior research funding from Amgen, Inc., and honoraria from Sandoz, Inc. and Amgen, Inc.

Appendix. Task Key

Task	Task Description
1	“Go to the internet.”
2	“Take a look at the website inscribed on your bracelet (or card) and find this website on the internet.”
	If the user cannot read the inscription on the bracelet, provide the business card with the website address instead.
3	“Please click on the Patient/Family link.”
4	“On this page, you will find a box where participants in a future research study will be able to enter a four-digit ID engraved on a medical alert bracelet similar to the sample bracelet you are using for this test. Looking at this page, point to where you think this ID is supposed to be entered.”
5	“Please locate the four-digit ID on the medical alert bracelet (or card) and read the ID back to me.”
	If the user cannot see the ID, provide the business card with the mock ID instead and repeat the task.
6	“Some visitors to this website will not be in a research study and will not have an ID to enter. Please point out on this page how these visitors can still enter this website.”
7	“Now, look again at the four-digit ID on your sample bracelet (or card) and enter this ID into the box on this page and then click on Submit.”
8	“You stated that you could find information about ___ (see #9, item 1 on list). Please, go ahead and see if you can find that information.”
9	“Now, please go back to the homepage.”
10	“You also stated that you might find information about ___ (see #9, item 2 on list). Please, go ahead and see if you can find that information.”
11	“If you thought the words on this page were too small, show me how you might increase the size of the words.”
12	“Now, please go back to the homepage.”
13	“On this page, there are two places where you can find a calculator to estimate your kidney function. Please point to those two places.”
14	“Please click on one of the links to the calculator.”
15	“Visitors to this website can estimate their kidney function by using this calculator. This calculator requires the following information: race, gender or sex, age, and the results of a routine blood creatinine test provided by a healthcare provider. Please enter the mock information on this sheet of paper (hand user the paper) into the calculator and then click on Calculate to obtain the results. When finished, tell me your results.”
	After participant finishes, probe for any questions or comments.
16	“Looking at the side panel on the left, please click on the link to a page that talks about safety concerns with fluids.”
17	“Please scroll to the bottom of the page to view the rest of the information on this page.”
18	“At the bottom of this page are three links to other websites that offer information. Please click on one of these links.”
19	“Please return to the Safe Kidney Care website.”
20	“If you were worried that a test or treatment might hurt your kidneys, where on this website would you go to find more information?”
21	“If you wanted to learn more about the kidneys, where on this website would you go to find more information?”

ID, identification.