Heuristic Evaluation of Online COPD Respiratory Therapy and Education Video Resource Center

Abstract

Purpose: Because of limited accessibility to pulmonary rehabilitation programs, patients with chronic obstructive pulmonary disease (COPD) are infrequently provided with patient education resources. To help educate patients with COPD on how to live a better life with diminished breathing capacity, we developed a novel social media resource center containing COPD respiratory therapy and education videos called “COPDFlix.” Methodology: A heuristic evaluation of COPDFlix was conducted as part of a larger study to determine whether the prototype was successful in adhering to formal Web site usability guidelines for older adults. A purposive sample of three experts, with expertise in Web design and health communications technology, was recruited (a) to identify usability violations and (b) to propose solutions to improve the functionality of the COPDFlix prototype. Each expert evaluated 18 heuristics in four categories of task-based criteria (i.e., interaction and navigation, information architecture, presentation design, and information design). Seventy-six subcriteria across these four categories were assessed. Quantitative ratings and qualitative comments from each expert were compiled into a single master list, noting the violated heuristic and type/location of problem(s). Results: Sixty-one usability violations were identified across the 18 heuristics. Evaluators rated the majority of heuristic subcriteria as either a “minor hindrance” (n=32) or “no problem” (n=132). Moreover, only 2 of the 18 heuristic categories were noted as “major” violations, with mean severity scores of ≥3. Conclusions: Mixed-methods data analysis helped the multidisciplinary research team to categorize and prioritize usability problems and solutions, leading to 26 discrete design modifications within the COPDFlix prototype.

Background

Chronic obstructive pulmonary disease (COPD) is a preventable and treatable disease, caused by a mixture of chronic bronchitis and emphysema that leads to prolonged coughing and dyspnea (i.e., shortness of breath) exacerbations that often go unreported.¹ An estimated 15 million U.S. adults have been diagnosed with COPD, and it is now the third leading cause of death in the United States.^2,3 Globally, it is estimated that COPD will become the third leading cause of death by 2030.⁴ COPD commonly affects older adults over the age of 50 years who have a history of smoking.⁵ Limited access to pulmonary rehabilitation programs prevents many COPD patients from obtaining patient education regarding effective self-management.^6–8

To improve online patient education resources for COPD patients, who often experience cognitive and attentive impairments,⁹ a multidisciplinary group of researchers, clinicians, information technology specialists, and multimedia designers created “COPDFlix” (www.copdflix.com) using an adapted version of the Web site developmental model for the healthcare consumer.¹⁰ COPDFlix was designed as a low computer-literate social media resource center providing online access to evidence-based COPD patient education videos.^11–13 Our Web platform used a broad and shallow navigational architecture to upload 167 brief videos into distinct COPD patient education categories. Videos were embedded using YouTube.

Prior to testing COPDFlix with patients, we assembled a small panel of usability experts to identify usability violations against a list of recognized Web design principles for older adults (i.e., “heuristics”).¹⁴ During a heuristic evaluation, experts identify system violations and assign each a severity score that is accompanied by a qualitative description of how to fix the problem. Two to five evaluators with usability expertise can detect 75% of usability problems.^15,16

Materials and Methods

Three usability experts in Web design and health communications technology were recruited as a purposive sample to conduct the heuristic evaluation. Each evaluator completed eight typical user tasks (Table 1), such as locating videos, posting comments, and contributing to online discussion threads. After completing the tasks, experts completed a survey that assessed 18 task-based heuristic criteria for evaluating Web sites for older adults.^17–19 We administered 76 subcriteria items related to four main categories: (1) 20 items on interaction and navigation (i.e., the way users work with the site), (2) 18 items on information architecture (i.e., organization of links and hierarchy of content categories), (3) 20 items on presentation design (i.e., graphical interface and visual elements), and (4) 18 items on information design (i.e., preparation of communication products to achieve specified performance objectives). Experts assigned ratings of all subcriteria items on a 4-point scale (where 1=“no problem” or satisfies the heuristic, 2=“minor hindrance” or possible issue that will probably not hinder the user, 3=“serious problem” that may hinder the user, or 4=“task failure” preventing the user from going forward). Each expert was also asked to provide explanatory qualitative commentary for each subcriteria rating.

Table 1.

Description of Eight Representative Tasks Completed by Each Usability Expert

TASK NUMBER	DESCRIPTION OF TASK
1	Create a new log-in that will allow you to sign into the Web site.
2	Review the video category labels on the left-hand side of the computer screen in blue and indicate which (if any) is unclear.
3	Play the “deep breathing” video under the blue “stress management” category label.
4	Post a one-sentence comment underneath the video indicating whether you found it informative.
5	Click on the “Talk to Other People with COPD” category label at the bottom of the page and post a one-sentence comment on the discussion topic of your choice.
6	Click on one of the videos that is “Recommended for You” and play the video.
7	Imagine you have just found this Web site online at your home or at the library. For the next 5 min or so, please explore the Web site however you would like.
8	Sign out of the Web site.

COPD, chornic obstructive pulmonary disease.

Data Analysis

Quantitative ratings and qualitative comments were compiled into a single master list, noting the violated heuristic and type/location of problem(s). Duplicate usability problems from multiple evaluators were grouped and treated as a single unit. SPSS version 22.0 software (SPSS Inc., Chicago, IL) was used to compute the mean (±standard deviation [SD]) severity rating scores for all 76 subcriteria. Mean severity scores of ≥3 and those with SD error bars exceeding 3 were categorized as “major” violations needing solutions. Heuristics with mean severity scores of >2 but <3 were described as “moderate” violations in need of further consideration. “Minor” violations were characterized by mean severity scores of ≤2 and were addressed if experts noted the violation as likely to hinder the user from successfully utilizing COPDFlix. Qualitative data were coded using Atlas.ti version 7.0 software (ATLAS.ti Scientific Software Development GmbH, Berlin, Germany) with a process of deductive constant comparison analysis,²⁰ which organized potential design solutions by each of the four heuristic categories.

Results and Discussion

When considering data from each of the three experts on the 76 subcriteria, we found that most were rated as a “minor hindrance” (n=32, 14%) or “no problem” (n=132, 57.9%). Only 2 of the 18 heuristic categories earned mean severity scores of ≥3 (i.e., “major” violations). However, evaluators did identify 61 violations across the four categorical domains (information design, n=18; interaction and navigation, n=17; information architecture, n=14; presentation design, n=12). The three most frequently violated heuristics were “Use the users' language; minimize jargon and technical terms” (n=8), “Include a site map and link to it from every page” (n=6), and “Make obvious what is clickable and what is not” (n=6) (Fig. 1). The heuristic for using nontechnical language was violated to a “moderate” degree (mean, 2.06; SD=1.16), whereas lack of a site map was noted to be a major violation (mean, 3; SD=1.32). Although lack of a site map reached the major violation threshold, it was noted by all evaluators that a site map was not likely to improve users' navigation of COPDFlix, given that the prototype already possessed a topic-delimited Web architecture. Experts did propose modifying video category labels to be written in plain language, however (e.g., “use ‘breathe’ instead of ‘exhale’”). The heuristic on “clickability” received a mean severity score of <2 (mean, 1.90; SD=1.04), indicating a minor violation. Although the raw number of violations on the “Provide feedback in other modes in addition to visual” heuristic was low (n=3), its mean severity score was high (mean, 3.67; SD=0.58). Experts explained that adding brief text-based descriptions of video content would reduce the seriousness of this major violation. The presentation design category had the fewest violations, suggesting that visual elements were appropriate for the intended audience. Overall, evaluators rated COPDFlix as possessing mainly inconsequential violations.

Fig. 1.

Severity of usability violations (n=61) by heuristic category. SD, standard deviation.

Quantitative severity ratings and qualitative technical feedback from experts helped prioritize usability problems and design solutions within the COPDFlix prototype. Of the 61 violations identified, 52 (85.2%) were addressed during the Web modification process. Table 2 lists expert comments used to inform 26 separate design modifications. Table 3 provides examples of identified violations along with descriptions of associated design solutions. One important limitation was lack of data from actual COPD patients. Think-aloud usability testing with COPD patients will follow the heuristic evaluation to provide complementary data that will inform more patient-centered improvements to COPDFlix.

Table 2.

Representative Qualitative Commentary from Expert Evaluators by Heuristic Category

HEURISTIC CATEGORY	EXPERT COMMENTS
Interaction and navigation	“The videos do not have meaningful captions. On YouTube, there's a way to add captions to videos, but this requires the user to import an accurate transcript. For a user to turn on captions, they have to click the 'CC' button on the video player, which then gives them several options, some of which aren't relevant…Manually captioning each video is a lot of work, but it might be beneficial for this audience. Another possibility would be to superimpose the captions on the video itself so that they are always on.” E1 H7 S1
	“For the majority of videos, I saw no captioning or alternative text. The COPD explanation animation on the Welcome page seemed to be more oriented to students or health professionals than patients—the language completely turned me off.” E2 H7 S1
	“While having links stay the same regardless of clicking makes for a cleaner looking site, older adults might need the visual reminders of changing link color so they know what they've visited in case they've forgotten.” E1 H1 S3
	“For the main links, yes, they change visibly. But for all other links on the page, like the sign-in link, recommended videos, links at the bottom, and links on the resources page, none of these things change when hovering with the mouse.” E1 H1 S7
	“They [category labels] show that they've been clicked while you're in that section, but when you go to another section, you can't see that you've already been to a particular section.” E2 H1 S3
	“The videos do not show whether they have already been viewed.” E3 H1 S3
Information architecture	“There is no site map. Even though most people don't use a sitemap, there should still be one, and this should be displayed in that bottom navigation bar.” E1 H11 S1
	“I don't see anything I'd identify as a site map, but I don't know how helpful that would be to older adults. Maybe something at the bottom of the page that would list all the categories and then what's available under each would help one re-locate videos that might be located under multiple categories.” E2 H11 S1
	“No site map is presented on the Web site, other than the menu itself.” E3 H11 S1
Presentation design	“White text on light blue—significant problem. Also, the gray 'play' button on the black border could be problematic.” E2 H15 S1
	“The category bars turn white, the recommended videos have a purple (narrow) border, and the play video button turns red. Could be problematic.” E2 H15 S4
Information design	“There doesn't appear to be a definitions page or anything. I couldn't even find written text for what 'COPD' stands for. The videos themselves, though, usually explain medical words or phrases in simple language.” E1 H18 S3
	“There is no safety information in written form. Some videos, like the Infection Treatment video, do provide appropriate safety information, but this should be supplemented with text outside the video.” E1 H18 S4
	“Seems like it would be helpful if the site had a glossary.” E2 H18 S3
	“For some low-income individuals, ‘exhale’ is probably not as clear as ‘breathe out.’” E2 H18 S5
	“There's very little text, but sometimes it goes by too fast in the videos.” E2 H18 S6
	“The language seems pretty straightforward and simplified.” E3 H18 S3

COPD, chronic obstructive pulmonary disease; E, evaluator; H, heuristic number; S, subcriteria number.

Table 3.

Examples of COPDFlix Prototype Modifications Based on Heuristic Evaluation Data

HEURISTIC CATEGORY	VIOLATED HEURISTIC	DESCRIPTION OF VIOLATION(S)	EXAMPLE OF DESIGN SOLUTION(S)
Information design	#1: Make obvious what is clickable and what is not (minor)	• Links do not change upon being clicked. Older adults will likely need visual reminders signified by changing link color so they remain oriented on the Web site. • All buttons must be labeled on the YouTube video media player. • Screen captured images do not capture what is portrayed in videos. • No indication of whether played videos have already been viewed during the session	Design solution: Developed new icons to make the “Sign-in” and “Recommended for You” buttons change and get bigger when the mouse is directed to hovers over the icons
	#7: Provide feedback in other modes in addition to visual (major)	• No meaningful captions or alternative text provided to users in addition to the videos • Some videos seemed more oriented to medical students or health professionals rather than patients, with non–plain language inappropriate for a patient.	Design solution: Developing short two-sentence captions that described content in each video and inserted them underneath the comment boxes beneath each video. Also developing transcripts of video content to import into YouTube for captioning the video in English
Interaction and navigation	#9: Clearly label content categories; assist recognition and retrieval rather than recall (minor)	• Order of videos on some of the pages could use some more structure to minimize scrolling. • Not all category labels would be easily recognizable.	Design solution: Combine common videos on medication use pages and label with commonly understood labels such as “Metered Dose Inhalers” instead of confusing prescription medication names. Design solution: Change video category labels from descriptive to action words (e.g., “Smoking Cessation” to “Quit Smoking”; “Stress Management” to “Managing Stress”; “Lifestyle” becomes “Living a Better Life.”
	#10: Implement the shallowest possible information hierarchy (minor)	• Available links were not cross-referenced. • Overlap in some video categories (e.g., lifestyle, exercise, moving effectively, and physical activity) could limit search success.	Design solution: Increase font size of subheadings for the “At-Home Exercise Workout” video page to help better subdivide the page.
Information architecture	#15: Make sure text and background colors contrast (minor)	• Overall color design is white text on light blue, which represents a significant problem. • Nothing visual to show people that the videos under “Recommended for You” are clickable	Design solution: Made “Recommended for You” icon larger and more striking with bright colors. Design solution: Change the white lettering on the light blue background to darker letters on lighter background colors.
	#16: Make it easy to find things on the page quickly (minor)	• No clear written instructions or directives to click on videos that recommended for the individual user	Design solution: Changed title icon to read “Click on Videos Recommended for You.” Also put category labels in alphabetical order on the homepage.
Presentation design	#18: Use the users' language; minimize jargon and technical terms (major)	• Opening video on pathophysiology of COPD unnecessarily complex	Design solution: Removed video under the Welcome tab and replaced with custom-made welcome video describing how to navigate the site and use the social media features.

COPD, chronic obstructive pulmonary disease.

Acknowledgments

This work was supported by the National Institutes of Health (NCATS) under CTSA awards UL1TR000064 and KL2TR000065 to the University of Florida.

Disclosure Statement

No competing financial interests exist.