Using Qualitative Methods to Create a Home Health Web Application User Interface for Patients with Low Computer Proficiency

Abstract

Introduction:

Despite the investment in public reporting for a number of healthcare settings, evidence indicates that consumers do not routinely use available data to select providers. This suggests that existing reports do not adequately incorporate recommendations for consumer-facing reports or web applications.

Methods:

Healthcentric Advisors and Brown University undertook a multi-phased approach to create a consumer-facing home health web application in Rhode Island. This included reviewing the evidence base review to identify design recommendations and then creating a paper prototype and wireframe. We performed qualitative research to iteratively test our proposed user interface with two user groups, home health consumers and hospital case managers, refining our design to create the final web application.

Results:

To test our prototype, we conducted two focus groups, with a total of 13 consumers, and 28 case manager interviews. Both user groups responded favorably to the prototype, with the majority commenting that they felt this type of tool would be useful. Case managers suggested revisions to ensure the application conformed to laws requiring Medicare patients to have the freedom to choose among providers and could be incorporated into hospital workflow. After incorporating changes and creating the wireframe, we conducted usability testing interviews with 14 home health consumers and six hospital case managers. We found that consumers needed prompting to navigate through the wireframe; they demonstrated confusion through both their words and body language. As a result, we modified the web application’s sequence, navigation, and function to provide additional instructions and prompts.

Discussion:

Although we designed our web application for low literacy and low health literacy, using recommendations from the evidence base, we overestimated the extent to which older adults were familiar with using computers. Some of our key learnings and recommendations run counter to general web design principles, leading us to believe that such guidelines need to be adapted for this user group. As web applications proliferate, it is important to ensure those who are most vulnerable—who have the least knowledge and the lowest literacy, health literacy, and computer proficiency—can access, understand, and use them.

Conclusions:

In order for the investment in public reporting to produce value, consumer-facing web applications need to be designed to address end users’ unique strengths and limitations. Our findings may help others to build consumer-facing tools or technology targeted to a predominantly older population. We encourage others designing consumer-facing web technologies to critically evaluate their assumptions about user interface design, particularly if they are designing tools for older adults, and to test products with their end users.

Introduction

Over the last decade, state and national public reporting programs have increasingly published data about health care quality and satisfaction to inform consumer choice and promote transparency.1 Despite the investment in public reporting for a number of health care settings and health care consumers’ stated desire to have access to this information,2,3,4,5 evidence indicates that consumers do not routinely use available data.6 This suggests that, although recommendations exist for consumer-facing reports or Web applications,7 reports may not adequately reflect consumer-centric principles. Consumers have expressed a desire for Web-based, customizable report platforms that are available when they are making important health care decisions.8

Limited financial resources have constrained the Rhode Island Department of Health’s efforts to create consumer-facing public reports. While the state was an early innovator in public reporting of quality and satisfaction data for hospitals, home health agencies, and nursing homes, it publishes static PDF files.9 The reports aim for a sixth grade reading level and use methods, such as provider ratings, that are intended to help consumers interpret results, but that do require basic literacy, health literacy, and numeracy. Program administrators and policymakers desired electronic user interfaces designed to meet consumers’ needs—particularly since health care decisions are often made quickly, under stressful conditions.

Healthcentric Advisors, the Rhode Island Department of Health’s public reporting contractor, partnered with Brown University to create a Web application that enables Rhode Island consumers to compare home health agencies based on information about the agencies, such as their services and health outcomes. We focused on the home health setting because of state and national policies focused on increasing the use of home and community-based services; because less is known about how consumers choose home health compared to other types of health care; and because home health consumers are often highly vulnerable because of age or their homebound status (an eligibility requirement for skilled home care).

Among the postacute care settings covered by Medicare, home health is the most frequently used, with over 40 percent of home health episodes occurring following a hospital stay.10 In 2010, 29 percent of these postacute home health episodes resulted in hospital readmission, and there is wide variation in readmission rates across regions and home health providers, suggesting opportunities to improve quality.10 Yet, despite variation in quality indicators, such as readmission, patients typically receive little guidance in choosing home health care during the hospital discharge process11 and there is no evidence about consumers’ use of quality data to choose home care.12

Ultimately, our goal was for the Web application to become part of the Department of Health’s website and for its functions and structure to be scalable for use in other health care settings in the future. Because this was our intention, we collaborated closely with the Department of Health to ensure that the application would adhere to state website requirements. While we collected information about consumers’ and case managers’ experiences selecting agencies and the data that would be helpful during that process, and published those results separately,13 this paper focuses on findings related to the design of the Web application for home health consumers. Our methods and findings may help others building consumer-facing tools or technology targeted to a predominantly older population, similar to home health consumers.

Methods

We undertook a multiphased approach to create the new Rhode Island home health Web application (Figure 1). First, we reviewed the evidence base to identify content and design recommendations for consumer-facing reports or Web applications. Second, we used knowledge gained from the evidence base to create a paper prototype of a home health Web application. Third, we vetted this prototype by conducting focus groups with people who had home health experience and by interviewing hospital case managers responsible for helping hospitalized patients select home health agencies at discharge. Fourth, we worked with interactive design experts to translate the paper prototype into a wireframe (described in more detail below) reflecting focus group and interview results. Fifth, we assessed ease or difficulty of use of the wireframe by conducting usability testing interviews with hospital case managers and with people who had home health experience or were ages ≥65 years old, while making iterative revisions to the wireframe based on usability testing results. Lastly, we worked with Web designers to build the final Web application. Methods for each phase follow in the next sections of the paper, along with qualitative findings from each phase. The study was approved by the Institutional Review Board of one of the local hospitals.

Figure 1.

Multiphased Approach for Creating a Consumer-Facing Home Health Web Application

Evidence Base Review

Although we did not conduct a systematic review, we did search PubMed and the Internet for relevant literature, using the following keywords: cognitive science, consumer reports, design, home care, home health, report cards, public reporting, public reports, quality measures, quality reports, use, usability, and website. We also reviewed the reference lists of identified materials and conducted targeted searches for individuals well-known to our team for their work in public reporting, including the investigators of other studies funded by the same Agency for Healthcare Research and Quality funding mechanism—Building the Science of Public Reporting. Our intent was not to conduct an exhaustive search of the evidence base, but to familiarize ourselves with existing recommendations and to incorporate relevant recommendations for report content and design.

Prototype Focus Groups and Interviews

We conducted consumer focus groups and hospital case manager interviews from April to July 2013, after developing a paper prototype of the Web application. Focus group participants responded to advertisements for volunteers and were eligible if they were English-speaking home health consumers (i.e., ages ≥21 years who either received home care within the previous three years or had cared for a family member who received such services within that time frame). We conducted focus groups with consumers because we wanted to facilitate dynamic conversations that allowed us to identify patient and family needs.

We also conducted structured interviews with hospital case managers ages ≥21 years who worked at one of five hospitals in Rhode Island and whose professional responsibilities included discharge planning. Although hospital case managers may be end users themselves, we included them as a secondary user group, primarily to improve our ability to understand and reach consumers. We believed that case managers’ experiences helping consumers select home care could enable us to capture a wider range of consumer needs than the limited number of focus group participants. We also thought that case managers, as the primary information conduit at hospital discharge, would be more likely to share the Web application with consumers if they found it useful. We chose to conduct individual interviews to maximize participation during work hours and to enable us to compare and contrast responses from different individuals.

During each focus group or interview, the facilitator asked participants to review the paper prototype and to provide input on the content and functions that they would find helpful when choosing an agency (consumers) or helping a patient to make that choice (case managers). One or two additional members of the research team observed each focus group or interview to record notes.

Usability Testing Interviews

We conducted usability testing interviews from March to April 2014. Consumer participants responded to advertisements for volunteers and (as with the focus groups) were eligible if they were English-speaking home health consumers. For this phase, participants were also eligible based solely on age (≥65 years), regardless of previous experience with home care, since older adults are the most likely to need home care and may need to help a family member choose services (e.g., a spouse or sibling). Case manager participants were eligible if they were ages ≥21 years, worked at one of three hospitals in Rhode Island, and their professional responsibilities included discharge planning. We narrowed our focus to three of the original five hospitals because we wanted to focus our resources on the following: (1) sites that would subsequently participate in a randomized, controlled trial of the Web application, and (2) the obtaining of direct usability feedback from our primary user group, consumers.

During the usability testing interviews, the facilitator (one of the authors) asked participants to test the electronic prototype and to describe their thoughts and reactions aloud, using a cognitive-based testing approach. She also asked questions to elicit additional information about how easy or difficult respondents found the application to use and to understand. One or two additional members of the research team observed each interview to record notes.

Qualitative Analysis

We audiotaped each focus group and interview. Three of the authors independently conducted repeated examination and content analysis using observers’ notes and data manually coded by research assistants from the audiotapes. (Although we collected demographic data about individual focus group participants, focus groups are dynamic interactions and our analysis was conducted at the focus group level, i.e., we looked for themes and representative quotes, but did not tally individual participants’ agreement.) We then met to reach agreement and settle any divergence in analysis. When identifying major themes, we considered the words, tone, context, nonverbal cues, internal consistency, frequency, intensity, and specificity of responses.

Results

We used a multiphased process to refine the user interface for our consumer Web application.

Evidence Base Review

We used the evidence base to inform content and design recommendations for consumer-facing reports or Web applications (Table 1). Although “content” could include data elements or quality measures incorporated into a report, in this paper we focus on the Web application’s design, including function, instructions, and layout. We found research that showed that consumers comprehend information best when they can choose what to view,14 e.g., information specific to their clinical condition,15 and that they generally retain only three or four pieces of information in their short-term memories.14 This suggests that interactive Web applications, like the one we designed, have significant potential to empower consumers if users have the ability to customize reports to view information tailored to their needs.14

Table 1.

Select Recommendations for Consumer-Facing Report Development

CONSIDERATION AND RECOMMENDATIONS	RATIONALE	SOURCE
ACCESSIBILITY
Avoid writing in all capital letters. Choose serif fonts. Enhance readability with high contrast. Use colors to draw attention to functions.	Adults ≥65 years old may experience changes in vision. Certain fonts and design elements can increase readability and therefore improve comprehension.	McGee, 201016
COMPUTER PROFICIENCY
Include detailed instructions. Provide prompts for functions.	Consumers’ limited knowledge of website functionality can be offset with contextual prompts and instructions.	Primary data collected by the authors during the study
FUNCTION
Allow users to choose what to view.	Consumers comprehend information best when they can choose what to view, e.g., information specific to their clinical condition.	Vaiana and McGlynn, 200214 Bardach et al., 201115
LITERACY AND HEALTH LITERACY
Write at a sixth grade reading level.	Consumers are more likely to comprehend less technical information targeted at lower education levels. Consumers are not well versed in clinical or technical terms. Providing definitions for key terms helps them to understand new concepts.	Kutner, 200718
Include definitions for medical terms. Limit the use of technical language.		Primary data collected by the authors during the study
NUMERACY
Group information to be compared.	Numeric information is more easily comprehended when presented simply, with fewer comparisons to evaluate.	Vaiana and McGlynn, 200214
Limit comparisons to 3–4 data points.	Cognitive science shows that people generally retain only 3–4 pieces of information at a time.	Peters, 200719
Directly label graphs (avoid footnotes).	People with low health literacy or numeracy skills may not understand that a symbol refers to information in a different location.	Peters, 200721

Other research findings centered on literacy and health literacy. A national assessment found that only 13 percent of American adults are proficient in the level of literacy critical to fully understand and use a comparative quality report.16 Reports need to be audience appropriate and easily comprehensible by a broad range of users; otherwise, consumers become confused and do not attempt to interpret the information they view.17 Recommendations to address literacy and readability include limiting the use of technical language, directly labeling graphs (instead of using footnotes or legends), and grouping information meant to be directly compared.14,18

Prototype

We created a paper prototype showing screenshots of our proposed sequence and function. The prototype illustrated how the Web application would allow users to narrow down a list of home health agencies to those meeting specific criteria—such as insurance, services, or service area—and would then allow them to customize the display of quality and patient satisfaction measures. They would be able to choose either to view all resulting agencies that met their search criteria or to compare three agencies at a time. They would also be able to choose to view all information and measures (e.g., quality measures and patient satisfaction) or to select a subset of measures to view. As recommended by others,15,17 we used side-by-side agency comparisons to help people to make meaningful distinctions between agencies.

Prototype Focus Groups and Interviews

During the prototype phase, we conducted two focus groups, with six and seven consumers, respectively, for a total of 13 consumers (Table 2) and 28 case manager interviews (Table 3). As mentioned above, we included case managers both as a user group and as a way to gather additional information about consumer needs, given their experience interacting with consumers choosing home care. Our findings about experiences selecting agencies and the data that would be helpful during that process are published separately, but notably indicate that both user groups believe a Web application would improve upon the current process, which involves consumers selecting agencies from paper-based lists of agencies.13 Consumers and case managers responded favorably to the prototype, with the majority commenting that they felt this type of tool would be useful. They also affirmed our decision to allow users to search for agencies that met specific criteria, spoke about the importance of being able to identify agencies that would accept a patient’s insurance and meet their specific needs, and agreed that limiting the comparison to three agencies at a time was appropriate.

Table 2.

Home Health Consumer Participant Characteristics

CHARACTERISTIC	PROTOTYPE FOCUS GROUP PARTICIPANTS (N=13)	USABILITY TESTING INTERVIEW PARTICIPANTS (N=14)
	n (%)
Home health experience*
Patient	1 (7.7%)	2 (14.3%)
Family member	10 (76.9%)	8 (57.1%)
Both patient and family member	2 (16.7%)	0 (0.0%)
Age ≥65 years	—	10 (71.4%)
Female	11 (84.6%)	12 (85.7%)
Race
Black	3 (23.1%)	1 (7.1%)
White	10 (76.9%)	13 (92.9%)
Other	0 (0.0%)	0 (0%)
Hispanic ethnicity	0 (0.0%)	0 (0%)
Correctly answered health literacy screen**	13 (100.0%)	12 (85.7%)
Participated in prototype focus group	—	1 (7.1%)

^*Percents may not sum to 100% due to missing information.

^**The health literacy screen is a single question asking patients to provide information about an exemplar medication label.

Table 3.

Hospital Case Manager Participant Characteristics

CHARACTERISTIC	PROTOTYPE INTERVIEW PARTICIPANTS (N=28)	USABILITY TESTING INTERVIEW PARTICIPANTS (N=6)
Female, n (%)	28 (100.0%)	6 (100.0%)
Race, n (%)
Black	0 (0.0%)	0 (0.0%)
White	27 (96.4%)	6 (100.0%)
Other	1 (3.6%)	0 (0.0%)
Hispanic ethnicity, n (%)	0 (0.0%)	0 (0.0%)
At a hospital with affiliated agency, n (%)	9 (32.1%)	0 (0.0%)
Years of experience helping patients choose home care, mean (median)	10.1 (7.0)	8.1 (8.0)
Participated in prototype interview	—	1 (16.7%)

Many participants commented that the patients who need home care are often very sick; they felt that patients’ family members were much more likely than patients themselves to use the Web application. They recommended that the application be optimized for mobile devices and that users be able to email search results, so that hospital visitors could use the application from tablets or smart phones and share their searches with others electronically.

The case manager interviews provided a nuanced understanding of hospital processes, resulting in two key changes to the prototype: the creation of a printer-friendly version and the addition of search criteria to the printout. All 28 case managers—despite working at five different hospitals—described providing patients with preprinted lists of home health agencies. They carried these preprinted paper handouts with them from their offices, and none had computer access while in patient rooms or on a hospital unit; as a result, they needed to be able to easily print search results to carry with them to the units. The case managers also spoke about using the list as a tool to uphold the Social Security20 and Medicare21 laws requiring Medicare patients to have the freedom to choose among providers: because their lists included all agencies in the state, their existing lists served as proof that no agencies were excluded from a patient’s selection. By including search criteria on the printout, the Web application could also clearly indicate that the case managers were upholding the laws.

Wireframe

We used the results of our evidence base review and prototype research to create a wireframe. A wireframe is a visual schematic that allows users to experience the Web application’s proposed sequence, navigation, and function, but does not include live data or graphic design. In other words, it focuses on what the application does, not what it looks like. Although we used our learnings from the to add features designed to improve accessibility and usability, such as making the application mobile friendly and capable of emailing and printing results, we retained much of the initial prototype’s proposed function and sequence.

Importantly, we added text showing the search criteria used to generate the report on our printer-friendly version, so that case managers could generate a list that met a patient’s specific needs without fearing they were breaking patient-choice laws. The primary difference between the prototype and wireframe, however, derives from our work with interactive design experts to create and sequence an interactive application. These experts added a five-question Web “wizard” to ask users about an individual’s home health needs, helping an individual to identify the subset of agencies that met his or her needs. The wizard included default options, so that users would not need to make a selection when unsure of a response. We also offered two options for detailed data displays, organizing information using either a vertical accordion menu (with collapsible submenus) or horizontal tabs.

Usability Testing

We conducted usability testing interviews with 14 home health consumers (Table 2) and 6 hospital case managers (Table 3). Case managers responded enthusiastically to the wireframe, commenting how the final Web application would address a longstanding need—to have a centrally maintained repository of up-to-date information. In contrast, consumers demonstrated confusion through both their words and body language. Many sat in front of the computer and removed their hands from the keyboard or mouse, either silently demonstrating their unease or explicitly asking how to proceed through the application.

The facilitator prompted consumers to navigate through the wireframe, advising them how to accomplish each step while eliciting their thoughts. For example, some of the pages required scrolling down to view all of the information. Many consumers did not realize that there was information that was not visible to them or that the Web browser had a scroll bar. Others did not realize that clicking on a button would submit their responses and lead to a subsequent page.

Because the wireframe had no live data or graphic design, we could easily modify its sequence, navigation, and function during usability testing. It underwent iterative modifications to respond to users’ suggestions and increase ease of use. Key changes included adding additional instructions at the beginning of the wizard, which helped users gain confidence in proceeding through the application; adding a pop-up box that reminded users to scroll down to view full pages, which helped individuals who had little familiarity with websites and Web applications; organizing each agency’s data into horizontal tabs to decrease the length of the page and allow users to see, at a glance, what was available; and modifying the comparison feature to “freeze” the comparison instructions and selected agencies at the top of the screen when users scrolled down. After each successive change, we tested the updated wireframe, sometimes showing participants two versions of a single feature. By the end of testing, we noted a marked decline in consumers’ confusion and questions.

Final Web Application

As a result of the modifications made to the wireframe during usability testing, the Web designers were able to easily translate the last version of the wireframe into the final Web application. This involved completing the programming necessary to incorporate live data and apply typographic style, color, and other graphic design elements and to optimize the website for use on mobile devices. We did not test the graphic design with end users, since the application was intended to become part of the Rhode Island Department of Health’s website and was, consequently, required to conform to state-mandated style sheets. We did, however, make minor design decisions based on the evidence base for people with visual impairment, such as ensuring that text had high contrast and that navigation buttons were brightly colored. We had previously included the ability for users to increase the font size.

Discussion

Although we designed our Web application for low literacy and low health literacy, using recommendations from the evidence base, we overestimated the extent to which older adults were familiar with using computer applications. In the early iteration of the design, usability testing demonstrated that consumers were confused and unsure about how to use the application, often stopping and looking to the facilitator for help. Many were hesitant to experiment by clicking on options or navigating between Web pages, preferring to wait for instructions or to take an action only when they were certain of the outcome. While they may have felt more comfortable exploring the application without being observed, these findings nonetheless attest to the importance of understanding the target audience and incorporating their input into any user interface design.

We concentrated on home health because of state and national policies focused on increasing the use of home- and community-based services and because much less is known about how consumers choose home health agencies compared to other health care. Designing a home health Web application allowed us to focus on people who are particularly vulnerable when selecting an agency: hospitalized patients who are eligible for home care. These vulnerable patients are experiencing stress because of their hospitalization and must be homebound (and therefore very sick) to qualify for home care. Yet they are routinely asked to choose a home health agency without any prior experience with home care or knowledge of the industry, and they receive very little information to inform their decision; we found they usually receive just a list of agency names.13 In creating a home health Web application, we also expanded on the data already available through Medicare’s Home Health Compare website by combining information about skilled services, which is included on the Medicare website, with information about insurance coverage, services offered, and nonskilled services, which are not included on Medicare’s site. We wanted to better enable consumers and case managers to identify providers that could meet patients’ needs.

Web applications are a cost-effective way to disseminate information because information can be updated centrally, does not need to be printed or mailed, and can be accessed on mobile devices; as a result, they are likely to proliferate, even if they are not ideally suited to a particular user group. We found that home health consumers, who are often ≥65 years old, may have difficulty using applications created using general Web design principles. For example, when writing for general Web audiences, the guiding principles are to use 50 percent fewer words than in a written document. In contrast, our older adult users preferred detailed instructions and prompts and seemed reluctant to experiment with the Web application in the absence of written directives. Although family members can (and often do) access technology on behalf of patients, and there may be a tipping point in computer proficiency as the United States population ages, we believe that low computer proficiency may be common among people ≥65 years old. As Web applications proliferate, it is important to ensure that those who are most vulnerable—who have the least prior knowledge and the lowest literacy, health literacy, and computer proficiency—can access, understand, and use them.

We recognized that hospital case managers—who interact with numerous patients and family members each day—could greatly expand our insight into consumers’ needs, and that they play an important role in conveying information to patients, so we included them in our study as an important secondary user group. There is little information available about providers’ use of public reports, although our prior research describes the fact that none of the 28 case managers who participated in the prototype interviews were aware of existing home health reports.13 By tailoring our application to the case manager user group, we were able to incorporate design elements that were not priorities for consumers (e.g., the printer-friendly option), but that will enable the Web application to be incorporated into hospital work flow. Case managers are the only reliable conduit of information for hospitalized patients making decisions about home health care, but they are unlikely to share the application if they are not able to easily incorporate it into their daily work and find it useful.

Although we conducted focus groups and interviews over several study phases, we focus here on usability testing results. Our usability interviews faced several limitations. First, as noted above, consumers may have felt evaluation apprehension. This could lead us to underestimate true computer proficiency. Second, participants were limited to home health consumers and older adults who were well enough and willing to participate in a research study. During usability testing, we may have observed the best-case scenario for comfort and confidence in using the application, leading us to overestimate true computer proficiency. Third, our results reflect consumers’ and case managers’ perceptions and preferences in a single state, Rhode Island. Preferences may differ in areas with different consumer demographics. Fourth, we conducted prototype interviews with case managers from five hospitals but usability testing interviews with case managers from only three of those five sites. Case managers’ input was relatively homogenous across five sites during the earlier prototype interviews, so we do not believe that limiting to three sites introduced a selection bias. Fifth, the majority of our participants were female. We do not know how preferences differ by gender, although we believe that our samples are generally representative of our two target populations: Twice as many women as men over 65 years of age receive home care;22 two out of every three caregivers are women;23 and, in our experience, nearly all case managers (most of whom are nurses) are women. Finally, as previously noted, our final Web application was required to adhere to state-mandated style sheets. Other states may have less stringent graphic design requirements or may want to test the impact of design requirements on users with low literacy, health literacy, and computer proficiency.

In order for the investment in public reporting to produce value, consumer-facing Web applications need to be designed to address end users’ unique strengths and limitations. Web applications have the potential to make public reports interactive and consumer driven, and may also be updated and disseminated in a more cost-efficient manner. However, they must be designed to meet end users’ needs in order to be useful and used.

Conclusions

We encourage others designing consumer-facing Web technologies to critically evaluate their assumptions about user interface design, particularly if they are designing tools for older adults, and to test products with their end users. Although we used the evidence base to create a home health Web application for low literacy and health literacy users, we initially overestimated the extent to which older adults were familiar with using computers, and we failed to incorporate functions necessary for hospital case managers. Prototype and usability testing enabled us to revise our approach, ensuring that our final Web application included changes tailored to our user groups’ needs. In a future phase of this study, we will evaluate how hospitalized patients’ choices and health outcomes differ if they use the Web application or experience the usual discharge process. Once that research is complete, the application will become part of the Rhode Island Department of Health’s website, and we will use Web statistics to monitor use over time.