Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Jan 11.
Published in final edited form as: Int J Hum Comput Interact. 2016 Jan 11;32(4):297–307. doi: 10.1080/10447318.2016.1140528

Designing an Educational Website to Improve Quality of Supportive Oncology Care for Women with Ovarian Cancer: An Expert Usability Review and Analysis

Molly A McClellan 1, Raghav Pavan Karumur 2, Rachel Isaksson Vogel 3, Sue V Petzel 4, Julie Cragg 5, Daniel Chan 6, Julie A Jacko 6, François Sainfort 1,7, Melissa A Geller 1,4,*
PMCID: PMC4838192  NIHMSID: NIHMS777339  PMID: 27110082

Abstract

A broad-based research team developed a Health Insurance Portability and Accountability Act (HIPAA)-compliant educational website for women with ovarian cancer to improve the quality of supportive oncology care. Prior to a randomized clinical trial of the website, initial usability testing was implemented to evaluate the website. The initial review found that 165/247 checklist items had sufficient information to allow for evaluation with the website achieving an overall score of 63%. By category, lowest scores were for the Home Page, Task Orientation, Page Layout & Visual Design, and Help, Feedback & Error Tolerance. Major issues thought to potentially impede actual usage were prioritized in redevelopment and the second usability review, conducted by the same expert, saw an improvement in scores. Incorporating usability concepts from the start of development, fulfilling the positive expectations of end-users and identifying technical and personal factors that optimize use may greatly enhance usage of health websites.

Keywords: Usability, Heuristic evaluation, Ovarian Cancer, Website, eHealth

BACKGROUND AND SIGNIFICANCE

Ovarian Cancer

Ovarian cancer is the most lethal gynecologic malignancy with an estimated 5-year survival of 44% for all stages of disease, decreasing to 18% for those diagnosed with Stage IV disease (American Cancer Society 2013). About 75% of women present with Stage III or IV disease. This disease affects older women; approximately 90% are diagnosed when they are older than 40 and the greatest number of cases are diagnosed in women over age 60 (United States Cancer Statistics Working Group 2013). Women with ovarian cancer are commonly impacted by symptom clusters that include fatigue, anxiety, pain, bloating, bowel disturbances, nausea and vomiting, anorexia, and depression (Bankhead, Kehoe et al. 2005). Risk for recurrence is 60–70% and women who recur currently cannot be cured.

Supportive Oncology Care

For individuals with metastatic disease and/or a high symptom burden, both of which are characteristic of women with ovarian cancer, the American Society of Clinical Oncology recommends early provision of palliative care, an interdisciplinary specialty focused on quality of life and symptom reduction (Smith, Temin et al. 2012). Enabling and supporting patients to make informed decisions about their care represents a core element of quality palliative cancer care (Ferrell, Paice et al. 2008). This includes education about their cancer and treatment options, making an advance healthcare directive, and accessing specialized palliative care professionals; patients’ active involvement in advance care planning is an important part of quality cancer care. Studies unfortunately report that palliative care is not offered as a consistent standard of care during cancer treatment and women with ovarian cancer tend to receive information about and access to palliative care late in their disease progression (Fauci, Schneider et al. 2012).

Information Sources and Cancer

Cancer patients describe obtaining cancer information primarily from their providers and secondarily through other relationships and the internet (Kirk, Kirk et al. 2004). A recent study of information-seeking for cancer on the internet suggests patients and their relatives may be increasingly turning to the internet for information on their particular disease (Ofran, Paltiel et al. 2012). In receiving palliative care information cancer patients emphasize the importance of pacing information in order to assimilate it, making information easy to understand, and that information needs change and diverge as the disease progresses (Kirk, Kirk et al. 2004). Research suggests that most women with ovarian cancer want detailed information about ovarian cancer through all phases of their disease and the more serious the illness, the more shared decision-making is desired (Stewart, Wong et al. 2000).

In addition to the increasing availability of cancer-related information on the internet, internet methods for delivering behavioral and health-related interventions (eHealth) are rapidly emerging. Reasons for delivering eHealth interventions include reduced delivery costs, user convenience, timeliness, reduction of stigma, increased user and supplier control of the intervention, and reduction of geographic, time and mobility-based delivery barriers (Griffiths, Lindenmeyer et al. 2006). While the literature is new, some published randomized trials of eHealth interventions have demonstrated positive results; though others have not (Strecher 2007, Paul, Carey et al. 2013). These interventions have focused on a variety of health conditions and behaviors, including alcohol abuse, nutrition, diet, exercise, smoking, and less frequently on chronic diseases such as asthma, diabetes, mood disorders and cancer (Strecher 2007, Paul, Carey et al. 2013). Research suggests that tailoring cancer information to individual patients can improve the persuasiveness of messages (Williams-Piehota, Latimer et al. 2009).

Personalized medicine and patient empowerment

A wave of consumer empowerment in healthcare is focused around Personal Health Records (PHRs). In addition to true PHRs, many hospital systems offer a patient portal, which gives consumers access to select information in their Electronic Medical Record (EMR). This access typically allows patients to schedule and cancel appointments, view lab and radiology results, post-visit summaries, current medication lists, order refills, and exchange secure messages with providers. Because of the increased consumer engagement in applications such as PHRs and patient portals, web based and mobile applications for chronic disease management are becoming increasingly popular. It is thought that stakeholders in the healthcare system will benefit when web-based applications leverage clinical data from the patient record to empower consumers to play a more active role in their healthcare (Detmer, Bloomrosen et al. 2008).

There are many informative websites and applications for patients with cancer, however few include cancer self-management applications, and none would be considered a true management suite for ovarian cancer. Few websites incorporate personalization, which is an important factor in the success of patient education programs (Doupi and van der Lei 2002). Personalization has been shown to increase patients’ satisfaction with education material, improve understanding of their condition or disease, and enhance patients’ ability to participate in disease management.

Many factors are influential in health self-management, particularly self-efficacy, referring to patients’ confidence in their own ability to take action (Bodenheimer, Lorig et al. 2002). Self-efficacy and active participation in disease management can be guided and assisted by improvements in health information technology. While technology-based interventions to promote self-efficacy and active participation are increasingly used in chronic disease settings, such interventions are uncommon in cancer (Kuijpers, Groen et al. 2013) and few have undergone rigorous testing and evaluation.

Usability evaluation

The purpose of usability evaluation of a website is to determine whether it is appropriate for its intended users. Conducting usability testing alongside functional testing as early in the life cycle as possible is a necessity when designing commercial websites, and can be even more important for health research websites. Websites used for a health intervention study may have more complexity than a consumer site. For example, if the website captures private health information (PHI), additional layers of security are needed and adherence to all Health Insurance Portability and Accountability Act (HIPAA) rules must be achieved.

Usability testing is conducted to establish that the product is useful or successful, and to identify where improvements can be made. For web design, many standards have been previously documented. Heuristic evaluation, developed by Jakob Nielson and Rolf Molich in 1990, is a popular, easy and systematic method for evaluating user interface design, including web-based design, and helps in rapidly optimizing the design using recognized usability principles (Molich and Nielson 1990, Nielsen and Molich 1990).

Usability evaluation includes four basic metrics: task success, comprehension/learning, satisfaction, and usefulness. The overarching goal is to improve the design of the website to provide the user with the feeling that the site is both useful and easy to use - the two pillars of the Technology Acceptance Model (Davis, Bagozzi et al. 1989). Our research suggests the importance of usability evaluation prior to implementation of an intervention. Changes can then be prioritized and accomplished prior to first release for formal research testing.

OBJECTIVE

There are numerous limitations to existing websites available for women with ovarian cancer. While there are many informational websites such as those sponsored by the National Ovarian Cancer Coalition, National Cancer Institute, and American Cancer Society, they present relatively heavy text-based information that is not tailored to the patient’s disease or learning/coping style (Miller 1995). In addition, most health websites are not designed to change health behavior and have not been evaluated in a randomized controlled trial. One exception is a recently published study by Donovan and colleagues (2014), which provided information on the development and evaluation of a symptom management website for women with recurrent ovarian cancer and found high usability and satisfaction (Donovan, Ward et al. 2014).

No web-based interventions to date designed for women with ovarian cancer have been directed at promoting advance care decision making. Therefore, our objective was to develop a comprehensive website for women with ovarian cancer with the primary goal of increasing advance care planning by encouraging completion of an advance healthcare directive and consultation with palliative care. Features of this website included personalized and tailored expert information, secure access to online forums, emotional and cancer-related symptom tracking and tools to promote advance care decision making. This study is novel in that it involved the development and assessment of a website surrounding advance care planning designed specifically for women with ovarian cancer, an older, ill patient population rarely studied with web interventions. Prior to initiating a randomized clinical trial of the intervention, we completed website usability testing to improve its design and usability for this population by ensuring that it followed both web and Technology Acceptance Model standards (Davis 1989). This paper focuses on the preliminary usability review conducted, subsequent changes, and final usability evaluation.

MATERIALS AND METHODS

Website development

Full details on the design, development, and results of the randomized clinical trial of our developed website are published elsewhere (Vogel, Petzel et al. 2013). Briefly, the research team followed an integrative and iterative process for system development (see Figure 1). It was integrative in that women with ovarian cancer, their families and providers were involved in the conceptual system design alongside the clinical, information and decision support experts in the research team. It was iterative in that several cycles encompassing cognitive walkthroughs, heuristic evaluations, and pilot testing were required to reach the final design in a rapid prototyping process (Nielsen 1994).

Figure 1.

Figure 1

Open in a new tab

A flowchart illustrating the process highlighting usability reviews during various phases.

Microsoft’s ASP.NET 4.0, a server-side Web application framework was used for designing and developing dynamic web-pages for this website and SQL Server 2008 was used for storing and maintaining the data generated during the use of this website. Programming languages such as C#, HTML, CSS, JavaScript, Jscript and AJAX 3.5 were used to develop the business logic and enhance the look and feel of the user interfaces.

The core components of the intervention website were designed to improve the quality of supportive oncology care (Ferrell, Paice et al. 2008). Basic modules included a Learning Library with information on various topics personalized to the user’s disease phase (newly diagnosed, remission, recurrent disease) and desired level of information; a tool guiding users through completing their own advance healthcare directive; a Palliative Care decision tool promoting setting up an appointment with a Palliative Care specialist; an Action Plan to track individual’s goals; a Distress Thermometer, widely used in cancer care to track emotional distress (Roth, Kornblith et al. 1998); a Journal to record individual’s private thoughts and a way to share them with their caregiver; the Discussion Forum where users could discuss personal issues anonymously with other users, all of whom had ovarian cancer; a ‘Question for your Provider’ form which allowed user’s to record their medical questions and print to take to their next healthcare appointment,

While the final version of the website had many patient-centered tasks, four tasks were designed as integral to the research goals. The first was for the user to read the information provided regarding completion of an advance healthcare directive, including the option to download and complete a form to give to their healthcare provider. The second was for the user to make an appointment with the Palliative Care team. This was encouraged through the website by providing information on the healthcare team and contact information for the clinic. Another important task was for the patient to read articles recommended for them based on their disease phase in the Learning Library. Finally, the fourth important task was for the user to report their distress and symptoms over time, which was captured at various time points by the website through completion of the Distress Thermometer tool.

Following University of Minnesota Institutional Review Board approval, a randomized clinical trial to pilot test the intervention website was conducted among women with ovarian cancer. Women with Stage III/IV or recurrent epithelial ovarian, primary peritoneal or fallopian tube cancer were recruited from the University of Minnesota’s Gynecologic Oncology clinic starting Fall 2012, with survey and medical record data collection through Spring 2013.

Upon initial login, participants completed a pre-study survey and then were randomized 1:1 to the intervention website or a control website. The control website included all usual care information documents converted to PDFs and were organized into general topics. Prior to initiation of this study, these documents were provided to patients in paper format only as part of a welcome folder and therefore the control website represented an electronic version of the standard of care in the clinic. Participants were asked to access the website 2–3 times per week for 60 days and then complete the post-study survey.

The primary study outcomes for the randomized clinical trial were completion of an advance healthcare directive and consultation with Palliative Care, assessed by self-report. Secondary outcomes included website use and satisfaction, emotional distress and attitudes about cancer information acquisition and coping. The measures of emotional distress and attitudes were not directly used to tailor the intervention; however, they were used to assess the impact of the information tailoring.

Usability testing

Following an initial one-day discussion group involving women with ovarian cancer, their families/caregivers, and healthcare providers and other stakeholders, the first website prototype was developed as described above. The prototype was then evaluated by a patient, caregiver, oncology nurse, and palliative care physician in order to get general feedback. No formal usability testing was done at this time, but each user provided general feedback as they explored the website and the functionality. The testers were each compensated for their time. Based on this session, changes were made to the website and a second prototype version was developed.

After the completion of the second prototype, a formal usability review (Usability Review I, Figure 1) was conducted by a health informatics specialist (M.M.). The review included an evaluation of the website and its usability against the best practice guidelines put forth by Travis (2009). The 247 guideline items are categorized by Travis as follows: 1) Home Page, 2) Task Orientation, 3) Navigation & Information Architecture (IA), 4) Forms & Data Entry, 5) Trust & Credibility, 6) Writing & Content Quality, 7) Page Layout & Visual Design, 8) Search, and 9) Help, Feedback & Error Tolerance. In each of these categories, multiple heuristics were scored as 1, 0 or −1. A score of 1 indicated that all requirements of the heuristic were met and no changes were needed. Scoring a 0 on an item indicated that some changes were necessary and a −1 indicated the item was attempted but was unsuccessful. An overall score along with individual scores for each of the nine categories were calculated. If an element related to a given item was missing entirely, or not applicable, it was not scored. For example, this website did not contain Search capability, therefore, all heuristics belonging to the Search section were not be scored and did not count negatively against the website. Missing or unscored categories such as Search were addressed in the notes provided to the entire research group with suggestions on how to implement the changes.

The majority of women diagnosed with ovarian cancer are over 60 years old; therefore age was an important consideration in determining the website’s usability. Research has demonstrated that older users often have different needs and characteristics than younger users, including changes in perceptual abilities, motor skills, cognitive abilities, mental models, and confidence in the use of technology (Lynch, Schwerha et al. 2013).

Chisnell and Reddish, commissioned by the American Association of Retired Persons, first put forth age-sensitive heuristics (Chisnell and Redish 2005). These heuristics complement basic web design heuristics (such as Nielsen’s ten heuristics) and can be combined to tailor websites appropriate for all web users, including an older audience. These heuristics were reviewed alongside the Travis (2009) guidelines to ensure that these particular user characteristics informed the design. More recently researchers put forth weighted heuristics (Lynch, Schwerha et al. 2013) and there appears to be consensus that designing for older adults requires additional guidelines based on abilities and preferences (Ellis and Kurniawan 2000, Romano Bergstrom, Olmsted-Hawala et al. 2013, Patsoule and Koutsabasis 2014).

Although the Chisnell and Redish heuristics were not specifically included in the metrics used to score the website, they were taken into consideration as appropriate when scoring the various categories. For example, Chisnell and Redish include the question “Do graphic buttons avoid symbols that will be unfamiliar to older adults who have low computer and Web expertise?” This question is similar to an item scored in the usability review, “The functionality of novel device controls is obvious.” For this study, the population is older adults, therefore when considering whether or not graphic buttons would be familiar one must consider whether or not they would be novel to older users. The remaining Chisnell and Redish heuristics generally mapped in this manner to the ones used in the usability review scoring. When scoring the website, each item in the review was evaluated considering the age and expected lack of computer experience of the population.

The initial score of the website and recommendations to improve usability were provided to the research team (detailed below in results). After completion of the randomized clinical trial, a second usability review (Usability Review II, Figure 1) was conducted by the same usability expert (M.M.) on the final website for comparison. The purpose of conducting the second usability review was to calculate the change in improvement between the initial review of the website and the redesigned final website based on recommended changes. Since not all recommendations were addressed prior to launch of the randomized clinical trial, it was important to conduct the secondary usability review to understand both the improvements made and the remaining limitations of the website.

RESULTS

Usability Review I (Figure 1) found that 165 out of 247 Travis checklist items had sufficient information to allow for evaluation (Table 1). It is noteworthy that some of the missing items represented important aspects of usability; for example, there was no search functionality on the site.

Table 1.

Usability Review I Results for Nine Usability Categories

Checklist Category Raw score # Questions # Answers Score
Home Page 2 20 19 55%
Task Orientation & Site Functionality 6 44 32 59%
Navigation & Information Architecture 12 29 23 76%
Forms & Data Entry 2 23 16 56%
Trust & Credibility 5 13 10 75%
Writing & Content Quality 13 23 21 81%
Page Layout & Visual Design 6 38 35 59%
Search 0 20 0 00%
Help, Feedback & Error Tolerance −1 37 9 44%
Overall score 247 165 63%
Open in a new tab

The overall score for the website using the Travis (2009) guidelines was 63%. The lowest category percentage scores for those evaluable during the initial review were for the Home Page, Task Orientation & Site Functionality, Page Layout & Visual Design, and Help, Feedback & Error Tolerance.

The team had a tight timeline, limited access to a programmer, and a minimal budget. Therefore recommendations were prioritized by the research team, favoring fixing major issues that would impede website use and small changes that would result in larger perceived improvement, focusing most on those related to the important user tasks. The major recommended changes and website redesign results are detailed below.

The Home Page was the primary focus of redesign. A screenshot of the Home Page design at the time of first usability testing can be seen in Figure 2. The Home Page is the most important page of a website as it holds the value proposition and is the main opportunity to engage the user and entice them to use the site. Most changes to this page required minimal coding to achieve large improvements. Prior to the redesign, the Home Page lacked a Welcome message and value proposition (e.g., what will the end-user get out of using this website), was too busy, and did not list navigation choices in a task-oriented manner. In order to address these issues, the developers simplified the Home Page while adding a Welcome Message with a clear value proposition. Accordions were added to provide instructions and organize content allowing the users to expand and display only the desired section(s). One of these accordions contained specific directions for getting started with the website. This was important as study participants were encouraged to complete actions on the website as part of the research goals. Navigation choices were also reordered on the menu bar so that the important tasks and most used features would be at the top. Figure 3 shows the new Home Page design (with accordions expanded) based on these recommended changes.

Figure 2.

Figure 2

Open in a new tab

A screenshot of the old home page.

Figure 3.

Figure 3

Open in a new tab

A screenshot of the new home page with accordions expanded.

The Task Orientation & Site Functionality category was another area where the website benefitted from simple programming changes. Items in this checklist addressed issues of information (cluttered pages) and activity overload (excessive registration steps) and included items surrounding streamlined design, presenting the most important information at the highest levels, and ensuring first time users could navigate and complete basic tasks. The biggest concern was that the website lacked a critical path for the important tasks. A feature called “My Action Plan” was also added and pre-populated with important user tasks for the patient to review and complete (Figure 4). In addition, a direct link to the Advance Healthcare Directive Form was added to the Home Page. To improve consistency, other pages were updated to include accordion instructions similar to the Home Page. Many of the pages required a lot of scrolling and recommendations for standardization of hyperlinks and reduction of scrolling required by users were made as well.

Figure 4.

Figure 4

Open in a new tab

A screenshot of the My Action Page with pre-populated suggested action items.

Navigation & Information Architecture are important aspects of usability for any website. They are especially important for a research study where one of the goals was to present useful information to patients. For this study, the recommendation to employ the following advice from Webcredible.co.uk was provided to the researchers: 1) Use clear and simple language, 2) Limit each paragraph to one idea, 3) Front-load important content, 4) Use descriptive sub-headings, 5) Bold important words, 6) Use descriptive link text, 7) Use lists, and 8) Left-align text. Following these recommendations, every page was revised to make the content much easier to read and follow web standards. This included removal of excessive capitalization, reduction and reorganization of materials, standardization of hyperlink colors, and consistent use of icons. In addition, standardized icons were implemented and warnings were added so that users would be aware if they were leaving the secured website for external content. Additionally, a site map was recommended to navigation of the complete site, though due to time constraints this was not implemented.

Finally, we addressed items related to Page Layout & Visual Design. This checklist covers topics related to placement of important information, visual consistency of components, and proper use of contrast and colors. On all pages, the most important information (such as frequently used topics, features and functions) needs to be presented on the first screen of information or “above the fold.” Topics and pages were reorganized to meet this recommendation (see Figure 5 for screenshots of pages for critical tasks following redesign). All button, hyperlinks and features were repaired to ensure consistency and as described above, information was rewritten to increase white space and provide information. While the color contrasts met standard web usability guidelines, the lightest colored text was replaced to improve contrast for this aged population.

Figure 5. Screenshots of website user critical tasks.

Figure 5

Open in a new tab

A) Advance Healthcare Directive completion; B) Palliative Care appointment; C) Patient Learning Library; D) Track symptoms via Distress Thermometer.

Based on Usability Review I, a large number of recommendations from every category were made to the research team and many of the recommendations were implemented to create the final version of the intervention website. Following completion of the randomized clinical trial, a second usability review (Usability Review II) was conducted on the final version of the website. Usability Review II achieved an improvement in the usability score from 63% to 73%. Scores for the Home Page, Task Orientation, and Navigation & Information Architecture checklists increased to over 80% (Figure 6).

Figure 6.

Figure 6

Open in a new tab

Comparison of percentage scores for the initial and final reviews by usability category.

DISCUSSION

With the use of standard website usability guidelines, we increased the overall usability of this research website from the initial design, with the greatest improvement in the Home Page usability. The final version of the website scored well; areas that still require either development of or improvement include: Search functionality, Page Layout & Visual Design, and Forms & Data Entry. The largest remaining issue was the lack of aesthetic and minimalist design. The pale colors and lack of color contrast were not the most appropriate choice for this older adult population. Web Content Accessibility Guidelines should be consulted to improve the accessibility and usability of websites and web applications for older people (Henry, Keith et al. 2010). The standard for text size states text can be resized without assistive technology up to 200 percent without loss of content or functionality. Minimum criteria for color require a contrast ratio of at least 4.5:1 for the visual presentation of text and images. Preferably, the contrast should be 7:1, which this website did not meet in many cases. As web pages are typically read in a “Z” formation, a left-to-right menu for the main navigation bar would have been more appropriate than a top-down menu.

Designing websites for older adults or redesigning websites to be acceptable for use by older adults provides developers and usability experts with an initial set of challenges to overcome. Furthermore, usability may impact optimal use in chronically ill populations, such as ours where users are older adults with life threatening illnesses, all which need to be considered during development. There is minimal available research on how chronic illnesses impact internet use, though studies on e-Health interventions are becoming more popular in the literature. Unfortunately, these studies mostly focus on cost effectiveness and health outcomes as opposed to usability or design principles (Eland-de Kok, van Os-Medendorp et al. 2011). This indicates that more research is needed in order to formulate heuristics appropriate for this population. Despite the medical differences between the diseases diagnosed, there are likely some uniting principles that could be identified to improve both usefulness and ease of use, the two most common themes of technology acceptance theories (Davis, Bagozzi et al. 1989, Venkatesh, Morris et al. 2003).

The software development process followed for this study was rapid iterative prototyping, a commonly used technique in design but not always the right fit for a specific project. In this case, the research team spent over a year in development. This meant that only some of the suggested changes from the usability review were implemented due to budget and time constraints for this pilot study. In this study, the research population was clearly defined as were the business requirements (website features and tasks) the population desired. Therefore, the research team could have instead followed a standard software design life cycle (SDLC) process that included a formal business requirements document, a functional design, and a technical design prior to development which may have improved efficiency. Future research studies should consider following a more structured development life cycle and incorporate usability reviews throughout the process.

Despite this, with minimal budget and time to implement recommended changes, the team was still successful in increasing the usability of the website. There were large improvements which highlights the importance of including usability concepts from the start of development.

CONCLUSION

Design and usability of a website can greatly impact usage. If a user experiences difficulty upon first login it is likely they will not return. Usability testing and evaluation are therefore imperative to the development of effective websites for chronic disease research and particularly so for specific cancer populations which vary widely in age of onset, symptoms and symptom management, disease severity, treatment, and prognosis. Women with ovarian cancer are facing a serious, life threatening disease. Incorporating usability concepts from the start of development, fulfilling the positive expectations of end-users and identifying technical and personal factors that optimize use may greatly enhance usage of eHealth tools, facilitating advanced care planning and improving patient support. This may aid in improving quality of life in addition to traditional sources of supportive oncology care.

As technology advances, many eHealth tools are being developed to allow patients access to more extensive and accurate health information and increase communication with their clinicians. Even with the greatest intentions to deliver patient centered information to patients, many websites are not formally tested and if they are, studies have found low use, especially for older patients with more complex or terminal diseases (Eysenbach 2005, Ruland, Maffei et al. 2013). This may occur when users who enroll do not immediately see the benefit of utilizing the tool. Unfortunately few studies have reported facilitators of Web-based use by chronically ill patients or patients’ experiences that might endorse the value their use has represented to them (Kuijpers, Groen et al. 2013). The research challenge remains delivery of a usable environment so that users find the website both useful and easy to use.

Acknowledgments

Funding

This study was funded by the Faculty Research Development Program of the Academic Health Center, University of Minnesota, NIH P30 CA77598 and the National Center for Advancing Translational Sciences of the National Institutes of Health Award Number UL1TR000114.

Biographies

Molly A. McClellan, PhD

Dr. McClellan is a co-owner and consultant with 2MD, a software design and consulting company comprised of global experts in Human Factors and Human-Computer Interaction. She is currently working as a Human Factors Engineer at Medtronic, Inc. Her research interests include usability in software development and human performance in medicine.

Raghav Pavan Karumur, MS

Raghav Karumur is currently a doctoral student in Computer Science at the University of Minnesota, Twin Cities. He received his M.S in Computer Science from University of Minnesota in 2012. His research focuses on user experience and usability, understanding crowd behaviors in online communities, design, data mining and information retrieval.

Rachel Isaksson Vogel, MS

Rachel Isaksson Vogel is a biostatistician with the Masonic Cancer Center of the University of Minnesota. She is primarily interested in research related to women’s cancer and collaborates with basic scientists and clinical doctors to design protocols, write research grants, conduct data analyses, and disseminate study results.

Sue v. Petzel, PhD

Dr. Petzel is a Health Psychologist in the Department of OB-GYN and Women’s Health, University of Minnesota. Psychological and social issues related to cancer represent the primary focus of her clinical and research interests. This includes adapting technology to optimize cancer education.

Julie Cragg, PhD

Julie Cragg holds a PhD in Health Informatics from the University of Minnesota. She has expertise in clinical informatics, patient-oncologist communication, and patient-centered care. Most recently, Julie has worked at Optum in Innovation, Research & Development, and at Medica in Health Management Analytics for improving outcomes.

Daniel Chan, PMP

With close to 25 years of IT experience, Daniel Chan has led numerous projects utilizing technologies that support databases, client-server, eCommerce, CRM and multi-systems connectivity. He embraces the ethos of placing users at the center of systems design and development to drive success in user adoption and ongoing systems operations.

Julie A. Jacko, PhD

Dr. Jacko is a Senior Partner at BioMedical Metrics LLC and an Adjunct Professor in Environmental Health Sciences at the University of Minnesota. She is a world-renowned expert and scholar in human factors engineering, human computer interaction, and health informatics.

François Sainfort, PhD

Dr. Sainfort is a Senior Partner at BioMedical Metrics LLC and an Adjunct Professor in Health Policy and Management at the University of Minnesota. His research expertise focuses on medical decision making, decision support systems design and evaluation, health outcomes modeling and measurement, and health-related performance measurement and analysis.

Melissa A. Geller, MD, MS

Dr. Geller maintains an active clinical practice as a Gynecologic Oncologist and has been awarded over one million dollars in research support including an NIH K12 award and an American Cancer Society Clinical Scholar Grant translating her basic science work with natural killer cells into a novel clinical trial.

Footnotes

Conflict of Interest

None of the authors declare a conflict of interest.

Contributions of Authors

MM conducted the usability evaluation and drafted the manuscript; RPK and DC developed the website; RIV conducted the statistical analyses; MM, RIV, SVP, JC, JAJ, FS, and MG contributed to website content and design; RPK, RIV and SVP provided significant edits to the manuscript; JAJ, FS and MM provided oversight of all phases. All authors approved the final version of the manuscript.

Reference List

  1. American Cancer Society. Ovarian Cancer Detailed Guide. American Cancer Society; 2013. www.cancer.org. [Google Scholar]
  2. Bankhead CR, Kehoe ST, Austoker J. Symptoms associated with diagnosis of ovarian cancer: a systematic review. BJOG. 2005;112(7):857–865. doi: 10.1111/j.1471-0528.2005.00572.x. [DOI] [PubMed] [Google Scholar]
  3. Bodenheimer T, Lorig K, Holman H, Grumbach K. Patient self-management of chronic disease in primary care. JAMA. 2002;288(19):2469–2475. doi: 10.1001/jama.288.19.2469. [DOI] [PubMed] [Google Scholar]
  4. Chisnell D, Redish J. Designing Web Sites for Older Adults: Expert Review of Usability for Older Adults at 50 Web Sites. 2005 Retrieved April 26, 2013, 2013, from assets.aarp.org/ www.aarp.org_/articles/research/oww/AARP-50Sites.pdf.
  5. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly. 1989;13(3):319–340. [Google Scholar]
  6. Davis FD, Bagozzi RP, Warshaw PR. User acceptance of computer technology: A comparison of two theoretical models. Management Science. 1989;35:982–1003. [Google Scholar]
  7. Detmer D, Bloomrosen M, Raymond B, Tang P. Integrated personal health records: transformative tools for consumer-centric care. BMC Med Inform Decis Mak. 2008;8:45. doi: 10.1186/1472-6947-8-45. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Donovan HS, Ward SE, Sereika SM, Knapp JE, Sherwood PR, Bender CM, Edwards RP, Fields M, Ingel R. Web-Based Symptom Management for Women With Recurrent Ovarian Cancer: A Pilot Randomized Controlled Trial of the WRITE Symptoms Intervention. J Pain Symptom Manage. 2014;47(2):218–230. doi: 10.1016/j.jpainsymman.2013.04.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Doupi P, van der Lei J. Towards personalized Internet health information: the STEPPS architecture. Med Inform Internet Med. 2002;27(3):139–151. doi: 10.1080/1463923021000014149. [DOI] [PubMed] [Google Scholar]
  10. Eland-de Kok P, van Os-Medendorp H, Vergouwe-Meijer A, Bruijnzeel-Koomen C, Ros W. A systematic review of the effects of e-health on chronically ill patients. J Clin Nurs. 2011;20(21–22):2997–3010. doi: 10.1111/j.1365-2702.2011.03743.x. [DOI] [PubMed] [Google Scholar]
  11. Ellis RD, Kurniawan SH. Increasing the Usability of Online Information for Older Users: A Casey Study in Participatory Design. International Journal of Human-Computer Interaction. 2000;12(2):263–276. [Google Scholar]
  12. Eysenbach G. The law of attrition. J Med Internet Res. 2005;7(1):e11. doi: 10.2196/jmir.7.1.e11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Fauci J, Schneider K, Walters C, Boone J, Whitworth J, Killian E, Straughn JM., Jr The utilization of palliative care in gynecologic oncology patients near the end of life. Gynecol Oncol. 2012;127(1):175–179. doi: 10.1016/j.ygyno.2012.06.025. [DOI] [PubMed] [Google Scholar]
  14. Ferrell B, Paice J, Koczywas M. New standards and implications for improving the quality of supportive oncology practice. J Clin Oncol. 2008;26(23):3824–3831. doi: 10.1200/JCO.2007.15.7552. [DOI] [PubMed] [Google Scholar]
  15. Griffiths F, Lindenmeyer A, Powell J, Lowe P, Thorogood M. Why are health care interventions delivered over the internet? A systematic review of the published literature. J Med Internet Res. 2006;8(2):e10. doi: 10.2196/jmir.8.2.e10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Henry S, Keith S, Roberts K. Web Content Accessibility Guidelines (WCAG) 2.0. Strategies, guidelines, resources to make the Web accessible to people with disabilities. 2010 Sep 22; 2010. Retrieved March 20, 2014, 2014. [Google Scholar]
  17. Kirk P, Kirk I, Kristjanson LJ. What do patients receiving palliative care for cancer and their families want to be told? A Canadian and Australian qualitative study. BMJ. 2004;328(7452):1343. doi: 10.1136/bmj.38103.423576.55. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Kuijpers W, Groen WG, Aaronson NK, van Harten WH. A systematic review of web-based interventions for patient empowerment and physical activity in chronic diseases: relevance for cancer survivors. J Med Internet Res. 2013;15(2):e37. doi: 10.2196/jmir.2281. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Lynch KR, Schwerha DJ, Johanson GA. Development of a Weighted Heuristic for Website Evaluation for Older Adults. International Journal of Human-Computer Interaction. 2013;29(6):404–418. [Google Scholar]
  20. Miller SM. Monitoring versus blunting styles of coping with cancer influence the information patients want and need about their disease. Implications for cancer screening and management. Cancer. 1995;76(2):167–177. doi: 10.1002/1097-0142(19950715)76:2<167::aid-cncr2820760203>3.0.co;2-k. [DOI] [PubMed] [Google Scholar]
  21. Molich R, Nielson J. Improving a human-computer diaglogue. Communications of the ACM. 1990;33(3):338–348. [Google Scholar]
  22. Nielsen J. Heuristic evaluation. New York, NY: John Wiley & Sons; 1994. [Google Scholar]
  23. Nielsen J, Molich R. Heuristic evaluation of user interfaces. Proc ACM CHI; Seattle, WA. 1990. [Google Scholar]
  24. Ofran Y, Paltiel O, Pelleg D, Rowe JM, Yom-Tov E. Patterns of information-seeking for cancer on the internet: an analysis of real world data. PLoS One. 2012;7(9):e45921. doi: 10.1371/journal.pone.0045921. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Patsoule E, Koutsabasis P. Redesigning websites for older adults: a case study. Behaviour & Information Technology. 2014;33(6):561–573. [Google Scholar]
  26. Paul CL, Carey ML, Sanson-Fisher RW, Houlcroft LE, Turon HE. The impact of web-based approaches on psychosocial health in chronic physical and mental health conditions. Health Educ Res. 2013;28(3):450–471. doi: 10.1093/her/cyt053. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Romano Bergstrom JC, Olmsted-Hawala EL, Jans ME. Age-related differences in eye tracking and usability performance: Website usability for older adults. International Journal of Human-Computer Interaction. 2013;29(8):541–548. [Google Scholar]
  28. Roth AJ, Kornblith AB, Batel-Copel L, Peabody E, Scher HI, Holland JC. Rapid screening for psychologic distress in men with prostate carcinoma: a pilot study. Cancer. 1998;82(10):1904–1908. doi: 10.1002/(sici)1097-0142(19980515)82:10<1904::aid-cncr13>3.0.co;2-x. [DOI] [PubMed] [Google Scholar]
  29. Ruland CM, Maffei RM, Borosund E, Krahn A, Andersen T, Grimsbo GH. Evaluation of different features of an eHealth application for personalized illness management support: Cancer patients’ use and appraisal of usefulness. Int J Med Inform. 2013 doi: 10.1016/j.ijmedinf.2013.02.007. [DOI] [PubMed] [Google Scholar]
  30. Smith TJ, Temin S, Alesi ER, Abernethy AP, Balboni TA, Basch EM, Ferrell BR, Loscalzo M, Meier DE, Paice JA, Peppercorn JM, Somerfield M, Stovall E, Von Roenn JH. American Society of Clinical Oncology provisional clinical opinion: the integration of palliative care into standard oncology care. J Clin Oncol. 2012;30(8):880–887. doi: 10.1200/JCO.2011.38.5161. [DOI] [PubMed] [Google Scholar]
  31. Stewart DE, Wong F, Cheung AM, Dancey J, Meana M, Cameron JI, McAndrews MP, Bunston T, Murphy J, Rosen B. Information needs and decisional preferences among women with ovarian cancer. Gynecol Oncol. 2000;77(3):357–361. doi: 10.1006/gyno.2000.5799. [DOI] [PubMed] [Google Scholar]
  32. Strecher V. Internet methods for delivering behavioral and health-related interventions (eHealth) Annu Rev Clin Psychol. 2007;3:53–76. doi: 10.1146/annurev.clinpsy.3.022806.091428. [DOI] [PubMed] [Google Scholar]
  33. Travis D. 247 web usability guidelines. 2009 Jul 6; 2009. Retrieved 04/01/2012, 2012, from http://www.userfocus.co.uk/resources/guidelines.html.
  34. United States Cancer Statistics Working Group. United States Cancer Statistics: 1999–2010 Incidence and Mortality Web-based Report. Atlanta, GA: Centers of Disease Control and Prevention; 2013. D. o. H. a. H. Services. [Google Scholar]
  35. Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: Towards a unified view. MIS Quarterly. 2003;27(3):425–478. [Google Scholar]
  36. Vogel RI, Petzel SV, Cragg J, McClellan M, Chan D, Dickson E, Jacko JA, Sainfort F, Geller MA. Development and pilot of an advance care planning website for women with ovarian cancer: A randomized controlled trial. Gynecol Oncol. 2013;131(2):430–436. doi: 10.1016/j.ygyno.2013.08.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Williams-Piehota P, Latimer AE, Katulak NA, Cox A, Silvera SA, Mowad L, Salovey P. Tailoring messages to individual differences in monitoring-blunting styles to increase fruit and vegetable intake. J Nutr Educ Behav. 2009;41(6):398–405. doi: 10.1016/j.jneb.2008.06.006. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES