A roadmap for the development and evaluation of the eHealthResp online course

Abstract

Background

Inappropriate antibiotic use constitutes one of the most concerning public health issues, being one of the main causes of antibiotic resistance. Hence, to tackle this issue, it is important to encourage the development of educational interventions for health practitioners, namely by using digital health tools. This study focuses on the description of the development and validation process of the eHealthResp online course, a web platform directed to physicians and pharmacists, with the overall goal of improving antibiotic use for respiratory tract infections, along with the assessment of its usability.

Methods

The eHealthResp platform and the courses, developed with a user-centered design and based on Wordpress and MySQL, were based on a previously developed online course. A questionnaire to assess the usability was distributed among physicians (n = 6) and pharmacists (n = 6). Based on the obtained results, statistical analyses were conducted to calculate the usability score and appraise the design of the online course, as well as to compare the overall scores attributed by both groups. Further qualitative comments provided by the participants have also been analyzed.

Results

The eHealthResp contains two online courses directed to physicians and pharmacists aiming to aid in the management of respiratory tract infections. The average usability score of the eHealthResp online courses for physicians and pharmacists was of 78.33 (±11.57, 95%CI), and 83.75 (±15.90, 95%CI), respectively. Qualitative feedback emphasized the usefulness of the course, including overall positive reviews regarding user-friendliness and consistency.

Conclusions

This study led us to conclude that the eHealthResp online course is not recognized as a complex web platform, as both qualitative and quantitative feedback obtained were globally positive.

Background

Antibiotic resistance is considered one of the major Public Health threats worldwide, with inappropriate use of antibiotics being one of the main concerns, especially for respiratory tract infections.^1–4 Considering: (i) that respiratory diseases are one of the leading causes of death and disability, (ii) the high incidence of respiratory tract infections, and (iii) the widespread overuse of antibiotics for these diseases, interventions to improve antibiotic use constitute an essential approach.^3–5 However, the effectiveness of antimicrobial stewardship interventions strongly depends on an adequate design, tailored to each setting.^6–8

Allying digital health tools to educational interventions for health practitioners can significantly improve healthcare quality,^9–11 going from the reduction of medication errors^12–14 to the improvement of antibiotic prescription quality.^15,16 e-Health instruments, especially clinical decision support systems (CDSS), comprise a multiplicity of tools that aid in clinical decision making, thus saving time needed to strengthen the relationship with patients and facilitating the act of providing care.^13,17–19 Hence, as increasingly new information on antibiotic use emerges, educating and informing both patients and health professionals becomes highly essential to enhance clinical practices, ensuring they are up-to-date. ²⁰

Our research group developed eHealthResp, ²¹ a digital platform comprising two online courses, one directed to primary care physicians and the other one to community pharmacists. Both courses address respiratory infections’ management, with the goal of improving healthcare quality,^9–11 and, ultimately, promoting adequate antibiotic use,^12–16 specifically for respiratory tract infections.

To improve the overall design and ensure the adequacy and user-friendliness of the eHealthResp online courses, the evaluation of their usability constitutes a critical step for the assessment of digital applications in human health. ²² Thus, the main goal of this study is to provide a description of the development and validation process of the eHealthResp online course, having as the main outcome the assessment of its usability by using the System Usability Scale (SUS).^23,24 Furthermore, as the web platform in which both online courses are embedded is the same, this study aims to compare the results obtained between the usability scores provided by physicians and pharmacists.

Methods

eHealthResp project and website

The eHealthResp website is part of a research project that comprises an educational intervention designed for primary care physicians and community pharmacists, which will be conducted through a cluster randomized controlled trial on the geographical area of Portugal's Center Regional Health Administration (ARS-C). This intervention consists of an online course and a mobile app composed by several algorithms for the management of respiratory tract infections in adults, serving as a useful aid to the clinical decision process.

The eHealthResp platform and the courses, were developed with a user-centered design, and were based on a previously developed online course. ²⁵ After a thorough bibliographic review and testing of different platforms, the re-organization of content and navigation structure has been conducted. The website was then developed on a Wordpress and MySQL based system, with Elementor and LifterLMS, a learning management system, as main plugins.

Web platform and course overview

The eHealthResp is a Wordpress-based web platform that contains two self-paced online courses directed to physicians and pharmacists aiming to aid in the management of respiratory tract infections. Additionally, the webpage serves as a host to the eHealthResp project's information, including a contacts section, a publications section, and a page for the download of the mobile app (see supplementary material S1).

The physicians’ course contains four sections, consisting of: 1) an introduction to the online course's contents and a brief overview regarding respiratory tract infections; 2) six modules on specific respiratory tract infections (namely i) acute otitis media, ii) acute rhinosinusitis, iii) acute pharyngitis, iv) acute bronchitis, v) community-acquired pneumonia, with an additional module for vi) differential diagnosis of COVID-19); 3) four clinical cases; 4) satisfaction questionnaire and course completion page. Similarly, the pharmacists’ online course contains the same structure, apart from having only three modules (specifically i) common cold and flu, ii) acute rhinosinusitis, acute pharyngitis, and acute bronchitis and iii) acting protocol), instead of six. Each group of health professionals has access to their reserved area.

Course and module structure

The module pages consist of a slideshow section, in which the user can navigate freely through the presentation. At the bottom of the slideshow section, the page presents a “Mark as complete” button, to register the module as completed thus granting access to the next module, and a “Download” button, which allows the users to save the presentation as a PDF file to their devices and access them offline. Furthermore, these pages also include two navigation buttons, to return to the previous module or to advance towards the next.

Content validation

The eHealthResp online courses’ contents have been subjected to content validation through a Delphi Method approach. ²⁶ For this study, several experts have been invited to help to improve both online courses’ contents, providing feedback regarding several clinical cases which were further included in the presentations and clinical cases sections. Besides the content validation, and since the eHealthResp platform has been developed with a user-centered design, two usability studies have also been conducted.

Usability testing

Six physicians and six pharmacists ²⁴ were recruited through a convenience sample to participate in a study aiming to validate the usability of the online course strictly directed to physicians and pharmacists, respectively. The participants were invited to participate in the study and asked to explore the site contents, with a special focus on the usability of the website.^27,28 To provide them with access to the restricted area, the website's URL was sent by e-mail, along with an individual username and password and requesting participants to fully explore the website and the online course. These credentials granted access to the course contents, providing them with full autonomy to explore the website.

Along with the access credentials sent by e-mail, a hyperlink to the usability questionnaire was also sent to each participant. This questionnaire was composed by ten mandatory closed-ended questions, based on the System Usability Scale, and an optional comment box, in which participants were able to provide comments about their user experience. Participants were given around two weeks to fully explore the website pages and to complete the online course.

In accordance with the General Data Protection Regulation (GDPR), participants provided their informed consent for the website credentials and questionnaire to be sent to their e-mails. Furthermore, each participant was informed about the objectives of this study and freely consented to participate in this study, providing their consent when answering the questionnaire.

System usability scale

The System Usability Scale (SUS) consists of a group of ten questions, in which participants should provide an answer based on a 5-point Likert scale numbered from 1 (“Strongly disagree”) to 5 (“Strongly agree”).^24,29–31 To calculate the usability score for each participant, odd-numbered questions (SOQ) scores and even-numbered questions (SEQ) scores were combined to obtain a 100-point scale.^23,32

Questionnaire's results analysis

Descriptive statistical analyses were conducted to evaluate the usability of the online course. To ensure the adequacy of the scale, internal reliability statistical tests were performed through the calculation of Cronbach's alpha.^24,29–31 As the variables did not follow a normal distribution, non-parametric tests were conducted. Hence, the differences between physicians and pharmacists were evaluated using the Mann–Whitney U test The outcomes were established as statistically significant at p < 0.05. Moreover, the research team analyzed the qualitative data obtained to better understand the final feedback about the course.

Results

Content validation and usability testing

After conducting the usability study with both physicians and pharmacists, the average score attributed by each group was of 78.33 (±11.57, 95%CI), and 83.75 (±15.90, 95%CI), respectively. The table 1 presented below compares the overall perception between physicians and pharmacists:

Table 1.

Comparison between physicians’ and pharmacists’ usability evaluation.

Item	Median (PCT25, PCT75)		Mann-Whitney (p-value)
	Physicians	Pharmacists
1. I think that I would like to use this system frequently.	3.50 (2.75, 4.00)	4.50 (3.50, 5.00)	0.13
2. I found the system unnecessarily complex.	2.00 (1.00, 2.50)	1.50 (1.00, 2.25)	0.70
3. I thought the system was easy to use.	5.00 (4.75, 5.00)	5.00 (3.75, 5.00)	0.59
4. I think that I would need the support of a technical person to be able to use this system.	1.00 (1.00, 2.00)	1.00 (1.00, 2.00)	0.82
5. I found the various functions in this system were well integrated.	3.50 (3.00, 4.00)	5.00 (4.00, 5.00)	0.02
6. I thought there was too much inconsistency in this system.	2.50 (2.00, 3.25)	1.50 (1.00, 3.25)	0.31
7. I would imagine that most people would learn to use this system very quickly.	4.50 (3.75, 5.00)	5.00 (3.75, 5.00)	0.70
8. I found the system very cumbersome to use.	1.00 (1.00, 2.50)	1.00 (1.00, 2.25)	1.00
9. I felt very confident using the system.	5.00 (4.75, 5.00)	4.50 (4.00, 5.00)	0.39
10. I needed to learn a lot of things before I could get going with this system.	1.00 (1.00, 1.75)	2.00 (1.00, 3.50)	0.39
Total score	85.00 (65.63, 88.13)	91.25 (64.38, 98.13)	0.49

As observed in table 1, most of the scores between physicians and pharmacists followed a similar distribution. However, a statistically significant difference was detected regarding the integration of the website's functions, where physicians tended to attribute a lower score than pharmacists. Nevertheless, the average overall score between physicians and pharmacists has differed less than 5 points.

Qualitative feedback

To complement the quantitative feedback obtained through the SUS's results, a comment box was included on the usability questionnaire sent to physicians and pharmacists. Though not all participants provided further comments on the eHealthResp online course, those who did have highlighted the eHealthResp online course usefulness and user-friendliness:

“I found the course very objective and practical. The search and navigability are not complex allowing good accessibility to the content.” – Pharmacist 1

“[The eHealthResp online course] is easy to use and very objective” – Physician 1

“Very educational. Clearly presented cases, without any doubts. Very useful for testing the quick thinking of diagnosis and treatment” – Physician 2

Still, some physicians have also suggested some improvements in the online course's contents:

“I suggest some corrections in the course contents, namely in the topic otitis media and acute pharyngitis.” – Physician 3

“Since the course is aimed at physicians, scientific language described in each pathology should be improve and adapted, as well as in the description of clinical cases.” – Physician 4

eHealthResp development roadmap

After the content validation and usability testing, the eHealthResp online course and contents have been readjusted by the research team. The Figure 1 illustrates a roadmap for the eHealthResp online course development, from its first development stages until its launch.

Figure 1.

Ehealthresp development roadmap.

Discussion

By following a user-centered design, the eHealthResp online course demanded a close interaction with the end-user throughout the development process. This iterative process, conducted through the validation of both the online courses’ contents ²⁶ and the usability of the web platform, constitutes one of the main strengths of this educational intervention. Overall, considering the highly positive feedback provided by the participants, the results obtained with the usability study reveal that the eHealthResp web platform does not show signs of inconsistencies, and is not perceived as a complex platform.

As most of the positive items have received a score above four, and most of the negative items had an average score of 2 or below, these outcomes reflect the user-friendliness of the online course, being in line with other usability studies for e-learning tools, especially with physicians, in which the usability score obtained has shown to be similar.^31,33,34

When comparing to the results obtained for the online course for pharmacists, only the question related to function integration has had a statistically significant lower score. Yet, the overall feedback on this statement has remained positive on both groups. Hence, the similarity between results and generally good scores gives us a sense of consistency and quality of both online courses and the web platform. As the reports provided by health professionals were positive towards eHealthResp and other systems, the notion that these tools can strongly improve clinical practice is here reinforced.^35–39

Moreover, it is important to note that participants using a tablet or cell phone to explore the eHealthResp website and online courses provided an average score significantly lower (> 20 points) than the scores provided by those using a computer/laptop. Despite these differences are in agreement with the literature, ⁴⁰ a possible explanation might be associated to the fact that most of the educational content is available in a slideshow mode, which might be more adequate for a computer/laptop screen. However, issues regarding scale and ease in navigation through the website might also contribute to the observed differences. Nevertheless, to tackle these difficulties, a “Download” button has been included, so contents can be easily downloaded as a PDF file, thus allowing to scale the contents easily, offline, and outside the browser.

Yet, although the SUS has several strengths, such as content validity and reliability,^23,41,42 and allows for a simple usability assessment, it only provides quantitative feedback. Hence, to tackle this lack of specificity, a commentary section has been added to the questionnaire, so participants could provide their qualitative feedback if they deemed necessary. Despite of being an optional evaluation parameter, five out of six physicians and one pharmacist have left some suggestions, most of them being related with technical corrections to the online course contents. However, when considering the comments provided by participants regarding the web platform per se, only positive feedback has been obtained, with emphasis on eHealthResp online course's easiness to use, overall aspect, and usefulness, which reinforces its viability.

Even though the end-goal of this study was to evaluate the usability of the eHealthResp online course, the technical comments on the contents were also taken into account, complementing the previously content validation study conducted by our group, ²⁶ and improving its overall quality.

Conclusions

The usability evaluation of the eHealthResp presented positive overall scores in terms of user-friendliness, complexity, and consistency. The eHealthResp online course aims to aid health practitioners to manage respiratory tract infections, and this study has allowed to obtain qualitative feedback from possible future users of the online course, which is currently being prepared for a pilot study involving a group of health professionals.

Usability is a very important dimension when developing digital educational contents. The validation of the online course eHealthResp, both in terms of its contents and usability, will support in the improvement of the educational intervention that will cover all primary care physicians and community pharmacists in the Center Region of Portugal, belonging to the ARS-C, as a cluster randomized controlled trial. We believe that this study may be an important description of the different phases that take place throughout an online course design and development, and thus hope to serve as a model to future educational interventions – not only for antibiotic resistances and respiratory diseases management but for clinical practice in general.