A web application to involve patients in the medication reconciliation process: a user-centered usability and usefulness study

Abstract

Objective

Medication reconciliation (MedRec) can improve patient safety by resolving medication discrepancies. Because information technology (IT) and patient engagement are promising approaches to optimizing MedRec, the SEAMPAT project aims to develop a MedRec IT platform based on two applications: the “patient app” and the “MedRec app.” This study evaluates three dimensions of the usability (efficiency, satisfaction, and effectiveness) and usefulness of the patient app.

Methods

We performed a four-month user-centered observational study. Quantitative and qualitative data were collected. Participants completed the system usability scale (SUS) questionnaire and a second questionnaire on usefulness. Effectiveness was assessed by measuring the completeness of the medication list generated by the patient application and its correctness (ie medication discrepancies between the patient list and the best possible medication history). Qualitative data were collected from semi-structured interviews, observations and comments, and questions raised by patients.

Results

Forty-two patients completed the study. Sixty-nine percent of patients considered the patient app to be acceptable (SUS Score ≥ 70) and usefulness was high. The medication list was complete for a quarter of the patients (7/28) and there was a discrepancy for 21.7% of medications (21/97). The qualitative data enabled the identification of several barriers (related to functional and non-functional aspects) to the optimization of usability and usefulness.

Conclusions

Our findings highlight the importance and value of user-centered usability testing of a patient application implemented in “real-world” conditions. To achieve adoption and sustained use by patients, the app should meet patients’ needs while also efficiently improving the quality of MedRec.

Introduction

Continuity of medication is a worldwide patient safety concern requiring information-sharing among providers, patients, and families across settings.^1–4 Medication discrepancies are unexplained differences between medication lists at different transition points of care.⁵ These discrepancies threaten patient safety.⁴,⁶,⁷

Medication reconciliation (MedRec) is the formal, collaborative process of obtaining and verifying a complete, accurate list of a patient’s current medication to ensure that precise, comprehensive medication information is transmitted consistently across transition points of care.⁸ MedRec makes it possible to resolve discrepancies. Although several leading organizations worldwide^9–13 have campaigned for the implementation of MedRec, improving MedRec remains challenging.¹⁴

The use of information technology (IT) seems a promising approach.³,^15–18 However, rigorous studies to evaluate the impact of electronic MedRec (eMedRec) are lacking.¹⁷,¹⁹

Patient engagement is increasingly recognized as a key component in the current redesign of health care processes. Accordingly, different research teams have developed electronic applications that allow patients to document medication lists.^20–26 These applications are generally standalone²⁶ or linked to a single source of information.²⁰,²²,²⁵ Moreover, all applications have been developed in the United States, where health systems, environments, and culture differ from Europe.

This study took place in Wallonia (a region in Belgium of about 3 600 000 inhabitants), where more than 1 million inhabitants are connected to the Regional eHealth Network (ReHN).²⁷ The ReHN allows the exchange of eHealth documents between healthcare professionals (HCPs) caring for a particular patient. For example, a cardiologist in hospital A can access the Summarized Electronic Health Record produced by the general practitioner (GP) and the discharge letter from hospital B.

The SEAMPAT project is a multidisciplinary research project aimed at developing an eMedRec process. The project chose to actively engage patients and to support HCPs—both in primary and secondary care—through two specific eMedRec applications¹⁸,¹⁹ interconnected with the ReHN: one for patients (“patient app”) and one for HCPs (MedRec App). Both were developed according to recommendations by health IT experts.^28–32 Requirements for the low-fidelity prototypes were defined on the basis of needs analysis and literature review.¹⁹,³³ These then evolved to medium- and high-fidelity prototypes using an iterative user-centered design with three main iterations. Details of the development and the first two phases of evaluation are reported elsewhere.³³

In the present study, we report the results of the third iteration phase with the high-fidelity prototype (ie a working prototype connected to real eHealth data) of the patient app. Our objective was to assess the usability and usefulness from the perspective of different categories of patients, with the aim of making recommendations that could lead to increased adoption and sustained use of the patient app and inform further research in the field.

Methods

System Details

The patient app (using HTML5/JavaScript technologies), usable on a desktop or laptop at home or in a hospital, lets patients document their medication lists. This IT system is protected by high-security mechanisms.³³ Patient authentication is by the Belgian federal electronic identity (eID) card and its PIN code. Only patients registered with the ReHN can use the app. The app (Figure 1) presents patients with a list of medications compiled from different sources available on the ReHN, using a specific algorithm to prevent conflicting information: the GP’s medication list, the hospital’s medication list, and the patient’s medication list (except for the first connection to the patient app). For each medication, patients indicate if and how they are taking it by choosing the appropriate option (taking as indicated, taking differently, not taking, unknown). Patients can also add medications and may also add an indication and name of prescriber (Figure 1). The patient medication list (Figure 2) is then updated on the ReHN.

Figure 1.

Patient app screenshots. Upper row: Left: Patient medication list = list compiled from different sources available on the ReHN. For each medication, patients can choose the right option: No, Yes, etc. Right: Patients can also add medications to the list by clicking on one of the icons. Finally they can add information concerning recent antibiotic intake and allergies. Middle row: If patients click on the magnifying glass, they can see details of date of prescription, indication, dose and frequency, larger pillbox picture, etc. Lower row: Left: After clicking on one of the icons (upper right), patients can start typing a medication’s name and the system shows existing medications and box pictures. Right: After selecting the right medications, patients can indicate for what purpose (indication) they are taking the medication and who prescribed it.

Figure 2.

Screenshots: a patient logged onto his/her Regional eHealth Network (ReHN) page. Upper row: Patient’s welcome page with his name in the upper right part of the page. Patient’s list of documents with patient’s medication lists (ie “schema de medication”) published on the ReHN by the patient app. Lower row: Patient medication list published by the patient. The column status shows the option chosen by the patient in answer to the question: “Are you currently taking this medication?”

At the time of our study, the MedRec app was still under development. The working prototype was not available yet; therefore ,HCPs could not use the MedRec app to look at the medication documented by the patient and to perform MedRec.

Study Design

This was a four-month user-centered descriptive study of usability and usefulness. In the context of our study, usability was defined according to ISO 9241-11 as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.”³⁴:

• Satisfaction: How satisfied is the patient with the tool?

• Effectiveness: How complete and correct is the patient medication list?

• Efficiency: What is the level of patient effort required?

Usefulness was defined as the degree to which a product enables users to meet specific needs and as an assessment of users’ willingness to use the product.³⁵ A table summarizing the different concepts evaluated, their definitions, and the measurement method is available in Appendix 1.

This paper follows the recommendations of the Statement on Reporting of Evaluation Studies in Health Informatics.³⁶ The study protocol was approved by the Ethical Committee of CHU UCL Namur.

Sampling/participants

We used a purposive sample of patients at high risk of medication discrepancies, who varied in terms of age, gender, education, illness, health literacy, computer literacy, setting, and medical encounters. Three groups of patients were defined: pulmonary transplant outpatients (who regularly have consultations with specialist physicians), outpatients visiting their GP at least once a month, and inpatients hospitalized for scheduled surgery. Eligibility criteria were: age older than 18 years; ability to speak and read French; use of ≥ 5 daily medications; and willingness to participate. Patients with cognitive or visual impairment were excluded unless they could be accompanied by a caregiver. Sociodemographic and computer knowledge data were provided by patients using questionnaires.

We aimed to recruit 15 participants per group. Pulmonary transplant outpatients and surgical inpatients were recruited in a rural teaching hospital (CHU UCL Namur, Belgium). Outpatients followed in primary care were cared for by GPs working in the same area as the teaching hospital and were recruited with their help. Patients were informed of the purpose and content of the study. A letter describing the study was sent to them and their principal physician. Every patient gave his/her informed consent. They were not compensated for their time, nor did they receive any incentive for participation.

Study Flow

Study flow proceeded in 4 steps (Figure 3). First, patients were contacted by phone for recruitment. Second, patients attended a “kick-off” session, for their first use of the app. We grouped 4 to 6 patients per session, in the presence of 3 to 5 researchers (at least 1 physician and 1 IT scientist). After an introduction, patients were invited to log into the patient app and document their medication list. Field notes were taken by researchers to record issues and challenges regarding the tool’s use. Critical usability issues were identified and immediately resolved. Third, patients could use the app at home over 3-4 months. They received a printed user guide and were instructed to use the app, if possible, before each new visit to any HCP. As their only reminder, patients received a phone call from the main researcher (SM) approximately 8 weeks after the kick-off. At any time, patients could ask for help by email or phone. For problems not solved remotely, a team member visited patients’ homes. Finally, every patient met the main researcher, at home or at the hospital, for a one-hour semi-structured interview.

Figure 3.

Patient flow and related measurements. Abbreviations: BPMH = Best Possible Medication History; eID = Electronic identity card; IT = information technology; ReHN = Regional eHealth Network; SUS = System Usability Scale.

Evaluation Criteria

As recommended by the Agency for Healthcare Research and Quality³² and by 2 usability textbooks,²⁹,³⁰ quantitative and qualitative data were collected at different times in order to gain an in-depth understanding of the app’s usability and usefulness (Appendix 1).

Quantitative measures

Usability and usefulness were assessed after the first app use and at the end of the study, using 2 questionnaires. To evaluate usability, the SUS (ten statements rated on a 5-point Likert scale) were selected because it has been successfully used in the medical domain, has been validated for a wide range of interface technologies, is quick to complete and easy to interpret, and can be applied to small sample sizes.^37–41 To evaluate usefulness, we developed a questionnaire based on previous publications.^42–44 The 7 statements were rated on a 5-point Likert scale.

Effectiveness was evaluated at the end of the study, by assessing the completeness and correctness of medication lists published on the ReHN using the patient app.⁴⁵ Firstly, completeness was analyzed in patients who used the app between 3 to 70 days before the final interview. This interval was selected in order to have a recent medication list updated by the patient. A shorter interval was not chosen, as some patients may not have frequent medication changes. Completeness was defined as the presence of name, dose, frequency, and medication status. All 4 pieces of information needed to be present for all medications on the list for it to be considered complete for each patient. Medication status corresponds to the option chosen by the patient for the question: “Are you currently taking this medication?” When no option was selected, the medication was considered as “without status” and the list, thus, incomplete. Status was considered to be essential information, as the information presented by the patient app on name, dosage, and frequency could possibly come from a HCP and not always from the patient. Secondly, correctness was evaluated by analyzing medication discrepancies for patients who documented the medication status of each medication. Discrepancies were defined as differences between the list documented by the patient and the best possible medication history (BPMH, gold standard), performed by a trained clinician during the final interview. Types and causes (Appendix 2) of discrepancies were analyzed, using a tool previously validated by our team.⁴⁶ Statistical analysis was performed using SPSS version 24 (eg continuous variables were compared between the start and the end of the study using a paired samples T-test, and between two groups using the Mann-Whitney U test, after assessing data normality; categorical variables were compared between groups using Pearson’s chi² test.)

It was technically impossible to collect quantitative data on efficiency.

Qualitative measures

Qualitative data concerning usability and usefulness were collected during discussion with patients and observations of patients interacting with the system (Appendix 1). At the kick-off session, patients performed specific tasks (ie log-in, documenting medications status, adding medication if relevant, log-out). They were asked to describe their first perceptions by writing down 3 positive and 3 negative aspects of the app. When researchers had contact with patients during the study, they took written notes of patients’ comments. During the individual interview at the end of the study, the main researcher met patients, using an interview guide. This guide was based on the literature and experts’ advice,^42–44,⁴⁷ was reviewed by the research team, and was tested with 1 patient (Appendix 3). The interview aimed to address user’s experiences and: (1) usability, (2) usefulness, and (3) patient’s usual medication organization (eg use of a pillbox, use of reminders) to understand how the app could fit in with the patient’s habits. Interviews were not recorded. During interviews the main researcher wrote keywords to capture patients’ comments. When very specific ideas were mentioned, these were transcribed verbatim.

Data were coded using a conventional qualitative content analysis approach. Interview notes, observation notes, and patients’ emails were triangulated to develop initial codes. Data were analyzed by 2 independent researchers (SM and DL) and organized according to the initial codes. Codes were then sorted into categories and subcategories to create meaningful results.⁴⁸,⁴⁹ A third researcher (AS) then checked categories and coding. Discrepancies were resolved by discussion between researchers. No specific conceptual framework guided the approach.

Results

Participants

A hundred eligible patients were invited by phone to participate. Forty-four agreed to participate. Reasons for non-participation are listed in Figure 3. The caregivers of 4 participating patients who asked to take part in the study and who met the inclusion criteria were also included. In total, 48 patients agreed to participate and 42 completed the study. Patient characteristics are presented in Table 1.

Table 1.

Characteristics of the study population at baseline (n = 48)

Sociodemographic variables
“Subgroup” (n)
Pulmonary transplants	16	33.3%
Ambulatory patients visiting their general practitioner at least once a month	20	41.7%
Patients hospitalized for scheduled surgery	8	16.7%
Caregiver of patient from another group	4	8.3%
Age (years; median [IQR])
	65	[60–69]
Gender (n)
Female	16	33.3%
Male	32	66.7%
Education
Middlea	15	31.3%
Highb	30	62.5%
Unknown	3	6.2%
Most frequent comorbidities
Hypertension	29	60.4%
Sleeping troubles (disorder)	18	37.5%
Cardiovascular disease	13	27.1%
Diabetes	11	22.9%
Medication variables at baseline per patient
Number of medications (median [IQR])	8	[5-11]
Pulmonary transplants	11.5	[10-13.5]
Ambulatory patients visiting their general practitioner at least once a month	7	[5-8]
Patients hospitalized for scheduled surgery	4.5	[4-7]
Caregiver of patient from another group	6	[5-8]
Over-the-counter medication (n)	31	64.6%
Use of IT at home
Regular use of a computer (>1x/week)	39	81.3%
Internet connection at home	45	93.8%
Knowledge of their eID PIN CODE	37	77.1%
Connected to other e-fgov applications (for example: Taxonweb to complete a tax form)	20	41.7%

Abbreviations: IQR = interquartile range; e-fgov application = electronic application of the federal government of Belgium.

Definitions: Education in Belgium was divided into two groups: ^aMiddle education level: primary or secondary school education; ^bHigh education level: professional higher education or university.

From the kick-off to the end of the study, the patient app was used 2 to 4 times by 20 patients of the 42 (47.6%), ≥ 5 times by 12 (28.6%), and once (during the kick-off meeting) by 10 (23.8%). The 32 who used the app at home (76.2%) made 147 connections (median: 3; IQR= 1.25-5). The number of connections was not significantly different between patient groups, except that the four caregivers tended to connect more frequently.

Usability

SUS questionnaire

The SUS questionnaire was filled in by 29 (91%) of the 32 patients who used the app at home. The median SUS score was 72.3 at the start and 75.0 at the end of the study. More than half of the patients (69%) rated the app above 70 both at the start and at the end³⁹,⁴⁰ (Appendix 4). No correlation was found between SUS scores and patients’ age, sex, group, previous IT use, or availability of a medication list provided by the GP or the hospital. The SUS score at the end of the study was slightly correlated with the number of connections, as high users (patients who used the app ≥4 times during the study) gave a median SUS score of 80 [70-83.8] versus 72.5 [67.5-75] for low users (p=.04).

Effectiveness: completeness and correctness

Completeness was evaluated for 28 patients (Table 2a). The medication list was complete for 7 of these 28 patients (25%). The main reason for incompleteness was the absence of medication status. This was mainly due to misunderstanding of the color code. For example, a patient viewing a green button – “Yes, I’m taking this medication as indicated” – did not realize the button was green (and the status, thus, already documented) because he/she had already selected this option during a previous connection. By clicking it again, the information on status was deselected and green changed to gray. Having a documented status for each medication was not influenced by patterns of use (high versus low), but it was influenced by the number of medications (the smaller the number of medications, the smaller the number of medications with undocumented status).

Table 2.

Completeness and correctness

2a. Completeness of the medication list for the 28 patients who had used the patient app between 3 and 70 days before the final interview (n=295 medications)
Medication name documented	N= 279 (94.6%)
Dosage and frequency documented	N=255 (86.4%)
Status documenteda	N=195 (66.1%)
2b. Correctness of the medication list: discrepancies between the list documented by the patient and the best possible medication history for the 13 patients who documented the status of each medication (n=97 medications)b
Prevalence per patient (n=13 patients), n (%)
Number of patients without discrepancies	3 (23.1%)
Number of patients having 1 medication with a discrepancy	4 (30.6%)
Number of patients having 2 medications with a discrepancy	4 (30.6%)
Number of patients having ≥3 medications with a discrepancy	2 (15.4%)
Prevalence per medication (n=97 medications), n (%)
Number of medications with a discrepancy	21 (21.7%)
Types (n=21 discrepancies in total)
Dosage and frequency	13 (61.9%)
Omission	5 (23.8%)
Other difference	1 (4.8%)
Therapeutic substitution	1 (4.8%)
Addition	1 (4.8%)
Generic-brand substitution	0%
Causes (n=21 discrepancies)
Technology-level	13 (61.9%)
Technical bug with information documented by the patient not recorded by the patient appc	9
Mis-clicking, mis-documentation by the patient in the free text box, or medication mis-chosen by the patient due to lack of clarity in the appd	4
Duplicates	0
Patient-level	8 (38.1%)
Self-medication	3
Intentional non-adherence	4
Other	1
System-level	0%

^aMedication status corresponds to the option chosen by the patient for the question: “Are you currently taking this medication?”

^bMedications with no status (taking as indicated, taking differently, not taking, unknown) documented by the patient could not be evaluated for correctness.

^cExample: Operating the patient app, concerning “Medrol 4 mg 2 tablets a day”, a patient had chosen “I’m taking the medication differently” and added in the free text box: “once a day in the morning”. Because information indicated in the free text box did not appear in the patient medication scheme published on the ReHN, what appeared was “Medrol 4 mg – taking differently – 2 tablets a day” instead of “Medrol 4 mg – taking differently – once a day in the morning.”

^dExample 1: Operating the patient app, concerning “Spidifen 400 mg” without any frequency indicated by a clinician, a patient had chosen “I’m taking the medication as indicated.” For the patient, it was not clear he had to specify the frequency.

Example 2: Operating the patient app, concerning “Cellcept 500 mg tablets,” a medication added by the patient himself. The picture associated with Cellcept was not clear enough. Therefore, the patient inadvertently selected, “Cellcept ampoules for injection of 500 mg.”

Correctness was evaluated for 13 patients (Table 2b). Ten patients had at least one medication with a discrepancy. For these 13 patients, 21 medications (21/97 = 21.7%) with a discrepancy were found (ie 1.6 per patient). Table 2 shows the prevalence, types, and causes of discrepancies. The main causes were technology-related. No significant correlation was found between the number of discrepancies and age, sex, group, SUS score, previous IT use, number of medications, or patterns of use. The median number of discrepancies was higher for patients with a medication list provided by the GP or the hospital than for patients who had to enter all medications by themselves (median of 2 and 1 respectively) but the trend was not significant (p = 0.06).

Qualitative data

Table 3 describes the barriers and strategies to optimizing the app, based on patient interviews. Table 3a describes usability items related to different aspects of the app such as connection, on-screen display, information on past medication, and completion checks.

Table 3.

Barriers and strategies to optimizing usability and usefulness, based on patient feedback

3a. Categories, subtopics, and quotes related to usability
Category	Subtopic	Sample quotes
Challenges for patients	Patients lack skills to connect to the patient app.	“The patient app requires better computer skills than I have.” (Patient 1)
		“I have never been able to connect to the patient app…. I'm not completely against the idea of using such a tool, but I would need much more support and more encouragement to use it.” (Patient 2)
		“I didn’t dare to connect to the patient app because I was afraid of doing stupid things” (Patient 3)
	Patients would like help of a caregiver or family member.	“The patient app is clearly needed, but maybe you should think about allowing someone other than the patient to document the medication list. For example, the caregiver, whether professional or not.” (Patient 19)
		“I would only use the patient app if my wife filled it in for me.” (Patient 22)
	Patients have difficulties using eID, but find it a reassuring security check.	“The ID card is a good level of security; I am reassured to have such a level of security.” (Patient 2)
On-screen display	Better overview of the medication list documented	“It should be possible to have an overview of your medication list before you log off. So you could check exactly what you noted.” (Patient 12)
	Need for “patient-friendly” medication names	“Improve the display of medication names. The patient doesn’t know details about boxes or types of generic. It is confusing to see too many details. It gives a feeling of insecurity.” (Patient 14)
		“I can’t always find the right box when adding medications [because the names are too long and confusing].” (Patient 25)
	Indications or reasons for stopping medication must be more clearly visible.	“It would be better to highlight the indication for which a medication has been prescribed, both for the patient and for other prescribers.” (Patient 15)
		“The start date and the end date, as well as the indication and the reason for stopping medication should be easier to see.” (Patient 42)
	Viewing or editing mode: need for distinct modes	“Having an editing and a viewing mode would make it easier to see a medication’s ‘status’ and whether you have already modified something or not.” (Patient 15)
		“It would be nice if the patient were welcomed with a personal opening page. This is his home and it would put him at ease and establish trust. He should be asked what he wants to do: consult his medications list, make changes, or delete a medication.” (Patient 14)
	Better highlighting of answers selected	“After selecting an answer, the other options should become gray, so we know we have already given an answer.” (Patient 12)
Past medication history data	Hide past medication history by default (risk of misunderstanding when presented by default)	“It is disturbing to see medications which aren’t taken. Archiving should be allowed, but optionally, by a specific button.” (Patient 14)
	Need for exhaustive data	“It is important to be able to access the entire history of taking medication, with a system of archiving to avoid confusion when using the patient app but being able to know which medication has already been tested, which one works or not.” (Patient 10)
		“Have a history of 6 to 12 months with access possible but not permanent, with the possibility of removing medications which aren’t taken.” (Patient 8)
Data exchange and connectivity	Add delivery data from the community pharmacy	“To be exhaustive, you should have all delivery data from the community pharmacy.” (Patient 8)
Miscellaneous	More possibilities to edit medication list posology (removing/suspending medication, undoing an action, etc.)	“I would like it to be easy to cancel an action.” (Patient 29)
		“I wish there was an option to definitively delete a medication from the list.” (Patient 4)
	Completion check	“The system should warn us when an answer is missing.” (Patient 24)
	Improve software functioning (performance, dependability, etc.)	“The waiting time to connect is too long. The cursor doesn’t click on the button I want to click on.” (Patient 21)
3b. Categories, subtopics, and quotes related to usefulness
Additional functionalities, to enhance usefulness/added value for patients	Possibility of printing a medication list or schedule	“Print the medication list and have the opportunity to make attractive medication schedules.” (Patient 38)
		“I would like to print my list by clicking on an icon or using a right mouse click.” (Patient 29)
	Reminders to take medications	“Reminders, I think that’s attractive, particularly for the day when I will have memory problems.” (Patient 30)
		“I think a reminder system, after having synchronized the medication list with your smartphone, would be necessary.” (Patient 33)
	Possibility of customizing the app to match patients’ needs	“Reminders could be helpful for some patients. But I think it is necessary to be able to cancel those options if we don't need them. Recurrent alerts will annoy us and make us ignore them!” (Patient 5)
		“Make the patient app more fun. The patient app has to be more user-friendly and maybe customizable for younger people!” (Patient 17)
	Reminder to log onto the app regularly	“Maybe also alerts to encourage people to log onto the patient app after several weeks of inactivity.” (Patient 33)
	Add educational information on medication, to improve health literacy	“I suggest a small description of each medication…. This description should contain: (1) why the drug is usually prescribed, (2) the most serious side effects with an order of ‘frequency’, (3) possibly contraindications and major interactions.” (Patient 5)
		“I think it could be useful to have links to popular websites containing information validated by health professionals about the expected effect, indications, and major side effects of medications.” (Patient 40)
Data exchange and connectivity	Transforming the patient app into a connected tool	“I would like to synchronize information with my smartphone. Not only to have reminders, but also to have my entire medication list on my smartphone.” (Patient 35)
	Offering suitable alternative electronic devices	“I would agree to use the patient app on a smartphone without the eID and eID card-reader, but not on a kiosk at the entrance to the hospital. I would rather opt for the employee at reception.” (Patient 9)
		“I would have liked to indicate my entire medical history or to be able to import or integrate the one provided by my GP. I think it's a pity to separate the two (my work on the patient app and the documents from the GP).” (Patient 19)
		“I think a discussion forum (with other patients or healthcare professionals) could be nice.” (Patient 39)
	Allowing patients to communicate about their health	“Sharing information about allergy and medication intolerance with all physicians is essential.” (Patient 15)
	Giving feedback based on graphs and statistics to patients so they can adjust their behavior to prevent adverse events	“I also imagine a system of graphs of consumption/intake of medications that can be compared to our symptoms. It would be interesting to produce statistics, but especially to understand and to anticipate. For example, within ten days of chemo it is usual to have a decrease in the number of white blood cells, followed by weight loss …so you could prepare yourself by a specific diet.” (Patient 33)

Usefulness

Questionnaire

Most patients thought the app could help them or their caregivers to save time and gain accuracy in keeping medication lists up to date. A large majority said the app could improve communication between HCPs (Table 4). No correlation was found between usefulness and patients’ age, sex, group, or frequency of use.

Table 4.

Patients’ agreement on seven perception statements about the patient app after first use (kick-off sessions) and last use (interviews) (N = 29)

		At kick-off	At the end of the study
		% (n) of patients who agreed or totally agreed	% (n) of patients who agreed or totally agreed
1.	I think by using the patient app I could save some time when updating my medication list.	89.7% (26)	79.3% (23)
2.	I think by using the patient app I could help my HCPs save some time when updating my medication list.	93.1% (27)	96.6% (28)
3.	I think the patient app could help improve my understanding of my medication list.	86.2% (25)	79.3% (23)
4.	I think the patient app could help my HCPs to have an accurate medication list.	96.6% (28)	100% (29)
5.	I think the patient app could improve communication about my medications between my HCPs and myself.	100% (29)	89.7% (26)
6.	I think the patient app could improve communication about my medications between HCPs.	100% (29)	93.1% (27)
7.	I would advise a friend to use the patient app.	89.7% (26)	82.8% (24)
	Percentage of patients who agreed or totally agreed with all seven statements	62.1% (18)	62.1% (18)

Notes: Percentage (and absolute number) of patients who agreed (score 4) or totally agreed (score 5) with the statement. Statement scoring: 1 = totally disagree, 2 = disagree, 3 = no opinion, 4 = agree, 5 = totally agree. The percentage of patients who (totally) agreed at the start and at the end of the study was compared for each item. P-values were all >0.05.

Abbreviations: n = absolute number of patients; SD = standard deviation.

Qualitative data ( Table 3b )

The overall perception of the app’s usefulness varied between patients. Some found it useful because it helped them to know their medication list better or to think about the reasons for their medications and because it gave them an opportunity to regain control of their treatment. In contrast, 16 patients (38%) perceived no added value, for various reasons. Firstly, some considered their medication management was good and therefore did not log onto the app. These patients had developed personalized approaches to managing medication, such as pillboxes near a printed list and smartphone reminders. They felt the tool would more likely benefit less-organized patients with frequent medication changes or with problems recalling medication. Secondly, some patients reported that their physicians already had in their computer all information concerning their medications. Finally, some were unwilling to use the app because of a lack of interest in IT use or because they were uncertain about security and privacy.

Half of the patients (23/42 = 54.8%) were convinced the app could be useful under certain conditions. First, patients asked for various additional functionalities (eg possibility of printing the medication list, reminders to take medications). Second, patients insisted on the need for integration and connectivity. The app would be much more valuable if all stakeholders participated actively. The list, moreover, should be part of a more global and shared overview of essential information concerning the patient’s health status. Finally, some patients thought the app should become a connected tool with possibilities for synchronization on smartphones, generating customizable and adjusted alerts (eg reminders).

Discussion

Our main results showed that a majority of the patients were satisfied with the app and would be willing to use it under certain conditions. Our study design made it possible to identify these conditions (eg additional functionalities, improved on-screen display, customization, and connectivity). However, our results also suggest that there may be inherent limits to how much a patient can clean up their own medication list. While improvements to the application may decrease the number of discrepancies, they will never be eliminated. Patients should be encouraged to do the best they can and to review their list with their GP or community pharmacist. Prompts could be sent to patients (eg text reminders on their mobile phones) to increase use of the app. It would then be easier to keep the list up-to-date, share it, and annotate the list whenever needed.

The quantitative and qualitative data on usability and usefulness were somewhat contrasting but highly complementary. The scores derived from the 2 questionnaires showed that most patients were satisfied and that most believed the app could improve the accuracy of medication lists and communication and reduce the time spent on reconciling medications. Nonetheless, several scores slightly decreased between the beginning and end of the study. It is possible that some patients were disappointed because of technical errors that occurred and because HCPs did not use the list generated by patients. Moreover, more than a third said during interviews that they would not use the app but usefulness is crucial for the adoption and sustained use of a tool.²⁸,³¹,⁵⁰ With regard to effectiveness, the medication list was complete for only a quarter of the patients and discrepancies were detected for approximately two-thirds. The types and causes of medication discrepancies differed from those reported in the literature.¹⁷,^51–55 Comparison must be made with caution, because the number of patients included in our study was low and because patients were updating their medication lists without any care transition triggering the process. The majority of discrepancies in the present study were technology-related. This confirms that IT approaches can introduce new types of errors and that analyzing the typology of errors is important when addressing usability.

Analyzing the quantitative and qualitative data, we identify 4 main areas for improvement for the future. The first is technology-related; the others apply to any approach to improving MedRec.

First, both the patient app itself and, to some extent, the eHealth network need improvement. For example, efficiency can be improved by decreasing the level of effort required to connect, without compromising security aspects. Such improvements are currently being implemented. Effectiveness could be improved through better on-screen display and editing features. Removing default statuses from a previous connection would probably greatly increase the number of medication statuses documented by patients. This would take a few more seconds of patients’ time, but would probably be worthwhile. Importantly, in order to improve usefulness – and therefore adoption – the app should better meet patients’ needs by offering additional features such as feedback, reminders, and appointment alerts. Such features were present in other similar applications.²⁶,⁵⁶,⁵⁷

Second, raising patient awareness of the epidemiology and dangers of medication discrepancies is essential. Many were unaware that their HCPs might lack information⁵⁸,⁵⁹ and unaware of the harmful impact of medication discrepancies and of their own potential role in optimizing MedRec.¹⁴,⁶⁰,⁶¹ Even though there have been previous efforts to raise patient awareness, greater efforts are needed, along with better communication on the roles and responsibilities of all stakeholders, including patients. We have now produced a video to contribute to meeting this need.

Third, better concordance between patients and physicians is required. Several discrepancies arose because patients documented the medication they were supposed to take rather than what they actually took. The main reasons were that patients did not want to be labelled as difficult or non-compliant or had previous experience with a prescriber not taking their perspective into account when (de)prescribing. Patient participation and shared decision-making are cornerstones of high-quality healthcare.⁶² This also applies to MedRec.

Finally, because the MedRec app for HCPs was still under development, the patient medication lists available on the ReHN were rarely used by physicians to reconcile medications. This was perceived by patients as a lack of support from their HCPs and certainly hampered patient app adoption. Indeed, the literature identifies providers’ endorsement as one of the most influential factors⁵⁸,⁶³,⁶⁴ in patients’ engagement with their health.

Comparison with Previous Work

There have been several previous reports on the evaluation of applications whereby patients document their medication lists in the context of MedRec. The majority – and we followed the same approach – asked patients to document medications from an existing list. Our work differs in several respects. The environment is different. Existing tools were developed in North America, in health organizations with closed IT systems, and used either no source of medication data or a single one (most often, medication refills). Our tool is, to the best of our knowledge, the second reported from a European country.⁵¹,⁵²,⁶⁵ However, the design, development, and usability assessment of the Spanish MedRec tool were never reported in the literature.

SEAMPAT has been developed to link and manage data from different providers, using different electronic tools and patient records and various sources of information. The patient app was evaluated with real patients, real data in a real workflow. This, therefore, brings added value to the existing literature. Finally, many existing tools provide additional functionalities (such as reminders or information on medications). Our data suggest that such functionalities may be critical for usefulness and therefore tool adoption.

With regard to results from evaluations, data on usability are available for some reports, but not all.²⁰,²²,²³,⁵⁶,^66–68 Moreover, we found none for which quantitative and qualitative usability measurements were collected through various methods at different time points in different settings.

Limitations

Our study had several limitations. First, the French version of the SUS questionnaire used in the present study has not been validated. Nevertheless, the French SUS questionnaire has been widely used, with conclusive results in many studies.⁶⁹,⁷⁰ Second, a selection bias in the sample cannot be ruled out. About half of those approached refused to participate, and we probably selected a sample with a greater interest in working with computers. However, we deliberately recruited patients from different groups of potential users and the study sample was quite diverse in age, sex, medical history, and IT use. In the context of a future study,⁶³ it would be useful to evaluate how patient health, eHealth, computer literacy, and levels and patterns of use influence research results. Third, interviews were not transcribed verbatim and most were done by just one researcher. We used reformulation to ensure that information was well understood by the researcher. Fourth, the interview guide was fairly close-ended, which could have influenced the kind of information we collected. Fifth, it was not feasible to evaluate efficiency in a quantitative way. This should be done in the future, as a tool will not be adopted if a burdensome number of clicks or too much time is needed to achieve the desired goal. Finally, there could be some debate about the definitions we used for usability and usefulness. Many different definitions exist. We selected definitions that are known worldwide, that have been used in published papers, and that made sense for the research team, given the objectives of the study.

Conclusions

Our findings highlight the value of a user-centered usability study in real-world conditions, collecting both quantitative and qualitative data. Our study also underlines the necessity of tailoring a patient application to patients’ real needs, while keeping its features in line with its primary objective, ie optimizing MedRec. Our results show that the patient app has valuable potential for optimizing MedRec and that it is worth developing further into a more mature system. A larger study could then be conducted to evaluate the effect of this eMedRec approach, using both the patient app and the MedRec app, on carefully selected outcomes, such as adverse events and hospital admissions.

Funding

This work was supported by the Région wallonne WBHealth program Grant 1318069 (principal investigator A Spinewine). The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Competing interest

None.

Contributors

Study concept and design: All authors. Acquisition of data: Marien. Analysis and interpretation of data: All authors. Drafting of the manuscript: Marien. Critical revision of the manuscript for important intellectual content: All authors. Final approval of the version to be published: All authors. Obtained funding: Spinewine. Administrative, technical, and material support: Marien, Spinewine, Ramdoyal, Ramon.Study supervision: Spinewine.

APPENDIX 1

Concepts evaluated, definitions, and methods of measurement

APPENDIX 2

Coding of types and causes of discrepancies

APPENDIX 3

Semi-structured interview guide

APPENDIX 4

Additional table: System Usability Scale (SUS) scores after first use (kick-off sessions) and at the end of the study (N = 29)