Identification and Classification Features of Patient Portal, A Systematic Review

ABSTRACT

Background and Aims

According to different theories in the field of using web technologies, it can be said that portals are one of the most appropriate tools to facilitate the process of care, communication, and education. Therefore, this systematic review aims to identify the features and functions of health portals that are critical for the success and widespread use of patient web‐based portals.

Methods

On August 28, 2022, a systematic search was conducted without a time limit in PubMed, Embase, Scopus, and Web of Science databases. Preferential reporting guidelines for systematic review and meta‐analysis studies were followed. Titles and abstracts were independently screened based on eligibility criteria. The checklist was used for data extraction, and finally, started to identify the functional features of patient portals based on the review of retrieved articles.

Results

We retrieved 1561 articles after initial screening and removing duplicate articles. After reviewing the title and abstract of the articles and applying the inclusion and exclusion criteria, finally, 22 articles were selected for full‐text review. The required data were extracted based on the checklist designed for this purpose. A total of 186 features were obtained for the design of the patient portal, which led to the identification of 6 main structural features and 30 content features for the design of the portal.

Conclusion

Identifying the structural and content features of patient portals can help healthcare providers and organizations in designing patient portals according to the needs of patients and considering the requirements comprehensively.

Summary

This study helps health information management managers and other health care workers to have a general understanding of the structure of the patient portal to provide better medical services in the age of technology.

1. Introduction

In the era of information and communication technology, digital tools play an important role in improving the patient care process [1]. Providing knowledge‐based care depends on patient participation, where patients have an active role in their care [2, 3]. According to different opinions on the use of web technologies, it can be said that portals are one of the most appropriate tools to facilitate the process of care, communication, and education. Patient Portal is a secure and private web‐based portal that enables patients and healthcare professionals to share their health‐related documents, images, lab results, and messages. Portals help patients become more involved in their care and provide them with a better understanding of their needs. Electronic health is one of the necessities of modern health care. The interaction of health information systems can help to improve the quality of efficiency and access to health care systems [4, 5]. Patient portals are considered a health information technology that promotes patient participation by providing online tools to patients to play an active role in their care [6] and as a technology platform for healthcare systems. Health professionals, including health care providers, act to communicate safely with patients and involve patients in research [7]. More advanced portals allow patients to send secure messages to clinical staff, access patient‐specific educational materials, schedule appointments, request prescription renewals, and manage billing. Patient portals can facilitate interactions between patients and providers that promote informed decision‐making and patient engagement. Therefore, patient portals are associated with improved communication and satisfaction with care [8]. Healthcare organizations have commonly adopted the use of patient portals as an essential strategy to provide patient‐centered care and patient engagement to improve clinical outcomes, which [9] can provide a platform to support disease management, including promoting medication adherence [10]. The need to classify patient portals, the widespread use of the Internet, and the advancement in the technical ability of patients have led to an increase in the use of patient portals in different countries [11, 12, 13]. However, there are countries where patient portals are still not widely used [6, 14]. In a study presented by Ammenwerth et al. [15] in 2020, the progress of eHealth in 14 countries across It analyzed the world with different health systems and different levels of economic development. This study found that the use of patient portals and the patient portal functions provided differ significantly between countries. While Finland and South Korea, for example, allow patients the best access to their health record data, 6 of the 14 countries analyzed in this study provide their populations with online access to their health data. They do not give [15]. Patient portals that are designed with appropriate quality and considering all patient requirements can effectively improve the quality of care and patient satisfaction and reduce treatment costs by facilitating interactions between patients and providers. The purpose of this systematic review is to identify the features and functions of health portals that are critical to the success and widespread use of web‐based patient portals. This study identifies and categorizes the features of patient portals, which can help healthcare providers in designing patient portals according to the needs of patients and considering the requirements comprehensively.

2. Methods

2.1. Study Design

Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) were used in this study to report the evidence obtained from the studies included in this systematic review [16]. Appendix A contains the PRISMA checklist. This study was approved by the Ethics Committee of Mashhad University of Medical Sciences (IR.MUMS.REC.1400.297).

2.2. Strategic Search

On August 28, 2022, we searched literature using PubMed, Embase, Scopus, and Web of Science databases. The below Medical Subject Headings and Emtree keywords and terms were used to search the databases:

(“Patient Portals” OR “Patient Web Portal” OR “Portal, Patient Web” OR “Portals, Patient Web” OR “Web Portal, Patient” OR “Web Portals, Patient” OR “Patient Internet Portals” OR “Internet Portal, Patient” OR “Internet Portals, Patient” OR “Patient Internet Portal” OR “Portal, Patient Internet” OR “Portals, Patient Internet” OR “Patient Web Portals” OR “Patient Portal” OR “Portal, Patient”) AND (“Scale” OR “Framework” OR “Tool” OR “Criteria” OR “Checklist” OR “Questionnaire” OR “Form” OR “Model” OR “Method”) AND (“Assessment” OR “Evaluation” OR “Design” OR “Implementation”). The search strategy for each database is shown in Appendix B.

2.3. Eligibility Criteria

The criteria for including articles in the study included the following:

• 1.Articles that include portal design features
• 2.Articles related to health portals and related to patients
• 3.The articles should be in English.
• 4.The full text of the articles should be available.

As exclusion criteria, studies such as Reports, Notes, Books, Book Chapters, Conference Papers, and Case Reports were excluded from the study, and articles that were not in the field of medical sciences were also excluded from this study.

2.4. Data Extraction and Synthesis

First duplicate articles were excluded from this review. Titles and abstracts were independently screened based on eligibility criteria. Articles that did not meet the inclusion criteria were excluded from this review. Then, the full texts were retrieved and screened by two separate researchers based on the eligibility criteria. Disputes were resolved by discussion, and in the event of disagreement, the third author would give the final opinion.

For data extraction, a checklist was used by two authors independently. In case of disagreement, the third author expressed the final opinion. The data elements in this checklist included study reference, publication year, type of portal, features of the patient portal, study objectives, and key findings.

2.5. Quality Assessment

The methodological quality of the included studies was critically appraised using the Downs and Black checklist [17]. Two researchers separately reviewed each included study using this scoring checklist. Quality scores of 20, 11–20, and less than 11 were considered good, moderate, and poor, respectively. The higher score represented a better methodology [17].

3. Results

3.1. Study Selection

The study selection process is illustrated as a PRISMA diagram in Figure 1. A total of 2398 documents that were found after searching the databases. The titles and abstracts of 1561 studies were examined after duplicates (837) were eliminated. A total of 1476 studies were excluded after the titles and abstracts of the studies were reviewed for relevance to the study's objectives. Eighty‐five articles were then chosen for full‐text evaluation. Ultimately, 22 studies that satisfied the inclusion requirements were included in this review.

Figure 1.

Flowchart of the study selection.

3.2. Quality Assessment

The mean score for methodological quality of the studies using the Downs and Black checklist was 13. The quality assessment of all articles was moderate (Appendix C).

3.3. Study Characteristics

Table 1 shows the characteristics of all included studies. Of the 22 studies included in this article, 11 studies were in the United States (11, 50%) [2, 8, 18, 19, 20, 21, 22, 23, 24, 25, 26], 5 studies were in Canada (5, 22%) [27, 28, 29, 30, 31], and 3 studies were in the Netherlands (3, 13.5%) [32, 33, 34], Other studies were conducted in Belgium [35], the United Kingdom [36], and India [37]. According to “The State and Outlook of the World Economy” by Belzi (2022), 21 of the studies were conducted in developed countries, and one of the studies was conducted in developing countries.

Table 1.

Features of entrance studies.

First author, year, country, reference	Type of portal	Features	Goals	Key finding
Ahmed 2016, Canada [1]	My Asthma Portal (MAP)	Information related to the patient's health status, List of medications List of patient problems, General information about the disease. List of allergies Articles	Comparison of the effect of access to a web‐based portal for asthma patients, known as MAP, compared with usual care on asthma control and patient quality of life.	This study supported the use of MAP to improve the quality of life of patients with asthma rather than asthma disease control as measured by a database, and concluded that implementation of MAP over 6 months with appropriate protocols for monitoring symptoms and Health behaviors improved people's knowledge and self‐management skills may lead to long‐term gains in asthma control.
Alpert 2016, USA [2]	Patient Portal	List of medications, Test results, Vaccination, diet, Lifestyle, information Editing section, page updates, Library, dictionary, Contact us	Evaluation of portals in transferring information to patients	Increased engagement and personalized messages can transform portals into customized, robust, easily accessible, and reliable information sources.
Geerts 2019, Netherlands [3]	Portal for Hematology	Test results, Imaging results, Patient's medical record, Reporting to other physicians, Make an appointment List of personal information, List of current medications Pharmaceutical history	Examining the wishes, expectations, and thoughts of patients with hematologic malignancy and their doctors according to the electronic patient portal	Patients and doctors appreciated the electronic patient portal. Both groups should be involved in further development of the portal to improve interaction by meeting patients' wishes, considering organizational and professional issues, and managing the expectations of both parties. To improve the performance of the portal, its design should be flexible and individual. Further research should focus on the perceived added value and impact on patient‐related outcome measures of portals
Griffin 2016, USA [4]	Patient Portal‐ University of North Carolina Health Care System (UNC)	Messaging, Medical advice, Lab results, List of medications List of problems Allergies, Vaccinations, Appointment details	Identification of patient characteristics associated with the use of the patient portal and determining the frequency of use of the usual features of the patient portal	The majority of patients who were given an access code for the UNC portal did not use it within 30 days of discharge. Of those who used the portal, active users were more likely to readmit within 30 days. Healthcare systems should consider strategies to: (1) increase the overall use of patient portals and (2) target patients with the highest comorbidity scores to reduce hospital readmissions.
Grossman 2018, USA [5]	Acute care patient portal	List of medications Care team, Test results Prescribed diet Vital Signs, Safety reminders Allergy Diagnoses/problem list Daily programs Future drug doses Future diagnostic tests Events added by the patient Main Page Discharge criteria/goals of care Educational materials podcast Video Patient‐generated content Notes (written, audio or video) Patient‐provider message Safety concerns Care goals Provider feedback Platform Feedback	Provide recommendations on how to most effectively implement the advanced features of acute care patient portals, including (1) patient‐provider communication, (2) care plan information, (3) clinical data visualization, (4) patient education, (5) patient safety, and (6) access. Health Caregivers (7) Hospital facilities	The design and implementation of acute care portal features depend on patient information needs, clinical judgment about the appropriateness of information release, potential impact on clinicians, forensic concerns, and technical feasibility. Each feature presents unique considerations and challenges for implementation, but also unique opportunities to deliver value to patients.
Kildea 2019, Canada [6]	Patient Portal	Main Page Make an appointment Notifications (eg, new document and new message), Posts (messages from the treatment team and announcements), Treatment status or treatment planning, Patient status chart, Diagnostic information, Archive of notices, Appointment app with appointment location maps, Request to change the appointment, Information about treatment or treatment planning, Access to doctors' notes and nursing notes (selected), Test results, Messages from the treatment team, Secure two‐way messaging with doctors, Outcome and satisfaction questionnaire reported by the patient. Phone directory and contact information, patient charter, Your feedback about the app or portal, Ability to report bugs in the program or portal. Educational materials (relevant and timely), Account Settings, User interface/multilingual, font size, The possibility of updating demographic information in the electronic medical record.	Sharing experiences in the design and development of a person‐centered patient portal using a collaborative stakeholder design approach.	Involving all stakeholders in the design and development of patient‐facing software can help ensure that the essential elements of person‐centeredness, physician acceptance, and informatics feasibility are achieved.
Walker 2018, USA [7]	Inpatient Portal	List of diagnoses and problems, List of medications Test results, Visit registration request, Educational materials, Care team member information, Messaging, Patient's notes, Services related to the patient, such as requesting physiotherapy, assistance	Creating a logical model that can help researchers and hospital managers in establishing and evaluating the impact of inpatient portals.	The model identifies priority areas for evaluation suggests applicable metrics and data sources for quality improvement and research, and captures the perspectives of multiple relevant stakeholders by describing organizational structures, processes, and outcomes related to patient portals. This model offers specific evaluation suggestions for hospital administrators looking to implement inpatient portals as well as for researchers looking to evaluate this new technology.
Munikrishna 2019, India [8]	Assisted Reproduction Center Portal	The possibility of receiving online advice through messages. Uploading patient reports to Receive doctors' opinions, View the date and time of Scheduled appointment, Online consultation with a doctor, Payment gateway, Upload patient reports	Development and effectiveness analysis of a web portal for a fertility help center in a tertiary care hospital in South India	Patients who registered on the portal found it user‐friendly. They found it easy to access all the information on the portal. Positive feedback and suggestions from patients indicate that the patient portal has been well received by them.
Hefner 2017, USA [9]	Inpatient Portal	Test results, List of diagnoses and Problems, Messaging, Educational materials, List of medications Care team information, Make an appointment	Assessing Attitudes and Perceptions About the Use of MCB (My Chart Bedside) and Staff Training for New Technology in a Health System in Five Hospitals	On average, 40%–60% of respondents in each group reported a positive orientation toward MCB technology and training received. This positive orientation was highest among support staff, lowest among nurses, and lowest among doctors. All differences based on the role of staff were statistically significant at p < 0.001. Additionally, 62% of respondents reported “inadequate” training.
Guy 2012, Canada [10]	Chronic Disease Management Portal	Messaging capability, Educational materials, Appointment	Feasibility assessment of two chronic disease management portals in terms of expectations, motivations, usability, and recommendations for the future	The findings indicate the usefulness of the features of this portal. Important issues for participants include accessing their medical records, communicating with healthcare professionals and other participants about topics of interest, and keeping up with the latest clinical studies.
Barbara 2016, Canada [11]	Aging Portal	Educational materials, Make an appointment Links to web resources.	Share the findings of the portal usability evaluation with a particular focus on content features for the general public and inform designers of health information websites and online resources for older people about key usability issues.	This study reinforces the importance of end‐user participation during the development of this unique, dynamic, evidence‐based health information website. Feedback was applied to improve the usability of the website iteratively. Designers of health‐related websites can use the findings of this study. Qualitative analysis revealed common themes regarding the portal's strengths and practical challenges. The website's strengths were reliability, usability, search functionality, design, and accessibility. Usability challenges included reluctance to register, registration process, search, terminology, and technical features.
Tanbeer 2021, Canada [12]	Portal for out‐of‐hospital patient care	Messaging capability, Educational materials, Make an appointment Care team information	Creating a patient portal (MyHealthPortal) aimed at nonprimary care, home care, and/or specialty health care for patients.	The results of the study showed that the patient portal is a practical solution for providing patients with up‐to‐date information about appointments, providers, and overall care.
Alturkistani 2020, UK [13]	Patient Portal	Test results, List of medications Make an appointment Allergy, Messaging capability	Synthesizing evidence on health outcomes and healthcare quality associated with patient portal use among adult patients	This systematic review provides an overview of health and healthcare quality outcomes associated with patient portal use among adult patients and provides detailed information on the performance of portals and their association with outcomes. This review could potentially contribute to patient portal evaluation studies by providing insight into outcomes associated with different patient portal functions.
Reynolds 2017, USA [14]	Patient Portal	Test results, List of diagnoses and problems, Make an appointment Messaging capability, Educational materials	Understanding patient questions about their medical records in a patient portal	This study provides evidence of opportunities for redesigning patient portals and may lead to improved patient portal design and, as a result, may lead to greater patient participation in health care, resulting in improved health status and reduced costs.
Wildenbos 2018, Netherland [15]	Older adults Portal	Test results, Messaging, Educational materials, Make an appointment	The objective is to gain insight into enrollment rates and experiences of elderly patients using a patient portal, 1 year after implementation in a university hospital.	Patient portal designs should meet the usability needs of the elderly. Portals should preferably include medical history information, physician notes, and the need for rapid response from providers.
Elizabeth 2018, USA [16]	CANCER Portal	Test results, List of medications Make an appointment Messaging, Educational materials,	Examining patient portals that serve as a tool for patient engagement by increasing access to electronic health care information and expanding communication channels with health care providers.	Meaningful use requirements have driven the design and development of patient portals in recent years. Patient engagement in oncology can improve quality of life and outcomes.
Van den Bulck 2018, Belgium [17]	Portal for Patient‐Centered Care	Allergies, Test results, Vaccination records, List of drugs, List of sick doctors, Family history, Medical problems, medical visits/surgeries/procedures, and lifestyle (examples: smoking and exercise), Patient searches on the Internet in the field of health, Adding notes to the medical record, Registration of appointments, referrals, re‐prescription, Communication with the doctor, Patient health insurance sick employer, Recording biological parameters, Record the overall costs of health care, Preventive medical reminders (e.g.,: flu shots), The possibility of registering to participate in medical research. The possibility of the guardian accessing the file of the children. Ability to communicate with other people with similar health problems. receiving educational materials related to health, Security and privacy of medical information, The possibility of recording the patient's understanding of his health. The possibility of registering the patient's consent, Recording the overall quality of healthcare	This study aims to (1) examine health information needs, expectations, and attitudes toward the patient portal and (2) evaluate whether determinants such as patient characteristics, health literacy, and empowerment status can predict two different variables including the following: 1) that is, the importance of individual characteristics for obtaining health information when using patient portals and 2) the expectations related to personal health care when using patient portals.	People want to receive alerts or some form of communication through the patient portal to take action when needed to manage their health information. Determinants such as education, the importance of shared decision‐making, difficulties in finding health‐related information, and the importance of asking questions about physicians' decisions should be considered in the design of patient portals.
Sakaguchi 2017, USA [18]	Lung Cancer Patient Portal	Patient education, Make an appointment, Record feedback, Fitness program, EMR access	The purpose of this study was to evaluate the feasibility and usability of the patient portal and to create preliminary evidence about its effectiveness.	The majority of responses (82%) were positive about using MyAVL, 69% saw it as a valuable adjunct to care, and 56% perceived increased control over their health. MyAVL appears to be a versatile, practical, and user‐friendly eHealth program for a select group of lung cancer patients. However, further improvements are needed to impact patient outcomes positively.
Burling 2014, USA [19]	Patient Portal	Test results, List of diagnoses and problems, List of medications Messaging, Educational materials, Make an appointment	Evaluate the full range of portal components to determine which capabilities are the best fit.	Features of the portal that are more attractive or useful to patients will draw patients to the portal, and once patients show interest in one feature, they are likely to start using other features.
McAlearney 2019, USA [20]	Inpatient portal	Providing services based on demand, Health information search, Communication with the care team, Secure Messaging	Improving self‐care understanding of the role of a portal in the context of inpatient care	Inpatient portals are recognized as tools that can enhance the delivery of patient‐centered care. In addition to empowering patients by increasing their sense of control, inpatient portals can support family members and caregivers during their hospital stay.
Jhamb 2015, USA [21]	Nephrology portal	Make an appointment Relevant medical information, Test results, Drug information, Request for medical advice	To describe the adoption of an EHR portal and to examine the association between portal adoption and blood pressure control.	While portal adoption appears to be increasing, more attention is needed to understand why vulnerable populations are not accessing it. Future research should examine barriers to the use of eHealth technologies in patients with CKD, interventions to address them, and their potential to improve outcomes.
Kuijpers 2018, Netherlands [22]	Portal to Empower Cancer Survivors	Make an appointment E‐consultation, Online community of patients Reporting the results by the patient. Registration of feedback by the user, An online rehabilitation program, Online psycho‐social management program,	Investigating cancer survivors' and health professionals' expectations of possible features of an interactive portal	Both groups of cancer survivors and health professionals were positive about the introduction of the portal, although their preferences for different features differed. These findings reflect their unique perspective and emphasize the importance of multistakeholder participation in the actual design process.

After reading 22 articles retrieved, 184 features for portal design were extracted from the articles. At this stage, to select the final features of the portal design, a 2‐h focus group meeting was held with the presence of experts in health information management, medical informatics, and immunology to check each feature in terms of suitability in the design of the portal, and finally, after categorizing common items, 30 items were selected for the design of the portal.

Classification of data according to the opinion of experts in health information management and medical informatics led to the identification of six main structural features including the main page (including notifications and posts, monolingual/multilingual user interface, entering the patient portal, calendar and engine search), patient profile (including registration in the patient portal, messaging, lab results, list of medications, list of allergies, diagnoses and problems, make an appointment), educational materials (including disease prevention, Diagnosis and treatment, signs and symptoms, epidemiology of the disease, access to the medical dictionary, educational videos and multimedia messages, access to articles related to the disease, information about international conferences related to the disease, care team (It includes the names of the care team and descriptions of each), about us (including mission and vision) and communication with us (including Phone number of the medical center, E‐mail, Communication networks, Address of medical centers, Map, Feedback about the application or portal, Ability to report bugs in the program or portal). Table 2 shows the final classification of structural and content features of the patient portal. Also, below are explanations of some of the features of the patient portal.

Table 2.

Final classification of structural and content features of the patient portal.

Structural features	Content features
Home page	Log in to the patient portal Posts: messages from the treatment team, general announcements, News and events Search engine Calendar User interface: monolingual/multilingual
Patient profile	Log in Messaging Test results List of medicines List of diagnosis and patient problems Allergies Appointment
Care team	Care team names and descriptions of which
Educational	Disease Prevention Diagnosis and treatment Signs and symptoms Epidemiology Access to the medical dictionary Videos and multimedia messages Access to articles and journals related to the disease Information about national and international conferences
Contact us	The phone number of the medical center E‐mail Communication networks Address of medical centers Map Feedback about the application or portal Ability to report bugs in the program or portal
About us	Mission and vision

3.3.1. Main Page

This section includes [37, 38]:

Notifications (new messages, posts, treatment team messages, and announcements) [30];

Calendar [30];

Search engine [25];

User interface: monolingual/multilingual [30];

Log in to the patient portal.

3.3.2. Patient profile

3.3.2.1. Registration in the patient portal

Patients can register in the portal related to their disease and update their information online [24, 35].

3.3.2.2. Messaging

This item is one of the most used items in portal design, which was mentioned in 19 articles [2, 8, 9, 18, 19, 20, 21, 22, 23, 27, 28, 30, 31, 32, 33, 36, 37]. Messaging is a feature that facilitates patient‐provider communication and provides an efficient way to exchange information with patients and other providers.

3.3.2.3. Lab results

According to the results of Table 2, out of 22 reviewed articles, 15 studies have considered the features of the lab results in the design of the portal [2, 8, 9, 18, 19, 20, 21, 22, 23, 24, 30, 32, 33, 36]. Grossman [2] has also stated in his study that electronic access to laboratory results may enable patients to share the results electronically with their doctor in future visits and avoid repeating lab results [2]. Patients want to know how the lab result affects them and whether they need more care after receiving the lab results or not. Notification of this information along with the result will be useful for patients. "For example, 'Your result is slightly outside the normal range' [39]." Eighty‐one percent of patients in Geerts's [32] study agreed to view their lab results on the portal. Also, the results of the patient survey about the features of the patient portal in the Kildea et al. study [30] show 84.6% agreement regarding the existence of this item [30].

3.3.2.4. List of Drugs

Ten out of 22 articles in the literature review used this feature in the design of the portal [2, 8, 9, 22, 24, 26, 31, 32, 36]. The results of Grossman et al.'s study [2] showed that all six educational centers investigated in this study have used the drug list feature in the portal. Also, 91% of patients in Geerts et al.'s study [32] survey wanted this function in the portal. The feature of the drug list may include the generic name of the drug, the brand name, the dose of the drug, the frequency of use, the method of administration, and the side effects of the drug, and ideally, the feature of the drug may show the logic of prescribing the drug, either in the physician's own words or from an external source explains to facilitate patient education [2].

3.3.2.5. List of Allergies

Four studies out of 22 reviewed studies have used the feature of the patient's allergy list in the design of the portal [2, 8, 32, 36]. In the study of Grossman et al. [2], four hospitals used this item in their portal. One hundred percent of active portal users in Griffin's study also wanted this feature in their disease portal [8].

3.3.2.6. List of Diagnoses and problems

Ten out of 22 reviewed articles used the diagnosis and problem list feature in the design of the portal [2, 8, 9, 20, 21, 23, 24, 30, 31, 32]. Ideally, the patient portal should display medical diagnoses in plain language and interpret unfamiliar terms [2].

3.3.2.7. Make an Appointment

The most common feature in all articles is make an Appointment, with 19 articles [8, 9, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 32, 33, 34, 36, 37]. Two hundred and thirty‐four out of 267 patients in the survey of the Kildea study wanted this feature [30]. This feature allows patients to view their previous and future appointments [8]. Making an appointment through the portal saves the patient's time. When the doctor's visit time is determined in advance, patients spend less time in the waiting room and spend more time taking care of themselves [24].

3.3.3. Educational material

Eighteen articles out of 22 systematic review articles included this item in the design of their patient portals [8, 9, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 38]. By uploading educational content about the disease, portals can help providers have extra time to respond to patients' messages and questions. Eighty‐six percent of patients in Geerts et al.'s study [32] asked for this feature, which shows the importance of this item in the portal. Providing educational materials, diagnostics, and treatment plans is difficult for patients to understand and remember. To strengthen this information, patients can access these materials online through the portal [24]. To provide valuable information to patients, educational features should be beyond what Google can provide [24]. In their study, Grossman et al. present this feature as a bookshelf where patients can read educational materials about their diagnosis, and the educational content is appropriate to the age and level of health literacy of the individual [2].

3.3.4. Care Team

Four articles out of 22 reviewed articles have used this feature in the patient portal [2, 20, 27, 38]. Displaying names, photos, and roles of care team members facilitates identification and communication. Portal users may wish to view a brief biography of each member of the care team as valuable information regarding the qualifications of each of those providers [2].

4. Discussion

4.1. Principal Finding

To create this classification, in the first step, we created an initial classification based on the literature review. Then, in the second step, using focus group meetings with relevant experts in this field, we created the final classification with 6 structural features and 30 content features. In this study, we were able to show that identifying and determining different features of the portal can help health information managers and other healthcare providers in selecting and designing portals for patients and meeting their needs. To date, studies describing portal implementation have focused on patient needs. Future research should examine what features portals have that improve communication between patients and health providers, as well as allow better comparisons between portals.

This systematic review provides an overview of some features of patient portals. Also, to investigate the differences between different features of the portals, we compared the related results based on the features of the patient portal, considering the disease or condition for which the portals are used. A major challenge in the current systematic review is the difference between the characteristics provided by each portal in each of the included studies. Some of the features presented in the patient portal can be more related to the portal designed for that particular disease. For example, Kuijpers et al. [34], in their study on designing a portal to empower cancer survivors, have included features such as an online rehabilitation program and an online psycho‐social management program in addition to other typical portal features. Barbara et al. [28], in the design of the portal for the elderly, included the usual features of a portal such as access to educational materials, appointment requests, and links to web resources in their portal, considering that the audience of this portal is elderly people and their skills in the field of technology and digital literacy may be limited, the existence of these features seems reasonable.

Also, to check the differences between different features of the portals, it is possible to compare the results related to the portals based on the health systems of different countries of the patient portal. Eleven of the studies reviewed in this systematic review were conducted in the United States [2, 8, 18, 20, 21, 22, 23, 24, 25, 26, 38]. In their study, Grossman et al. [2] provide recommendations on how to implement the most effective advanced features of acute care patient portals in the United States, including patient‐provider communication, care plan information, clinical data visualization, patient education, patient safety, access to Health care providers, and hospital facilities provided [2]. The use of HIT in the United States has grown over the past decades, especially after the implementation of the HITECH Act. Similarly, nearly three‐quarters of Americans use the Internet to track their health, and 40% access their health records. On the consumer side, the Internet has become a source of information about health care (and even misinformation), and providers can communicate with patients via E‐mail or web portals [40]. Finney Rutten et al. [41] also reported in their study that those who had access to the Internet (range, 69.8%–81.5%) used the Internet to search for information about health or health care. Therefore, the large number of studies conducted in this country seems natural considering the growing demand for the use of technology for health interventions.

On the other hand, Canada's performance in using information and communication technology to provide health is average compared with many other developed countries [42]. Five of the 22 reviewed articles were conducted in Canada [27, 28, 29, 30, 31]. Most doctors in Canada do not offer health information technology services to patients such as electronic communication, online appointment and prescription renewal, and online test viewing. According to a 2018 survey, the use of electronic appointment booking among Canadians was very low (8%), and visiting a healthcare provider virtually via video was almost rare (3%). While there have been advances in patient access to medical information online in Canada, such as the Carnet santé Québec and Alberta's MyHealth Records, these are not yet widely available across Canada [42]. Kildea et al. [30] shared their experience in designing and developing a person‐centered patient portal using a collaborative stakeholder design approach in Canada, and the results of this study showed that patients overwhelmingly felt that digital access to their PHI was important, and Most patients wanted instant access to all their data. Therefore, it seems necessary to pay attention to this category of patients' needs through the portal.

In recent years, the Netherlands has consistently ranked in the top five countries in Europe for fast broadband internet [43]. Also, some health challenges in the Netherlands are related to fundamentally favorable trends, such as increased life expectancy as a result of demographic changes and better diagnostic and treatment options (e.g., cancer is becoming a chronic rather than a fatal disease) [43]. Three studies out of 22 reviewed studies investigated the features of the patient portal for cancer patients in the Netherlands [32, 33, 34]. The patients in these studies were satisfied with the various features provided by the portal and stated that they have a better overall health status [32, 33, 34]. Therefore, according to the level of digital literacy in the Netherlands, the use of various patient portals seems to be very popular.

The results of the comprehensive review conducted in this research have collected rich sources of portal design features, although some of the features proposed for the portal are a series of common and customary features among web design professionals, which are used in every portal and website design should be taken into consideration, but some items were specifically suitable for health portals. In general, based on the reviewed sources and according to the results of Table 2, the most emphasis is on the existence of features such as making an appointment, access to educational materials, messaging capability, access to lab results, medication list, list of diagnoses and problems, information about the care team and the list of allergies. Considering that nowadays the internet environment is widely used by the general public, including patients, and considering the sensitivity of medical information and people's health, it is necessary to consider the development of patient interaction systems with health care providers as a basic principle.

Regarding the existence of the feature of access to educational materials, the goal of patient education through the portal is to help patients take responsibility for their care after discharge and by receiving reliable education from care providers to address patient concerns especially to reduce the condition of his illness. Provider‐based education is complementary and reduces patients' uncertainty about their condition [2]. In other studies, conducted among patients, the findings show that portals involve patients in the process of improving their health by accessing appropriate educational items [2, 30, 35]. Lum et al. [44] also stated in their study that patient portals can provide patients access to web‐based education, interactive tools, and secure patient‐provider electronic communications, which together can lead to improved patient‐centered care. Walker et al. [38] also in their study, pointed out that the patient portal can provide the necessary training to the patient, and this access to knowledge may lead to improved patient satisfaction because patients may feel more involved, empowered, and informed in having their care program. One of the potential challenges regarding access to the educational materials available in the portal is the limitations related to the level of health literacy and the level of education of the portal users (patients) [45]. The patient's health literacy level is an aspect that is considered a strong influencing factor on the patient's interest and ability to use the patient portal, and its low level causes the portal to be ineffective [33]. As a way of working, it seems that teaching patients before using the patient portal, especially patients who have little knowledge in the field of technology, makes them better understand how to work with the portal.

The presence of the feature of making an appointment can be justified according to the results of Table 2, and in the study conducted by Kildea, the results of the survey about the possible features of the portal from the patient's point of view show that 87.6% of the participants want the feature of make an appointment feature in the patient portal [30]. Also, 91 patients in Geerts et al.'s study [32] agreed to have to make an appointment feature in the patient portal.

In their study, Ahern et al. [46] divided portal services into three main categories, and messaging is one of the main specialized services of the patient portal [24]. Also, patient triage and answering patient phone calls can take up a large amount of healthcare workers' time. Activating the messaging module in the patient portal can reduce the waiting time of patients and allow the patient to effectively and quickly communicate with their referring doctors by exchanging information [24]. Patients also communicate with providers for a variety of reasons, including communicating and clarifying concerns, and needs, requesting clinical information updates, coordinating care, and providing feedback [2]. Implementing a messaging module is potentially useful, but it can also be destructive. Providers fear constant interruptions and excessive patient contact, as well as unreasonable patient expectations and potential legal liability, Patients want informative and quick answers [2]. Physicians also worry about spending more time answering patients' questions [18].

In justifying the existence of the test results, the results of the studies showed that visual representations, instead of tables, help the patient to understand better. Also, in their study, Grossman et al. recommend that visual representations include real numbers, instead of just indicating the normality or abnormality of laboratory values [2]. 84.6% of patients in John Kildea's study wanted to see the feature of viewing test results in the patient portal [18, 35]. However, in general, it is difficult to interpret laboratory data for patients, especially the elderly or patients with a lower education level [18, 20]. Integrating the results measured in different laboratory systems, displaying the results in different time intervals, and conveying the clinical significance of each result are some of the challenges raised in the field of laboratory test results that may be used in the interpretation of diagnostic laboratory information of patients with to face problems [2].

Other features such as a list of medicines, a list of diagnoses and problems, care team, and list of allergies are among those mentioned in articles and various designed portals and can be considered for designing the portal. Patients and their families are key sources of information that care providers rely on to obtain information, but families are not always present and patients are not always able to communicate. Personalization tools such as portals offer a unique opportunity to operationalize the concept of patient knowledge. Therefore, it can be said that patient portals are a potentially powerful way for care team members to effectively communicate with patients using the various features available in the portal [47]. For example, a doctor can make a virtual appointment in his office or personal home at any hour of the day, without direct contact with the patient, adjust the patient's medication prescriptions, or answer the patient's questions through the portal [32, 35, 36]. It is important to adapt the functions of the portal to the needs and capacities of patients, to facilitate the use of this technology and improve its dissemination. In particular, overcoming ethnic and literacy barriers to portal use is an essential goal for creating more equitable, effective, and safe healthcare systems [48].

A comprehensive review of this research has provided a set of features used in the design of health portals, and the use of these items in the design of patient portals by researchers, specialists, and designers in this field leads to the creation of methodical and efficient systems. Which can attract the trust of its users who are patients. Finally, it is suggested to use the features identified in this study in the design of patient portals. On the other hand, due to the increasing use of portals, the issue of integration of these technologies with other health information technologies such as electronic health records, hospital information systems, registries of diseases and health outcomes, and such matters are of interest to designers. The developers of these systems can access a rich source of quantitative and qualitative data regarding lifestyle, frequently asked questions, and results of various interventions and treatments in different groups of patients, which are obtained using health information management techniques such as text mining.

Advances in artificial intelligence also have the potential to revolutionize many aspects of healthcare, enabling a more personalized, accurate, predictable, and portable future. Side effects related to drug reactions, drug errors, predicting disease risks among critically ill patients, and predicting test results and vital signs that affect patient safety outcomes are among the topics that are addressed using artificial intelligence algorithms at the clinical level, Machine learning, and natural language processing techniques can be used in the patient portal [49].

5. Strengths and Limitations

We developed our initial classification based on an extensive international literature review that identified patient portals from six countries. It is possible that studies with low patient portal recruitment will not report or publish their study findings. One way to deal with publication bias is to include published and unpublished literature such as gray literature. However, due to time constraints, this review only identified characteristics reported by the included studies. It is also challenging to conclude which patient portal functions contribute most to improving health outcomes, as some studies only reported overall portal use.

Another limitation of this review is that half of the studies were from the United States, which has a diverse healthcare system that includes private healthcare organizations, nonprofit organizations, and government agencies. The way the healthcare organization is organized may limit the applicability of the findings of this review to other healthcare settings and systems.

6. Conclusion

Identifying the structural and content features of patient portals can help healthcare providers and organizations in designing patient portals according to the needs of patients and considering the requirements comprehensively.

Author Contributions

Reyhane Norouzi Aval: conceptualization, investigation, writing – original draft, methodology, software, data curation, resources, visualization. Houshang Rafatpanah: writing – review and editing, validation. Masoumeh Sarbaz: writing – review and editing, validation. Seyyedeh Fatemeh Mousavi Baigi: data curation, resources. Khalil Kimiafar: conceptualization, investigation, writing – review and editing, supervision, project administration.

Ethics Statement

The Mashhad University of Medical Sciences ethics committee granted approval for this study (approval number: IR.MUMS.REC.1400.297).

Conflicts of Interest

The authors declare no conflicts of interest.

Transparency Statement

The lead author Khalil Kimiafar affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

1.

See Table A1.

Table A1.

PRISMA checklist.

Section and topic	Item #	Checklist item	Location where item is reported
TITLE
Title	1	Identify the report as a systematic review.	1
ABSTRACT
Abstract	2	See the PRISMA 2020 for Abstracts checklist.	2
INTRODUCTION
Rationale	3	Describe the rationale for the review in the context of existing knowledge.	3
Objectives	4	Provide an explicit statement of the objective(s) or question(s) the review addresses.	4
METHODS
Eligibility criteria	5	Specify the inclusion and exclusion criteria for the review and how studies were grouped for the syntheses.	5
Information sources	6	Specify all databases, registers, websites, organisations, reference lists and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted.	5
Search strategy	7	Present the full search strategies for all databases, registers and websites, including any filters and limits used.	5
Selection process	8	Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process.	5
Data collection process	9	Specify the methods used to collect data from reports, including how many reviewers collected data from each report, whether they worked independently, any processes for obtaining or confirming data from study investigators, and if applicable, details of automation tools used in the process.	5
Data items	10a	List and define all outcomes for which data were sought. Specify whether all results that were compatible with each outcome domain in each study were sought (e.g., for all measures, time points, analyses), and if not, the methods used to decide which results to collect.	6–8
	10b	List and define all other variables for which data were sought (e.g., participant and intervention characteristics, funding sources). Describe any assumptions made about any missing or unclear information.	N/A
Study risk of bias assessment	11	Specify the methods used to assess risk of bias in the included studies, including details of the tool(s) used, how many reviewers assessed each study and whether they worked independently, and if applicable, details of automation tools used in the process.	N/A
Effect measures	12	Specify for each outcome the effect measure(s) (e.g., risk ratio, mean difference) used in the synthesis or presentation of results.	N/A
Synthesis methods	13a	Describe the processes used to decide which studies were eligible for each synthesis (e.g., tabulating the study intervention characteristics and comparing against the planned groups for each synthesis (item #5)).	5
	13b	Describe any methods required to prepare the data for presentation or synthesis, such as handling of missing summary statistics, or data conversions.	5
	13c	Describe any methods used to tabulate or visually display results of individual studies and syntheses.	5
	13d	Describe any methods used to synthesize results and provide a rationale for the choice(s). If meta‐analysis was performed, describe the model(s), method(s) to identify the presence and extent of statistical heterogeneity, and software package(s) used.	5
	13e	Describe any methods used to explore possible causes of heterogeneity among study results (e.g., subgroup analysis, meta‐regression).	5
	13f	Describe any sensitivity analyses conducted to assess robustness of the synthesized results.	5
Reporting bias assessment	14	Describe any methods used to assess risk of bias due to missing results in a synthesis (arising from reporting biases).	N/A
Certainty assessment	15	Describe any methods used to assess certainty (or confidence) in the body of evidence for an outcome.	N/A
RESULTS
Study selection	16a	Describe the results of the search and selection process, from the number of records identified in the search to the number of studies included in the review, ideally using a flow diagram.	14
	16b	Cite studies that might appear to meet the inclusion criteria, but which were excluded, and explain why they were excluded.	14
Study characteristics	17	Cite each included study and present its characteristics.	15
Risk of bias in studies	18	Present assessments of risk of bias for each included study.	N/A
Results of individual studies	19	For all outcomes, present, for each study: (a) summary statistics for each group (where appropriate) and (b) an effect estimate and its precision (e.g., confidence/credible interval), ideally using structured tables or plots.	15–20
Results of syntheses	20a	For each synthesis, briefly summarise the characteristics and risk of bias among contributing studies.	N/A
	20b	Present results of all statistical syntheses conducted. If meta‐analysis was done, present for each the summary estimate and its precision (e.g., confidence/credible interval) and measures of statistical heterogeneity. If comparing groups, describe the direction of the effect.	N/A
	20c	Present results of all investigations of possible causes of heterogeneity among study results.	N/A
	20d	Present results of all sensitivity analyses conducted to assess the robustness of the synthesized results.	N/A
Reporting biases	21	Present assessments of risk of bias due to missing results (arising from reporting biases) for each synthesis assessed.	N/A
Certainty of evidence	22	Present assessments of certainty (or confidence) in the body of evidence for each outcome assessed.	N/A
DISCUSSION
Discussion	23a	Provide a general interpretation of the results in the context of other evidence.	20
	23b	Discuss any limitations of the evidence included in the review.	20
	23c	Discuss any limitations of the review processes used.	20
	23d	Discuss implications of the results for practice, policy, and future research.	20
OTHER INFORMATION
Registration and protocol	24a	Provide registration information for the review, including register name and registration number, or state that the review was not registered.	N/A
	24b	Indicate where the review protocol can be accessed, or state that a protocol was not prepared.	N/A
	24c	Describe and explain any amendments to information provided at registration or in the protocol.	N/A
Support	25	Describe sources of financial or nonfinancial support for the review, and the role of the funders or sponsors in the review.	1
Competing interests	26	Declare any competing interests of review authors.	1
Availability of data, code and other materials	27	Report which of the following are publicly available and where they can be found: template data collection forms; data extracted from included studies; data used for all analyses; analytic code; any other materials used in the review.	1

Appendix B.

Search strategy of PubMed

No	Searches	Results
1	((“Patient Portals”[MeSH Terms])	N = 669
2	(“Patient Portals”[Title/Abstract] OR “Patient Web Portal”[Title/Abstract] OR “Portal, Patient Web”[Title/Abstract] OR “Portals, Patient Web”[Title/Abstract] OR “Web Portal, Patient”[Title/Abstract] OR “Web Portals, Patient”[Title/Abstract] OR “Patient Internet Portals”[Title/Abstract] OR “Internet Portal, Patient”[Title/Abstract] OR “Internet Portals, Patient”[Title/Abstract] OR “Patient Internet Portal”[Title/Abstract] OR “Portal, Patient Internet”[Title/Abstract] OR “Portals, Patient Internet”[Title/Abstract] OR “Patient Web Portals”[Title/Abstract] OR “Patient Portal”[Title/Abstract] OR “Portal, Patient”[Title/Abstract])	N = 1539
3	(“Scale”[Title/Abstract] OR “Framework”[Title/Abstract] OR “Tool”[Title/Abstract] OR “Criteria”[Title/Abstract] OR “Checklist”[Title/Abstract] OR “Questionnaire”[Title/Abstract] OR “Form”[Title/Abstract] OR “Model”[Title/Abstract] OR “Method”[Title/Abstract])	N = 7,471,918
4	(“Assessment”[Title/Abstract] OR “Evaluation”[Title/Abstract] OR “Design”[Title/Abstract] OR “Implementation”[Title/Abstract])	N = 3,757,841
5	2 AND 3 AND 4	N = 306
4	1 OR 5	N = 878

1. Search Strategy of Embase

No	Searches	Results
1	‘patient portals’:ti,ab,kw OR ‘patient web portal’:ti,ab,kw OR ‘portal, patient web’:ti,ab,kw OR ‘portals, patient web’:ti,ab,kw OR ‘web portal, patient’:ti,ab,kw OR ‘web portals, patient’:ti,ab,kw OR ‘patient internet portals’:ti,ab,kw OR ‘internet portal, patient’:ti,ab,kw OR ‘internet portals, patient’:ti,ab,kw OR ‘patient internet portal’:ti,ab,kw OR ‘portal, patient internet’:ti,ab,kw OR ‘portals, patient internet’:ti,ab,kw OR ‘patient web portals’:ti,ab,kw OR ‘patient portal’:ti,ab,kw OR ‘portal, patient’:ti,ab,kw	N = 2117
2	'scale’:ti,ab,kw OR ‘framework’:ti,ab,kw OR ‘tool’:ti,ab,kw OR ‘criteria’:ti,ab,kw OR ‘checklist’:ti,ab,kw OR ‘questionnaire’:ti,ab,kw OR ‘form’:ti,ab,kw OR ‘model’:ti,ab,kw OR ‘method’:ti,ab,kw	N = 9,763,674
3	'assessment’:ti,ab,kw OR ‘evaluation’:ti,ab,kw OR ‘design’:ti,ab,kw OR ‘implementation’:ti,ab,kw	N = 5,083,190
3	1 AND 2 AND 3	N = 502

2. Search Strategy of Scopus

No	Searches	Results
1	TITLE‐ABS‐KEY ((“Patient Portals” OR “Patient Web Portal “ OR “Portal, Patient Web” OR “Portals, Patient Web” OR “Web Portal, Patient” OR “Web Portals, Patient” OR “Patient Internet Portals” OR “Internet Portal, Patient” OR “Internet Portals, Patient” OR “Patient Internet Portal” OR “Portal, Patient Internet” OR “Portals, Patient Internet” OR “Patient Web Portals” OR “Patient Portal” OR “Portal, Patient”))	N = 2064
2	TITLE‐ABS‐KEY (“Scale” OR “Framework” OR “Tool” OR “Criteria” OR “Checklist” OR “Questionnaire” OR “Form” OR “Model” OR “Method”)	N = 35,656,981
	TITLE‐ABS‐KEY ((“Assessment” OR “Evaluation” OR “Design” OR “Implementation”))	N = 6,479,381
3	1 AND 2 AND 3	N = 753

Search Strategy of Web of Sciences

No	Searches	Results
1	TS= (“Patient Portals” OR “Patient Web Portal” OR “Portal, Patient Web” OR “Portals, Patient Web” OR “Web Portal, Patient” OR “Web Portals, Patient” OR “Patient Internet Portals” OR “Internet Portal, Patient” OR “Internet Portals, Patient” OR “Patient Internet Portal” OR “Portal, Patient Internet” OR “Portals, Patient Internet” OR “Patient Web Portals” OR “Patient Portal” OR “Portal, Patient”)	N = 1296
2	TS= (“Scale” OR “Framework” OR “Tool” OR “Criteria” OR “Checklist” OR “Questionnaire” OR “Form” OR “Model” OR “Method”)	N = 15,200,012
3	TS= (“Assessment” OR “Evaluation” OR “Design” OR “Implementation”)	N = 6,479,381
4	1 AND 2 AND 3	N = 265

Appendix C.

See Table C1.

Table C1.

Downs and Black checklist.

REPORTING		[1]	[2]	[3]	[4]	[5]	[6]	[7]	[8]	[9]	[10]	[11]	[12]	[13]	[14]	[15]	[16]	[17]	[18]	[19]	[20]	[21]	[22]
Q1	Hypothesis/aim/objective clearly described	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
Q2	Main outcomes in Introduction or Methods	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
Q3	Patient characteristics clearly described	0	0	1	1	1	1	0	0	1	1	0	1	1	1	1	1	1	1	1	1	1	1
Q4	Interventions of interest clearly described	1	1	1	1	1	1	1	1	1	1	1	0	1	1	1	0	1	1	0	1	1	1
Q5	Principal confounders clearly described	0	0	0	0	0	0	1	0	0	0	0	0	0	1	0	0	0	0	0	0	1	0
Q6	Main findings clearly described	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
Q7	Estimates of random variability provided for main outcomes	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown
Q8	All adverse events of intervention reported	0	0	0	0	1	0	0	0	1	0	1	1	0	1	0	0	1	1	0	0	0	0
Q9	Characteristics of patients lost to follow‐up described	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Q10	Probability values reported for main outcomes	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
EXTERNAL VALIDITY
Q11	Subjects asked to participate were representative of source population	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown
Q12	Subjects prepared to participate were representative of source population	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown	Unknown
Q13	Location and delivery of study treatment was representative of source population	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
INTERNAL VALIDITY – BIAS & CONFOUNDING
Q14	Study participants blinded to treatment	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Q15	Blinded outcome assessment	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Q16	Any data dredging clearly described	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
Q17	Analyses adjust for differing lengths of follow‐up	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Q18	Appropriate statistical tests performed	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
Q19	Compliance with interventions was reliable	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
Q20	Outcome measures were reliable and valid	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
Q21	All participants were recruited from the same source population	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
Q22	All participants recruited over the same period	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
Q23	Participants randomized to treatment(s)	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Q2	Allocation of treatment concealed from investigators and participants	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Q25	Adequate adjustment for confounding	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Q26	Losses to follow‐up taken into account	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Q27	Sufficient power to detect treatment effect at a significance level of 0.05	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
Total		12	12	13	13	14	14	13	12	14	14	13	13	13	15	13	12	14	14	12	13	14	13

Data Availability Statement

The data that supports the findings of this study are available in the supplementary material of this article.