Synthesising evidence on patient portals: a protocol for an umbrella review

Abstract

Introduction

Over the last two decades, patient portals have emerged as a noticeable eHealth strategy. To date, research on patient portals has been rapidly increasing. Our umbrella review aims to provide a meta-level synthesis to make sense of the evidence on patient portals from published systematic reviews (SRs).

Methods

We will employ a modified version of the Joanna Briggs Institute umbrella review method. The search strategy encompasses multiple databases. The inclusion criterion is specific to SRs focused on patient portal. Patients or public were not involved in this work.

Analysis

Two researchers will independently screen titles/abstracts and then full-text articles against the inclusion/exclusion criteria. Methodological quality of included reviews will be assessed and data will be extracted from the final selection of reviews. These reviews will be categorised into quantitative, qualitative and/or mixed-synthesis groups based on information about the design of primary studies provided in the reviews. Correspondingly, we will create quantitative, qualitative and/or mixed-synthesis Excel data-extraction tables. Within each table, data will be extracted with the reference to primary studies as reported in the reviews, and will be synthesised into themes and then a smaller number of findings/outcomes. Modified Grading of Recommendations Assessment, Development and Evaluation (GRADE) and Confidence in the Evidence from Reviews of Qualitative research (CERQual) tools will be applied to assess the strength of evidence at the level of each finding/outcome. The output of our umbrella review will consist of summary of findings tables and evidence profile tables. A narrative meta-level synthesis will be provided. We will use the clinical adoption meta-model as an organising framework.

Ethics and dissemination

As an outcome of this review, we will create a guidance and roadmap to be used in a future Delphi study to gather feedback from Canadian eHealth stakeholders. We will also present at conferences and publish the final report. The umbrella review does not require ethical approval.

PROSPERO registration number

CRD42018096657.

Strengths and limitations of this study.

• Through the application of Grading of Recommendations Assessment, Development and Evaluation (GRADE) and Confidence in the Evidence from Reviews of Qualitative research (CERQual), this work provides an evaluation of the strength of the quantitative evidence and confidence in the qualitative evidence.

• We apply Sandelowski et al’s conception of logic (ie, aggregation and configuration) underlying included reviews as an early step in umbrella reviews, to determine the approach to data analysis and synthesis that preserves the integrity of findings reported in included reviews.

• Our umbrella review offers a recommended, but seldom-used approach to managing overlaps in included reviews underpinned by the logic of aggregation, namely, elimination of duplicates at the level of primary studies.

• While selected elements of the Joanna Briggs Institute (JBI) umbrella review method will be used, we are not adhering to this method as a whole. Our methodological modifications of the JBI approach include (a) extracting data at the level of primary studies as reported within reviews underpinned by the logic of aggregation and (b) using CERQual tool developed by the Cochrane GRADE group.

• Only systematic reviews published in English will be included.

Introduction

During the past two decades, many western countries have introduced eHealth strategies and programmes to support patients through a variety of electronic health technologies such as the patient portal.^1–3 For example, the Patient Portals & e-Views project funded by Canada Health Infoway was designed at the jurisdictional level to enable patients to assume an active role in their own health.⁴ In England, the National Health Service Patient Online programme allows patients to securely communicate with their health providers, schedule appointments and view their GP record.² The US Office of National Coordinator for Health Information Technology has introduced the Patient Engagement Playbook as a web-based resource guide for healthcare providers and administrators to engage patients in their health and care through such technologies as patient portals linked to an electronic health record.⁵

Patient portals are a secure interface that provide patients with 24-hour online access to their personal health information such as recent doctor visits, discharge summaries, medications, allergies, immunisations and lab results.^{6 7} Some portals also enable patients to communicate with their care providers through secure email/text messaging as well as to schedule appointments and request medication refills online. Patient portals, also known as tethered personal health records, are maintained by healthcare organisations.

Organisations responsible for consumer-focused eHealth technologies tout the benefits of patient portals including improved communication with care providers, better access to health information and services, higher satisfaction level and quality of care and increased motivation and confidence in managing one’s health.^{4 5 8} For example, results from a patient survey (n=1000) during a 6-month Canadian pilot project on the implementation of the ‘Citizen Health Information Portal’, suggested improved patient care and provider–patient relationships.⁹ Similarly, empirical studies have identified the benefits of patient portals.^10–15 However, other studies cautioned about barriers to the use of patient portals among different user groups. Factors influencing utilisation of portals among patients include health literacy, technological proficiency, educational level and socioeconomic status.^16–18 Provider-specific factors include concerns about workload and personal attitudes and perceptions influencing adoption of portals among health providers.¹⁹ Despite these mixed responses, promised benefits of portals such as an enhanced patient engagement and improved health outcomes seem to generate growing interest in this technology among various stakeholders.

Alongside policy conducive to the implementation and uptake of eHealth such as the US Meaningful Use legislation,²⁰ research on the introduction, use and impact of electronic patient portals has been rapidly increasing. In addition to hundreds of original research articles, multiple systematic reviews (SRs) on patient portals have been published in the past decade. These reviews are focused on diabetes care,²¹paediatric population,²² impact,²³ patient and provider attitudes,¹⁹ facilitators and barriers²⁴ and technical development.²⁵ Thus, the evidence on patient portals is dispersed across many publications. Moreover, the empirical evidence on portals is mixed. For instance, studies have reported varying results as to whether utilisation of patient portals results in a decrease, increase or no difference in the number of patient visits.^26–28 These accumulating disparate findings have made it difficult for those involved with, or affected by, patient portals to form a coherent view on the current state of evidence on the introduction, use and effects of these technologies.

With the volume of SRs on eHealth technologies rapidly growing, a higher or meta-level synthesis is required to make sense of the evidence from published reviews in a given domain such as patient portals. The need for a SR of reviews on the topic of patient portals is confirmed by our preliminary literature search, which identified one meta-level review explicitly referring to patient portals.²⁹ However, this integrative review by Jilka et al ²⁹ is based on 10 reviews published prior to 2015 and specifically focused on patient-accessible electronic health records among adult populations. Thus, reviews on patient portals were a subset of articles on patient access to electronic records. In light of these limitations, there is a necessity for a current and more comprehensive SR of reviews addressing the increasing utilisation of patient portals. To address this knowledge gap, we will conduct an umbrella review synthesising present-day evidence on patient portals.

Our decision for selecting an umbrella review approach for this SR of reviews was made following a scan of published higher-level reviews and relevant methodological literature.^{30 31} The literature scan revealed a disunity of terminology for labelling higher-level reviews: umbrella review, overview, meta-review, review of SRs, review of reviews and so on. Meta-review label is often applied to SRs of published meta-analyses, or reviews that employ statistical analyses of data pooled from randomised controlled trials (RCTs) or observational intervention studies; meta-reviews themselves may or may not employ statistical analyses.^32–35 Reviews of SRs and overviews of reviews tend to focus on quantitative SRs not exclusive to meta-analyses of RCTs. Some authors reserve the term overviews for syntheses of Cochrane SRs only.³⁶ In contrast, umbrella reviews and reviews of reviews are usually more inclusive of different types of SRs. In particular, an umbrella review ‘focuses on broad condition or problem for which there are competing interventions and highlights reviews that address these interventions and their results’ (Grant and Booth, p95)³⁷ to integrate evidence from multiple SRs based on primary studies of various designs into one handy document.³⁸ In fact, the Joanna Briggs Institute (JBI) claims that their umbrella review methodology is ‘the first to consider reviews that report other than quantitative evidence’. (Aromataris et al, p132)³⁹ Our review will include reviews of quantitative, qualitative and mixed-method primary studies, and thus the JBI approach to umbrella reviews offers a useful guidance.⁴⁰ However, we adopt selected elements of this approach while modifying other elements of the JBI method. Our methodological decisions are explained below.

We anticipate that our substantive and methodological contribution will be manifold. This umbrella review will consolidate aspects of the current state of knowledge about patient portals. Given the rapidly rising volume of SR literature to date, the umbrella review method is the next logical step to synthesise the review literature on portals in a more timely and efficient manner. Moreover, we aim to apply a novel approach to appraising quantitative evidence that supplements Grading of Recommendations Assessment, Development and Evaluation (GRADE) criteria modified by the evidence-based practice centres programme^{41 42} with a vote count (described below). Further, we demonstrate the usefulness of Sandelowski et al’s⁴³ conception of the logic of aggregation or configuration underpinning included reviews (addressed in more detail below). Next, we offer an approach to managing overlaps in reviews by eliminating duplicates at the level of primary studies. Further, as far as we know, our application of Confidence in the Evidence from Reviews of Qualitative research (CERQual) criteria⁴⁴ to evaluate qualitative evidence will be the first attempt to use this tool in the context of umbrella reviews. Additionally, the application of GRADE and CERQual to rate the quality of eHealth evidence will contribute to the health informatics discipline in terms of both growing the evidence base and providing guidance on evidence review methods.

Review methodology

Umbrella reviews, or overviews of reviews of qualitative, quantitative and mixed-method studies is a growing genre in health sciences,^{36 45} and several protocols have been recently published.^46–48 In 2016, Pollock et al ⁴⁹ identified as many as 52 guidance documents produced by 19 research groups on how to conduct overviews of reviews. The most consistent recommendations are that umbrella reviews include published SRs with the aim to synthesise findings from included reviews; that these SRs are retrieved through comprehensive searches using more than one database; and that the methodological quality of reviews is assessed. The most consistent challenge that these guidance documents point out is that overviews are limited by the methods, reporting and coverage of their included SRs. Further, Pollock et al ⁴⁹ found that the guidance documents present limited and inconsistent recommendations in respect to procedures for evaluating confidence in evidence, managing overlap among reviews and analysing and synthesising data from SRs that include primary studies of various designs. Moreover, Pollock et al ⁴⁹ indicated that the guidance documents do not address several important logistical challenges (eg, the extent of turning to primary studies vs remaining at the level of included SRs). Indeed, this diversity or absence of guidance is reflected in the methodological variation observed in recently published umbrella reviews.^50–52

Our survey of several published protocols for umbrella reviews identified a protocol by Rouleau et al ⁴⁷ that illustrates how researchers conducting a review of mixed-synthesis reviews grapple with some challenges listed above (eg, evaluating quality of evidence, managing overlaps, synthesising data from mixed-synthesis reviews). Rouleau et al’s protocol⁴⁷ is also distinct for its recognition of (a) the element of emergence in umbrella reviews (ie, an open-ended nature of the data extraction process that makes it counterproductive to preselect all phenomena of interest at the outset) and (b) the importance of both inductive and deductive analysis when using a preselected theoretical framework. We anticipate that these challenges and insights will be applicable for our work.

Our umbrella review will use a modified version of the JBI umbrella review methodology as defined earlier, and more details are provided below. This protocol adheres to the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols guidelines⁵³ and has been registered in PROSPERO (http://www.crd.york.ac.uk/PROSPERO/).

Objective and questions

The objective of this umbrella review is to summarise the aspects of the current state of evidence on patient portals reported in published SRs. Based on this summary, our future step is to provide guidance and a roadmap for stakeholders involved with this eHealth technology, specifically in Canada. Our findings will be of interest not only to eHealth managers/directors, health providers and researchers but also to patients and families affected by the introduction of patient portals. The questions addressed in this umbrella review are:

1. What are the characteristics of the patient portals being introduced and used in different settings?

2. What is the impact of patient portals on clinical outcomes of care?

3. What are the system-related, health provider-related and patient-related factors that influence the introduction, use and impact of patient portals?

Conceptual framework

We will use the clinical adoption meta-model (CAMM)⁵⁴ as a framework to organise and make sense of the umbrella review findings. The CAMM is a maturity model used to understand, describe and explain the introduction, use and effects of eHealth systems over time. It is a temporal model with five dimensions of availability, use, clinical/health behaviour, outcomes and time. In this review, availability refers to the ability of users to access the patient portal. System use refers to user interaction and experience with the portal. Clinical/health behaviour refers to changes in user behaviours from interacting with the portal. Outcomes refers to effects of portal use, which can be at the patient, provider, organisation or population level. Time refers to the transition periods across the four dimensions.

Modifications to the JBI umbrella review method

Umbrella review method is intended to provide an overall examination of a body of information that is available for a given topic.⁴⁰ We have adopted selected key features of the JBI approach to umbrella reviews:

1. Compiling evidence from multiple research syntheses that may be quantitative and/or qualitative in nature.

2. Including reviews based on empirical studies rather than theoretical speculations or opinion (even if the review itself is titled theoretical or critical).

3. Summarising evidence from existing reviews without retrieving and reanalysing primary studies.

4. Publishing a protocol prior to conducting the umbrella review.

5. Including at least two researchers to conduct the umbrella review.

6. Using a standard JBI critical appraisal checklist to assess the methodological quality of the included reviews.

7. Applying an established tool to assess the overall strength of the evidence.

8. Presenting a summary of findings table and an evidence profile table.

The following five features are unique to our review and constitute a modification of the JBI approach to umbrella reviews. First, we will use Sandelowski et al’s⁴³ classification of reviews (ie, the logic of aggregation or configuration underpinning SRs [To prevent any possible confusion, we would like to emphasise that Sandelowski et al’s ideas presented in this 2012 article, differ from both her earlier conceptions of aggregation and the JBI’s terminology used in the context of mixed-method reviews. Importantly, the logics of aggregation and configuration are not tied exclusively to any one side of the qualitative/quantitative binary. Eg, narrative qualitative meta-synthesis can be based on the logic of aggregation]) as a guidance for data analysis and synthesis (explained below). Second, although we will summarise data from included reviews without retrieving and reanalysing primary studies, our Excel data-extraction tables will list the primary studies referenced in each review that aggregates primary quantitative, qualitative and/or mixed findings, as a support for relevant pieces of data. This step will enable us to reconcile the primary studies across the reviews to eliminate duplicates (described below). In addition, this step is a prerequisite for the application of GRADE and CERQual criteria at the level of individual outcome/finding. Third, we will apply both the GRADE criteria modified by the USA Evidence-Based Practice Centres programme^{41 42} and vote counting⁵⁵ as ways to determine the strength of evidence synthesised from aggregative reviews that include quantitative primary studies. Fourth, we will apply the CERQual criteria to determine the confidence in the evidence synthesised from aggregative reviews that include qualitative primary studies. Fifth, we will apply the CAMM⁵⁴ to organise and make sense of the umbrella review findings.

Our SR of reviews will reflect methodological recommendations outlined by Pollock et al ³⁰ and Smith et al.³¹ Of note is that these recommendations reinforce those presented in the JBI approach to umbrella reviews.^{39 40} Particular attention will be paid to what Pollock et al ³⁰ identified as eight methodological challenges affecting the quality of reviews of reviews: (a) overlap between reviews (studies appearing in more than one review); (b) outdated reviews; (c) ‘systematic reviews’ that do not meet expected methodological standards; (d) assessment of methodological quality of reviews; (e) quality of reporting within reviews; (f) applying GRADE; (g) potential for publication bias; and (h) summarising key findings in brief accessible format suitable for informing decision-making. Each of these areas will be addressed either below or in the final review report, as appropriate.

Search strategy

An academic librarian developed a search strategy and assisted with searches. Two search terms, (a) patient portal and (b) SRs, were used in combination and adapted according to the databases, MeSH terms and Boolean rules, and other library best practices to maximise the retrieval of relevant citations. For example, synonyms for patient portal included patient web portal and tethered personal health record. Multiple search terms for SRs are listed in the following section. We searched multiple databases on 20 April 2018: Ovid MEDLINE, Embase, CINAHL Plus with Full Text, Web of Science Core Collection, Scopus, the Cochrane Database of Systematic Reviews, PROSPERO registry, the JBI Database of Systematic Reviews and Implementation Reports, and Proquest Dissertations & Theses. A MEDLINE search strategy is included as an online supplement.

A preliminary scan of retrieved citations (after eliminating duplicates) identified ~40 citations meeting inclusion criteria at a glance. We anticipate that after a rigorous application of the inclusion/exclusion criteria and a methodological quality appraisal, we will have a smaller, manageable number of reviews. A preliminary scan also revealed two other important feature of SRs candidates for inclusion in our umbrella review: (a) the majority of SRs synthesise quantitative, qualitative and/or mixed-method primary studies within each review and (b) none of a few purely quantitative SRs perform meta-analyses with statistical pooling of findings. Thus, SRs candidates for inclusion all appear to synthesise their findings narratively.

We restricted our searches to reviews published since the year 1990 in English. Patient portals appeared in the 1990s, and the policy attention fueled their development and use in the 2000s. Incidentally, during this time, various kinds of SRs and overviews of SRs started to flourish. In our preliminary searches, the bulk of retrieved citations fell within the last decade. Due to the recent emergence of patient portals, the issue of outdated reviews (ie, Pollock et al’s³⁰ challenge #2) will likely be irrelevant in our umbrella review. We are planning to supplement the above searches by examining the reference lists of all included reviews for additional studies. We will also search the first 100 citations in Google scholar for missed reviews. The searches will be rerun during an analysis stage to identify reviews published since the initial search. In addition, at that time, we will expand our search to SRs published in grey literature such as reports commissioned by governmental agencies and non-governmental organisations, and retrieved from a Google search.

Inclusion criteria

The overarching inclusion criterion is SRs focused on patient portals as the topic. The types of reviews may include systematic reviews, meta-analysis, narrative reviews, descriptive reviews, qualitative reviews, theoretical reviews, realist reviews, critical reviews, literature reviews, mixed-methods reviews, and qualitative evidence synthesis.^{37 38} To be included, these reviews must synthesise findings from empirical studies (ie, the review authors must indicate that their review synthesises primary research studies; if in included aggregative reviews we come across an occasional non-empirical primary source or a SR, we will delete this primary source). Because scoping reviews tend to include broader, non-empirical literature, they will be excluded. Inclusion will be limited to reviews published in English since 1990.

We will use the PICOS/PICo framework to provide explicit criteria on the types of population (P), intervention (I), comparison (C), outcome (O), study design (S) and context (Co) for inclusion⁵⁶ as described below.

• Population—patients regardless of demographic and disease characteristics, and also health providers, consumers, researchers, educators, policy and/or decision makers and the public.

• Intervention/exposure—patient portal; patient web portal; tethered personal health record.

• Comparison—primary studies in included SRs can be intervention versus a non-exposed control group, pre versus post, user versus non-user, and single cohorts only, as well as qualitative designs not mentioning any comparison.

• Outcome—any types of effects including attitudes/behaviours, utilisation, facilitators and barriers, care processes, economic value, health outcomes or policies.

• Study design—any types of SRs summarising empirical studies (eg, meta-analysis, narrative review, descriptive review, qualitative review, theoretical review, realist review, critical review, literature review, mixed-methods reviews, and qualitative evidence synthesis). Reviews can include empirical primary studies of any design: experimental, quasi-experimental, cross-sectional surveys, mixed and qualitative designs.

• Context—any organisational and practice settings in countries including but not limited to the USA, UK, Canada or Netherlands, except those locations explicitly labelled in SRs as low- or medium-resource countries.

Exclusion criteria

• Reviews with multiple eHealth technologies where portals are just one of many technologies examined.

• Reviews that include standalone (ie, not tethered) personal health records controlled by patients (this topic will be addressed in a separate umbrella review).

• Reviews that explicitly identify in the title or abstract their focus on low- and medium-resource countries (this is a topic for a separate umbrella review).

• Reviews in languages other than English.

• Reviews not based on primary empirical studies, for example, scoping reviews as well as higher-order reviews such as reviews of reviews.

• Reviews that do not provide a complete list of included primary studies.

• SRs that do not describe (at a minimum) the search strategy and explicit inclusion criteria. This inclusion/exclusion decision will happen at the stage of full-text screening or quality evaluation, and will address Pollock et al’s³⁰ challenge #3.

Review selection

Citations retrieved via searches of electronic databases will be imported to Covidence (https://www.covidence.org/home), a Cochrane-supported software designed for conducting SRs. Two researchers will independently proceed through a series of steps: (a) screening the titles and abstracts against the inclusion criteria and (b) screening the full-text articles that met the initial screening step, against the inclusion criteria. Excluded articles and the reasons for exclusion will be logged. Discrepancies will be resolved by consensus between the two researchers and/or by a third researcher.

Methodological quality assessment

Typically, the methodology for conducting review of reviews presupposes that the quality of included reviews rather than the quality of primary studies be assessed. The purpose of quality assessment is to assess methodological quality, risk of bias and reporting quality.

As a preliminary scan of retrieved citations revealed, candidates for inclusion in our umbrella review are mostly mixed-syntheses reviews that narratively aggregate findings of quantitative, qualitative and/or mixed-method primary studies within each review. To assess the quality of SRs included after screening the full text, we will apply the JBI critical appraisal checklist for SRs.⁴⁰

While several critical appraisal instruments exist^{40 57–61} and they are based on common principles, the JBI checklist is the only tool designed for evaluating both quantitative and qualitative reviews. There are 11 questions in the JBI checklist each with a possible response of yes, no or unclear. For example, Q5 asks ‘were the criteria for appraising studies appropriate?’ and requires that the included review provided details of the appraisal in either the Methods section, an appendix or an online supplementary file. By tallying all yes responses, a review can have a score range of 0–11, with 11 being the highest quality.

We will develop a rubric explicating how to interpret each of the tool’s criteria for this specific review. In addition, we will determine the cut-off score for eliminating low quality reviews. Using the agreed-upon rubric, one researcher (FL) will assess the quality of all included reviews, whereas the second and third researchers (MA and OP) will each assess at least 30% of reviews selected randomly. Any discrepancies will be discussed by all three researchers and resolved by consensus.

Overall, our team’s approach to assessing methodological quality of reviews recognises the issue of the absence of a universally accepted quality assessment instrument and the ensuing attempts by reviewers to mitigate this challenge by acknowledging the subjective component in applying quality assessment tools⁶² and by modifying existing tools.⁶³ By paying close attention to the process of quality assessment, we are aiming to address Pollock et al’s³⁰ challenges #4 and #5.

Patient and public involvement

Patients and public were not involved at this stage. Patient groups will be included in a future Delphi study.

Data analysis

Prior to data extraction, we will separate included SRs into distinctive groups based on design of primary studies comprising those reviews (ie, purely quantitative, purely qualitative or mixed synthesis). For the purely quantitative reviews, we will ascertain their approaches to data synthesis, for instance, meta-analysis with statistical pooling of findings or narrative synthesis. For qualitative and mixed-synthesis reviews, we anticipate some kind of narrative synthesis reported by the authors of those reviews. As explained above, our preliminary scan of review papers that were candidates for inclusion has shown that the majority of reviews are mixed syntheses while a smaller number of reviews synthesise quantitative primary studies; and that all reviews employ narrative synthesis. This grouping has implications for our subsequent analysis and synthesis.

As the next analytical move, we will apply Sandelowski et al’s⁴³ ideas about the type of logic—aggregation or configuration—that can underpin review syntheses (irrespective of the design of primary studies comprising those reviews). The logic of aggregation is evident when a review simply amasses findings of primary studies of various designs in an additive manner. In other words, aggregation is merging thematically similar findings into a pooled summary (Sandelowski et al, p323).⁴³ In contrast, the logic of configuration is evident when a review develops a synthesis exceeding any specific findings of primary studies. In other words, configuration is meshing thematically diverse findings into a theory or model. (Sandelowski et al, p323)⁴³ A significance of this move is that narrative aggregative syntheses can be disaggregated into the level of primary studies (for our Excel data-extraction tables) without detracting from the integrity of SR findings. On the other hand, syntheses underpinned by the logic of configuration should not be pulled apart into their component findings, as this can detract from the integrity of a theory or model. (Sandelowski et al, p323)⁴³

Based on the above groupings, we will determine what Excel data-extraction tables are necessary in our umbrella review. Examples of data-extraction tables are quantitative, qualitative and/or mixed synthesis. Narrative aggregative SRs included in these tables will be analysed at the level of primary studies. If necessary, we will separately extract any theories reported in reviews underpinned by the logic of configuration.

As mentioned earlier, this analytical process will enable us to achieve three important goals: (a) not to retrieve and reanalyse primary studies while at the same time tracking their findings; (b) to manage overlaps in reviews by removing duplicate primary studies from each table so that they do not contribute the same finding more than once; and (c) to apply GRADE and CERQual at the level of individual outcome/finding from the included reviews.

Eliminating duplicates

Duplicates are identified as an important issue in umbrella reviews, and metrics for calculating the degree of an overlap have been suggested.⁶⁴ As described above, our approach to managing an overlap among included reviews is to filter out duplicate primary studies so that they only appear once. The goal of removing duplicates is ‘to preclude the double counting that overstates the evidence’. (Pieper et al, p374) ⁶⁴

On the other hand, we will aim to avoid an overestimation of the degree of overlap.⁶⁴ This happens when different reviews include the same primary studies, but extract non-overlapping data from those primary studies. Our Excel data-extraction tables will list both the primary studies and the finding from these studies reported in reviews, so that we will only eliminate fully overlapping findings originating from the same primary study and reported in different reviews.

Data extraction

Excel data-extraction tables described above will be initially piloted by at least two reviewers. Extracted information will include:

1. Characteristics of included reviews: review reference (author–year–country), date of search (years that the review covers), objective of review, types of studies/designs included in review, number of included studies and country of included studies.

2. Setting; focus of the review; study population and participant demographics and baseline characteristics (participants included in review; number of participants included in review; target condition being addressed in the review); interventions included in review (a thorough description of the features of the patient portal); comparisons included in review if applicable; suggested mechanisms of interventions included in review; outcomes included in review; statistical data from quantitative studies reported in review such as effect size, CIs, and positive and negative predictive values if applicable; themes from qualitative studies reported in review; study limitations reported in review.

Items in group B will be extracted line by line from the reviews’ Findings/Results sections and recorded in the relevant Excel tables by the primary studies from which these findings originate, as reported in the reviews. One researcher (FL) will extract all data independently. Two other researchers (MA and OP) will each crosscheck at least 30% of the extracted data against review articles. All three researchers will compare the outputs for consistency and resolve discrepancies through discussion.

Data synthesis

If applicable, statistical meta-analysis and subgroup analysis will be performed depending on the homogeneity in the scope of the intervention, population or outcomes, using the information provided in included reviews of RCTs and observational intervention studies.⁶⁵

Within each data extraction table, data will be synthesised into descriptive themes, then analytical themes,⁶⁶ and then higher-order domains (or evidence findings), and the final summary of findings tables will be presented. We will use the CAMM as an organising framework to narratively report the findings based on the temporal implementation stage of patient portal in health organisations. Within each implementation stage, a narrative can be structured around the type of evidence, selected population characteristics and type of outcome.

One researcher (FL) will conduct the synthesis, which will be checked by the other two researchers (MA and OP). Discrepancies will be discussed to reach consensus among the three researchers.

Rating the evidence

We will apply modified GRADE and CERQual tools to assess the strength of the quantitative evidence and the confidence in the qualitative evidence, respectively, at the level of each individual finding. The output of this process will be evidence profile tables.

For quantitative findings, we will apply the GRADE method to determine the strength of evidence for each outcome. The GRADE method will follow the updated guidance from the USA Evidence-Based Practice Centre⁴¹ as used by Gibbons et al ⁴² in their evidence review of consumer eHealth technology. Specifically, we will assign a score to each outcome according to the five domains: study limitations, directness, consistency, precision and reporting bias. Then, an overall grade—high, moderate, low or insufficient—will be assigned to reflect the level of confidence that the estimated effect of the outcome is close to the true effect.⁴¹ We will supplement the GRADE method with vote counting^{55 67} to quantify the evidence for each outcome. This will be done by tallying the number of positive/neutral/negative results for each outcome based on the significant differences reported in the reviews. An outcome will be considered positive if at least 50% of the results are positive and statistically significant. While vote counting does not show the magnitude of effect, it can reveal the overall direction of the effect for a given outcome. We will also record the sample size of each primary study if mentioned in the reviews and use this information alongside the vote count to make sense of the outcome qualitatively.

For qualitative findings, we will apply GRADE–CERQual criteria to determine the confidence in evidence for each outcome.^68–73 We will assign a score to each outcome according to the four domains: methodological limitations, coherence, relevance and adequacy. Then, an overall grade—high, moderate, low or very low confidence—will be assigned to reflect the level of confidence that the estimated effect of the outcome is close to the true effect.^68–73

The steps of eliminating duplicates and rating the evidence aim to address Pollock et al’s³⁰ challenges #1, #6 and #7.

Ethics and dissemination

A future Delphi study

As an output of this review, we will create a guidance and roadmap to be used in a future Delphi study^{74 75} to gather feedback from Canadian eHealth stakeholders (number and roles of stakeholders to be decided). A guidance will consist of a set of suggested actions on how a healthcare organisation may achieve the optimal effects based on the evidence available and our personal field experiences, when implementing a patient portal. A roadmap will visually represent suggested actions based on CAMM⁵⁴ stages.

Our findings will be of interest not only to eHealth managers/directors, health providers and researchers but also to patients and families affected by the introduction of patient portals. We will also present at conferences and publish the final report in a peer reviewed, preferably open access journal.