Validating a framework to guide the implementation of high-quality virtual primary care: an international eDelphi study protocol

Abstract

Background

There is an urgent need to support primary care organisations in implementing safe and high-quality virtual consultations. We have previously performed qualitative research to capture the views of 1600 primary care physicians across 20 countries on the main benefits and challenges of using virtual consultations. Subsequently, a prototype of a framework to guide the implementation of high-quality virtual primary care was developed.

Aim

To explore general practitioners’ perspectives on the appropriateness and relevance of each component of the framework’s prototype, to further refine it and optimise its practical use in primary care facilities.

Methods and analysis

Participants will be primary care physicians with active experience providing virtual care, recruited through convenience and snowball sampling. This study will use a systematic and iterative online Delphi research approach (eDelphi), with a minimum of three rounds. A pre-round will be used to circulate items for initial feedback and adjustment. In subsequent rounds, participants will be asked to rate the relevance of the framework’s components. Consensus will be defined as >70% of participants agreeing/strongly agreeing or disagreeing/strongly disagreeing with a component. Data will be collected using structured online questionnaires. The primary outcome of the study will be a list of the essential components to be incorporated in the final version of the framework.

Ethics and dissemination

The study has received ethical approval conceded by the Imperial College London Science, Engineering and Technology Research Ethics Committee (SETREC) (reference no .6559176/2023). Anonymous results will be made available to the public, academic organisations and policymakers.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• This study will validate and refine a framework to support the implementation, evaluation and monitoring of the use of virtual consultations in primary care, using a codesign and participatory approach.

• An international recruitment strategy will enable the capturing of insight of general practitioners practising in various countries.

• Limitations include general limitations of the Delphi methodology, including attrition and selection biases.

Introduction

The primary care landscape has undergone an intense transformation since early 2020, further hastened by advancements in digital technologies expedited by the COVID-19 pandemic.^{1 2} Specifically, virtual consultations (ie, remote synchronous consultations performed using telephone or online conferencing software) emerged as an innovative approach to patient–provider interaction, with the potential to transform care delivery by improving effectiveness and timeliness of care.³

As part of the emergency response during the pandemic, implementation of virtual consultations was considered essential for the safety of both patients and healthcare staff. Due to the urgency of the situation, this implementation was often carried out rapidly, as healthcare systems focused on the necessary immediate response to the outbreak.⁴ For example, in 2020, in England’s National Health Service (NHS), a rapid procurement was run nationally for online consultations systems in primary care to support general practitioner (GP) practices in moving immediately to the total triage model advocated at the time by NHS England. Subsequently, NHS England established the Digital First online consultation and video consultation framework to provide greater assurance of online consultation products and centralise processes for management of serious incidents relating to their use.^5–8

The post-pandemic context presents a new opportunity—and ethical imperative—to revisit the implementation of virtual consultations, and co-design strategies that support both providers and patients towards a better, safer and more equitable use.^{4 9 10} In this context, a structured and systematic framework to guide, evaluate and monitor the smooth implementation of virtual consultation into the existing primary care infrastructure is crucial.

There have been several previous attempts to develop guidance material to support the implementation of various virtual care and telemedicine tools. In 2022, the WHO published the Consolidated Telemedicine Implementation Guide, a comprehensive document intended on supporting countries on how to best plan, implement and maintain telemedicine, as well as incorporating lessons learnt from the recent global COVID-19 pandemic.¹¹ In the UK, the General Medical Council outlined some broad considerations for healthcare providers on when best to use virtual versus face-to-face consultations.¹² Similar efforts by the Royal College of General Practitioners and other health organisations have resulted in an assortment of guidance, literature and toolkits which stipulated how best to conduct remote consultation effectively and aided in safer virtual care practices.^{13 14} Though these pieces were valuable in offering high-level, general guidance for policymakers and healthcare administrators, as well as some practical tips for healthcare providers, the lack of data provided to substantiate these recommendations limited their ability to inform the myriad of policies necessary for a more systemic integration of remote consultation tools in the context of the wider national health system.^{11 12}

Greenhalgh et al, proposed an innovative framework which incorporated seven main factors (the reasons for the consultation, clinical relationships, the patient, their home and family, technology, staff, the healthcare organisation itself and the wider system) and explored how these elements influence each other and affected digital maturity in the NHS.¹⁵ However, much of the nuance discussed was mainly applicable to the UK health system, with findings and subsequent recommendations less generalisable to other healthcare contexts. Consequently, there is a clear knowledge gap for data-driven guidance on safe and effective implementation of virtual consultations which is appropriate for an international audience.

Based on qualitative research capturing the views of more than 1600 GPs across 20 countries, we have previously developed the prototype of a data-driven framework to guide the implementation of high-quality virtual primary care. The framework includes several domains (contextual considerations, technology infrastructure, awareness & experience, safety & risk management, strategic planning and supporting policies), each one including several components (figure 1).^{16 17} This framework aimed to provide healthcare professionals, primary care organisations and policymakers with a methodical, stepwise approach for implementing, planning and evaluating the implementation of virtual consultations. However, there is a need to bridge the gap between research and real-world application, aligning the framework with those who will use it in general practice settings.

Figure 1.

Prototype framework for the implementation of high-quality virtual consultations in primary care, as previously developed by the research group. Adapted from Li et al.¹⁶

This work aims to obtain GPs’ perspectives on the prototype and specifically to reach consensus on which components should be prioritised for the final version of the framework. Through this collaborative and iterative process, we aim to develop a practical, field-tested tool that truly reflects the needs, challenges and perspectives of primary care physicians—and involve them as active stakeholders leading change.

Methods and analysis

Study design

The study will use a Delphi methodology to seek consensus among a group of experts.¹⁸ The technique seeks the opinion of a group of experts to assess the extent of agreement and to resolve disagreement on an issue and been used successfully to establish consensus across a range of subject areas.^19–21 Originally developed by Dalkey and Helmer, the Delphi methodology includes two fundamental steps.²² First, the overarching approach is based on a series of ‘rounds’, where a group of experts are asked their opinions on a particular issue. Second, participants can see the results of previous rounds through the process allowing them to reflect on the views of others and reposition their own opinions accordingly. The number of Delphi rounds depends on the amount of time and resources available, the number of questions and consideration of levels of sample fatigue.²³ The literature demonstrates that the classic Delphi technique had four rounds.²³ The findings of each round are always shared with the broader group anonymously; this avoids any bias that might result from participants being concerned about their own views being viewed negatively, or from their own opinions being biased by personal factors.

The electronic Delphi questionnaire (‘eDelphi’) helps in achieving a good representation and diversity of panel members, saves time and hastens the survey rounds using technology without physical voting²⁴; it has been widely used in healthcare research.^25–28

In this study, standard eDelphi methodology will be used with an expectation to run up to four rounds, including a pre-round. During the pre-round, items will be circulated for initial feedback and to capture additional items. Participants will be subsequently asked to enrol in subsequent rounds, in which they will describe their level of agreement with each individual statement of the framework. Three to four rounds are expected to be performed; however, the number may be lower in case consensus is achieved earlier. Before each round, a summary of the group’s overall response, as aggregated data, will be shared with each participant, allowing for individual responses to be reconsidered towards the establishment of consensus. An overview of the study design is provided in figure 2.

Figure 2.

Overview of study design.

Expert panel recruitment

Throughout this study, ‘experts’ will be defined as primary care physicians with working experience delivering virtual consultations (ie, video or telephone), working in one of the 20 countries involved in the development of the prototype. The number of participants who will be approached for the study will be between 50 and 150; generally, a number of participants close to 30–50 is considered acceptable in concluding rounds for a homogeneous Delphi.^{29 30} Participants will be recruited using both convenience and snowball sampling from contacts known to the research team via the European General Research Network, as well as personal contact networks, and invitation to participate will also be shared in Facebook/LinkedIn profiles of the research team members.

Survey development

The components of the framework were previously published (figure 1),¹⁶ as part of previous work evaluating the adoption of virtual care in general practice across 20 countries (Australia, Brazil, Canada, Chile, Colombia, Croatia, Finland, France, Germany, Ireland, Israel, Italy, Poland, Portugal, Slovenia, Spain, Sweden, Turkey, the United Kingdom and the United States).

Participant demographic information will be collected, including gender, age, hours of work per week, teaching activities, country of residence and type of setting (urban/rural/mixed). In the pre-round, participants will be asked to provide feedback on the individual components and whether there are any additional components that should be included (free text). During subsequent rounds, participants will be asked to rank the relevance of each component, by responding the question: ‘How essential is this aspect for the implementation of high-quality virtual consultations in primary care?’. Responses will be captured by a 5-point Likert scale (1=not useful; 2=may be useful, but not essential; 3=somewhat essential; 4=very essential; 5=completely essential).^{31 32} Prior to use, the questionnaire will be piloted with the respective national leads to ensure the appropriateness of its contents, clarity in the questions asked, and whether it adequately accounted for contextual nuances present in the different participating countries. The final paper resulting of this research will also interpret and contextualise the implications of the findings across diverse countries. The questionnaire is presented in online supplemental file 1.

Data collection

An initial invitation email will be sent to possible suitable participants. Following confirmation of participation, and to conduct the eDelphi, an electronic invitation containing the participant information sheet, the consent form and links to the surveys, will be sent to the participants for each round.

Data collection will take place between January and March 2024. The survey will be made available for agreed participants on a secure online survey platform (Qualtrics) accessed through Imperial College London. The survey associated with each of the eDelphi rounds will remain open for two weeks for each round, and participants will receive weekly reminders. In every round, participants will have access to the group results from the previous rounds, to allow them to review their answers at every stage. Each survey will take approximately 10–20 min to complete.

Data analysis

Participants’ information will be analysed using descriptive statistics, including absolute (n) and relative frequencies (%) for categorical variables and mean and SD for continuous variables. The Delphi round responses (ie, ranking of relevance of components) will be presented using mean, median, SD and IQR. Consensus will be determined when >70% of participants agreeing/strongly agreeing or disagreeing/strongly disagreeing with a given component, as similar Delphi studies have previously considered this level of agreement appropriate.³³

Data management and governance

All collected data and accompanying analysis will be stored in a secure Imperial College Shared Drive and held in accordance with General Data Protection Regulation (2016), Data Protection Act (2018) and Imperial College Data Protection Policy, and only accessed by the research team. Data will be stored for a minimum of 10 years after completion of the study or longer if needed for further reference.

Patient and public involvement

Public partners will be involved in the interpretation of results, in the co-development of a dissemination strategy, as well as in the writing of lay summaries and reports that summarise our research findings to increase consciousness and foster public engagement on this matter. Additionally, representative leads will be prompted to disseminate results through their universities and health facilities in newsletters, magazines, meetings and other appropriate means.

Ethical approval

Ethical approval was provided by the Science, Engineering and Technology Research Ethics Committee (SETREC) at Imperial College London (reference number: 6559176/2023).

Discussion

Considering the key role that digital health can have in boosting healthcare systems, we believe this framework can serve as a guide for primary care personnel to effectively introduce and expand digital health interventions with a potential to enhance overall quality of care. This study aims to further refine it and improve its practical use to assist the implementation of virtual consultations in primary care facilities, by applying a co-participatory approach.

Strengths and limitations

The use of the eDelphi method is an effective means to assemble a diverse group of experts to reach a consensus on a topic of interest and ensure that decisions are based on evidence and informed. It will also allow participants to anonymously address controversial topics, creating a safe environment with which to capture honest responses. The interactive process with multiple rounds enabling participants to iteratively refine their answers may also contribute to potentially reducing the biases. As mentioned previously, the eDelphi approach consisting of a fully online data collection along with our proposed sample collection methodology will contribute to gathering geographically dispersed responses. Piloting the questionnaire with national leads also aids in ensuring the relevance of the contents and its appropriateness given the diverse national contexts of the study participants.

It is possible that a few limitations may influence the results. The study will only focus on collecting quantitative measures of individuals’ views in the form of Likert scales, which could mean that granular details, and nuance of relevant aspects to the refinement of the framework may be missed. The Delphi technique has also been previously identified as a cause of attrition due to its time-consuming nature and the lack of ownership of ideas, resulting in participants potentially disengaging with the study prematurely. To address this, the research team will closely monitor any participant concerns raised and ensure the timing and duration of each round of questioning are cautiously deliberated. Additionally, GPs who use virtual care more often might be more likely to participate, thus introducing selection bias. Lastly, as our goal is to provide a list of essential components to be incorporated into a framework, the study will not address the implementation and effectiveness of the framework in real practice.

Implications for policy and practice

Virtual consultations have significant potential to improve efficiency in the delivery of healthcare in resource-limited health systems under ever increasing pressure of demand. It is imperative that the implementation of these new models of care is provisioned in an accessible, equitable, sustainable and safe way. Having captured the views of a large group of GPs across 20 countries, we have developed a prototype framework which we believe has the potential effectively to support such implementation across a wide range of national health settings.

Capturing primary care physicians’ consensus on components critical for inclusion will enable the refinement and focus of this prototype framework, resulting in a tool that is practical to health professionals and policymakers across primary care settings in informing, evaluating and monitoring the implementation of virtual consultations. Future work will also aim to incorporate the views of patients.

Ethics statements

Patient consent for publication

Not applicable.