Key stakeholders’ perspectives and experiences with defining, identifying and displaying gaps in health research: a qualitative study

Abstract

Introduction

Mapping the current body of evidence including what is missing helps provide a better understanding of what research is available, ongoing and needed and should be prioritised. Identifying research gaps can inform the design and conduct of health research by providing additional context information about the body of evidence in a given topic area. Despite the commonly used term ‘research gap’ in scientific literature, little is written on how to find a ‘research gap’ in the first place. Moreover, there is no clear methodological guidance to identify and display gaps.

Objective

This study aimed to explore how key stakeholders define research gaps and characterise methods/practices used to identify and display gaps in health research to further advance efforts in this area.

Design

This was an exploratory qualitative study using semistructured in-depth interviews. The study sample included the following stakeholder groups: researchers, funders, healthcare providers, patients/public and policy-makers. Interview transcripts were subjected to thematic analysis.

Results

Among the 20 interviews conducted (20 participants), a variety of research gap definitions were expressed (ie, five main themes, including gaps in information, knowledge/evidence gaps, uncertainties, quality and patient perspective). We identified three main themes for methods used to identify gaps (primary, secondary and both primary and secondary) and finally six main themes for the methods to display gaps (forest plots, diagrams/illustrations, evidence maps, mega maps, 3IE gap maps and info graphics).

Conclusion

This study provides insights into issues related to defining research gaps and methods used to identify and display gaps in health research from the perspectives of key stakeholders involved in the process. Findings will be used to inform methodological guidance on identifying research gaps.

Strengths and limitations of this study.

• This study used qualitative methodology that provided an in-depth understanding of key stakeholders’ perspectives and experiences in identifying, describing and displaying gaps in health research.

• The study benefited from having a variety of different stakeholders participating in semistructured interviews, which provided a wider scope of perspectives and experiences in identifying, describing and displaying gaps in health research.

• This study could have benefited from involving patient/public perspectives to inform the design of the study to improve the importance and relevance of the findings for this population.

• This study used qualitative methodology that provided an in-depth understanding of key stakeholders’ perspectives and experiences in identifying, describing and displaying gaps in health research.

• The study benefited from having a variety of different stakeholders participating in semistructured interviews, which provided a wider scope of perspectives and experiences in identifying, describing and displaying gaps in health research.

• This study could have benefited from involving patient/public perspectives to inform the design of the study to improve the importance and relevance of the findings for this population.

Background

Identifying research gaps can help inform the design and conduct of health research, practice and policies by providing a better understanding of the current body of evidence. Healthcare decisions for individual patients, public health policies and clinical guidelines should be informed by the best available research while taking into account research gaps.

The identification of research gaps has no well-defined process, although research gaps serve as the basis in developing a new research question and informing future research, healthcare delivery and health policies. In addition, research gaps in healthcare do not necessarily align directly with research needs. Hence, research gaps are critical in that knowledge gaps substantially inhibit the decision-making ability of stakeholders such as patients, healthcare providers and policy-makers, thus creating a need to fill the knowledge gap.¹

Moreover, identifying and characterising research gaps often highlight multiple competing gaps that are worthwhile to be explored.¹ Initiatives such as the James Lind Alliance (JLA), UK Database of Uncertainties about the Effects of Treatments, Cochrane Agenda and Priority Setting Methods Group, and Evidence-based Research Network are some examples of existing efforts to identify and prioritise research gaps in health.²

The term ‘research gap’ is not well defined, and its meaning can differ depending on the researcher and research context. A recent scoping review of methods used to identify, prioritise and display gaps in health research reported 12 different definitions related to gaps in health research, each describing research gaps differently.² This finding shows the ambiguity of the term ‘research gap’ and the different practices it may relate to.

As a basis for further exploring and understanding ‘research gaps’, we start from the definition given by the National Collaborating Centre for Methods and Tools in Canada based on the work by Robinson et al, whereby a research gap is defined as a topic or area for which missing or insufficient information limits the ability to reach a conclusion for a question.³ Given the different meanings and definitions of research gaps identified in the scoping review,² we considered it important to further explore key stakeholders’ perspectives to better understand the topic area. Clearly defining the type of research gap can help determine how to better identify, characterise, prioritise and address research gaps.

Different methods for identifying research gaps reported include scoping reviews and umbrella reviews for mapping and summarising evidence. These methods have an explicit aim of identifying research gaps in a broad area compared with systematic reviews, which focus on answering a specific research question.^4–8 Furthermore, investigating experiences with practices/methods used to identify research gaps can inform explicit methodological approaches in identifying and describing research gaps. This investigation can enhance practices of different stakeholder groups (ie, health professionals, health commissioners, researchers, patients/public and decision-makers) when addressing areas of uncertainty within the research problem and topic area.⁹

The specific objectives of the study were to (1) investigate key stakeholders’ knowledge and perceptions of and experiences with defining research gaps and (2) characterise methods/practices used to identify and display gaps in health research.

Methods and analysis

Qualitative study design

We conducted an exploratory qualitative study using semistructured interviews. This method was selected to provide an in-depth understanding of key stakeholders’ perspectives, experiences and practices in defining, identifying and displaying research gaps. This method also ensured that we explored key stakeholders’ understanding and practices related to identifying research gaps through a variety of lenses from different stakeholder groups. In turn, this process provided multiple facets of research gap definitions and methodological practices to identify and display gaps.¹⁰

Study sample and recruitment

We used purposive sampling to ensure that the perspectives of all identified stakeholder groups were represented. Purposive sampling is widely used in qualitative research to identify and select information-rich cases. The study sample included the following stakeholder groups: researchers, funders, healthcare providers, patients/public and policy-makers. The stakeholder groups were determined according to the findings of a previously conducted scoping review² and organised into three main categories focusing on the use of evidence to inform health policy, health practice and health research (table 1). A detailed description of participant categories was given in the previously published study protocol.¹¹ Study participants were recruited via contacts and organisations identified in the scoping review, relevant scientific publications, existing professional networks (eg, H2020 International Training Network ‘Methods in Research on Research’) and contacts from conference attendance (eg, Evidence Live and Cochrane Colloquium).

Table 1.

Participant characteristics (n=20)

Category	No. (% of total)
Researcher	9 (45)
Methodologist	5 (25)
Data visualisation	3 (15)
PhD student	1 (5)
Health practitioner	6 (30)
Healthcare provider	5 (25)
Public health professional	1 (5)
Oversight bodies	3 (15)
Health policy-maker	2 (10)
Funding body	1 (5)
Patients/public	2 (10)

The main inclusion criteria for the study were as follows:

1. Adults aged ≥18 years (researchers, funders, healthcare providers, patients/public and policy-makers).

2. Experience with the use of evidence to inform health decisions/choices, policy, practice or research.

3. Ability to converse in English.

4. Consent for research.

The sample size for qualitative studies usually depends on the point when data saturation is reached (ie, the point when new data do not add to a better understanding of the studied phenomenon but rather repeat what was previously expressed¹²). Considering that the point of saturation cannot be specified in advance, we planned to conduct between 14 and 28 interviews, owing to usual points of data saturation reported in qualitative studies.¹¹ The point of data saturation was determined based on the seven parameters identified by Hennink et al,^{13 14} including the study purpose, population, sampling strategy, data quality, type of codes, code book and saturation goal, and focus retrieved from the study. These parameters were discussed throughout the study primarily between the lead researcher (LN) and the senior researcher (DH).

Data collection and recording

Semistructured interviews were used for this study. The main reason for selecting semi-structured interviews was to allow for specific areas to be addressed while giving the interviewees the opportunity to reflect on their experiences and perspectives related to defining, identifying and presenting research gaps that are relevant to them and that may not have been explored or anticipated by the researcher(s).¹⁵

The guide was developed by focusing on exploring key stakeholders’ perspectives and experiences with the following key areas:

1. Participant background information.

2. Definitions of research gaps.

3. Knowledge and perceptions of and experiences with methods/practices used to identify and display gaps in health research to inform further health policy, practice and research.

These three categories were developed with information from the scoping review to guide the questions. The interview topic guide was piloted before data collection. It was also adapted according to key stakeholder groups to ensure that it was meaningful to their background and to gather more relevant information based on their experiences and knowledge.¹⁶

The semistructured interview guide contained two levels of questions: main themes and follow‐up questions. The main themes covered the general content of the research gaps aimed at encouraging participants to speak freely about their perceptions, experiences and practices. The follow-up questions were used as prompts and probes aiming to follow respondents’ answers and to investigate the issues raised more in depth. The interview guide covered the main topics of the study, providing a focused structure for the discussion during the interview.¹⁷

We conducted in-person, telephone and teleconference interviews. In-person interviews were conducted with participants residing or reachable in London, UK, and other participants were interviewed via telephone or teleconference (for the interview guide for both in-person and teleconference interviews, see online supplemental appendix).

All interviews were digitally recorded, transcribed verbatim and anonymised. The lead researcher (LN) transcribed two interviews to help inform the analytical process, and the other audio files were transcribed by a professional transcription agency licensed from the University of Liverpool.

Data analysis

We used analytical categories to describe and explain definitions, experiences and practices reported among the groups of participants. All data relevant to each category (defining research gaps, experiences with methods/practices used to identify and display gaps in health research) were identified and examined to ensure that each data item was checked accordingly.

Our approach was based on the thematic analysis outlined by Braun and Clarke.¹⁸ The steps included the following: (1) transcription and checking transcripts with recordings for accuracy, (2) open coding from interview responses performed by two researchers independently (LN and DH), (3) agreement of initial codes discussed among the researchers and an initial codebook developed, (4) developing the code structure used for analysing the remaining responses with openness that included new codes and refined existing ones and (5) themes and subthemes identified from the final code structure and their relationships presented.¹⁸

The initial coding framework for our analysis started from broad categories identified in the previous scoping review with which the interviews were structured. Within these broad categories (ie, defining research gaps, experiences with methods/practices used to identify and display gaps in health research), analytical categories were inductively derived from the data. In this sense, our approach includes both top-down and bottom-up development of analytical categories and themes.

QSR International’s NVivo V.12 qualitative data analysis software was used for data management and analysis.

Ensuring study quality

To further ensure rigour and trustworthiness, the study was guided by Lincoln and Guba ’s concepts for defining and investigating quality in qualitative research that can be considered parallel to quantitative research concepts of validity and reliability.^{13 19 20} The concepts include credibility, transferability, dependability, confirmability, audit trails and reflexivity. They are interrelated, and thinking through them from the onset and incorporating them into a study improve the study’s rigour.

The main researcher’s (LN) past experience as a Public Health Advisor at a National Institute of Public Health in Europe influenced the conceptualisation and conduct of this study, including the interviews. Her previous role focused on knowledge production for the health sector and providing knowledge about the health status of the population, influencing factors and how the status can be improved. She recognised the need for evidence to inform research planning, implementation and evaluation. Therefore, the design and conduct of this study were informed by her previous role and influenced the development of the interview guide, and interpretation and reporting of study findings. Throughout the different steps of the study, she consulted a senior researcher (DH) to discuss all matters related to the study design, conduct and reporting.

Patient and public involvement

Patients and the public were not involved in the design or analysis of this study. However, we involved them as study participants and will disseminate the study findings that pertain to them using a patient/public online platform, peopleinresearch.org.

Results

Among the 30 key stakeholders contacted, 20 agreed to participate in the study. Hence, we conducted 20 interviews with 20 participants involved in using evidence for informing health policy, practice or future research (table 1).

Definitions of gaps in health research

We first explored what participants reported as gaps in health research. Given the nature of our interest, all participants’ answers were grouped under a single theme ‘Definitions of Gaps in Health Research’. However, the focus of the definitions differed, and within this main theme, we identified five subthemes related to gaps in health research described by the participants (ie, gaps in information, knowledge/evidence-related gaps, quality of evidence, uncertainties and patient-related gaps; summarised in figure 1). The discrepancies and similarities of terms used are further illustrated in the online supplemental appendix. Terms ranged from lack of information/insufficient information, known unknowns/unanswered research questions and evidence uncertainty to treatment uncertainty, among others.

Figure 1.

Reported descriptions of gaps in health research.

We identified some similarities among the participants on how they defined research gaps, for example, researchers and oversight bodies mainly defined gaps in health research as a lack of information/insufficient information, known unknowns and no primary studies (more information can be found in online supplemental appendix). Patient/public participants defined research gaps in a much more literal manner, for example, ‘The gap is to get more patients involved in doing … clinical trials; have [someone] at the beginning introduce me, [educate me], [provide] awareness [because] I didn’t know what [a clinical trial] was. I [didn’t] know what they’re talking about’ (patient/public person, PPI01) and ‘Get me involved in co-production. That is the gap that is missing in clinical research’ (patient/public person, PPI01). The most common description research participants provided was the absence of scientific information to answer a research question, for example,

An area where there is missing or … insufficient information. And because of this … you cannot reach a conclusion for a question. So … it is a field, it is an area, a question an issue to which you don’t have an appropriate answer because there is missing … information or the research that still needs to be done in that particular area. (Funding body, F01)

One participant related research gaps to quality of evidence by use of Grades of Recommendation, Assessment, Development and Evaluation (GRADE), an approach for rating the quality of evidence and grading the strength of recommendations in healthcare.²¹ Another participant emphasised the importance of public and community involvement in gap identification to ensure that it takes into account their perspectives and contributions to the research ecosystem:

existing knowledge but not documented is of key importance in understanding the current body of knowledge on a particular topic area …. Evidence gaps need to be defined not only by [the] research community but also according to the key stakeholders including community members. Community knowledge is of key importance to inform the evidence base. Further evaluation on research findings to characterise the nature of research gaps can be carried out by evaluating community perspectives and local evidence to confirm scientific evidence. (Health research PhD student, R01)

We identified variability in participant responses on how to define gaps in health research; this variability was mainly observed in individual responses for the three main categories (research, practice, and policy and funding).

Methods to identify gaps in health research

Participants reported a range of applicable methods to identify gaps in health research (eg, surveys, reviews, syntheses, priority-setting partnerships and assessments) as shown in figure 2. The methods were also characterised by the different research methodologies used (ie, primary, secondary and both). Participants also expressed their difficulty in identifying research gaps, for example,

Figure 2.

Methods used to identify gaps in health research.

It is really difficult to identify research gaps. Lots of people you know will try and use the discussion section from research, [whereas] other authors have asked for further research, but in my experience that has not been a very useful method because sometimes authors will write that you know without really seeing or understanding that there has been something similar done in that field. (Health research methodologist, R02)

The variety of identified methods reflected the state of the field in the sense of the wide array of methods currently used, in line with the variety of specific goals of studies on research gaps (figure 1). The difficulty in identifying research gaps raised by participants, together with the plurality of definition of gaps and range of methodologies, may, however, also reflect a possible lack of consensus and guidance on what method would be best suited for a given objective.

Methods to display gaps in health research

Participants referred to a number of different methods used to display gaps in health research (ie, forest plots, diagrams/illustrations, evidence maps, mega maps, 3IE gap maps and info graphics) (figure 3). Participant perspectives varied; one of the interviewees pointed out,

Figure 3.

Methods used to display gaps in health research.

I think with the growth of technology, it is very important to use sophisticated methods to better communicate evidence for policy-making and decision-making. I think the availability of evidence is not enough on its own and finding different methods to communicate is important, not only the analysis and findings but also sharing it in different platforms online for a greater audience. (Health policy and guideline developer, P02)

Another participant highlighted that one of the key benefits of visually presenting research is being able to immediately see what information is available and missing.

The participants mainly expressed the importance of using data visualisation in research; there was a common understanding on the use of data visualisation as a whole, particularly with the growth of technology and the need to capitalise on it. The main challenges expressed were how to identify an appropriate visualisation to present the research and also how to effectively present data. We summarise these general experiences with data visualisation in health research in figure 3 and the online supplemental appendix.

Discussion

This study provides insight into issues related to defining, identifying and displaying research gaps in health from the perspectives of key stakeholders. The findings indicate several definitions of gaps in health research and methods used to identify and display research gaps.

Our study confirmed the ambiguity in defining research gaps and methodological approaches to identify^{3 22} and display research gaps.² The methods used to identify research gaps were closely linked to the definition of research gaps. For example, the JLA method of gap identification and setting priorities for research begins by clearly defining what the alliance refers to as evidence uncertainty, that is, there is no up-to-date, reliable systematic review of research evidence addressing the uncertainty or showing that uncertainty.²³ This step further informs the rest of the methodology used and is critical in identifying treatment uncertainties and determining future research priorities. This method combines both primary and secondary approaches and not only identifies research gaps but also verifies them across different relevant stakeholders, including researchers, patients, their carers and clinicians, to ensure the relevance and potential benefit to them.²³ This verification is important, given that some research gaps may be of key interest to researchers but have little relevance and importance to patients or the public, who should be the main beneficiaries of research to improve their health and well-being.

The overall method to identify research gaps involved primary, secondary or both approaches (figure 2). Most of the participants mentioned the use of secondary research methods; this is in accordance with the research that has been conducted on research gaps, which has also primarily focused on the use of secondary research and developed frameworks for identifying research gaps.2 3 8 24 25 The most commonly adopted framework involves identifying research gaps from systematic reviews using the Population, Intervention, Comparison and Outcome framework to characterise a research gap.³ The other framework involves identifying research gaps in qualitative literature reviews.²⁵ In addition, the GRADE approach for rating the quality of evidence and grading the strength of recommendations in healthcare²¹ presents the use of a prominent framework for evaluating the certainty of evidence that can inform the research gap and characterise it.²⁶ Moreover, scoping reviews are commonly used, and the definition includes aiming to identify research gaps by mapping the current body of evidence. These examples focus on the use of secondary research methods, but we lack studies that specifically explore the use of primary or both primary and secondary methods to identify research gaps, yet these methods equally exist and are being used. Additional exploration of applicable methods for identifying gaps can improve their usefulness and relevance in health research.

In summary, this study showed that research gaps need to be defined by researchers and confirmed by different research stakeholders such as patients and the public to ensure societal relevance and importance.¹ We also found that clearly defining research gaps can provide information on the most appropriate methodological approach to adopt in identifying and displaying gaps, for example, for exploring research gaps in a specific or broad area. For a specific area, a systematic review can be considered, and within a broad area, an umbrella review can be considered. The study also showed that the use of both primary and secondary methods (JLA method) to identify gaps is the most robust method for gap identification. The main reported advantage of this method is that it identifies gaps (treatment uncertainties) and involves different stakeholders, including patients and the public, to confirm and prioritise gaps. The main disadvantage is that it is labor-intensive (requires a team of different specialists) and expensive (administrative support, meeting rooms and catering, among others) compared with secondary methods (evidence synthesis) or primary methods (survey).

Participants mainly expressed the importance of data visualisation in communicating research; no specific methods or formats to present gaps were expressed. Thus, the use of data visualisation is desirable among different stakeholders, particularly researchers, when communicating research, although we found few examples of experiences with developing and using data visualisation. The participants mainly expressed their difficulty in finding the right tool to use to present research findings.

Finally, although scientific articles often refer to the existence of research gaps in studies, few respondents were able to define research gaps, unless contextualising them within a specific study or area, or methods of identification. Fully understanding research gaps in health research and adequately addressing them are difficult. In this study, we highlighted three key items on the topic: (1) clearly defining research gaps provides a context to understand better what the gaps are and what they are caused by; (2) a clear definition of research gaps can inform the methods used to identify research gaps, similar to how a clear research question can inform the research study methodology; and (3) on adopting the most appropriate methods to identify research gaps, finding the right visualisation to communicate them effectively is important. Last but not least, public involvement, when applicable, is needed to verify that gaps are important and relevant to the public.

To conclude, our study found that various methods can be used to identify gaps (ie, primary, secondary and both primary and secondary). Of all the methods used to identify gaps, secondary methods are the most common, specifically systematic reviews, which are considered the gold standard in that they address a highly focused question related to the existing evidence and thus present difficulties for explicitly identifying research gaps in a general area.^{3 8 27} Other secondary research methods reported were overviews of reviews, also known as umbrella reviews, scoping reviews and evidence mapping. Overviews of reviews focus on a much broader area, compiling evidence from multiple reviews into one accessible and usable document and highlighting other reviews within the specified topic area.^{28 29} Given the resource requirements of formal evidence reviews, topic prioritisation is needed to best allocate resources to those areas deemed the most relevant for the health system. Regardless of the topic, the prioritisation process is likely to be stakeholder-dependent. Priorities for evidence synthesis will vary depending on the mission of the healthcare system and the local needs of the healthcare stakeholders.¹ Hence, using both primary and secondary methods is the most robust because it involves the participation of patients, caregivers and healthcare and social care professionals in identifying research questions and then prioritising them using a combination of primary and secondary research.^30–49

To advance efforts in identifying research gaps, further work and different study designs are needed to take this work to the next step, to find consensus on definitions and different practices for methods in identifying research gaps. Subsequently, also assessing the best methods according to different stakeholders will be informative and important.

One of the main challenges of this study was that because the topic area is still very vague and unclear, the recruitment and interview process was challenging. Therefore, this study was primarily limited to what participants were familiar with and not necessarily representative of the full scope of the status of health researchers, health practitioners, oversight bodies and patients/public. A more generalisable understanding of this topic area would require a larger sample of participants and methodology, such as a Delphi survey, and/or a priority-setting partnership with representatives using evidence to inform policy, practice and research. This study would also have benefited from widening the scope of the stakeholder categories (use of evidence to inform health policy, health practice and health research).² This would have enriched our study findings and provided a wider view of stakeholder experiences outside our categories. Another limitation of this study is not including patients/public in designing the study. Including patient/public perspectives would have benefited the study design by being able to improve the importance and relevance of the findings for this population.

One of the main strengths of the study is improving the definition of research gaps and subsequently improving the accurate reporting of research gaps to elucidate the characteristics, which can help in evidence-based decisions. For example, a decision based on a research gap contributing to lack of primary research on a specific health problem can differ from the one based on a research gap related to lack of secondary research summarising the research. Hence, all these factors regarding research gaps need to be highlighted if they are known and made explicit when disseminating and communicating research. In addition, providing more information on what the gap represents may inform users of evidence of more specific information about the research gap and how it can be addressed more accurately.