Scanning the horizon: a systematic literature review of methodologies

Abstract

Objectives

Society is confronted with the rapid emergence of innovation in science and technology. To manage this, horizon scanning is being adopted globally to identify, assess and prioritise innovations and trends at an early stage of their development. This enables decision-makers to be better informed and to prepare for change. The aim of this paper is to systematically identify and evaluate horizon scanning methodologies employed in the healthcare and biomedical fields.

Methods

A systematic literature review was performed using PubMed and Embase and was supplemented with grey literature searches (2008–2018). The principal methodologies used in horizon scanning were extracted.

Results

Approximately 100 articles were summarised in a literature map. The search revealed many examples of horizon scanning across disciplines. Challenges, such as the need to refine prioritisation criteria, manage uncertainty inherent in the findings and improve the dissemination of identified issues, have been highlighted.

Conclusion

Horizon scanning, when performed appropriately, is a flexible and potentially reliable tool, with a wide variety of methods. Horizon scanning can inform and influence decision-making, through identifying opportunities and challenges, from an organisational to an international level. Further research to identify the most effective methodologies available would add depth to this landscape and enable the evolution of best practice to most efficiently anticipate novel developments and innovations.

Strengths and limitations of this study.

• This systematic review offers an up-to-date perspective on horizon scanning methodologies—incorporating practices from a number of different fields; however, as reviews into horizon scanning in healthcare have been undertaken previously, some duplication of findings was inevitable.

• A detailed evaluation, and a more practical guide to all the methodologies, could not be performed for practical reasons; specifically, the inconsistent reporting of the horizon-scanning details and their efficacy and the continually evolving approaches employed.

• The paper may be subject to omission bias as the literature reported systematic methodological aspects of horizon scanning; therefore, some ad hoc horizon-scanning methods may not be captured.

Introduction

Across organisations worldwide, the rapid emergence of high impact innovation is a major challenge faced by decision-makers.¹ To respond, the identification of future innovations and trends is being undertaken in a comprehensive, systematic and sustainable manner so that policymakers, and other stakeholders, can respond appropriately and enable innovations to reach the market with minimal developmental, legal, regulatory, process or procurement bottle-necks. To catalyse the achievement of this objective, horizon scanning is emerging as a valuable and viable strategy. This is particularly true in the health sector, where the European Medicines Agency (EMA) is seeking to promote the availability of innovative medicines using horizon scanning. This foresight will in turn inform the Agency’s Regulatory Science Strategy and the European medicines regulatory network strategy.^2–4

Horizon scanning has been in use for many years, initially by commercial organisations and later by public bodies; Japan was an early adopter of foresight methodologies in the 1970s.^{5 6} Since then, horizon scanning has been used across diverse sectors to aid financial, policy, process and research planning.^{7 8} There are many definitions of horizon scanning,^{9 10} but most can be captured by its generic characterisation as a systematic examination of information sources to detect early signs of important developments. The approach generally targets the early lifecycle of technologies—that is, in an early phase of adoption before their introduction onto the market—but may also scan for broader trends, challenges and opportunities. It provides an early warning of ‘signals’, rather than a comprehensive study of their impact.

Horizon scanning generally follows a process of signal detection, filtration, prioritisation, assessment and dissemination (figure 1). Its use is growing across sectors,¹¹ and this risks the duplication of efforts (both in design and execution) as similar stretches of the horizon are scanned by many.

Figure 1.

Common stages of horizon scanning from the EuroScan network.⁹ This figure is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).

There have been no recent reviews of horizon scanning methods used in the health sector,^{9 12} or those looking at broader biomedical sectors.6 7 12–14 As a result, the aim of this literature review is to systematically identify and evaluate horizon scanning methodologies employed in the healthcare and biomedical fields. The overall goal is to broaden and update knowledge on the methodologies used, and through mapping and evaluation, provide a useful guide for the establishment and optimisation of future horizon scanning initiatives. This includes the activities of the EMA’s recently established Regulatory Science Observatory, as well other international efforts to reduce duplication including the EU Innovation offices Network, the European Network for Health Technology Assessment and the International Coalition of Medicines Regulatory Authorities (ICMRA).^{4 15}

Methods

A systematic review of the literature was performed to capture and map the use of horizon scanning activity and the widespread use of modern IT/web capabilities, over the period from 2008 to 2018 (see online supplementary file 1).¹⁶ Three researchers were involved and a systematic protocol was followed to minimise inter-rater bias; this involved cross-validation between two researchers, with diverging opinions then arbitrated by the third. This systematic approach also permitted gaps and inconsistencies in the field to be identified. Data collection followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines¹⁷: the literature was first screened and filtered using inclusion and exclusion criteria; the accepted papers then underwent data extraction and analysis, and, finally, the scanning methodologies were mapped.

Search strategy

Medline and Embase bibliographic databases were searched to identify research papers on the use of horizon scanning and the methods used for this purpose. The final search took place on 4 July 2018. Grey literature and bibliographies of the most relevant research papers supplemented this search. The primary search terms used were derived from previous literature: ‘horizon scan*’ OR ‘strategic foresight*’ OR ‘systematic early dialogue*’ OR ‘early warning and alert system*’. All literature, of which the title or abstract contained any of the keywords above, was flagged.

Inclusion and exclusion criteria

Only articles published in English or Italian from 2008 to 2018 were included. In the first round of quality appraisal and screening (see online supplementary file 1), the publication abstracts or title had to indicate:

1. A methodology for horizon scanning or strategic foresight, or a discussion or experience provided of horizon scanning;

2. A breadth of horizon scanning of the relevant field or address a methodological aspect which may be generally applicable across different fields.

In the second round, the full texts of the selected articles were then reviewed according to additional inclusion criteria:

1. The horizon scan or foresight methodology was detailed.

2. The priority areas included relevant science and/or technology.

3. A collaborative/international approach was used

4. The horizon scanning undertaken scanned a period of between 2 and 15 years ahead.

Alternatively, the paper was required to demonstrate methodological aspect(s) of foresight or horizon scanning of potentially general applicability.

The foresight period of 2–15 years reflected the fact that signals suggesting impact in less than 2 years concern innovations that are already in late-stage development, while those anticipated to ‘mature’ in 15–20 years’ time are too distant and uncertain to be useful. A collaborative/international approach was sought because of the global nature of innovation and change.

The mapping was elaborated using the EuroScan International Information Network method, a scientific association of member organisations and individuals for the exchange of information on important emerging new drugs, devices, procedures, programmes and settings in healthcare (EuroScan), and novel methodological aspects from the searched literature.⁹

Patient and public involvement

No patients or public were directly involved in the design or conduct of the study.

Results

There were 413 papers identified through the initial database searches and grey literature, of which 252 were removed due to duplication or failing to meet the inclusion criteria. Sixty papers were excluded because of language issues or lack of access to full text. One hundred and one publications were included in this study after reading the full text as indicated (figure 2).

Figure 2.

Literature selection process flowchart.¹⁷

Consistency between the selection of papers by the two researchers, who undertook the screening, was fair (Cohen’s unweighted Kappa=0.28).¹⁸ A third researcher therefore screened the excluded literature according to the criteria to ensure that all relevant papers were captured (and resulting in one further publication being selected).

Most of the studies included in this review address horizon scanning as whole, following the process outlined in figure 1, and of these, many operate in a national context. A few papers also focus on the use of specific methodological aspects such as Delphi techniques. Given the databases interrogated, the most prevalent priority areas identified were environmental issues,⁹ gene therapy (8), oncology (9) and clinical practice. Public sector bodies in the UK, USA, Sweden and Australia published most often on the topic. Internationally, EuroScan was repeatedly referenced for its role in harmonising horizon scanning methodology, supporting its members and encouraging international collaboration.

Risk of bias was assessed in accordance with the BMJ guidelines and the Cochrane risk of bias tool.¹⁹ While bias was not typically found in these non-clinical qualitative studies, there were three exceptions:

• A form of publication bias was likely in which only horizon scanning undertaken in organisations with a strong background in publishing academic publications and transparency were discovered. It was not possible to correct for this.

• Omission bias may have occurred as the papers reported systematic methodological aspects of horizon scanning; however, some horizon scanning may occur in an ad hoc manner, for example, a signal discovered by word-of-mouth.

• The competing interests of the authors were not considered beyond the standards of the source journals. This was because it was not thought highly relevant to the reporting of methodologies.

Mapping

The process outlined in the EuroScan Method toolkit was used as a basis and novel methodological aspects found in the literature were added.⁹ The resulting map is segmented into: signal identification and detection, criteria and methods of filtration and prioritisation, assessment, dissemination and updating of information and overall evaluation of the process. Notable references are given and the full data set is provided in the online supplementary file 2.

Information sources and signal detection

Signals are detected from manifold sources in a horizon scanning exercise; box 1 summarises the most common.

Box 1. Information sources used for signal detection in horizon scanning.

Information sources (signal detection)

• Scientific/biomedical literature review

• Patents

• Input from industry and industry associations

• Other observatories

• Media

• International institutions and forums

• Individuals, committees and expert groups

• Surveys

• Government bodies

• Meetings and conferences

• Other organisations

• Grey literature

Review of the scientific and biomedical literature is perhaps the most common place to identify innovation. Searches can be structured, using systematic and validated strategies, for broad or targeted topic areas^20–22; a two-step approach, first to survey the field and, second, for a ‘deep dive’, can be used. Recently issued patents and published patent applications (eg, the European Patent Office)²³ represent an alternative source of early signals, particularly of innovations originating in industry (large, medium and small). Systematic and/or ad hoc scanning of direct or indirect information about new findings from industry or industry associations^{24 25} (such as investment of venture capital in SMEs) is also useful for monitoring research pipelines. Similarly, other sources, such as clinical trials databases^{26 27} and intelligence gathered from research infrastructures and consortia, and from university and research institute technology transfer offices, are valuable.

Additional observatories of potential value include repositories of innovation and trends generated from the horizon scanning outputs of international regulators and the committees and expert groups of governmental bodies, such as OECD and EuroScan.^28–30 The media—print, electronic and social—generate substantial topic specific and commercially relevant information, available via RSS feeds,¹¹ Twitter, Facebook and so on.³¹ Social media also provide signal suggestions from stakeholders. Workshops can also be useful to bring together diverse experts (chosen on the basis of their area of specialisation, breadth of knowledge, publications and commitment to the process) to discuss areas of novel science and technology and to collaboratively scan the horizon from different points of view.³² These ‘sand-pit’ exercises can be supplemented by participants from the scanning organisation itself, appropriate stakeholder groups, external consultants and policy-makers.^{33 34} For sustainable and continuous horizon scanning, it may prove valuable to create a steering committee, think tank or ‘idea radar’ including representatives of the aforementioned participants.^{12 35}

Delphi studies are widely used to pool knowledge and build consensus around emerging issues. There are two or more rounds involved. In the first, participants identify relevant issues, which are then pooled and ranked; the second round sees these issues discussed followed by their re-ranking. This process is iterated until a consensus is reached. Several Delphi variations have been described, from more conventional workshop formats to the use of online tools such as Nvivo (quantitative analysis of text) or Wordle.net (a word cloud tool).³⁶ The design of a Delphi study should take into account the sample size and confounding factors, such as the level of conformity in the group.^{7 37 38}

Surveys, conducted via the web or by mail,^{12 25} enable staff of an organisation, stakeholders and the public to be asked to identify new technologies or trends.³⁹ These may be most useful when horizon scanning in well-defined fields.⁷ Semistructured interviews covering a standard set of questions can also be used, with similar outcomes.²¹ However, public input was not found to be hugely productive in topic identification.⁴⁰ Likewise, an attempt to establish a Wikipedia community has been largely unsuccessful.⁴⁰

Finally, a number of additional sources have been identified including draft legislation and policy papers from governmental bodies, the proceedings of scientific conferences and symposia, professional and scientific societies, interest groups, think tanks and research funders (government, charities, venture capital, etc), the so-called grey literature where global shifts that influence society, the economy and the environment—megatrends—⁴¹ are sometimes foreseen. Google alert queries, Google Trends, Google News Timeline, Google Insight and blogs were also mentioned.^{40 42}

Filtration criteria and methods

Table 1 presents the criteria and methods commonly used to discard irrelevant signals. Several key criteria concerning a signal’s potential impact were used12 22 43–45 including what are the costs, and the cost–utility ratio, of resource consumption and what are the implications in terms of quality of life, burden of disease and patient safety^{45 46}?

Table 1.

Filtration criteria and methods used in horizon scanning to discard irrelevant signals

Filtration criteria (discarding irrelevant signals)	Filtration methods
Potential impact Size of affected population or global relevance Novelty Level of innovation Evidence Organisational impact Plausibility Levels of stakeholder and media interest Policy priority Stage of development Ethical and social issues Within time frame of 2–15 years	Classification criteria Automated text-mining tools Individual and group filtration Peer review Expert participation

The level of evidence is a further important criterion that has been ranked using a simple traffic light system,⁴⁷ where green denotes sufficient evidence to support the uptake of the signal, yellow indicates insufficient evidence to support uptake but the evidence may constitute useful information and red implies unsupportive or insufficient evidence.

In terms of filtration methods, these may be separated into those which tag signals according to the criteria in a binary yes/no fashion, or those which use distinct or graduated categories, for example, confirmed, likely, potential, unlikely and questionable.²⁷ Automated text-mining tools can be used with databases to enable the identification, tagging and categorisation of signals and facilitate clustering and filtering.⁷ Individual or group filtration may be performed by organisational staff, who can also undertake some initial peer review that is ultimately performed in depth by external experts. The latter can also contribute to, or determine, the weighting of signals, according to the criteria, using an evidence framework.^{22 32}

Prioritisation criteria and methods for assessing signals

The signals which have met the filtration criteria can then be prioritised. The prioritisation criteria which must be met, and the methods used to do so, are collected in table 2.

Table 2.

Prioritisation criteria and methods used in horizon scanning to assess signals

Prioritisation criteria (assessing signals)	Prioritisation methods
Potential impact on outcomes Size make-up of the affected population Expected variation of impact Likely time frame Evidence of effectiveness Relevance to strategic and political priorities Effect on other related policies Desirability Factual basis Requirement of availability of expertise Novelty	Qualitative approach Quantitative or semiquantitative approaches Rating and ranking Best-worst scaling Risk analysis Standardisation of signals Delphi approach Public consultation Engagement of experts Mixed methods

Logically, the criteria consider the potential impact on outcomes, a clear example being resource consumption, and the cost implications.^{12 43} The size and composition of the affected population are therefore important factors,12 46 48 49 as well as the expected variation that may be observed between different subsets.^{43 48} For the signal to be prioritised, the time frame must be realistic^{8 12 50} and there must be a clear, factual indication of true novelty and desirability. In addition to evidence of effectiveness, consideration must be given to the relative added value over current practice¹² and whether this sufficient to satisfy strategic and/or political priorities and policies (eg, reduction in inequality).⁴³

With respect to prioritisation methods, a simple qualitative approach uses short summaries of the signals as a basis to prioritise.⁵¹ Quantitative or semiquantitative approaches are obviously more rigorous and typical.

There were several novel Delphi approaches developed, for example, to acquire expert input online in a continuous feedback forum or market place. Here, participants prioritise, or purchase, a limited number of signals which then accrue a ‘price’ that can ultimately be used to prioritise those of greatest value.³⁵ Controls are possible to counteract the possibility of scoring fatigue.⁵²

Finally, it should be emphasised that the engagement of experts for prioritisation must ensure diverse participation from different sectors, geographical regions, disciplines and demographics.^{31 44 53} Public consultation is a valuable asset to provide input and involvement from citizens and users in prioritisation and can be achieved in person, via email or online.^{12 40}

Signal assessment and methods

The signals which have met the prioritisation criteria are then assessed. The factors assessed and the methods employed to do so are in table 3 (eg, in terms of resource implications and broad financial perspectives).

Table 3.

Signal assessment and methods used in horizon scanning

Signal assessment	Assessment methods
Impact, for example, resource (financial) implications Level of innovation Expected utilisation and diffusion Risk assessment Actions required and time to impact Legal and ethical issues Barriers to market Stakeholder perception	ExpertLens Driver analysis Scenario planning Expert, user and policymaker participation Peer review

A key factor to consider in the assessment of any signal, of course, is the resource implications. The expected utilisation and availability of the innovation across different geographical regions is also important,^{12 34 46} as is an assessment of risk. A number of practical issues must also be considered, including actions needed to translate the signal into use (such as further research, the development of new processes and whether complementary technology, eg, is essential to realise the value of the signal), the time and investment required to do so, the need for new or specialised training of personnel involved,⁸ the cooperation and acceptance of key stakeholders, any ethical issues, access to the necessary experts⁷ and the intellectual property associated with the signal and whether legislative or regulatory guideline changes are required. As always, impact on the market must be taken into consideration. Is the innovation likely to have a disruptive effect, will it encounter reimbursement barriers, what are the timelines and milestones,³⁵ etc? A consensus level of innovation can be sought (eg, important, moderate, modest).⁵⁴

Insofar as the methods used for signal assessment are concerned, a number of approaches are available.

Dissemination and evaluation of the results of horizon scanning

The key elements involved in disseminating and evaluating the results of horizon scanning are listed in table 4.

Table 4.

Dissemination and evaluation of the results of horizon scanning

Dissemination	Evaluation
Format Methods Audience Frequency Updating	Short, medium and long term Process and output audit Validation and sensitivity Focus groups Metrics Access to database

In terms of dissemination, the assessment of an individual signal can be summarised in a document with the following elements: authors, lay summary, assessment objectives and methods, background and current practice, signal description, impacts and other issues, estimated time to impact, comparator signals (innovations), expert opinion and declaration of any conflict of interests. It may also be beneficial to include policy recommendations which are linked to decision-making priorities, structures and individual and cross-cutting policies.20 32 39 46 49 Dissemination can be achieved, when a new report is available, via numerous pathways, including email, social media, notification of target groups,^{40 44} public events involving the participation of policymakers,^{7 12} publicly accessible repositories of data or outputs that are clearly indexed, easily searchable and categorised, for example, by level of evidence and other metrics.^{42 55}

Dissemination of any new report should be made systematically through diverse platforms⁴⁴ and shared directly with relevant organisations.⁴² The frequency of dissemination depends on circumstances.

A related activity that bridges dissemination and evaluation is the updating of horizon-scanning information. This comprises four essential elements: (1) continually checking and pruning sources based on their usefulness, relevance and evolution,¹² (2) monitoring and updating changes in signals by periodically refreshing the horizon scan,⁴⁹ (3) reassessment of signals when sufficient new data are available or a step change in technology has occurred⁵⁶ and (4) validating annually, for example, the horizon-scanning update by a team of expert researchers, practitioners and journalists.³¹

Evaluation of the results of horizon scanning can be performed in the short, medium and long term.¹² A short-term evaluation may involve the following actions: survey of an appropriate audience on the usefulness of horizon scanning in decision-making; use of metrics (eg, provided by Google Analytics), such as number of downloads, page views, average session duration, citations in publications and funding applications⁵⁷; reports of failures; consistency with other horizon-scanning methods. In the medium term, an evaluation would include the responsiveness of the horizon-scanning team to requests, the ability to keep the horizon-scanning content up-to-date, comparing findings with other horizon-scanning agencies/databases (eg, EuroScan) and measuring sensitivity and associated predictive value. Finally, a long-term evaluation assesses the usage of horizon-scanning information in arriving at decisions, the accuracy of projections, the timeliness with which new technologies were detected and the prioritisation criteria which best signalled the impact of the technology.⁵⁴

A process and output audit represents another approach to evaluation and ensures the completeness of the search record, records of external input and expert contact details, clear filing of information used and a clear statement of the innovation in the briefing.⁶ A focus group of users can be employed to review the information input and dissemination and to develop a user-friendly interface through which to access a database.^{40 58}

Discussion

There was a wide variety of sources and methods used to identify new and emerging issues. However, it was common to use scientific literature, individuals, committees and expert groups, the web and Delphi methodologies. That the scientific literature dominates is expected as innovation often begins in an academic environment and because widely accessible bibliographic databases have powerful search and filtering capabilities. The frequent use of the Delphi methodology may be explained by its ability to ‘crowd-source’ information and build a consensus among participants in a relatively short time frame. This consensus, particularly expert consensus, adds weight to the conclusions drawn from horizon scanning.

Overall, the majority of the methods used were manual or semiautomated, with relatively few automated aspects. This could be due to the limited availability of software and budget constraints. Complex filtration, prioritisation and assessment criteria are some of the barriers to full automation that may be resolved in the not-too-distant future by the rapidly evolving fields of machine learning and artificial intelligence.⁵⁹

Dissemination of horizon-scanning reports appear to have rarely fed directly and systematically into policymaking. This may simply be a reflection of the unpredictable and political nature of policymaking, as well as a mismatch with the longer time scale of horizon scanning. Equally, it is probably fair to say that the information gathered by horizon scanning lacks, at least to some extent, the conventional measures of credibility and authority required to influence policymaking.³² New tools and approaches (eg, via generation of complex scenarios and the clear weighting of evidence)^{32 60} are probably needed to enable horizon scanning to be considered more seriously by policymakers.

The distance of the horizon scanned was also found to be a tricky balance between the need to assess signals as early as possible to inform decision-makers and the limited information available at an early stage.⁵⁶ There were many different evaluation methods employed, covering different time-spans, reflecting inter alia cultural differences, resource limitations and a time lag between horizon scanning and its critical evaluation (eg, a high false-positive rate of horizon scanning implied the need for tighter filtration criteria).¹⁴

From a public policy standpoint, horizon scanning has both informative and creative functions, alerting policymakers to emerging issues and providing new, plausible policy options.⁷ This use of horizon scanning is well established for identification of emerging issues, both positive and negative, in global conservation and biological diversity.⁶¹ In some contrast, however, in the biomedical field, in which this review has concentrated, horizon scanning is biased towards identification of positive, innovative signals as those with low value inevitably have little impact.⁶² Finally, in addition to its institutional value, horizon scanning can significantly help related stakeholders, such as technology developers or civil society; it can reveal barriers to innovation and allow proactive engagement to reduce these barriers.⁴³

Limitations

As systematic reviews into horizon scanning in healthcare have been undertaken previously, some duplication of findings was inevitable6 7 13 14 63; however, this review offers an up-to-date and wider perspective and includes methodologies from beyond the health field, for example, conservation. Resource limitations have precluded evaluation of horizon scanning in other, related sectors and consideration of material in languages other than English and Italian. Lastly, a detailed evaluation and a more practical guide to all the methodologies could not be performed for practical reasons: the inconsistent reporting of the horizon-scanning details and their efficacy and the continually evolving approaches employed. This effort must be viewed as a ‘snapshot’, therefore, of a rapidly moving target.

Conclusions

To respond to accelerating innovation, horizon-scanning methodology is being adopted both nationally and internationally, particularly in the public sector. The range of methods used and the limited assessment of their performance render recommendation of a single approach premature and explains why combining two or more techniques makes sense for validation and for improving the accuracy of predictions.^{6 7}

Undoubtedly, automation and the development of artificially ‘intelligent’ horizon scanning, which self-assesses and improves its signal management, are short-term milestones that will significantly improve the process, enhancing the evidence base, disseminating the acquired outputs efficiently, and facilitating decision-making. Self-evidently, given the need for horizon scanning across diverse disciplines, involving large numbers of interested stakeholders with related information needs, the process can only benefit from international collaboration. To this end, initiatives are underway, including within the ICMRA in which the EMA is taking an active role.^{15 64–66} Of course, scanning the horizon for signals is not an endpoint, in and of itself, but rather a window through which current and future opportunities and policies can be linked.⁶⁷ It is essential, therefore, that further research be performed to develop, assess and ultimately implement the most efficacious methods of scanning and to ensure their acceptance and uptake by relevant stakeholders.