RAMPVIS: Answering the challenges of building visualisation capabilities for large-scale emergency responses

Abstract

The effort for combating the COVID-19 pandemic around the world has resulted in a huge amount of data, e.g., from testing, contact tracing, modelling, treatment, vaccine trials, and more. In addition to numerous challenges in epidemiology, healthcare, biosciences, and social sciences, there has been an urgent need to develop and provide visualisation and visual analytics (VIS) capacities to support emergency responses under difficult operational conditions. In this paper, we report the experience of a group of VIS volunteers who have been working in a large research and development consortium and providing VIS support to various observational, analytical, model-developmental, and disseminative tasks. In particular, we describe our approaches to the challenges that we have encountered in requirements analysis, data acquisition, visual design, software design, system development, team organisation, and resource planning. By reflecting on our experience, we propose a set of recommendations as the first step towards a methodology for developing and providing rapid VIS capacities to support emergency responses.

1. Introduction

Visualisation and visual analytics (abbreviated as VIS) has been used extensively in many mission-critical applications and healthcare applications. Since the emergence of COVID-19, data visualisation has been widely visible in traditional and online media for disseminating information related to COVID-19. Meanwhile what has not been obvious to the public is the fact that VIS techniques can and should be used to help healthcare scientists and experts in combating COVID-19. In particular, since epidemiologists and modelling scientists encounter a huge amount of collected data and simulation data on a daily basis (Shadbolt et al., 2021), it is not difficult to infer that we need to provide epidemiological modelling workflows with as much VIS support as possible. However, there have been some challenges for many epidemiologists and modelling scientists to receive adequate VIS support. These challenges include:

• a.Epidemiologists and epidemiological modelling scientists are not accustomed to receiving VIS support systematically. In some disciplines, modelling scientists have received VIS support systematically. For example, in visualisation journals and conferences, there are hundreds of research papers on VIS support for computational fluid dynamics, including many surveys (e.g., McLoughlin et al. (2010)). For the past a few years, the topic of providing machine learning workflows with VIS support has been growing rapidly (Sacha et al., 2019). In contrast, VIS papers on supporting epidemiological modelling are very rare. This suggests that scientists in epidemiology may not be used to the notions that they could visualise their data at their fingertips, could have visualisation experts to design visual representations specifically for their models, and could monitor and analyse the behaviours of their models and parameters dynamically.
• b.Visualisation is widely mistaken only for information or knowledge dissemination. In many modelling applications, VIS techniques are commonly used for scientific and public dissemination, but seriously underused in all other stages of a modelling workflow, which typically consists of a set of iterative processes, such as (a) data collection and observation; (b) hypothesis formulation and causality analysis; (c) model development, testing, validation, and comparison; and (d) model deployment, monitoring, and improvement. Ideally, modelling scientists and epidemiologists could have a quick glance of dynamic data anytime when there is a need (cf. stock brokers observing stock market data), access effective overviews of spatiotemporal patterns of the disease development and control (cf., meteorologists observing satellite images, contour maps, etc.), be provided with external memorisation of data to stimulate hypotheses and contemplate various decisions (cf. a general pacing around in a war room in front of many maps), and receive advice from an ensemble of analytical algorithms and visualisations about similarity, anomalies, clusters, correlation, causality, and association hidden in the data (cf. a CEO consulting specialists).
• c.There are not enough visualisation researchers around to support epidemiologists and modelling scientists. Mathematically, deriving an optimal model to forecast the contagion patterns of COVID-19 in different conditions (e.g., geographical, social, seasonal variation; different human intervention; etc.) is an intractable problem. It is an absolutely vital strategy to involve many modelling scientists and epidemiologists to develop different models because probabilistically, many developed models can produce sensible forecasts under some conditions. The more modelling scientists and epidemiologists can observe real-world data, examine model behaviours in different conditions, and compare the quality of different models, the more likely they can gain a better understanding and improve model performance in varying conditions. In 2020, there were some 100 university teams in the UK working on different epidemiological models. In comparison, there are only around a dozen of VIS teams in the UK. It is not feasible to pair a VIS team and a modelling team individually.
• d.There is a lack of a VIS infrastructure that can quickly be adapted to support epidemiologists and modelling scientists. The most cost-effective way to deliver VIS support to many modelling scientists and epidemiologists would be to have a technical infrastructure, which would host many applicable VIS techniques, and enable modelling scientists and epidemiologists to visualise any relevant data and analytical recommendations at their fingertips. Such an infrastructure could potentially support many other operations for combating COVID-19. Of course, it is understandable that we did not have such a VIS infrastructure ready in anticipating the COVID-19 pandemic. Ideally one could clone and re-purpose an existing VIS infrastructure, and adapt existing VIS tools for analysing and visualising epidemiological data. However, partly because of (b), there has not been adequate investment in the past for developing such a VIS infrastructure for any application. Consequently, during the emergency, there was not an existing infrastructure to clone, re-purpose, or adapt.

Nevertheless, “complaining does not work as a strategy” (Pausch, 2008), and “every challenge [we] face today makes [us] stronger tomorrow” (Bennett, 2020). RAMPVIS (RAMP VIS, 2020) is a group of 22 VIS volunteers, who answered a call in May 2020 to support the modelling scientists and epidemiologists in the Scottish COVID-19 Response Consortium (SCRC) (The Scottish COVID-19 Response Consortium, 2020a), which was part of the rapid responses organised by the Royal Society (UK) (The Royal Society, 2020). A few more volunteers joined the RAMPVIS group in the summer and autumn of 2020. As a volunteering operation in an emergency context, the VIS volunteers encountered many challenges. For example, the time urgency demanded rapid development of useable VIS tools, the travel restriction and the domain experts’ heavy workload hampered in-depth requirements analysis, and parallel developments of pandemic models and data infrastructure entailed delays in accessing data to be visualised. In addition, there was a shortage of skilled developers for designing and engineering a VIS system, and a fair amount of uncertainty in organising and scheduling volunteering resources.

The VIS volunteers made time urgency as their top priority, and were grouped into seven teams according to the available VIS expertise as well as different VIS needs in the SCRC modelling workflow. The grouping also enabled each team to progress independently in terms of requirements analysis, visual design, and system engineering. A few volunteers took part in different teams, facilitating knowledge sharing and collaboration among the teams. To our best knowledge, the experience of RAMPVIS volunteers is unprecedented in the VIS literature and epidemiology literature. In this paper, we describe our effort during 2020 to address the challenges in providing VIS support to epidemiologists and modelling scientists. By reporting and reflecting on our experience, we highlight the need for developing and providing VIS capacities to support a data-intensive emergency response.

2. Related work

In this section, we review the application of VIS to emergency response and healthcare and discuss the existing methodologies that may be used to develop VIS in such applications.

VIS for emergency responses.

Emergency response has been a regular theme in VIS since 2005 (Dusse et al., 2016). Kwan and Lee (2005) incorporate geospatial visualisation into a real-time 3D emergency response system to support quick response to terrorist attacks. Chittaro et al. (2006) introduce VU-Flow, a 3D visual environment that provides navigation guidance to users during emergency simulations. Based on the data collected from large community disaster events (e.g., 9/11 and Hurricane Katrina), Campbell et al. (2008) use visualisation-based interactive simulation for training emergency response teams. Natarajan and Ganz (2009) introduce distributed visual analytics for managing emergency response between geographically dispersed users. Waser et al. (2011) incorporate visual designs into simulation-based investigation of flood disasters to recommend appropriate response strategies. Maciejewski et al. (2011) introduce PanViz, a VIS toolkit providing decision support for simulated pandemic scenarios. Ribicic et al. (2012) develop a VIS interface to provide flood simulations to non-expert users. Konev et al. (2014) incorporate interactive visual designs into a simulation-based approach for flood protection planning. Gelernter et al. (2018) provide visualisations to guide first responders at a crisis scene. Visualising social media data is commonly used to support emergency responses for situational awareness (MacEachren et al., 2011), resource allocation (Jeitler et al., 2019), critical infrastructure management (Thom et al., 2016) and post-disaster analytics (Hornbeck and Alim, 2019, Medoc et al., 2019, Nguyen and Dang, 2019). Whitlock et al. (2020) integrate VIS tools with mobile and immersive technologies to support critical operations during emergency response.

VIS plays a critical role in mission-critical applications such as space missions. Abramyan et al. (2012) develop an immersive visualisation environment for controlling space robots remotely on the Earth. Edell and Wortman (2015) introduce advanced visualisations to assist the diagnosis of operational problems and failures for Van Allen Probes, a NASA space mission.

Furthermore, the IEEE Conference on Visual Analytics Science and Technology (VAST) host an international challenge workshop annually since 2006. Competition entries demonstrated novel VIS solutions for epidemic spread (Grinstein et al., 2010, Grinstein et al., 2011a), illicit activities (Grinstein et al., 2011b, Cook et al., 2018a, Cook et al., 2018b), security streaming data (Cook et al., 2016a, Cook et al., 2016b, Cook et al., 2016c) and natural disasters (Cook et al., 2019).

Unlike most of the prior work generally carried out in preparation for a future emergency, this work was conducted during the period of an emergency response to the COVID-19 pandemic.

VIS for healthcare.

The healthcare industry benefits from the adoption of Visualisation and Visual Analytics. Rind et al. (2013) review VIS tools developed for the exploration of electronic health records. Carroll et al. (2014) review 88 articles on VIS tools for infectious disease. Gotz and Borland (2016) discuss challenges unique to the healthcare industry and the critical role that VIS plays in the domain. McNabb and Laramee (2017) conduct an extensive survey of surveys including the adoption of VIS in the healthcare sector. Preim and Lawonn (2020) survey the use of VIS for supporting decision making in the public health sector.

The VIS techniques used in healthcare often incorporate analytical techniques. For example, Event Sequence Simplification is used to reduce visual complexity of sequential clinical events (Wongsuphasawat et al., 2011, Monroe et al., 2013, Gotz and Stavropoulos, 2014, Guo et al., 2019), and support a more efficient decision support process (Augusto, 2005). Natural Language Processing is used to extract textual data from raw clinical datasets (Zhang et al., 2011, Glueck et al., 2017, Sultanum et al., 2019). Machine Learning help automating the processing of clinical data and providing guidance to clinicians and researchers, including Active Learning (Bernard et al., 2015), Support Vector Machine (Trivedi et al., 2018), Topic Modelling (Glueck et al., 2018) and Recurrent Neural Networks (Kwon et al., 2019).

The main difference between this work and the prior work is that we had to develop multiple VIS capacities rapidly to address different VIS needs concurrently.

Methodologies for VIS applications.

Design study methodology (Sedlmair et al., 2012) builds on the nested model of design and validation (Munzner, 2009) to provide guidelines, pitfalls, and a process that help visualisation researchers design systems in applied contexts. This methodology uses the metaphor of “the Trenches”, which is where we found ourselves, and so, just as others have adapted its concepts and processes across varied settings (Lam et al., 2018, Syeda et al., 2020), this paper reports our approach that adopts, adapts and sometimes contradicts established guidance in the context of rapid emergency response.

Workshops (Kerzner et al., 2019, Knoll et al., 2020) can speed up requirements gathering. The five-design sheets methodology (Roberts et al., 2016) can structure the sketching process. Collaborative design methodologies (Losev et al., 2020) can address issues due to travel restrictions, and assist rapid visualisation design processes (Dixit et al., 2020). Some prior work in VIS advocates different software engineering methodologies (Brooks Jr, 1975, Hunt and Thomas, 1990). The agile approach (Kanban boards, SCRUM, etc.) is particularly suitable for developing VIS systems in applications with changing characteristics of data, users, and tasks. A recently-proposed method based on the cost–benefit analysis can potentially be used to discover shortcomings in a VIS workflow and explore potential solution systematically (Chen and Ebert, 2019). We have drawn inspiration from these methodologies in our work.

3. Formulating the RAMPVIS approach

The Scottish COVID-19 Response Consortium (SCRC) (The Scottish COVID-19 Response Consortium, 2020a) was established in April 2020 by researchers in three Scottish organisations in response to a call from the Royal Society for Rapid Assistance in Modelling the Pandemic (RAMP) (The Royal Society, 2020). The goal of the Consortium was to develop a more robust and clearer understanding of potential medium- and long-term strategies for controlling the COVID-19 epidemic in Scotland and in the UK. The Consortium currently has over 150 members from 36 organisations.

On 14 May, Dr. Richard Reeve, the SCRC modelling coordinator, first met a VIS scientist. They discussed the SCRC’s overall requirements for visualisation. As shown in the first sketch in Fig. 1(a), SCRC initially only required assistance for visualising the results of modelling, reflecting the widespread perception of visualisation as a tool only for information or knowledge dissemination. The VIS scientist described how different VIS techniques could enable domain experts to observe data quickly, analyse data with the aid of data mining algorithms, and improve their models through, e.g., visualisation of ensemble data, parameter space, and results from sensitivity or uncertainty analysis. Dr. Reeve embraced the idea of integrating VIS techniques throughout the modelling workflow, and revised the original sketch soon after (Fig. 1(b)). The VIS scientist (referred to as VIS coordinator hereinafter) indicated the need to enlist the help from many VIS experts.

Fig. 1.

Two sketches illustrate the major change of the role of VIS in the SCRC modelling workflow during the initial discussion. The symbols “V” in pink circles indicate the needs for VIS. The discussion helped establish the need for visualisation capacities not only for dissemination of modelling results but also, perhaps more importantly, for many processes during model development and improvement.

The following day (15 May), the VIS coordinator sent an email call for VIS volunteers to many VIS scientists, researchers, and developers in the UK, some of whom forwarded the call to others. By June 1, 22 VIS volunteers (including the VIS coordinator) answered the call. There are 19 faculty members, two industrial researchers, and one academic research officer. Among them, 14 indicated being able to prototype VIS software, and seven indicated willingness to engineer VIS systems. By June 2, the coordinator held meetings with all 21 volunteers individually or in small groups. The VIS volunteers had been using a diverse range of programming platforms. The most common denominator is D3.js (Bostock et al., 2011). Five VIS volunteers had experience of coding in D3.js (one became unavailable a few weeks later).

At that time, several teams in the SCRC were working on six different epidemiological models and one team on inference and model validation, while substantial effort was devoted to the development of a data infrastructure for storing modelling results as well as captured data related to COVID-19 spread in Scotland. For the VIS volunteers, there were many unknowns, such as what data might be available, how it may be retrieved, what were the requirements of individual domain experts and individual models, and so on.

While the agile methodology in software engineering (Larman, 2003) and the nested model (Munzner, 2009) in visualisation advocate the necessity of iterative requirements analysis and software evaluation, they do not prescribe a full requirements analysis and software evaluation within a single iteration. Otherwise, they would be similar to the waterfall methodology. In the VIS literature, many application papers indicate that it usually takes many months to acquire a meaningful set of requirements (e.g., six months in Abdul-Rahman et al., 2013, Fang et al., 2017 and 12 or more months in Lloyd and Dykes, 2011, Elshehaly et al., 2021). To support an emergency response, a lengthy delay due to requirements analysis would not be acceptable. Hence we had to complement user-centred requirements analysis with the existing knowledge documented in the VIS literature, and commenced the development as soon as we had understood a partial set of requirements.

As reviewed in Section 2, many papers in the literature reported VIS techniques and tools for supporting healthcare applications, model development, and mission-critical operations. If one can identify the data types, user tasks, and user knowledge in an application, one can relate them to the requirements in previously reported applications that featured similar characteristics of data, tasks, and users. During the two weeks when we were recruiting VIS volunteers, we gained our understanding of:

• •Datatypes — Based on several briefs from the SCRC modelling teams, we quickly learned that there would be a huge amount of time series data, and some geographical data (e.g., maps), network data (e.g., contact tracing), and multivariate data (e.g., demographic data). Building on our knowledge of VIS literature, we anticipated that some other types of data that might result from analytical algorithms, such as similarity matrices.
• •User tasks — Building on our knowledge of other VIS applications, we quickly established that there would be a need for viewing time series in different ways for observation and comparison, in order to evaluate a model run against captured data, other runs, and other models. We anticipated that some analytical tasks would benefit from data mining algorithms, and at a later stage, domain experts would become interested in ensemble data visualisation and parameter optimisation.
• •User knowledge — Building on our experience working with other domain experts, we anticipated that (i) domain experts were highly knowledgeable about their own models, but could not avoid frequent observation of captured data and model results; and (ii) they were familiar with the major geographical locations in Scotland, but would need to incorporate map-based visualisation for smaller regions in Scotland and other UK regions.

Meanwhile, we also consulted the abstracted theories and methodologies in the VIS literature. We observed that there was a need for all four levels of visualisation (Chen and Golan, 2016). Fig. 2 is an abstract representation of the workflow in Fig. 1. Although disseminative visualisation was the initial requirement, from Fig. 2, we can easily anticipate the needs for performing other types of visualisation tasks:

Fig. 2.

Visualisation tasks can be categorised into four levels according to the complexity of their search spaces. The workflows for epidemiological modelling, especially during an emergency, can benefit significantly from observational, analytical, and model-developmental visualisation in addition to disseminative visualisation. While visualisation can significantly reduce the time required for information acquisition, it can also enable experts to reason with the data in conjunction with their knowledge about, for instance, related facts and events in the reality, shortcomings in data collection and processing, and scientific understanding about the models concerned.

• •Observational visualisation. During an epidemiological emergency, many epidemiologists and modelling scientists need to observe a huge amount of data routinely, such as viewing daily updates in different regions, simulation results of different models or different versions of a model, and previously captured or simulated data (to aid memory recollection). For example, one of the commonly encountered data types is time series. Given a time series with $m$ data values, observing these values in a time series plot is much quicker than reading $m$ values. Hence, for $K$ domain experts to observe $N$ time series routinely, enabling them to perform observational tasks at their fingertips can collectively save a huge amount of time at the scale of $KN$. Our initial requirement analysis did not identify the existence of any efficient and systematic support for such observational tasks. It would mean that domain experts either had to spend a lot of valuable time and cognitive resource to fiddle with spreadsheets or other tools to create visualisation plots or had difficulties to see enough data or view data frequently enough.
• •Analytical visualisation. Many visualisation tasks are about identifying or determining the relations among the data objects being depicted without explicit visual confirmation of such relations. As the number of possible relations or grouping patterns is a combinatorial function of the number of data objects, such analytical tasks are usually more complex and time consuming when there are many data objects in visualisation imagery. For example, given $N$ time series (each with $m$ data points) representing situations in $N$ regions, visually determining the similarity among these time series or grouping them according to their similarity is more complex than observing each time series independently. In terms of the space complexity in computer science, the search space has a polynomial growth in relation to $mN$. Hence, when $mN$ is a big value, it is highly desirable to use algorithms to reduce the search space by for instance, ranking the similarity or recommending clusters. Such algorithmic suggestions can be conveyed to the domain experts through various types of visual illustration, such as connecting, ordering, highlighting, etc. Meanwhile, it is necessary for the domain experts to be aware that different algorithms may yield different algorithmic suggestions and algorithms may not have considered all variables necessary for correct suggestions. In an emergency scenario, it is typically uncertain about the best algorithm or missing variables. Hence, VIS tools should try to offer different algorithms and empower domain experts to judge the quality of algorithmic suggestions. In general, a reasonable algorithm is expected to reduce the search space significantly even if it may not result in an absolutely correct suggestion.
• •Model-developmental visualisation. Developing epidemiological models is a mission-critical operation in combating COVID-19 (Marion et al., 2021). There is a diverse range of models in terms of their epidemiological conceptualisation, mathematical specification and computational structure. Each model typically has many parameters, and different parameter combinations effectuate different model behaviours. Hence, the search space for an optimal model is usually intractable. For example, given $n$ parameters, each with $k$ possible values, there are $k^n$ combinations of parameter values. Some models have over 100 parameters, and many parameters are real numbers. Hence, it is unlikely that one can explore all $k^n$ combinations. It is crucial to empower model developers to use their knowledge to explore the search space intelligently, effectively, and efficiently. In some disciplines, such as computational fluid dynamics and machine learning, VIS researchers commonly design and develop model-specific VIS tools to enable model-developers to explore their search space effectively and efficiently (e.g., McLoughlin et al., 2010, Sacha et al., 2019). Such effort requires close collaboration between VIS researchers and model-developers as it takes time for VIS researchers to gain adequate understanding about a model and a model-developmental workflow, and for model-developers to appreciate how VIS techniques may help without the intelligence and knowledge similar to model developers.

We therefore concluded that although disseminative visualisation was the initial requirement, the above considerations about data, tasks and users confirmed that the priority should be given to observational, analytical, and model-developmental visualisation (Chen and Golan, 2016). We also anticipated that the increasing complexity from observational to analytical and to model-developmental visualisation compels increasing depth of collaboration, time needed for requirement analysis, creativity in visual designs, and effort for iterative design evaluation and optimisation. Meanwhile, we could not and should not delay the infrastructural development, which demanded rather-scarce skills and experience of designing and engineering deployable VIS systems.

The VIS volunteers were thus organised into several teams, including a generic supporting team (focusing on observational visualisation), an analytical support team, four modelling support teams, and a disseminative visualisation team. We placed all D3.js developers into the generic support team, and distributed other VIS volunteers according to their expertise and time capacity. In the following four sections, we report the activities of these teams, including further requirement analysis conducted by each team.

4. Generic support and RAMP VIS infrastructure

Infrastructure setup.

The generic support team consists of mainly VIS volunteers who can program in D3.js and have developed and deployed VIS systems. Our requirement analysis indicated that domain experts were not able to observe data regularly. Using the recently-proposed method for optimising VIS workflows (Chen and Ebert, 2019), we quickly identified that this was caused by the cost of reading and visualising data using spreadsheets, and a solution is to develop a VIS infrastructure closely coupled with the SCRC data infrastructure that was being developed. The goal of the team was to enable observational visualisation for every piece of data held by the data infrastructure.

The development of the SCRC data infrastructure started several weeks before VIS volunteers joined SCRC. A group of professional research software engineers (also volunteers) have carried out the design and implementation since. The goal is to capture the provenance of models and their results, enabling all contributing elements traceable from results to models and the conclusions drawn. Transparency is thus a key principle. All models and core software components are open-source (The Scottish COVID-19 Response Consortium, 2020b). Hence the VIS infrastructure has also been developed in the open.

The UK Science and Technology Facilities Council (STFC) provides the data and VIS infrastructure featuring virtual machines on the STFC cloud service, including a chat platform for collaboration and a data registry for web applications. The readiness of STFC for emergency responses enabled the hardware for the VIS infrastructure to be available within 24 hours after our request.

As emergency responses, the SCRC data infrastructure, VIS infrastructure, and six epidemiological models were developed in parallel. While the generic support team was waiting for the measured data, we had access to some Scotland data in three spreadsheets, which contained over 300 time series and a few data tables. We anticipated that there would be at least thousands of time series when data from other regions and model runs became available. Such scale would be a challenge to the domain experts as well as the VIS developers. Users would need to assess the relevant plots quickly, while developers would need to adapt each visualisation program (referred to as a VIS function) to other applicable data with minimal development effort.

User interface.

To address the need of the domain expert users, the RAMPVIS server provides the following facilities (Fig. 3):

Fig. 3.

The RAMPVIS server (vis.scrc.uk) provides users with a user interface (with search facility, personal portal, etc.), a collection of dashboards (with clickable visual and data objects), and various plots for visualising the data hosted by the SCRC data infrastructure.

• •A user interface (UI) with a side bar for accessing visualisations organised in categories;
• •A multi-keyword search facility;
• •A personal portal for storing frequently-used visualisations;
• •Because each visualisation is given a unique URL, users can also tag frequently-used visualisations on a web browser;
• •A collection of dashboards, each providing links to other dashboards and visualisation plots.

On the RAMPVIS server (Nguyen et al., 2020), there are broadly two types of visualisations, dashboards and plots. Each dashboard is designed to show key indicators and/or summary plots that some domain experts need to view frequently. For example, it may show the daily data of a region or a summary of a model run (Fig. 3(b)). Some information may serve as overviews while others serve as detailed views. The data objects, visual widgets, or summary plots on a dashboard are all clickable, providing a gateway to fuller or more detailed visualisations or other dashboards when required. In addition to a set of pre-defined dashboards, the generic support team provides a service to SCRC domain experts for constructing new dashboard whenever needed.

The team have developed a variety of visualisation plots. As shown in Fig. 3(c), some plots feature interactive capabilities, some compare multiple data streams, some convey the analytical results produced by data mining algorithms, and some display modelling results with estimated uncertainty.

Architecture and ontology.

To address the aforementioned second need for propagating each VIS function (either a dashboard or a plot) to other applicable data, the team has designed and developed an ontology- and agent-based architecture for the RAMPVIS server (Khan et al., 2022b, Khan and Nguyen, 2020). When a VIS developer fetches a task of writing a VIS function in D3.js, an infrastructure manager creates a program template with one or more example data streams. The binding of the VIS function and the given data streams results in a unique web page. Once the development completes, the VIS function can be reused by replacing the sample data stream with other applicable data streams. Through a simple UI, the infrastructure manager queries a Search Service to find all suitable candidate data streams, finalises a collection of data streams for propagation, and calls a Propagation Agent to create new bindings (and web pages) for these data streams automatically (Khan et al., 2022a).

As illustrated in Fig. 4, in the VIS infrastructure, an ontology provides the vital support to the search facilities that enable users to find desired dashboards and plots and the infrastructure manager to find applicable data streams for propagation. The ontology is a graph data structure that stores the relationships among all VIS functions, all data streams, and all data-VIS bindings (and the resulting web pages). Because we modelled the ontology using a document data model (Chodorow, 2013), we implemented the ontology using three MongoDB database collections, which are:

Fig. 4.

The architecture of the RAMPVIS server, featuring its ontology, agents, and services, and their relationships with the UI, web pages (dashboards and plots), VIS functions (programs and test data), fetched and derived data streams, and SCRC data infrastructure.

• •OntoVis for defining and keeping the records of all VIS functions and their metadata;
• •OntoData for storing the records of all data streams and their metadata;
• •OntoPage for maintaining the binding points between VIS functions and data streams, their metadata and the URLs of the corresponding web pages.

As shown in Fig. 4, in addition to the aforementioned Search Service and Propagation Agent, there are:

• •A Downloader Agent for fetching data from dynamically- changing data automatically from the SCRC data infrastructure;
• •A set of Analytical Agents, each applies an analytical function or data mining algorithm to a predefined collection of data streams and generates derived data to be rendered, e.g., a similarity matrix of a collection of time series (see also Section 5);
• •An Indexing Agent that periodically scans the database operation logs and updates various textual descriptions in the ontology, which may be used in search or displayed by VIS functions;
• •A Thumbnail Agent for creating and updating the thumbnails of dashboards and plots that may change due to the dynamic change of the underlying data.
• •A Bookmark Service for managing bookmarks in users’ portals;
• •An Authentication Service for approving a user’s login action;
• •An Authorisation Service for distinguishing ordinary or administrative users.

The RAMPVIS server was implemented with backend microservices using two state-of-the-art REST-API frameworks: JavaScript-based NodeJS and Python-based Flask. The I/O-intensive operations (e.g., database or file-system access) are performed asynchronously and are implemented with a NodeJS microservice. The CPU intensive operations (e.g., running Analytics Agents) are implemented with a Flask microservice. The Flask framework also provides Analytics Agents with some off-the-shelf analytical libraries, e.g., NumPy, SciPy, scikit-learn. The SCRC data infrastructure provides the Downloader Agent with Python APIs to fetch H5 data.

5. Analytical support

As exemplified by numerous visual analytics papers, VIS applications with a large data repository are expected to employ both data analysis algorithms and visualisation techniques. For example, analytical tasks have been critical for assessing model performance and uncertainty (e.g., comparing predictions to observations and comparing multiple model runs) (Konyha et al., 2006), and for exploring epidemiological data (e.g., identifying areas, time periods, or demographic groups exhibiting similar trends in outbreak progression) (Bhaskaran et al., 2013). With the huge number of time series to be hosted by SCRC data infrastructure, we anticipated that simply relying on observational visualisation might not be efficient or effective. We thus grouped several VIS volunteers with strong data mining experience into the analytical support team.

After several attempts to acquire detailed requirements for analytical visualisation capacity, we learned a high-level requirement from some modelling scientists, i.e., “cross-model validation” would be needed at some stage. With their expertise and initiatives, the team members anticipated that comparing time series would be an unavoidable analytical need because of the sheer volume of dynamically expanding time-series data. The initial data available to the team contained hundreds of time series for different regions of Scotland, different indicators (e.g., test, case, hospitalised, and fatality), different genders and age groups, and so on. While waiting for modelling data to be prepared for comparative analysis, the team started to develop visual analytics techniques for summarising, simplifying, and comparing time series and for searching and visualising patterns and structures in the data. Building on the VIS literature on time-series (Aigner et al., 2011), we anticipated the following analytical tasks: (i) discovering recurring trends, (ii) looking for outliers, (iii) identifying clusters, and (iv) measuring similarities according to one or more characteristics (e.g., scale, gradient, time lag, etc.). The team’s anticipation was confirmed when more data and specific requirements arrived several months later.

Guided primarily by these tasks, we experimented with a number of analytical visualisations in conjunction with an analytical toolbox that consists of different metrics and algorithms for quantifying the characteristics of individual time series, computing pairwise similarities, and transforming the time series to feature spaces that enable their similarities and clustering to be visualised.

Analytical toolbox.

We started developing the analytical toolbox by creating a library of low-level analytical filters that treat time series as 1-D signals. Since time series may contain noise, various types of filters (e.g., flat, Hanning, Hamming, Bartlett, and Blackman windows) can be used to smooth time series when required. We then added a comprehensive library of analytical metrics for measuring the distance, difference, similarity, or error between two time series (e.g. mean square error and many of its variations, Pearson correlation coefficient, structural similarity index measure, mutual entropy, Spearman correlation coefficient, Kendall’s tau, peak signal-to-noise ratio, $F$-test, and so on). We further included algorithms such as dynamic time warping (DTW) and dimensionality reduction methods such as principal component analysis (PCA) and multi-dimensional scaling (MDS).

Analytical visualisations.

We also experimented with a number of visual representations, focusing on comparing $N$ time series with $T$ data points. If one needs to determine any group of $k$ segments of time series that may be similar, the number of possible groups to be observed would be at the level of $O\left(N^k{T}^k\right)$, hence using analytical algorithms to narrow down the search space is highly beneficial (Chen and Golan, 2016). However, relying on metrics alone is not sufficient since time series could be similar/dissimilar due to factors not encoded in the data (e.g., differences in terms of demography and intervention). In that respect visualisation provides ways for domain experts to incorporate their knowledge when analysing and comparing different time-series. Fig. 5 shows one set of our experiments for analysing the time series associated with the 14 regional health boards in Scotland.

Fig. 5.

The analytical support team develops and experiments with different analytical algorithms and visual representations (top left) and selects some designs to be deployed on the VIS infrastructure by the generic support team (top right). The lower part of the figure shows examples of analytical visualisations for time series analysis, including time-lag plot, force-directed graph, scatter plot, and heatmap matrix.

• •A time-lag visualisation compares two time series by registering them using a cross-correlation that computes the displacement of one time series relative to the other. A viewer can foresee what the future may look like in one board if it follows the same trends as another board, but with a delay.
• •A heatmap matrix shows pairwise similarity scores among all $N$ time series reported by different health boards. Row or column headers can be accompanied by time-series profiles for detailed observations. This is especially important when the similarity/difference measures are difficult for viewers to interpret.
• •A force-directed graph produces a layout where the $N$ nodes represent $N$ time series, and the length of each edge encodes the similarity/difference (short/long) between a pair of nodes. This visual representation is particularly useful for users to discover clusters of similar time series and outliers.
• •A chord diagram, which is shown in Fig. 3, places $N$ time series as segments/nodes along a circle, and uses the thickness or colour of each chord to encode the similarity/difference measures.
• •A scatter plot compares $N$ time series in their feature space. Typically, two most important features (e.g., principal components computed using PCA) are selected as the axes of a 2-D space, and each time series is positioned as glyphs in the space according its feature coordinates.

When the analytical support team examined these experiment results with the domain experts, one domain expert commented “These give us a lot to think about. It is not that we do not require these. We were just overwhelmed by what visualisation can do”.

Infrastructural support.

As shown on the right of Fig. 5, following the experiments, the analytical support team selects analytical algorithms and visual representations to be integrated into the VIS infrastructure maintained by the generic support team. Each analytical algorithm becomes an analytical agent, while each visual representation becomes a plot. Because many data streams in the infrastructure are updated dynamically, each analytical agent is scheduled to recompute various measures and generate derived data automatically. In this way, when an analytical plot is called, it always displays the analytical results based on the latest data.

6. Modelling support

Mathematically, finding an optimal model to forecast the contagion patterns of COVID-19 in different conditions (e.g., geographical, social, seasonal variations, different human interventions, etc.) is an intractable problem. Nevertheless the effort to develop better models and improve existing ones is both necessary and desirable (Marion et al., 2021). When VIS volunteers were first gathered together, we anticipated that supporting the model development in SCRC would be the most challenging undertaking, because VIS would have to support the search for better models in an NP space (i.e., the EXPSPACE class) (Chen and Golan, 2016). We therefore organised some 10 VIS volunteers into four modelling support teams (referred to as Teams ${\mathbf{M}}_A$, ${\mathbf{M}}_B$, ${\mathbf{M}}_C$, and ${\mathbf{M}}_D$ below), providing opportunities for each team to focus on supporting one or two SCRC modelling teams through close collaboration.

6.1. Team M${}_A$: Supporting 1-km${}^2$ spatial simulation

Model Simulation.jl (Harris et al., 2020) simulates the spread of COVID-19 based on spatial proximity and its effect on the local population according to its demographic structure, over time. The inputs are population counts in 10-year age-bands in 1 × 1 km${}^2$ to 10 × 10 km${}^2$ grid cells across Scotland. Given an initial set of “seed” locations on day 0, the model outputs the number of people in different COVID-affected categories for each subsequent day by age group in the same grid cell. Fig. 6 shows a set of simulation results.

Fig. 6.

Model output showing proportion of population by COVID-affected category for the whole of Scotland (left) and more detail in SE Scotland (right). Top: glyphs show change in proportion of COVID-affected population (top-bottom) from day 0 (left) to day 60 (right). Bottom: proportion of COVID-affected population (left to right) for different age groups (young-old; top-bottom).

The epidemiological modellers and VIS researchers began their collaboration as the first version of the model was being created. There was an urgent need to visually inspect the relative proportions of COVID-affected individuals in different categories over time and space. When discussing the high-resolution model outputs and strategies one might use to summarise them to validate model outputs and (later) to compare different modelling scenarios, the need to freely explore these prior to establishing fixed tabular summaries became apparent. As this need was so urgent, we quickly established two VIS requirements: to enable (i) studying the relative proportions of COVID-affected individuals in different categories over space and time, and (ii) exploring the results at different scales from Scotland-wide to 1-km${}^2$ neighbourhoods.

Our solution to these challenges was to use interactive “tilemaps” (Slingsby, 2018) (also known as “glyphmaps” (Wickham et al., 2012) and “embedded plots” (Grolemund and Wickham, 2015)) with (a) glyphs representing multiple aspects of the modelled output together and (b) on-the-fly interactive gridding of the output at a suitable resolution in response to zoom/pan user interaction. The technical challenge was to make interaction, with this very high-resolution data quick and responsive enough to facilitate, rather than impede, exploration.

On the left of Fig. 6, data at a coarse spatial resolution is superimposed on a map of Scotland. The glyphs in the top, left image show the aggregated temporal trend as the population moves through the COVID-affected categories, with significantly lower proportions of affected populations in the less populated areas of NW Scotland. The image below this is a snapshot on day 32, showing the different rates at which the virus is affecting the population. The right column is a more detailed view of SE Scotland, as a result of zooming/panning. The snapshot on day 32 (bottom right) shows that the virus is affecting different age groups differently. This is largely due to differences in resident population structure. Domain experts with knowledge of population density can see that low-density areas seem to act as “firebreaks”. Although this is the inner working of the spatial spreading algorithm, its appearance in the visualisation started a debate among the domain experts, influencing the next stage of model development — to investigate the importance of population density on speed of disease spread. The interactive tilemaps with glyphs have provided a basis for the ongoing work for evaluating the relative importance of different factors in the modelling and comparison of different lockdown scenarios.

6.2. Team M${}_B$: Supporting simple network simulation

The Simple Network Simulation (SNS) (Enright et al., 2020) is a disease state progression model that computes numbers of people in a series of states for specified age bands, for different geographic units, over time. The inputs are population counts in each of the age bands, spatial units, social contact data capturing interactions between age groups and a spatial interaction matrix that determines the extent of the likely flows of people, and thus transmission, between each pair of spatial units. The model calculates the number of people in different states of the disease for each age group for each spatial unit on a daily basis. These outputs vary according to a number of model parameter settings.

When we connected with the SNS team, they were considering generating a lockdown spatial interaction matrix by grouping daily travel-to-work flows for those in different employment sectors according to the likely effects on their jobs of the “work from home” edict of April 2020. Their graphical approach used origin–destination matrices of data recorded in the 2011 UK Census of Population. However, flow quantities, group differences, and any geographic variations or effects of scale were difficult to see (Fig. 7, left). The effects of these factors, or those relating to other model inputs, on the model outputs were unknown.

Fig. 7.

Left to right: Original origin destination (OD) matrices — travel to work between local authorities in Scotland (top) and those in employment sectors likely to be operating under lockdown (bottom). Asymmetric flow map — all travel to work between local authorities shown in the spatial context, with line widths representing number of daily flows. Flow grid map — all travel to work, shows major flows and has the space to show internal flows. OD grid map — shows differences between travel to work flows in the ‘All’ and ‘Likely Operating’ employment sectors. Mini maps show incoming (and internal) flows for each local authority. Blues are negative (lower proportions of workers), reds are positive (higher proportions of commuting). Darker colours show larger differences. Differences are predominantly local at this scale, but strong spatial patterns are revealed.

We collectively identified the opportunity to apply established VIS principles to ongoing efforts to visually explore two specific domain questions: (a) which types of workers should we include in the input network (and what difference does this make to outputs)? (b) what do model outputs look like (and how do they vary over time and by age-group)?

These gave plenty of scope for using judicious visual design and interactive methods for filtering, highlighting and selection to develop elegant answers to challenging questions. Further questions that we hoped to address at a higher level were: (c) how do the answers vary with scale and geography? (d) is visualisation effective in answering these questions (in this context)?

Our solutions were developed as data sketches (Lloyd and Dykes, 2011) embedded in structured documents for discourse around data and design (Wood et al., 2018) and through regular video conferences and online discussion in a series of tight redesign loops (McCurdy et al., 2016). The discourse resulted in some preliminary answers to these questions. For instance, for questions (a, c), the nature of the input network does have an effect on outputs, with distinct spatial variation and greater effect at smaller scales. We redesigned graphics, allowing us to identify the areas most affected by the employment sector selection (our naive modelling of lockdown), areas with high out-of-area epidemiological importance, and the sources of those who visit them (Fig. 7). These views were informative but do not dictate the scale or nature of the network we should use. We began by looking at NHS boards, but increased resolution to local authorities and then the higher resolution ‘Middle Layer Super Output Areas’ (MSOA) for selected areas as our answers, our knowledge of the kinds and scales of effect and their likely geographies developed. The application of VIS principles was informative and effective, and included guidelines and designs known to, used by, and sometimes developed by, the team, for example:

• •GridMaps (Eppstein et al., 2015, Meulemans et al., 2017) — to add spatial information to origin–destination matrices and address the occlusion that occurs when flows are shown between variously sized geographic units using standard projections.
• •OD Maps (Wood et al., 2010) — to show the geographic variation of the effects of the different input networks.
• •Animated transitions (Slingsby et al., 2009) between alternative layouts — to relate spatial and semi-spatial geographic projections.
• •Interactive selection and filtering — to vary view parameters and support exploration. Asymmetric link curves (Fekete et al., 2003, Wood et al., 2011) to clearly show bi-directionality and asymmetry in the spatial interactions.

For (b) and (c), initial model outputs were not particularly varied by space, time or attribute, but we do have plausible candidate methods that will allow such variation to be detected and assessed when the SNS models are finalised. Importantly, the visualisation work resulted in better knowledge of the input network and its lockdown characteristics, and an emphasis on smaller scale “data zones”. The collaboration therefore enables us to be more confident about injecting visualisation into the modelling process (d), to shine light on the models as they are developed, tested, and parameterised, in ways that had not been considered by the modellers (Fig. 1).

6.3. Team M${}_C$: Supporting contact tracing modelling

By the time VIS volunteers joined the SCRC, the contact tracing model (Mohr et al., 2020) was already in development. The model simulates the spread of COVID-19 through a dynamic network that encodes the potential contacts among millions of individuals. These simulations result in some very large temporal networks (Holme and Saramäki, 2012).

We immediately established bi-weekly meetings with the modelling scientists. It quickly became clear that the domain experts did not have access to bespoke network visualisation tools, and were primarily relying on standard plots to view simulation results as disease progression curves and some summary measures such as the reproduction number $R$. They were also comparing different intervention policies through such plots. We noticed that the temporal networks, which were used to derive the summary information, were never visualised. Hence, the first urgent requirement for VIS was to enable domain experts to observe such networks in order to gain an intuitive understanding about the temporal and topological behaviours of the model. The visualisation would also assist domain experts in communicating modelling results and informing policy making.

We addressed the requirement by using existing network visualisation tools to minimise the delay due to software development. This allowed us to build familiarity with the model and the data, while providing example visualisations to stimulate our discussions with the domain experts. We then progressed to more advanced VIS techniques, e.g., employing scalable graph-drawing techniques (Simonetto et al., 2018, Simonetto et al., 2020), geographic-inspired metaphors (Gansner et al., 2010), and graph-theoretic analysis for derived metrics to complement network visualisation (Fig. 8).

Fig. 8.

Some of the largest infection chains visualised — with asymptomatic transmissions highlighted (left), and with index nodes and locations of infections expressed along with a visual indication of the sizes of the chains (right). The visual analysis was also enriched with extracted graph-theoretic metrics (left-bottom).

With bi-weekly collaboration meetings providing continuous feedback and ideas, we improved our prototypes iteratively through a web-based “project diary” and an open software repository (Turkay et al., 2020). As the collaboration matured over the period, we observed a trend that both domain experts and VIS volunteers actively contributed to the discussions on model building and visual design together. It became difficult to label whether a discussion was about visualisation or the model itself, and there were more discussions on generating insight than on producing software.

6.4. Team M${}_D$: Supporting inference and model assessment

One group of SCRC modelling scientists focused on quantifying epidemic characteristics, effect of intervention, model performance, result uncertainty, and parameter sensitivity (Swallow et al., 2021). From the onset, the VIS volunteers for supporting these modelling activities anticipated the use of ensemble and uncertainty visualisation. Uncertainty is a common property of epidemiological models. The basic uncertainty visualisation typically depicts the main computational results (e.g., an aggregated time series) in conjunction with other visual information for conveying the uncertainty (e.g., some amorphous pattern behind the main time series, or additional time series representing the 25 and 75 percentiles of the ensemble). Such visual representation are commonly used in modelling. While the generic support team was developing these basic plots (e.g., top-right image in Fig. 3), Team ${\mathbf{M}}_D$ focused on sensitivity analysis that helps modelling scientists to discover the reasons behind uncertainty.

During an initial collaboration meeting, domain experts confirmed the need for visualising the sensitivity of model parameters. At that time, the domain experts were working on their models in parallel, thus VIS volunteers were not able to obtain multi-run simulation data in the early months of the collaboration. VIS Team ${\mathbf{M}}_D$ took initiatives to study two COVID-19 models in the public domain, attempting to generate multi-run simulation data. Before this attempt could yield useful output, one SCRC model, ABC-smc (Porphyre et al., 2020) produced multi-run simulation data for uncertainty visualisation and parameter space analysis. By attending modelling scientists’ meetings, the team were able to observe the interactions between the perspectives of modelling and uncertainty quantification, and established a set of requirements.

Similar to most multi-run simulation problems, VIS needed to support the analysis of many sets of model parameters and outputs. In this case, each dataset consists of some 200 time series and their corresponding parameter sets. One obvious requirement is to visualise the uncertainty featured in the set of time series. As this is a common requirement for all models with time series outputs, we passed the requirement to the generic support team (see Fig. 3). The team focused on the more complex tasks, i.e., (i) to identify input/output relationships, (ii) to determine key curve features such as maximum or largest slope, and (iii) to compare outputs from a number of different model runs. Immediately after the requirements analysis, we started to develop a VIS system iteratively, with increasing facilities for analytics, visualisation, and interaction.

Fig. 9 shows the current prototype after several iterations. We used the design approach of coordinated multiple views (Boukhelifa et al., 2003), the visual analytics approach for computing a set of curve features (Tam et al., 2011), a parallel coordinate plot (Inselberg and Dimsdale, 1990) for viewing and filtering multi-dimensional parameter sets and curve features, and aggregated curves summarising the outputs of the selected parameter sets. We are in the process of introducing new facilities, such as slicing the multi-dimensional parameter space (Torsney-Weir et al., 2017), using functional box plots (Whitaker et al., 2013, Mirzargar et al., 2014) to summarise many curves, and the “contribution to the sample mean” plot (Bolado-Lavin et al., 2009) to show the sensitivity of outputs to input parameters.

Fig. 9.

A prototype system for analysing and visualising ensemble data. The parallel coordinate plot (right) allows one to filter input parameters, select a subset of time series to be aggregated, and display the aggregated curves (left).

7. Disseminative visualisation

While most VIS volunteers were distributed among the aforementioned six teams, we also created a small team for disseminative visualisation since public information dissemination was the original overall requirement. One epidemiology researcher with interest in data visualisation also joined the team. The team explored the prevailing approaches, in the UK and internationally, in public-facing visualisations related to the pandemic. This ranged from those produced by a number of governments (e.g., the four home nations in the UK), organisations (e.g., WHO, UK ONS), universities (e.g., Johns Hopkins dashboards), media outlets (e.g., FT Coronavirus tracker), and non-commercial web services (e.g., Worldometers).

The team concluded that we should complement, but not duplicate, the existing effort, and defined our goal as to inform the public about activities of SCRC through storytelling visualisation. We identified the following requirements: (i) to maintain scientific rigour, (ii) to retain scientific language, and (iii) to abstract visualisation output. For example, one of our initial designs used football result prediction and weather forecasting metaphors. Our rigorous consideration indicated that the former might be associated with gambling, whereas the latter could be perceived as inaccurate, and they could have negative connotations to how epidemiological modelling would be perceived.

Fig. 10 illustrates our process for creating storytelling visualisation. It started with an ideation phase where we elaborated preliminary concepts and ideas. This was done using a combination of the five-design sheets (Roberts et al., 2016) methodology (Fig. 10(a)), animated PowerPoint mock-ups, and web-based prototyping. These sparked off other explanatory forms, e.g., infographics and slide packs, such as in Fig. 10(b), which were used to describe each epidemiological model and the corresponding visualisations. As it would be difficult to apply a unified narrative to every model, we opted for allowing each story to be developed independently.

Fig. 10.

Storytelling visualisation creation process. (a) Initial concepts and ideas were explored using a combination of the five-design sheets (Roberts et al., 2016) method. (b) Following the progress of the modelling activities, these were transformed to animated presentations and infographics, which incorporated visualisations from other teams (in this example, from Team ${\mathbf{M}}_A$). High-resolution images of the design sheets and the presentations are available as the supplementary materials.

Current storytelling visualisations have been implemented as: (a) web-based presentations using the Reveal.js framework, with SVG-based animations and the potential for directly feeding into them visualisations created by other teams, (b) as video outputs of animated presentations, and (c) as infographics, created using graphics editors and creative design tools. We are in the process of creating a public web server for hosting these storytelling visualisations.

8. Reflections and recommendations

In this section, we reflect on our experience of developing VIS capacities for emergency response, and translate our reflections to a set of recommendations as a step towards a new methodology.

Reflection on general perception of VIS as a dissemination tool.

Expert users often see visualisation as “for informing others” rather than “for helping myself”. This can be a big stumbling block during requirement analysis. As illustrated in Fig. 1, Dr. Reeve’s response during the first meeting helped overcome this stumbling block, shortening the delay in requirements analysis by months. Meanwhile, to the disseminative visualisation team, creating such visualisation has not been an easy journey, especially without the advice from an expert on public engagement.

Reflection on requirements analysis.

During the pandemic, domain experts were extremely busy. Different teams did not follow the same formula for requirement analysis. Modelling support teams ${\mathbf{M}}_B$ and ${\mathbf{M}}_C$ followed the recommended method for user-centred requirement analysis, and benefited from frequent engagement. The generic support team and team ${\mathbf{M}}_A$ identified urgent requirements quickly and began their development without much delay. The analytical support team, team ${\mathbf{M}}_D$, and disseminative visualisation team had to use their knowledge to anticipate and analyse the potential requirements. In emergency responses, all of these are valid methods. Several teams had positive experience in using quickly-produced visualisations to stimulate requirement analysis. Team ${\mathbf{M}}_B$ made good use of several communication mechanisms, including searchable threaded chat streams, structured feedback, and design exposition. A few VIS volunteers also found it rewarding to attend domain experts’ meetings that at first appeared irrelevant.

Reflection on team organisation.

The categorisation of visualisation tasks based on the complexity of the search space of the possible solutions is relatively new (Chen and Golan, 2016). It informs us that model optimisation is an NP process in general, and it requires all three levels of visualisation, i.e., observational, analytical, and model-developmental visualisation, which correspond to solution spaces of complexity $O\left(n\right)$, $O\left(n^k\right)$, and EXPSPACE (an NP class). The complexity is likely to impinge on the effort for identifying VIS requirements. Hence, having a VIS team working with each modelling team was necessary for establishing such understanding. All teams quickly identified and addressed the observational requirements related to individual models, and some have started to address the requirements for model analysis and model optimisation. Meanwhile, the generic support team progressed to the development stage quickly because of not only the necessity but also the less complex search space.

Reflection on VIS resources.

Using volunteer effort is not an ideal solution for emergency responses. It would be more efficient if we could utilise an existing technical and knowledge infrastructure for such an emergency response, if such an infrastructure had existed for other operations and had an advanced VIS server and a team of VIS developers who were knowledgeable about different levels of visualisation tasks. Our volunteering effort was a make-shift solution, which benefited strongly from the academic knowledge infrastructure in the UK. Its progress could be more rapid if there were more development resources. The organisation of VIS volunteers partly reflects the need to concentrate most development resources in the generic support team. Nevertheless, the outcomes delivered by the VIS volunteers between June and December 2020 without any funding are unprecedented. This demonstrates the importance of VIS as well as volunteering effort in emergency responses.

Reflection on VIS in epidemiological modelling.

From the perspective of epidemiological modelling, the RAMPVIS effort allowed us to appreciate how VIS may be used in many aspects of modelling workflows, in addition to disseminating modelling results. Developing epidemiological models during an epidemic or pandemic is a not a new phenomenon. To have epidemiology teams develop multiple models is not uncommon, but activities by SCRC and RAMPVIS have demonstrated that the scale, complexity and dynamic nature of the many datasets required to support COVID-19 modelling benefits enormously from infrastructural support featuring data management, analysis, and visualisation capabilities. Success in an emergency response, such as combating COVID-19, rarely hinges on a single correct model or a critical visual design. More likely, it will benefit from the collective effort of continuously developing, evaluating, and improving models, and from the capabilities for enabling rapid observation of numerous pieces of data and analytical results throughout the modelling workflows. The RAMPVIS effort showed that collective VIS support is necessary and feasible, and such capabilities should be available to epidemiological modellers from the very beginning in future emergency responses.

Recommendation.

Our approaches, experience, and reflections may be translated to the following recommendations for future VIS applications in emergency responses:

• •In December 2014, US President Barack Obama spoke to the National Institutes of Health (USA): “There may and likely will come a time in which we have both an airborne disease that is deadly. And in order for us to deal with that effectively, we have to put in place an infrastructure” (The White House, Office of the Press Secretary, 2014). Shadbolt et al. outlined the future need for a data ecosystem (Shadbolt et al., 2021). VIS should be part of any data ecosystem, and be closely coupled with or integrated into data infrastructures. The “readiness” of VIS technical and knowledge infrastructures will make a difference. While it may not be feasible to build an infrastructure for every type of potential emergency, we can benefit significantly to have a few VIS infrastructures that are ready to be cloned, re-purposed, and adapted for different emergency responses.
• •We need to make a serious effort to redress the common misconception that visualisation is only a dissemination tool. In particular, we need to help modelling scientists to become accustomed to use VIS techniques throughout their workflows. Perhaps the best way to broaden the uses of VIS in modelling workflows is to enable more collaborative research between VIS scientists and modelling scientists in different disciplines.
• •While the agile principle (Munzner, 2009) fits well with VIS development for supporting emergency responses, one should be open-minded about different approaches. The diverse approaches taken by different VIS teams in the RAMPVIS effort indicated that standard practice might not always be applicable. VIS development in emergency responses can benefit tremendously from the existing VIS knowledge, in the form of theories, methodologies, literature, and personal experience. The VIS community should improve its “readiness” by advancing abstract VIS knowledge in the form of theories and methodologies.

9. Conclusions

In this paper, we have reported the work carried out by a group of VIS volunteers to support modelling scientists and epidemiologists in combating COVID-19. Our approaches to the challenges that we have encountered are rare and valuable contributions to the first step towards a methodology for developing and providing VIS capacity to support emergency response. In November 2020, the UK Research and Innovation awarded funding to the group, transforming the volunteering effort to a more structured VIS operation in 2021. This allows us to develop a VIS infrastructure that can be deployed to support some ongoing modelling effort as well as be served as a major example to influence VIS infrastructure. Meanwhile, we continue to strengthen the collaboration between VIS researchers and epidemiological experts, developing more domain-specific VIS techniques for supporting epidemiological modelling workflows.

CRediT authorship contribution statement

M. Chen: Conception and design of study, Analysis and/or interpretation of data, Writing – original draft. A. Abdul-Rahman: Conception and design of study, Acquisition of data, Analysis and/or interpretation of data, Writing – original draft. D. Archambault: Conception and design of study, Analysis and/or interpretation of data, Writing – original draft. J. Dykes: Conception and design of study, Acquisition of data, Analysis and/or interpretation of data, Writing – original draft. P.D. Ritsos: Conception and design of study, Analysis and/or interpretation of data, Writing – original draft. A. Slingsby: Conception and design of study, Acquisition of data, Analysis and/or interpretation of data, Writing – original draft. T. Torsney-Weir: Analysis and/or interpretation of data, Writing – original draft. C. Turkay: Conception and design of study, Acquisition of data, Analysis and/or interpretation of data, Writing – original draft. B. Bach: Conception and design of study, Acquisition of data, Analysis and/or interpretation of data, Writing – review & editing. R. Borgo: Conception and design of study, Analysis and/or interpretation of data, Writing – review & editing. A. Brett: Conception and design of study, Writing – original draft. H. Fang: Acquisition of data, Analysis and/or interpretation of data, Writing – review & editing. R. Jianu: Acquisition of data, Analysis and/or interpretation of data, Writing – review & editing. S. Khan: Acquisition of data, Analysis and/or interpretation of data, Writing – review & editing. R.S. Laramee: Analysis and/or interpretation of data, Writing – review & editing. L. Matthews: Acquisition of data, Analysis and/or interpretation of data, Writing – review & editing. P.H. Nguyen: Acquisition of data, Analysis and/or interpretation of data, Writing – review & editing. R. Reeve: Conception and design of study, Acquisition of data, Analysis and/or interpretation of data, Writing – review & editing. J.C. Roberts: Acquisition of data, Analysis and/or interpretation of data, Writing – review & editing. F.P. Vidal: Acquisition of data, Analysis and/or interpretation of data, Writing – review & editing. Q. Wang: Acquisition of data, Analysis and/or interpretation of data, Writing – original draft. J. Wood: Analysis and/or interpretation of data, Writing – review & editing. K. Xu: Acquisition of data, Analysis and/or interpretation of data, Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary material related to this article.

MMC S1

This file contains the large images of the five-design sheets shown in Fig. 10(a).