Ontario’s COVID-19 Modelling Consensus Table: mobilizing scientific expertise to support pandemic response

Abstract

Setting

COVID-19 has highlighted the need for credible epidemiological models to inform pandemic policy. Traditional mechanisms of commissioning research are ill-suited to guide policy during a rapidly evolving pandemic. At the same time, contracting with a single centre of expertise has been criticized for failing to reflect challenges inherent in specific modelling approaches.

Intervention

This report describes an alternative approach to mobilizing scientific expertise. Ontario’s COVID-19 Modelling Consensus Table (MCT) was created in March 2020 to enable rapid communication of credible estimates of the impact of COVID-19 and to accelerate learning on how the disease is spreading and what could slow its transmission. The MCT is a partnership between the province and academic modellers and consists of multiple groups of experts, health system leaders, and senior decision-makers. Armed with Ministry of Health data, the MCT meets once per week to share results from modelling exercises, generate consensus judgements of the likely future impact of COVID-19, and discuss decision-makers’ priorities.

Outcomes

The MCT has enabled swift access to data for participants, a structure for developing consensus estimates and communicating these to decision-makers, credible models to inform health system planning, and increased transparency in public reporting of COVID-19 data. It has also facilitated the rapid publication of research findings and its incorporation into government policy.

Implications

The MCT approach is one way to quickly draw on scientific advice outside of government and public health agencies. Beyond speed, this approach allows for nimbleness as experts from different organizations can be added as needed. It also shows how universities and research institutes have a role to play in crisis situations, and how this expertise can be marshalled to inform policy while respecting academic freedom and confidentiality.

Introduction

The COVID-19 pandemic has presented unprecedented economic, social, and health challenges. In response, jurisdictions around the world have implemented public health interventions designed to contain the spread of the virus and ensure adequate health system capacity. The availability of epidemiological and health system modelling, and corresponding estimates of case numbers, hospitalizations, and deaths have shaped and supported such interventions.

However, the rapid onset of COVID-19 necessitated an acceleration in both knowledge production and translation. Traditional mechanisms of commissioning research (e.g., through funding competitions) or conference presentations are ill-suited to guide policy during a rapidly evolving pandemic. At the same time, another common mechanism—contracting with a specific centre of expertise—has been criticized for failing to reflect the challenges inherent in specific modelling approaches and creating concerns over transparency (Hamzelou, 2020; Saltelli et al., 2020). This commentary describes the creation of an alternative approach in the form of an explicit partnership between the decision-makers in Ontario and a group of largely university-based modellers.

Mobilizing expertise in a crisis

During a pandemic, decision-makers need access to credible epidemiological models to inform public health measures, health system interventions, and broader public policy as well as to inform the public of the current and likely direction of the pandemic. This requires access to evidence and experts with the requisite range of skills, along with timely data, open communication, and a clear process for sharing insights and translating findings. Although seemingly straightforward, this can be challenging to put into practice.

During the 2003 SARS epidemic, “Canadian researchers were hamstrung by patient care and scientific advisory responsibilities, a lack of data, infighting about data access, limited research funds, and the need to obtain ethics approvals at multiple institutions.” (Government of Canada, 2003). Similarly, while external experts were brought together to advise the Ontario government, an inquiry found their efforts were hampered by a lack of supporting infrastructure, knowledge translation mechanisms, and clarity on role and responsibilities (The SARS Commission, 2005). Challenges around data access, collaboration, and methodological disagreement beset public health and health services research at the best of times and become more acute during times of crisis.

Seventeen years later, mobilizing expertise is still an issue. At the start of the COVID-19 pandemic, the Ontario government had access to epidemiological models through a variety of sources, including government agencies such as Public Health Ontario, hospitals, public health units, academic researchers, and even the media. However, there was no formal mechanism for coordinating these disparate sources, nor for dealing with conflicting data or interpretation of results. This led to three key problems. First, different modelling exercises produce different numbers, which is to be expected, but when non-expert audiences are presented with different numbers, it can lead to confusion and delay. It can also lead to over-reliance on the results produced by one approach or model as a way to simplify this challenge. Second, multiple modelling exercises create inefficiencies. Insights gained by one group do not necessarily transfer to other groups, meaning there is likely substantial wasted effort when time is of the essence. This is particularly critical in an emerging pandemic where there is little known about the disease and existing datasets may be difficult to use. Third, modelling exercises by multiple groups are challenging to coordinate and it is hard to guarantee that policy decisions are based on the most reliable estimates, particularly when it is not possible to identify what makes a particular model optimal until well after the pandemic has passed.

To address these challenges, the idea arose to convene a group of modelling experts to advise government. Each expert would build their own models and approach data in different ways, but the group as a whole would work to develop consensus epidemiological estimates and communicate these quickly to government.

The COVID-19 Modelling Consensus Table

Ontario has long built innovative structures to provide advice and improve the translation of research to guide policy development and implementation (Lomas & Brown, 2009). Building on existing structures, a group of academics and public servants started the Modelling Consensus Table (MCT) on March 26, 2020. The goal was to enable rapid creation and communication of credible consensus estimates of the impact of COVID-19 on health system capacity, morbidity, and mortality, and to accelerate learning on how the disease was spreading and what could slow its transmission. Roughly 1 week later, the MCT’s first estimates were used by government to communicate with the public on the seriousness of the pandemic and to underscore the importance of public health interventions (see Figure 1).

Figure 1.

MCT milestones overlaid on timeline of COVID-19 cases in Ontario

Core objectives of the MCT are listed in Text Box 1. These were developed collaboratively by modellers and the Table’s sponsors: the Ontario Ministry of Health, Ontario Health, and Public Health Ontario. The MCT is chaired by two senior academics, membership is entirely voluntary, and all are bound by a formal Terms of Reference. Members are mathematicians, epidemiologists, health services researchers, statisticians, and senior decision-makers.1 In selecting members, a diversity of modelling expertise was sought to strengthen the quality of the consensus and to answer different types of questions. Members were added over time to address different needs.

Text Box 1 Core objectives of COVID-19 Modelling Consensus Table

1. To offer best possible guidance to decision-makers and health system leaders about the dynamic and evolving state of the COVID-19 pandemic in Ontario, including consensus estimates of the incidence, prevalence and spread of COVID-19 in Ontario and other key parameters of use to decision-makers. 2. To leverage the latest knowledge and advances in techniques and data sources. 3. To rapidly fulfill related data needs from data partners across the health system. 4. To rapidly respond to a prioritized list of research questions that can support the current fight against COVID-19 in Ontario. 5. To leverage and combine the collective analytic expertise in the Ontario Ministry of Health, Ontario Health, and Public Health Ontario with scholars and experts working on COVID-19 models.

Since March 26^th, the MCT has met once per week to share results from different modelling exercises of disease transmission and the impact that different public health interventions might have; to generate consensus judgements of the likely future impact (4 to 6 weeks out) of COVID-19 in Ontario; and to discuss decision-makers’ priority topics for further work. All discussions are held under the Chatham House Rule, where comments made during a meeting can be shared publicly but may not be attributed to any individual or organization.2

In addition to the opportunity to inform important policy decisions, a key draw for the table is data availability and expedited research ethics board approval. Past examinations of pandemic responses have found that a lack of consistent epidemiological and health system data hampered informed responses and inhibited research from being published expeditiously (Xing et al., 2010).

The Ministry of Health supplied a wide range of epidemiological, clinical, laboratory, health system, and public health data for modelling activities and arranged for data sharing agreements that permitted dissemination and publication within 24 h to match the speed with which information is being produced on the pandemic. Patient-level data are anonymized prior to their release to protect privacy. The data released represent the largest set of databases made available to a broad group of researchers for public health research to date in Ontario.

The MCT adhered to the WHO principles laid out for research conducted during a pandemic, which includes the need for an independent ethics review (WHO, 2020). Since ethics review can be time-consuming and challenging—it proved to be a rate-limiting step to publication of findings during the SARS epidemic—the Table’s co-chairs arranged for umbrella expedited review from the University of Toronto’s Research Ethics Board. This greatly reduced the time required to begin research and disseminate findings.

The MCT strives to reach consensus estimates (not necessarily unanimous) on forecasts for key indicators such as basic and effective reproduction numbers, mortality rates, and the number of hospital beds and ventilators that will be used by COVID-19 patients. Consensus estimates developed at the meetings are strictly confidential and communicated only to the sponsors who then share these estimates in briefings and communiques. However, the research underpinning these estimates and based on the data provided are disseminated or published widely, with only a requirement for 24-hour notification to the Ministry. Some MCT members initially publish their findings on preprint servers so knowledge can quickly be shared with colleagues while traditional peer-review and publication processes continue in parallel.

According to MCT members, benefits of this structure include:

• Unprecedented, swift access to data for participants.

• Strong leadership and structure to support consensus for decision making.

• Credible models to inform health system preparedness and capacity planning.

• Rapid, focused debate and dialogue, fostering enhanced scientific quality, relevance and shared understanding of both disease patterns and source data.

• Increased transparency in the public daily reporting by government of COVID-19 data.

• Access to credible scientific experts who can explain relevance and contextualize findings.

• Administrative infrastructure to support governance and meeting logistics.

Issues to consider

Convening an expert consensus table may seem simple and a regular feature of academic and policy work. But the demands brought on by the COVID-19 pandemic for rapid, constant, and evolving advice put significant pressures on this approach. Furthermore, the MCT model raises additional issues because it relies solely on scholars’ intrinsic motivations.

The first issue is academic freedom. Most of the teams working on COVID modelling are led by academics doing grant-funded research. These scholars need to continue their work and universities have policies on academic freedom that protect their ability to research and publish independently. Policy structures must allow for this, otherwise academics will be reluctant to participate. This can create challenges as decision-makers need to consider and plan carefully for how they want to share information to ensure the effectiveness of public health and other interventions.

The second issue is the need to maintain confidentiality for senior decision-makers, particularly politicians whose communication cycle requires almost daily briefings. These decision-makers need to see information as it becomes available, need the ability to consider their options, and need to be able to do so without the constant risk of being second-guessed by outside parties. They should not be constrained in how they view and commission modelling estimates. Separating out a stream of useful policy advice from published academic work is challenging and is made all the more so by the preprint and rapid publication culture that has developed during the pandemic (Mehta, 2020).

To address the above issues, the MCT adopted terms of reference that clearly state there are no limitations on academic publication or sharing of results from individual groups. Standard principles of authorship are followed for individual works and each publication includes a reference to the COVID-19 MCT. Advice from the Table is communicated quickly to decision-makers; however, the MCT itself is not part of the decision-making process, thereby maintaining a separation between advising and policymaking.

The third issue is the current regime of physical distancing and self-isolation. Many of the ways that researchers would interact and compare results are lost without the usual pattern of contacts at meetings, conferences, or hallway conversations. The current pace of work as clinicians and managers respond to the pandemic exacerbates this problem, leading to even less interaction among different experts and researchers. A related challenge is overcoming disciplinary boundaries to enable diverse researchers to bring their expertise to bear. To counter this, the MCT included academics from a broad range of disciplines and invited guest researchers to present their data and insights.

A multi-faceted strategy was employed to overcome the challenges of adopting a collaborative approach across centres in the midst of public health restrictions. A small secretariat with a part-time academic administrator and administrative support was created to facilitate logistics, scheduling, and communications. The secretariat established an online document and collaboration platform. This infrastructure has been instrumental in ensuring productive interactions and that members remain connected between meetings. This model of common support could be employed to facilitate the participation of smaller centres or even interprovincial collaboration.

A fourth issue is interpretability and trust. Understanding and making meaningful use of epidemiological models can be a daunting prospect for senior decision-makers who are faced with the need to act upon their findings. The Centers for Disease Control and Prevention does an admirable job in routinely documenting a wide range of case and death projections from dozens of academic groups (CDC, 2020). This exercise in comprehensiveness and transparency is a valuable resource but potentially less useful for decision-makers because of the lack of interpretation and context for local settings. The MCT combines the consensus efforts used by the CDC with the benefit of interpretation and guidance by a group of experts.

The co-chairs sought to achieve consensus on all estimates provided to government. The goal was to always ensure that a majority, if not all, of the members agreed with the estimates being put forward. The group did not employ formal methods of consensus such as the Delphi technique, instead opting for consensus through discussion. This informal approach of building consensus allows for the voicing of different perspectives but does rely on a strong facilitator to ensure all viewpoints are heard and incorporated into discussion (WHO, 2021).

Finally, jurisdictional issues may arise. In the case of COVID-19, different roles and responsibilities are held by federal, provincial, and local levels of government. Experts may be called upon by different agencies, creating potential conflicts or risk of burnout. Ontario’s approach has again benefited from the academic model, whereby MCT members are free—and indeed encouraged—to participate at other levels. While the focus of MCT is on the province of Ontario, there is a regular sharing of interests, outputs, tools, and interpretations with groups working at local, federal, and international levels.

Discussion

Academics create new knowledge, and the academic process is a constant contest and collision of ideas. The continual debate over conflicting ideas and approaches is the basis of scientific inquiry but during a pandemic, government policy-makers and public health officials lack the time to effectively synthesize potentially conflicting and competing findings. The need for consistency of presentation and established communications protocols is critical due to the intense scrutiny that public policies undergo, both within and outside of government. The volume and scope of misinformation during the COVID-19 pandemic have made it even more important to ensure the quality and clarity of scientific information being put forth to help guide pandemic responses. The MCT was created as a way to purposefully connect a structured, organized community of modelling experts and government decision-makers.

The Kirby (Senate of Canada, 2003), Naylor (Government of Canada, 2003), Walker (Government of Ontario, 2004), and Campbell (The SARS Commission, 2006) reports all found opportunities for improvement following the 2003 SARS epidemic. Challenges included the ad hoc assembly of external experts, the lack of availability of data for epidemiological research, unclear roles and responsibilities, the lack of effective knowledge translation, and the absence of supporting infrastructure for the group. Ultimately, they called for an infrastructure that would support rapid assembly of expertise and data in a model of open policy development.

Canada has produced several important reports and knowledge translation initiatives which helped set the stage for the MCT. The Pan-Canadian Strategy for Public Health Workforce Education outlines different reports and initiatives which highlight the need to build competency and skill in building research into practice (Spasoff, 2005). Provincially, Public Health Ontario and the Ontario Ministry of Health both have purposely-built programs designed to foster connections between practitioners and researchers (Public Health Ontario, 2021; Government of Ontario, 2021).

The MCT was a rapid response to the challenges identified in these reports. Ultimately, it may be hard to maintain the depth and diversity of expertise provided by the MCT except in academic settings, as the costs of developing such expertise are high. Although it is unfortunate and all too often tragic, it is difficult for governments to justify large public health expenditures necessary for pandemic readiness outside of times of crises.

Despite these challenges, the MCT provides one way to quickly draw on scientific advice outside of governmental or public health agencies. Its value is demonstrated through the rapid knowledge translation that it facilitated. For example, over a period of 8 days, a draft publication was shared with the modelling group, used to brief senior decision-makers, published on a preprint server, and used as part of an Ontario Ministry of Health COVID-19 technical briefing on modelling (Fisman, 2020; Government of Ontario, 2020). The speed of this cycle is astonishing by traditional academic standards.

Lessons learned

The Ontario experience suggests several lessons for jurisdictions looking for ways to mobilize expertise during a pandemic.

• For evidence-informed decision making during a pandemic, it is important to ensure that a range of scholarly (expert) voices are heard, both to gain a broader range of input and to counter limitations of relying on a single centre of expertise. This means it is important to work with multiple scholars and academic institutions, not just rely on one group as the source of evidence.

• Academic experts need to work together with a focus on evidence generation to guide pandemic control decisions—something that is not always easy across institutions—so that their advice is more coherent and less susceptible to attack or denigration. This means that the more that their work can line up with typical academic practices and principles like publication of results and academic freedom, the better they will be able to work together.

• It is difficult to maintain capacity in public health agencies due to constant budgetary pressures. However, there is expertise in universities and research institutes that can be harnessed to offset this. Accessing this expertise, however, requires give-and-take including the sharing of data by government and its agencies and some adaptation to academic principles.

• Universities and research institutes have a role to play in crisis situations that extends beyond teaching and research. This should be recognized, both by external parties and by universities. It will be important for academic institutions to consider how they want to recognize the key and important service that scholars can provide through their research when making decisions about promotion, tenure, and other awards. If universities fail to adapt following this pandemic, scholars may be less likely to participate in responses to the next pandemic.

Conclusion

Ontario’s COVID-19 MCT provided a conduit between senior governmental and public health decision-makers and the scientific community. The table’s principles of open discussion, dissension, suggestions, and eventually consensus continue to provide robust scientific advice to government. The feedback loop established between policy making, public health measures, and epidemiological estimates of impact is certainly one of the important factors that led to better management of COVID-19 in Ontario.

Implications for policy and practice

What does this study add to existing knowledge?

• The study documents how a knowledge translation process, developed in the midst of a global pandemic, accelerated the delivery of epidemiological and health system findings directly into government policy discourse.

• The study identifies factors for successfully bringing together a diverse group of academics and policy-makers, particularly regarding access to data, canvassing and coordinating a diverse set of experts, and balancing the need for academic freedom with the necessity of confidentiality required by decision-makers.

What are the key implications for public health interventions, practice, or policy?

• Rapid evidence synthesis and knowledge translation during a pandemic is critical.

• For evidence-informed decision making during a pandemic, it is important to ensure that a range of scholarly voices are heard, both to gain a broader range of input and to counter the limitations of relying on a single centre of expertise.

• Collaborative expert bodies, backed with administrative support, data, and clear channels of communication with government, can help ensure policy-makers receive trusted, timely, and relevant evidence to inform rapid decision making on important issues such as health system capacity and public health guidance.

Author contributions

All the authors contributed to the paper. Research, data collection, and analysis were performed by Michael Hillmer, Patrick Feng, John McLaughlin, and Adalsteinn Brown. The first draft of the manuscript was written by MH and PF, with co-authors providing comments and feedback on versions of the manuscript. All the authors read and approved the final manuscript.

While several MCT members contributed to this paper (MH, JM, KM, BS, AG, AB), the views expressed here are those of the authors, not the Table. Any errors or omissions are the responsibility of the authors, not our colleagues on the MCT.

Declarations

Conflict of interest

Three of the authors (MH, AG, AB) have roles in the Ontario Government or Ontario Health. In addition, most of the authors are members of the MCT. While there is no actual conflict of interest, we recognize there may be a perceived conflict of interest when authors comment on the activities with which their organizations are involved. For the sake of clarity, the opinions expressed here are those of the authors, not of their organizations.

Ethics approval

Ethics approval was not required as this paper does not involve research with human subjects.

Consent to participate

N/A

Consent for publication

N/A