Identifying gaps in COVID-19 health equity data reporting in Canada using a scorecard approach

Abstract

Objective

To assess health equity-oriented COVID-19 reporting across Canadian provinces and territories, using a scorecard approach.

Methods

A scan was performed of provincial and territorial reporting of five data elements (cumulative totals of tests, cases, hospitalizations, deaths, and population size) across three units of aggregation (province or territory level, health regions, and local areas) (15 “overall” indicators), and for four vulnerable settings (long-term care and detention facilities, schools, and homeless shelters) and eight social markers (age, sex, immigration status, race/ethnicity, healthcare worker status, occupational sector, income, and education) (180 “equity-related” indicators) as of December 31, 2020. Per indicator, one point was awarded if case-delimited data were released, 0.7 points if only summary statistics were reported, and 0 if neither was provided. Results were presented using a scorecard approach.

Results

Overall, information was more complete for cases and deaths than for tests, hospitalizations, and population size denominators needed for rate estimation. Information provided on jurisdictions and their regions, overall, tended to be more available (average score of 58%, “D”) than that for equity-related indicators (average score of 17%, “F”). Only British Columbia, Alberta, and Ontario provided case-delimited data, with Ontario and Alberta providing case information for local areas. No jurisdiction reported on outcomes according to patients’ immigration status, race/ethnicity, income, or education. Though several provinces reported on cases in long-term care facilities, only Ontario and Quebec provided detailed information for detention facilities and schools, and only Ontario reported on cases within homeless shelters and across occupational sectors.

Conclusion

One year into the pandemic, socially stratified reporting for COVID-19 outcomes remains sparse in Canada. However, several “best practices” in health equity-oriented reporting were observed and set a relevant precedent for all jurisdictions to follow for this pandemic and future ones.

Supplementary Information

The online version contains supplementary material available at 10.17269/s41997-021-00496-6.

Introduction

Early reporting by regional and provincial jurisdictions in Canada suggests that, as has been the case in other countries such as the United States (USA) (Chen et al. 2020; Moore et al. 2020), social inequities in COVID-19 outcomes have emerged. In Ontario, for instance, higher rates of COVID-19 incidence, hospitalization, and death have been observed in lower-income areas and areas with higher densities of immigrant and racialized residents (Chung et al. 2020). Toronto has reported that 83% of COVID-19 cases with available race/ethnicity data, identified between mid-May and mid-July 2020, occurred among racialized residents, despite these residents representing 52% of the city’s population (Toronto Public Health 2020).

These early reports of social inequities in COVID-19 outcomes beg several questions for public health policy and intervention. Since the identification of these inequities is predicated on the availability and release of COVID-19 surveillance data according to social markers, one fundamental question is how Canada is doing, overall, in health equity-informed COVID-19 data reporting across jurisdictions? Knowing which inequities have emerged, and where, is a necessary first step in planning health equity-informed health policy and interventions (Blair et al. 2018; Frank and Matsunaga 2020; Moore et al. 2020).

Indeed, public release and reporting on surveillance data have been essential to inform epidemiologic research and modelling and public health interventions since the start of the COVID-19 pandemic. Presenting data disaggregated by social markers, such as sex or race/ethnicity, can ensure that social and political responses to the health crisis are sensitive to and designed to be effective against social disparities in outcomes (Childs and Palmieri 2020). Data transparency also serves to protect the public’s trust in public health guidelines and ensure accountability (Brison 2018). However, given that provincial and territorial rather than federal public health authorities are the primary entities collecting and reporting on health data in Canada, public-facing output on local- or social marker-disaggregated data can vary across Canadian jurisdictions. An assessment of both overall and health equity-oriented COVID-19 data reporting in each Canadian province and territory is needed to identify both best practices and reporting gaps.

The objective of this study was to perform an environmental scan of COVID-19 data reporting across Canadian provinces and territories and to assess health equity-focused reporting using a scorecard approach. Scorecards can be used to help track health-related trends or the quality of data reporting across jurisdictions (MHASEF Research Team 2018). Here, we build on the USA-based Coronavirus in Kids (COVKID) Tracking and Education Project’s recently proposed COVKID State Data Quality Report Card (Pathak et al. 2020) which was designed to identify gaps in COVID-19 surveillance in children. We propose the Canadian COVID-19 Health Equity Data Scorecard as an evaluation framework for the Canadian context.

Method

Data

A detailed environmental scan of official provincial and territorial public health websites and published reports was performed to identify data reporting content. Reference websites used were those provided by the Public Health Agency of Canada on their centralized reporting website (Public Health Agency of Canada 2020a). Provincial and territorial websites were searched for data summaries, figures, and tables as well as downloadable reports (most often available in portable document (PDF) format), by navigating through websites and downloading and reviewing reports. The results are accurate up to December 31, 2020. This scan represents a summary of Canadian reporting as of approximately one year after the identification of the first cases of COVID-19 (World Health Organization 2020).

Scorecard indicators

Based on the minimum data requirements proposed by extant COVID-19 data quality assessments, such as the COVKID Project Data Quality Report Card (Pathak et al. 2020), we assessed provinces’ and territories’ reporting of five data elements: cumulative totals of tests performed, case counts, hospitalizations, and deaths, as well as the availability of data on the size of populations of interest. Population size was not included in the COVKID Report Card assessment. However, it is included here insofar as it is necessary for rate estimation and relative comparisons across jurisdictions and groups.

As the COVKID Report Card does for each state, we assessed the availability (and the availability of explicit operational definitions) of these five data elements across provinces and territories, overall. We also assessed reporting on two additional levels of population aggregation: health region or unit level, and Forward Sortation Area-level or small neighbourhood area equivalent. Reporting on various levels of spatial aggregation was assessed given that transmission epidemiology and distributions of risk factors can vary across jurisdictions and localities.

Across these levels of aggregation, we also assessed data reporting across eight equity-related indicator strata. Building on the COVKID Report Card’s assessment of reporting across age and race/ethnicity, we assessed reporting across individual-level exposure to four congregate living and institutional settings that are vulnerable to COVID-19 outbreaks (long-term care and detention facilities, homeless shelters, and schools) (Blair et al. 2020; Hsu et al. 2020; Public Health Ontario 2020a; Richard et al. 2021) and eight individual-level social markers (age, sex, immigration status, race/ethnicity, healthcare worker status, occupational sector, and income and education groups). The latter social markers have been identified as key social determinants of health and infectious disease burden (Semenza et al. 2016; Solar and Irwin 2010). These individual-level social, economic, and occupational data would typically be obtained at testing, during case interviews, or via data linkage to existing provincial or territorial health and social administrative databases. With the five data elements across three units of population aggregation—overall and across twelve social strata—195 indicators were used (5 * 3 * (1 “overall” population-level stratum + 12 social strata) = 195).

As done for other scorecards, these indicators were selected for being measurable, relevant for health equity surveillance, actionable, and interpretable (Institute for Clinical Evaluative Sciences 2018). Indeed, precedent exists for surveillance reporting across all social markers used, including by race/ethnicity (CDC 2020; New Zealand Ministry of Health 2020) and occupational sector (Iowa Department of Public Health 2020)—if not for COVID-19, for other common health outcomes (Agic et al. 2013; Public Health Agency of Canada 2020a; Stachenko 2008).

Analysis

For each of the 195 indicators, 1 point was awarded if raw, anonymized individual case-delimited data were released and publicly available (i.e., where each case represented one data row, available in a downloadable, and editable file format, such as in Comma Separated Values (.csv) format). A total of 0.7 points was awarded if summary statistics were reported for the indicator, but no raw case-delimited data were publicly available, and a score of 0 was awarded if neither information was reported nor data made publicly available. To contrast surveillance reporting across jurisdictions at a national scale, points were only awarded if the data element was available for the entire jurisdiction (i.e., not if data were only available for certain regions).

We used a near-complete (0.7 points; intentionally higher than a half-point) and complete (1 point) scoring system rather than a binary (present/absent, 0 vs. 1 point) method to acknowledge the relevance of summary statistic reporting, while rewarding jurisdictions that opted for full data transparency for public use—as done in peer nations such as the United Kingdom (UK Data Service 2020) and the USA (USA Facts 2020). Raw data sharing has been identified as a best practice in supporting innovation and research, advancing government accountability and evidence-informed decision-making (Brison 2018; Lindquist and Huse 2017; Roy 2014). It also allows for an intersectional assessment of indicators. For example, the child health-focused COVKID scorecard found that though many states report on COVID-19 outcomes by age and race, a limited number of states report on the race of cases by age group (thereby allowing for an assessment of racial disparities among children) (Pathak et al. 2020). The sharing of case-delimited data allows users to pursue these more precise lines of inquiry.

For each level of aggregation, a percent score was estimated based on overall population-level data availability (score out of 5 data elements) and based on “equity” data across social strata (5 data elements * 12 social strata = score out of 60). A summary percent score was computed for population-level data overall (5 data elements * 3 population aggregation units = total score out of 15) and for equity-related data (5 data elements * 3 population aggregation units * 12 social strata = total score out of 180).

We adjusted score denominators to take into account that reporting on some of the indicators, such as the cumulative total of deaths or hospitalizations, is less relevant in jurisdictions without any recorded cases or when case numbers are so low (i.e., n < 5) that reporting may jeopardize patient confidentiality. When the total number of observations needed to estimate the indicator was less than 5, the indicator was removed from the score’s denominator total. In that way, if one jurisdiction had not recorded any COVID-19 cases, for example, it was not penalized for not reporting on cases by age and sex. Last, the following letter grades were associated with documented percent scores: 0–39% as “F” (very poor), 40–59% as “D” (poor), 60–69% as “C” (fair), 70–79% as “B” (good), 80–89% as “A” (very good), and 90–100% as “A+” (excellent).

Results

Scores were estimated for each province and territory, and Canada overall (Fig. 1, with detailed scores and sources in Supplementary File, eTable 1). On average, over half (58%, “D” score) of the data elements were available at the overall jurisdictional, health region, and local neighbourhood levels, while just under one in five (17%, “F” score) equity-related data elements were available.

Fig. 1.

Scores for multi-level population and equity-related data in Canada, by province and territory. PEI, Prince Edward Island. NWT, Northwest Territories

Though all jurisdictions reported on the types of tests used to identify SARS-CoV-2 infections, clear case definitions were not always available, and definitions of what was counted as a COVID-19 hospitalization or death were often missing (eTable 2).

Overall population-level data reporting

By province and territory

At the province and territory level, data availability scores ranged from “C” (62–68%) for Nova Scotia, Prince Edward Island, Newfoundland and Labrador, Nunavut, and the Yukon to “A” (82–88%) for British Columbia, Alberta, and Ontario. On average, 73% (“B”) of data elements were available at the province and territory level in Canada (Fig. 1).

All provinces and territories reported on the total number of tests, cases, and deaths (Fig. 2), with British Columbia, Alberta, and Ontario providing case-delimited data for all cases observed (Fig. 2b). Alberta and Ontario were the only two provinces that also reported on the outcomes (recovery or death) for each case, in a downloadable case-delimited format. Though most jurisdictions reported on the total number of hospitalizations that have occurred, these data were not provided by Nova Scotia, Prince Edward Island, Newfoundland and Labrador, nor the three territories (Fig. 2c). Population denominators for all jurisdictions were available through the Canadian Census.

Fig. 2.

Overall, province- and territory-level reporting (data availability) on the cumulative total of tests (panel a), cases (panel b), hospitalizations (panel c), and deaths (panel d)

By health region

For reporting by health regions within jurisdictions, population data availability scores ranged from “F” (25–34%) for Nova Scotia, Nunavut, and the Yukon to “A” (82–88%) for British Columbia, Alberta, and Ontario (Fig. 1). On average in Canada, 59% (“D”) of data elements were available for regions within jurisdictions.

The overall number of tests conducted per health region was available for 8 out of 13 jurisdictions (Fig. 3a). Quebec, one of the provinces hit hardest by the pandemic (Public Health Agency of Canada 2020a), did not report on total tests conducted per region. In contrast, reporting on the total number of cases per health region was more complete, with British Columbia, Alberta, and Ontario standing out as provinces that provide data on the region of residence for all identified cases (Fig. 3b). Overall, most provinces that reported on overall hospitalizations (Fig. 2c) also provided summaries of hospitalizations per health region (Fig. 3c)—New Brunswick was the exception to this rule. Except for Nova Scotia, data on deaths per health region were available for all jurisdictions reporting over five deaths (Fig. 3d). Last, population denominators for all health regions within jurisdictions were available from Statistics Canada’s Canadian Census.

Fig. 3.

Reporting (data availability) on the cumulative total of tests (panel a), cases (panel b), hospitalizations (panel c), and deaths (panel d), overall, for each health region or health unit within the province/territory

By Forward Sortation Area or the local neighbourhood area equivalent

Population denominators are made available by Statistics Canada for all Forward Sortation Areas in Canada. Overall, no jurisdictions reported on the cumulative total of hospitalizations at the local area level. None but Nunavut reported on the overall number of tests undertaken at the local level. Very few jurisdictions reported on the number of cases or deaths by local area. Exceptions were Alberta, with its reporting on the number of cases and deaths for all local areas of residence, and Ontario’s case-delimited dataset, which included the postal code associated with each death.

Overall, data availability scores for population data at the local area level ranged from “F” (20–25%) for most jurisdictions to “D” (48%) for Alberta and “C” (60%) for Ontario. On average in Canada, 27% (“F”) of data elements were available for local areas within jurisdictions (Fig. 1).

Equity-oriented reporting by social markers and vulnerable settings

On average in Canada, 17% (“F”) of data elements were available across identified equity-oriented social markers and vulnerable settings at the province and territory level (Fig. 1). Scores were lower for equity data reporting across health regions and local areas (average scores of 16% and 15%, respectively, for Canada overall).

By age and sex

Information on population sizes by age and sex overall, and for regions and local areas is made available by Statistics Canada, through the Canadian Census. Of all the social markers studied, age and sex were the characteristics for which COVID-19 data reporting was most common. All provinces except for New Brunswick reported on cases’ age and sex distribution at the population level (Fig. 4). None of the territories reported on cases’ age or sex (Fig. 4). British Columbia, Alberta, and Ontario were the only three provinces that provided age and sex characteristics of all cases, in a case-delimited format.

Fig. 4.

Reporting on the cumulative total of tests, cases, hospitalizations, and deaths by social markers and settings. BC, British Columbia; AB, Alberta; SK, Saskatchewan; MB, Manitoba; ON, Ontario; QC, Quebec; NB, New Brunswick; NS, Nova Scotia; PE, Prince Edward Island; NL, Newfoundland and Labrador; NU, Nunavut; NT, Northwest Territories; YT, Yukon; LTC, long-term care settings

In contrast, age- and sex-related information was sparser for testing, hospitalizations, and deaths across all jurisdictions overall and by health regions. Only Ontario consistently reported on all data elements by age and sex at the overall provincial and health region level (Fig. 4).

Sex disaggregation was mostly done according to designations of “male” and “female.” However, Ontario also included categories of “Other” and “Unspecified” and British Columbia included the category “Unknown” (eTable 2). Age disaggregation was mostly done for 10-year age groups—consistently from ages <20, 20–39, 40–49, etc. to 80 years and above (eTable 2). New Brunswick, Quebec, Manitoba, Alberta, and British Columbia also included categories for those under 10 years—with Alberta also reporting on data for additional age ranges of <1, 1–4, and 5–9 years (eTable 2).

Immigration, race/ethnicity, income, education

Though information on population sizes by immigration status, race/ethnicity, income, and education is available through the Canadian Census for jurisdictions overall, as well as by region and local area, no province or territory reported on any of the data elements according to these social markers (Fig. 4).

Healthcare worker status and occupational sector

At the overall provincial level, British Columbia, Alberta, Saskatchewan, Manitoba, and Ontario provided information on cases among essential healthcare workers—with Alberta and Ontario providing this information at both the province and territory and regional levels (Fig. 4). However, the total number of healthcare workers at the provincial or territorial level, or by region or local area, was missing for all jurisdictions.

Only Ontario provided details on the number of cases across occupational sectors beyond the healthcare setting (e.g., farm, food processing, retail) (eTable 1).

Vulnerable settings (long-term care and detention facilities, homeless shelters, schools)

Ontario, Quebec, Newfoundland and Labrador, British Columbia, Saskatchewan, and Manitoba reported on the number of cases associated with long-term care (LTC) facilities—with British Columbia, Ontario, and Quebec also reporting on deaths within these facilities. Ontario and Quebec listed precise facilities (that could be geolocated in local areas) that had or were experiencing outbreaks. Saskatchewan provided current case counts for precise LTC facilities on a weekly, rather than cumulative, basis. In Ontario and Quebec, the total number of tests and hospitalizations recorded for patients or staff in these settings was missing. Only Ontario provided data on tests within LTC facilities, as well as the total number of beds per facility experiencing an outbreak—which could be used as a proxy for patient population size.

Quebec and Ontario were the only provinces that reported on the total number of cases for each provincial detention facility. Of these two, Quebec was the only province to report on the total number of tests, deaths, number of prisoners, and cases among staff per facility. Missing, however, was information on cases’ potential hospitalization status.

For school settings, only case totals were reported in Ontario and Quebec. Ontario was also the only province to explicitly report on case totals associated with homeless shelters.

Discussion

This paper provides the first summary of health equity-related COVID-19 data reporting in Canada within the first year since the first COVID-19 case was identified in Wuhan, China. In Canada, information on cases and deaths was more complete than that for tests, hospitalizations, and population denominators for all indicators. Jurisdictions tended to report more completely on overall statistics than on information for regions or local areas, or according to population subgroups. The scan suggests that large gaps in reporting remain, even for more standard social disaggregation markers such as age and sex. Though relatively uncommon across the country, certain “best practices” in reporting emerged. For example, three provinces (Alberta, British Columbia, and Ontario) provided case-delimited data on cases for external users to study. Alberta and Ontario provided case-delimited data that included cases’ local area-level identifiers, enabling localized spatial analyses. Almost half of the provinces provided case counts associated with LTC settings, with Ontario and Quebec listing individual facilities that had or were experiencing outbreaks in the province—which can enable the precise geo-location of facilities within neighbourhoods, for use in socio-spatial analyses of transmission risk. Ontario stood out in its reporting of cases across the largest range of equity strata, including across schools, homeless shelters, and occupational sectors outside of healthcare. Last, though Ontario and Quebec both provided details on cases within provincial detention facilities, Quebec was alone in providing detailed information on COVID-19 tests, deaths, prisoner population size, and cases within staff populations per detention facilities. These examples set important precedents and guidance for other jurisdictions to follow, especially as emerging evidence suggests that if COVID-19 outcomes are properly examined across population subgroups, underlying inequities can be revealed and addressed (Chung et al. 2020; Toronto Public Health 2020).

Heterogeneities in reporting observed across Canada are aligned with previous findings that public health surveillance infrastructures and capacities tend to vary across jurisdictions in Canada—which had been identified as an area of concern for pandemic planning and preparedness following the SARS outbreak in 2003 (Naylor 2003). This variability in resources across jurisdictions may limit capacities to collect necessary social data and report on findings across settings or social markers. Since the availability of individual-level equity-related data would typically be obtained at testing, during case interviews, or via data linkage to existing administrative databases, several situations may jeopardize equity-related data collection and reporting. For example, there may be provinces or territories for which these types of databases may not exist or be limited in scope, or for which COVID-19 testing intake or case interview questionnaires are missing items on social, economic, or occupational characteristics. When faced with elevated testing and caseloads and limited time to collect and report on a wide range of indicators, lower personnel capacity can require the prioritization of a subset of case interview elements that exclude equity-related items. For settings such as long-term care homes, detention facilities, schools and homeless shelters, collection and reporting on cases and population sizes requires intersectoral collaboration and communication between public health and other governmental sectors, as well as with the private or community sectors.

To overcome these challenges, formal exchange of promising practices, through meetings at federal and provincial and territorial levels, between public health, healthcare, and community-level stakeholders may be beneficial. These can foster communication on existing barriers to equity-related data collection and reporting, as well as promising practices—be it on equity-related data collection and reporting guidelines (Government of Ontario 2019), questionnaires for social data collection (Agic et al. 2013), intersectoral collaboration strategies, or data communication. National guidance can be developed to ensure the quality and comparability of data collected across jurisdictions, which respects fundamental principles such as those pertaining to Indigenous data sovereignty. An example of this is the Canadian Institute for Health Information’s Proposed Standards for Race-Based and Indigenous Identity Data Collection and Health Reporting in Canada (Canadian Institute for Health Information 2020). Indeed, the Public Health Agency of Canada’s COVID-19 Case Report Form was updated in October 2020 to include more detailed items on cases’ dwelling type (including correctional facility, long-term care, homeless shelter), race/ethnicity, temporary foreign worker status, and occupation, which can be used across provinces and territories for more detailed reporting (Public Health Agency of Canada 2020b). Additional equity-informed elements (e.g., education level) could be added to this case report form or those for use in future pandemics. It will also be essential to assess how these guidelines and tools are applied across jurisdictions, to allow for national reporting on social inequalities.

In the context of other major public health phenomena, such as rising diabetes rates in Canada, successful strategies to guide public health practice included the development of a national action plan, which yielded the development of performance measures, knowledge mobilization strategies, funding mechanisms to support research and community programs, a joint surveillance plan with First Nations, Métis, and Inuit partners, and feasibility studies of the use of health administrative data linkages for surveillance (Public Health Agency of Canada, Health Canada, & Canadian Institutes of Health Research 2013). Similar steps may also be fruitful for determining what data should be collected, and how they can be collected and shared to improve equity-informed reporting and decision-making on COVID-19 and other health phenomena.

The scorecard approach presented here can be used for continued assessments of COVID-19 surveillance reporting or adapted for use in future infectious disease outbreaks. The COVKID scorecard database has been updated twice (August and May 2020) (Pathak et al. 2020). An update of a scan and scorecard such as the one presented here would be beneficial every year, at minimum, which corresponds to the frequency of reporting for infectious disease-related outcomes and targets in Canada (Public Health Agency of Canada 2020c). Updating the scan as of December of every year would provide a useful portrait of reporting progress since the first cases of COVID-19 were identified on December 31, 2019 (World Health Organization 2020).

However, the scorecard approach used has certain limitations. For one, a restricted list of social marker indicators was used. Future expanded versions of an equity-oriented scorecard could assess COVID-19 outcome reporting according to indicators such as preferred language, year of immigration, disability status, sexual orientation, household crowding, gender, or Indigenous identity (Agic et al. 2013). Second, by evaluating provincial or territorial reporting, this scorecard assessment did not address more detailed reporting efforts in specific public health units within jurisdictions. For instance, detailed neighbourhood-level reporting efforts have been made by Montreal Public Health (Direction de la santé publique de Montréal 2020) and several public health units in Ontario (Public Health Ontario 2020b), including Toronto Public Health’s reporting on cases by income and race/ethnicity (Toronto Public Health 2020). In Manitoba, First Nations partners have collaborated to provide a report of tests, cases, hospitalizations, and deaths among First Nations people living on and off reserve (Manitoba First Nations COVID-19 Pandemic Response Coordination Team 2021). The present scan was restricted to provincial and territorial reporting to contrast among jurisdictions on a national scale. Future scans of best data collection and reporting practices across levels of governance may be warranted. Further, this scan excludes information sharing by federal bodies, such as the Correctional Service of Canada’s reporting on cases within federal penitentiaries (CSC 2020). Future assessments of federal-level reporting may also be warranted.

Conclusion

Though several “best practices” in health equity-oriented reporting were observed in Canada, equity data reporting is sparse and large gaps remain. Since jurisdictions that have explored potential social inequities in COVID-19 indicators have found stark gradients in outcomes across individual- and local-area level characteristics, the absence of reporting of data according to vulnerable settings or social markers may be concealing broader COVID-19-related inequities in Canada. The proposed scorecard format and examples of “best practices” identified herein can be used to guide surveillance and reporting during this pandemic and in future ones, and monitor progress on health equity-informed reporting overall.

Author contributions

AB, AP, and AS designed the scorecard framework. KW, HN, WB, and AB performed the environmental scan and collected the data for the study. AB drafted the manuscript, which was revised by AS, AP, KW, HN, and WB.

Funding

AB receives postdoctoral funding from the Fonds de Recherche du Québec-Santé. AS is supported by the Canada Research Chair in Population Health Equity.

Data Availability

Results are summarized in the Supplementary File (eTable 1).

Code Availability

Not applicable.

Declarations

Ethics approval

None required.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Conflict of interest

The authors declare no competing interests.