Multi-province epidemiological research using linked administrative data: a case study from Canada

Abstract

Background

Canada has a publicly-funded universal healthcare system with information systems managed by 13 different provinces and territories. This context creates inconsistencies in data collection and challenges for research or surveillance conducted at the national or multi-jurisdictional level.

Objective

Using a recent Canadian research project as a case study, we document the strengths and challenges of using administrative health data in a multi-jurisdictional context. We discuss the implications of using different health information systems and the solutions we adopted to deal with variations. Our goal is to contribute to better understanding of these challenges and the development of a more integrated and harmonized approach to conducting multi-jurisdictional research using administrative data.

Context and model

Using data from five separate provincial healthcare data systems, we sought to create and report on a set of provincially-comparable mental health and addiction services performance indicators. In this paper, we document the research process, challenges, and solutions. Finally, we conclude by making recommendations for investment in national infrastructure that could help cut costs, broaden scope, and increase use of administrative health data that exists in Canada.

Conclusions

Canada has an incredible wealth of administrative data that resides in 13 territorial and provincial government systems. Navigating access and improving comparability across these systems has been an ongoing challenge for the past 20 years, but progress is being made. We believe that with some investment, a more harmonized and integrated information network could be developed that supports a broad range of surveillance and research activities with strong policy and program implications.

Introduction

The Canadian healthcare system is a universal one that is mandated by the Canada Health Act 1984. Although the system is funded at both the provincial and federal government level, responsibility for managing healthcare delivery falls to the provinces and territories (P/Ts). Each P/T, in turn, has developed its own information systems to assist in managing and disbursing payments related to the delivery of healthcare in its jurisdiction. Administrative data are generated at every encounter with the health system and, although these data are generated for billing and administrative purposes, they can be used to study (and thus, improve) the system of care (1).

Overall, healthcare administrative data are population-based, timely, generally accessible, and offer a large sample size and broad jurisdictional coverage so they can be used to measure performance of the healthcare system and, for many health conditions, the health of the population. The most sophisticated systems exist in Nordic countries, where the centralized government-funded health systems and history of record-keeping and individual linkage has led to nation-wide, linked population-based registries (2). For example, the Danish National Patient Registry (DNPR) has complete coverage of all Danish hospitals since 1978 and includes diagnoses, treatments and examinations (3). Furthermore, the Danish government has compiled over 200 databases with information on socioeconomics, medical records, employment, and more – all of which can be linked by a 10-digit personal identification number (4). These data systems allow for studies of healthcare utilization and how healthcare planning may affect patient outcomes; they can support evidence-informed clinical and policy decision-making.

Canada’s federal structure creates complexity for nation-wide data collection and reporting. Although aspects of the healthcare system differ by province, common methods in data collection within the national setting allows for reasonable comparisons across jurisdictions (5). Some health administrative data, such as inpatient hospitalizations, are standardized across most provinces and territories (1). For example, discharge abstract databases (DAD) are created in all Canadian hospitals and include clinical and demographic information from charts after a patient has been discharged (includes deaths, sign-outs and transfers) (6). The Canadian Institute for Health Information (CIHI) houses this information for most of the Canadian population outside of Québec (7). In Québec, acute inpatient separations are reported to the Hospital Morbidity Database (HMDB). The National Prescription Drug Utilization Information System Databases (NPDUID), which is managed by CIHI, includes pan-Canadian information on public drug benefit plans. The National Ambulatory Care Reporting System (NACRS) contains data for hospital-based and community-based ambulatory care at the national level including day surgery, outpatient clinics and emergency departments (8). However, there are some major gaps in provincial coverage for hospitals reporting to NACRS; reporting is mandated province-wide in only Ontario, Alberta and Yukon, but the number of hospitals included is steadily increasing each year (7, 9).

While robust hospitalization data are available at the federal level, most health services are delivered in ambulatory care settings. The nature and extent of these administrative data vary by province or territory, making inter-jurisdictional comparisons difficult (10). All provinces have a fee-for-service billing system that provides information on the services provided by physicians. Information in these claims tend to be similar across provinces but there are important variations. For example, British Columbia has PharmaNet and Manitoba has the Drug Program Information Network (DPIN), both of which include province-wide information about all pharmaceutical dispensations, while most other P/T’s prescription data are restricted to those covered by a public drug benefit plan. Ontario collects more detailed information on psychiatric hospitalizations through their Ontario Mental Health Reporting System (OMHRS) database, but this creates comparability issues because in all other provinces, psychiatric hospitalizations are collected in the DAD.

Availability of data for health system performance analysis and cost to access such data can vary widely across provinces. Wait-times for access to data range from months to years, and costs range from zero to tens of thousands of dollars for the same type of data request (11, 12). Data quality checking also varies by province, with wide variation in the capacity for these and other data management activities (12). Because of these barriers and inconsistencies many researchers avoid cross-jurisdictional comparative research or alternatively use data from only a few provinces, or data that are several years old (11). Researchers have documented the need for innovative thinking to help investigators tap into the potential of Canada’s rich sources of administrative health data (11).

A range of research projects have been conducted using administrative data in Canada during the past few decades (11, 13). For example, Grundfeld et al.(14) tested the feasibility of using administrative data to measure quality indicators for end-of-life care in two Canadian provinces. The study involved using comparable provincial databases including physician claims, hospitalizations, prescriptions, and palliative care databases. Out of a list of 19 possible indicators, seven were determined to be feasible, valid and comparable. O’Donnell et al. (15) assessed the feasibility of using administrative data in Canada to compare prevalence of osteoporosis and incidence rates for related fractures across five provinces. A validated algorithm showed consistent age and sex patterns across provinces, and proved suitable for national monitoring of diagnosed osteoporosis. Vasiliadis et al. (16) obtained aggregate data from four Canadian provinces’ health administrative databases to establish the yearly prevalence and incidence of treated attention-deficit hyperactivity disorder (ADHD).

While a handful of studies have tested the feasibility of data harmonization and validation for inter-provincial research on a host of health-related issues, very few studies have documented in detail the process, challenges, and solutions required to achieve their results. This can result in inefficiency for subsequent projects as research teams re-engineer solutions to problems that have previously been addressed. In Canada, research using data sources from multiple P/Ts would allow for a process of developing standardized data by revealing differences in data collection and seeking solutions to address these. There are now several suggestions to create pan-Canadian research and surveillance infrastructure in Canada (12, 17, 18). It is therefore essential to document lessons learned to guide the focus and priorities of infrastructure and data policy development.

In this paper we describe a multi-jurisdictional project that examined mental health and addiction performance measures in five Canadian provinces (accounting for approximately 90% of the total Canadian population): British Columbia (BC), Alberta (AB), Manitoba (MB), Ontario (ON) and Québec (QC) (19). We briefly outline the five stages of the feasibility project, documenting some of the major challenges and complexities. Our hope in discussing these is to inform a better understanding of the problems inherent in multi-jurisdictional administrative data research, and contribute to the development of a more harmonized and integrated cross-provincial administrative data research and surveillance environment in Canada.

About the research study

Using data from five provincial administrative healthcare data systems, we sought to test the feasibility of generating provincially-comparable mental health and addiction services performance indicators. A team of scientists from all five provinces developed and generated the measures, where possible, for individuals ages 10 years and older. The Centre for Applied Research in Mental Health & Addiction (CARMHA), a research centre based at Simon Fraser University in BC, coordinated the project.

The study was conducted in five steps: (1) refinement of the research questions to decide on a set of indicators to be measured across provinces; (2) iterative development of a detailed specifications document that could be used by each province; (3) applying for, and obtaining, data within each province; (4) analysis of the data by each province in a manner consistent with the specifications document; and (5) compilation and interpretation of data across provinces, in a manner acceptable to the researchers and other stakeholders involved in the project. The final report, published in 2017, presents comparative results for six mental health indicators by province (19). To our knowledge, this was the first project to have reported extensive and comparable data on the performance of mental health services in multiple provinces in Canada.

1. Deciding on the indicators

The goal for this step was to find indicators that were of interest to the mental health community and could be constructed from existing data. The initial set of indicators was developed by CARMHA in collaboration with scientists from the Institute for Clinical Evaluative Sciences (ICES) in Ontario. It was not the aim of this study to develop a comprehensive list of indicators, but rather to test the feasibility of measuring indicators from administrative databases. Investigators in each province reviewed the indicators to determine feasibility of analysis within their jurisdiction. This process resulted in the following six indicators:

1. Access to the same primary care physician over time for people diagnosed with a mental disorder or addiction.

2. First treatment contact for a mental disorder or addiction was in an emergency department.

3. Physician follow-up after hospital discharge for a mental disorder or addiction (within 7 and 30 days).

4. Rates of suicide attempts among people diagnosed with a mental disorder or addiction.

5. Suicide rates among people diagnosed with a mental disorder or addiction.

6. Mortality among people diagnosed with a mental disorder or addiction.

A seventh desired indicator presented a specific challenge. For this indicator we wanted to examine the proportion of individuals who had received treatment concordant with clinical guidelines for depression (e.g. guideline-concordant prescription of medication). Guidelines for depression exist but they do not translate easily to measures that can be obtained from administrative data. Even if a measure could be derived (e.g. 4 follow-up visits in a year after a depression diagnosis) the application of a specific guideline to cases drawn from administrative data based on diagnostic code(s) was problematic. The diagnostic coding for depression varies drastically across provinces in Canada (20), raising significant methodological and interpretability issues related to the measurement of guideline-concordant care. The team decided not to pursue this indicator but documented it in the technical report to highlight the difficulties encountered in doing this kind of cross-jurisdictional research, and to make the case for improving performance measurement in the mental illness primary care sector. The other six indicators, while imperfect, turned out to be reliably measured and comparable across provinces.

Data for the six indicators came from provincial physician billing claims, in-patient hospitalizations, emergency department visits, and death registers.

2. Development of a specifications document

Once the initial suite of indicators was decided, the next step was to define the process by which these indicators could be measured in each of the provinces. An indicator specification working group was set up which included at least one representative from each of the five provinces with detailed knowledge of their provincial data systems and sources. The project coordinator organized meetings and took on documentation responsibilities. The goal was to develop a detailed specifications document that could be used by all sites to guide their data abstraction, programming, methods, and reporting format so that they were conducted in a similar manner.

We began with a set of analytic plans that had been developed by ICES to evaluate the six indicators. However, it quickly became apparent that these could not be operationalized, in identically the same way, across all provinces primarily due to data differences, and that adjustments and work-arounds were needed in most provinces. Regular working group meetings were critical for identifying and resolving issues associated with indicator specifications. For example, one of the indicators measured whether first contact for a mental disorder or addiction was in an emergency department. It was decided that the indicator numerator would include individuals treated for a mental disorder or addiction in an emergency department who had not been seen by any other service provider in the previous two years. It was determined by the working group that the provinces’ ability to measure “any other service provider” varied considerably. For example, Alberta could include mental health teams and community mental healthcare (with additional compilation of data), but the other four provinces could not. As such, the numerator data sources were restricted for provincial comparability.

Another limitation was that Québec, Ontario and Alberta were the only provinces with complete provincial coverage for emergency department data. The suicide attempts indicator included only those who had been hospitalized for the attempt, due to the extreme variation in emergency department data access and coverage. This indicator would benefit greatly from the inclusion of emergency department data and the hope is that highlighting this limitation will lead to further adoption of the National Ambulatory Care Reporting System (NACRS) (8).

Details of each decision were recorded by the national project coordinator and noted in a revised version of the specifications document that was recirculated to the group after each meeting. The result of this process was a specifications document that described the indicators in detail, the steps to be used to abstract the data for the numerator and denominator, and the format of the tabulation to be used for analyzing and reporting the results. In other contexts this process has been referred to as developing a common analytic protocol (18). Table 1 presents the provincial coverage for each of the six indicators. The lesson learned was that, while it would have been convenient if we could simply apply the indicators developed in one province to all the others, this was not feasible.

Table 1: Provincial data available for each indicator.

Indicators	BC	AB	MB	ON	QC
Regular access to a primary care physician	✓	✓	✓	✓
First contact was in an emergency department	✓	✓	✓	✓	✓
Physician follow-up after hospital	✓	✓	✓	✓	✓
Suicide rate	✓		✓	✓	✓
Suicide attempts	✓	✓	✓	✓
Mortality rates	✓	✓	✓

External consultation was sought, which helped to strengthen the project documentation. The work was reviewed by CIHI’s Indicator Methodology Committee and the Michael Smith Foundation for Health Research (MSFHR). The team also reported to an Advisory Committee which included representatives from provincial governments, national agencies, and advocacy groups. The Advisory Committee provided important feedback for knowledge translation (KT) activities and strategies to promote active engagement in indicator uptake at the policy and practice levels.

3. Data access processes

Each provincial team lead was responsible for preparing and submitting their own provincial application for data which included ethics approvals, data requests, liaising with their provincial data stewards, running the analysis and transferring the aggregate data to CARMHA. In current legislative interpretation for some provinces, administrative data cannot cross provincial jurisdictional boundaries, requiring that linkage and analyses take place province-by-province (12). Early in the project, the national coordinator asked teams to complete a brief questionnaire to assess the status of data accessibility and document some of the foreseeable challenges. This included questions about the current political context, recent mental health indicator activity, process for obtaining and analyzing administrative data, and potential challenges. This information was used to help estimate time and costs required for each provincial application for data.

British Columbia, Ontario and Manitoba were experienced in handling administrative data requests, and so the path forward for these provinces was relatively predictable. Population Data BC, the Manitoba Centre for Health Policy, and ICES in Ontario are coordinating agencies with the mandate and authority to receive data from the provincial governments and provide access for researchers. These agencies have personnel with data expertise and comprehensive knowledge of the provincial data sources; the ability to determine scientific validity of study designs; provision of education, training, guidance and support for researchers developing data applications; a method for transfer of the data or a secure research environment for data analyses; and a standardized access procedure for applying for the use of data.

Alberta was in the early stages of developing capacity for assistance with access and analysis through the CIHR-funded Alberta SPOR (Strategy for Patient-Oriented Research) SUPPPORT Unit (AbSPORU) initiative (21). In this case, the provincial lead worked with the provincial government, AbSPORU data and methods platforms and her affiliated university. AbSPORU is continuing to provide services to researchers on an ongoing basis, including no-fee access for students or research teams without funding (working towards a cost-recovery model). This mechanism made an enormous difference in Alberta’s ability to contribute both data and expertise to the project; but additional infrastructure similar to that in British Columbia, Manitoba and Ontario could help to ensure sufficient and sustainable capacity to optimize use of health administrative data in Alberta going forward.

Finally, Québec had comparable sources of data, but does not currently have infrastructure to systematically coordinate applications for access to linked health administrative data or provide analytic support to researchers who may wish to use the data. In this case, the team worked collaboratively with the Institut national de santé publique de Québec (INSPQ), an organization that had developed an ongoing relationship with the provincial government for their own research and surveillance work. INSPQ was able to provide access to the Canadian Chronic Disease Surveillance System in Québec (QICDSS) (22) which, while comparable to the data sources in other provinces, provided less flexibility and opportunity for data manipulation and analysis.

Time required to access the data varied greatly (from 4 to 9 months), and thus there were sequencing challenges, with some provinces being well into the analysis stage while others were still waiting for data. Cost of data (or analytic support) also varied across provinces, although not as dramatically, from $12,000 – $15,000, which in turn was partly constrained by available project funding.

4. Data analysis

Upon completion of the application processes, each team obtained and prepared the analytic summaries for their own province, based on the common specifications developed by the working group. Access to human resources (analyst time) also varied across provinces and in some cases led to delays in data abstraction. Unanticipated issues did arise despite the comprehensiveness of the indicators specifications document. For example, how to handle physician billing claims made within an episode of hospitalization, or how to suppress small cell entries in summary tables, were not addressed in the early stages. An issue would often be raised by a single jurisdiction, and then solutions needed to be determined and communicated to the other jurisdictions for consistency of approach. As provinces were often at different stages of the analysis process, care was needed to ensure that decisions did not negatively impact work that had already been completed.

5. Compilation and interpretation of the results

Summary results were sent to CARMHA for aggregation into the final report. Borrowing from our previous approach, a data interpretation working group was established to assist with the interpretation of results. The goal of this project was to demonstrate the ability to compare results of mental health indicators across provinces in a meaningful way. This was successfully achieved but an important next step would be to generate hypotheses about what the observed provincial variations represent. Informed interpretation that considers contextual variation is needed for the indicators to have meaningful impact on mental health policy and practice decisions. Examples of contextual considerations include important health service policy and practices changes, transient populations (e.g. Alberta has a high percentage of transient young people due to the oil industry), Indigenous peoples, refugees, and rural service delivery challenges.

Panel 1: Stages of the project process.

Stages	Lessons and Recommendations
1. Deciding on the indicators	Identify one lead agency or organization to ensure national coordination of activities. Establish a Data Specifications Working Group with members from each participating jurisdiction. Experts determining indicator feasibility should be well-versed in their jurisdiction’s administrative data systems, sources, and linkage capacity.
2. Indicators specifications	Develop a detailed indicators specifications document which can be used by all sites to guide data abstraction and analyses. The document should be a “living” document, managed by the lead agency to reflect current consensus decisions and rationales. Seek and facilitate external consultation where possible (e.g. government, research and policy agencies, advocacy groups, etc.).
3. Data access	This process should be led or overseen by the provincial data experts. The ethics and data access applications process differ across jurisdictions. Nonetheless, the team should work towards a common, realistic deadline for data abstraction.
4. Data analyses	Establish clear communication streams between those abstracting and analyzing the data and those who developed the indicator specifications. If one jurisdiction faces an analytic problem that is not covered in the indicator specifications document, take care to assess whether the resolution to that issue will affect the analytic work in the other jurisdictions. Ensure the team has agreed upon a standardized file format to simplify multi-jurisdiction data merging after analyses are complete.
5. Compilation and interpretation of results	Invite health professionals, people with lived experience, and other experts in service delivery and local/provincial policy to assist with data interpretation alongside researchers. A detailed technical report describing the methodology should be sufficient to permit replication of the work.

Key learnings and conclusions

The overall objective of this project was to demonstrate that it is possible to generate mental health and addiction measures from administrative data that can be compared across Canadian provinces. Our experience was challenging because provinces do not have equal capacity for participation in projects like this one; administrative data access procedures, data sources, and availability for research differ substantially. With investment in a broader scheme for sustainable cross-provincial data work, the indicators developed for this report could more easily be repeated over time, which might enhance the impact they have on health service delivery or policy. As stated earlier, the indicators were selected based on feasibility and comparability. It would be ideal to increase the data sources available as well as capacity to collect and report. Some provinces are pursuing promising intra-provincial data linkages to increase the capacity to measure mental health and addictions system performance to better represent the entire mental health sector. Currently however, coverage of data sources is limited, as is access to emergency department and community mental health data. Data standardization would also facilitate inter-provincial comparisons. Superficially, all provinces routinely collect administrative data that reflects the care provided under universal coverage. However, the variations in data revealed by this project underscore the need for agreed standards if inter-provincial performance measurement is to become routine.

This type of research work requires detailed coordination and extensive cooperation from all involved. The provincial teams consisted of members who have expertise in mental health and addictions, a long history of working with administrative data, and, in some cases, experience working together on previous projects or committees. This eased the process of the group coalescing into a team, and by leveraging this experience, we could quickly overcome many of the obstacles that might have stymied a less experienced group. While often not identified, the usefulness of good team dynamics should be stressed as a central component to the success of any such endeavor. Critical to this process was a coordinating site with expertise in both health administrative data and cross-provincial project coordination.

This project demonstrated the value of comparable data infrastructure. While Canada has world-class potential, the Council of Canadian Academics suggested that “this potential is not being realized due to an incoherent maze of rules, procedure and practices and a general tendency of many data custodians to err on the side of caution when it comes to granting access.” (23) Common approaches that clarify responsibilities and streamline processes such as ethics approvals, would improve timeliness. A framework for measuring and scoring data accessibility across the provinces is also recommended.

Building from experience: models supporting data integration

The benefits of large-scale investment in data linkage infrastructure have been shown elsewhere. As a federation, Australia faces similar data linkage and harmonization challenges to Canada. Western Australia has demonstrated laudable progress. The Western Australia Data Linkage System (WADLS) was established in 1995 to link up 40 years of data for over 30 data sources (24). A report that measured the outputs of the WADLS, identified several benefits including adding meaningfully to existing data assets, increasing cost efficiency of health research, conservation of patient privacy (due to reduced need for personal identifying information), community development, contributions to scientific knowledge, and improvements to public health (data-linkage research has been directly linked to reforms in mental health legislation in Australia). The success of WADLS provided the impetus for a $20 million Australian government investment to create the Population Health Research Network (PHRN), established to help facilitate cross-jurisdictional data integration. The national linkage infrastructure has been used to join individual data from multiple datasets and helps account for the 'mobile' population in Australia which crosses jurisdictional boundaries (25, 26). The PHRN is expected to become the world’s largest population database supporting health research, policy and planning (27).

In Canada, there are several models that could provide an example of the real potential for sustainable, accurate, and timely multi-province epidemiological research. The Canadian Chronic Disease Surveillance System (CCDSS), a collaborative P/T network supported by the Public Health Agency of Canada (PHAC), uses linked administrative data sources from every province and territory to estimate the incidence and prevalence of chronic conditions, including mental disorders, and health service use and outcomes (28, 29). Individual-record linkage and analyses is completed by the P/Ts using analytic code developed by PHAC and the aggregate data are collected at the national level (30). In this case, abstracts of the data needed in each province have been combined into a common data model, so that a single software solution can be executed across all provinces. Unfortunately, this national infrastructure is not currently available to the research community, as it is funded and focused on the specific surveillance mandate that it serves. A different approach has been adopted by the Canadian Network for Observational Drug Effect Studies (CNODES), an academically-based network of researchers and data centres from seven provinces and three countries focused on studying the safety and effectiveness of drugs including adverse reactions (18). CNODES develops a common analytical protocol but allows each province or country to adjust it according to the data needs in that jurisdiction. The differences this introduces are both scrutinized and minimized as well as adjusted for in a meta-analytic protocol. CNODES has clearly defined principles of operation designed to achieve high internal validity. The use of meta-analysis to analyze data across jurisdictions within the same research project is a unique approach in our experience.

Many of the disadvantages of multi-province projects using healthcare administrative data, such as potential coding errors and inconsistencies, can be managed by developing national standards and protocols, as well as software tools that are shared for data cleaning and validation. CCDSS and CNODES are promising models, but are still limited in that the networks are focused on a set of specific issues, and the aggregate information collected from the P/Ts may not be suitable for other research purposes. Building on the success of the CCDSS and CNODES, Canada needs a common data model to support a wide range of surveillance and research studies. Such an approach would allow for the development of analytic protocols in one province or territory to be run in other jurisdictions. This would lay the foundation for the expansion of research and surveillance that takes advantage of the rich administrative data available in this country.

Funding Statement

Funding for the project Toward Quality Mental Health Services in Canada: A Comparison of Performance Indicators across 5 Provinces was provided by the Graham Boeckh Foundation, Royal Bank of Canada Foundation, Echo Foundation, Ontario Mental Health Foundation, Mental Health Commission of Canada, and the Michael Smith Foundation for Health Research. In-kind support was provided by the Canadian Institute for Health Information (CIHI) and partner organizations in some provinces.