Health Systems Performance Measurement Systems in Canada: How Well do They Perform in First Nations, Inuit, and Métis Contexts?

Abstract

Objective

To inventory and describe currently available health performance measurement systems for First Nations, Inuit, and Métis people in Canada to identify why current measurement systems are inadequate to inform community or regional level health planning.

Methods

Inventory, classification, and synthesis of strengths and weaknesses among existing health system performance measures through systematic literature review and key informant interviews.

Results

Indigenous-specific health indicators are available at national, provincial, regional, and community levels, but there is a paucity of data for non-registered First Nations, Métis, and Inuit people. Barriers to the effective use of these indicators include: indicator selection driven by accountability rather than public health requirements; poor data quality; inadequate infrastructure and human resources; minimal information returned to communities; tension between agencies collecting universal indicators and Indigenous-determined processes; and, mistrust by Indigenous communities of externally imposed processes. The focus on national systems results in greater attention to disease-based measures and less focus on regional cultural diversity and Indigenous-specific values and priorities.

Conclusion

Indigenous health system performance measurement infrastructure in Canada is underdeveloped, particularly at the local level, and hence deficient in its ability to support community or regional health planning.

Introduction

Levels of ill health among Indigenous communities in Australia, (Research Agenda Working Group [RAWG], 2002; Miller and Torzillo, 1996; Australian Bureau of Statistics, 2001, 2005; National Aboriginal and Torres Strait Islander Clearinghouse, 2000; Australian Medical Association, 2001) Canada (Smylie, 2000; First Nations and Inuit Regional Health Survey, 1999; MacMillan et al., 1996; Department of Indian and Northern Development, 2001), and New Zealand (Public Health Intelligence, 2001; Ministry of Health, 2003) are disturbing from a global health perspective because they are elevated despite the relative affluence and excellent health status enjoyed by the general population of these nations. In Canada, Indigenous people are referred to in the Constitution as “Aboriginal,” thus this is the term used in the majority of government documents and programs. According to the 2006 Canadian census 3.8% of Canada’s population is Aboriginal, indicating ancestry that is North American Indian, Métis, Inuit, or multiple Aboriginal groups (Statistics Canada, 2008). The Aboriginal population of Canada is growing at a faster rate than the general population, and is significantly younger with the average age of Aboriginal people being 27 years compared to 40 years for the non-Aboriginal population.

Currently available First Nations, Métis, and Inuit health indicators highlight striking and persistent disparities in health status when compared to the non-Aboriginal population (Health Council of Canada, 2005; First Nations and Inuit Regional Health Survey, 1999; First Nations Centre, 2005a). Health measurement systems have been ineffective in providing health systems that address these inequities. The situation is complicated by the fact that the organization and delivery of health care varies for different Indigenous groups, as well as within groups, depending on place of residence. All secondary and tertiary level care is delivered by the provincial/territorial governments, but primary care and public health is funded by First Nations and Inuit Health Branch (FNIHB) of Health Canada to registered First Nations and Inuit people living in First Nations or Inuit communities. Communities have varying levels of control over how care delivery based on individual transfer agreements with FNIHB. Primary care and public health is delivered by the province/territorial governments for all non-registered First Nations and Métis people, as well as registered First Nations and Inuit people living outside of First Nations or Inuit communities. Thus health measurement, planning, program and service delivery, and evaluation are carried out at multiple levels by multiple jurisdictions.

The Canadian Institute for Health Information has developed a Health Indicator Framework which guides the development and organization of their annual reports (Canadian Institute for Health Information, 2008). The sections of the framework include health status, nonmedical determinants of health, health system performance, and community and health system characteristics. While we are interested in all of the aspects of this framework and how it performs with regards to Indigenous people, particular focus will be on the health status and nonmedical determinants of health, core to population health assessment and community or regional level primary care and public health planning.

The purpose of this study was to identify why current health measurement systems are ineffective in contributing to Indigenous health planning at the community or regional level. Specifically we sought to understand the underlying goals, current applications, data sources and gaps, as well as the barriers to more effective application of current health system performance measures.

Methods

The research questions to be addressed included:

• What Indigenous health indicators and measurement systems are currently in use?

• What are the underlying goals of existing measures/measurement systems?

• How do these currently used indicators and systems relate to Indigenous understandings of health?

• How have Indigenous people been involved in their development and use?

• What are the gaps and barriers to the application and use of current indicators and systems?

Information was collected through systematic literature review, key informant interviews, and consultations with leaders in the field. The Canadian study was nested in a broader study of Indigenous health measures that included parallel assessments of Indigenous health measurement in Australia and New Zealand (Smylie et al., 2006).

Published references were identified by searching Medline, Embase, Cumulative Index to Nursing and Allied Health Literature, Allied and Complementary Medicine, the Cochrane Database, and social science databases (Expanded Academic ASAP Plus, Sociological Abstracts, Science Citation Index Expanded, Social Sciences Citation Index, and Arts and Humanities Citation Index). The following MeSH headings were used: “health surveys” or “health indicator” and “community health services” or “community-based” and “American Native Continental Ancestry Group” or “health services, indigenous” or “aborigin$ or indigenous or Inuit or Eskimo or first nations” and “Canada.” Ancillary and unpublished references were identified through review of article references, website reviews of health agencies, and by recommendation of experts identified by the team of Canadian investigators. Identified articles were reviewed according to the following criteria:

• What health indicators or health measurements were used?

• What populations are included in the measurements (registered or non-registered First Nations, Inuit, Métis)?

• Who designed or selected the health indicators used?

• Was the local Indigenous community involved in any aspect of defining, collecting, or analyzing the health indicators?

• Were the results used by the local Indigenous community in any identified way?

Eleven key informants were deliberately sampled to represent First Nations, Inuit, Métis, non-Aboriginal government, and nongovernmental organizations. Informants were sampled from national, provincial, regional, and community levels. Some informants represented more than one of the perspectives we sought to obtain. The main criteria for selection included: (1) being employed by one of the sectors previously mentioned; and, (2) working specifically in the area of First Nations, Inuit, or Métis health or health measurement and thus knowledgeable to discuss the subject area. The interviews were performed either face-to-face or via telephone, and were recorded and transcribed. Interviews were semistructured. The interview guide is found in Table I.

Table I.

Questions Asked of Key Informants

	Key Informant Questions
1.	What community health system performance measurement systems do you currently use/are you developing? Probes:
	What are the goals/objectives of this system?
	What are the benefits of the system?
	What are the problems with the system?
	How were Aboriginal individuals or communities involved in the development of the system?
	How do the indicators relate to local Indigenous perspectives of health?
	How has/does the system contribute to capacity building at community level?
2.	What individual and community health status measurement systems do you currently use/are you developing?
3.	What about measures of wellness or well-being?
4.	What about broader determinants of health? (i.e., housing, education)

Transcripts of the key informant interviews were thematically analyzed (Smylie et al., 2008; Bell, 1999) by Marcia Anderson, Janet Smylie, and an additional external researcher. Each reviewer analyzed the transcripts independently and identified emergent themes. The three reviewers then met to review and group their themes. A group consensus regarding the key themes was quickly achieved and this preliminary coding schema was applied by Marcia Anderson in a more detailed analysis of the transcript texts. The final categorical coding schema with examples of supporting text is found in Table 2.

Table 2.

Coding Schema with Examples of Supporting Text.

Theme	Examples of Supporting Text
1. Currently collected data is driven by accountability requirements.	“They always have the data to justify why to give them the money but there’s no real plan to actually use the data for anything real.” “Some of the regional type data may be based on accountability documentation that is sort of a requirement of regional health authorities…because government is spending, they’re trying to tie to indicators and accountability indicators.” “Public health information has not been clearly defined. Even at the headquarters level or at the regional level. So because of this vacuum of expertise in public health we haven’t really focused on public health indicators that we need to do public health.”
2. The quality of existing data is poor.	“We have very little data, basic data, vital statistics, denominators, birth, death. We have very, very little data and it’s probably not usable.” “There is data we don’t have that, because it doesn’t exist, or if data exists and it’s not valid, or it’s not complete, or it’s not usable, or it’s not pertinent. There is data that is complete and valid somewhere but it’s not aggregated. There’s data that is aggregated but not analyzed. There is data that’s analyzed but not interpreted, and there’s data that’s interpreted but not fed back to the communities to contribute to their needs assessment process.”
3. The availability of data is limited by the lack of opportunity to self-identify when encountering the health care system.	“They said it’s too difficult; it takes too long to go through the process, to negotiate the data linking…. But you could actually change all that with including some ethnicity identifiers within all databases.” “The other issue is the limited opportunity for Metis to self-identify as Metis.”
4. Data quality is affected by the multijurisdictional nature of health care for Indigenous peoples.	“We don’t have links to vital statistics, to the provincial systems and that’s a very, that’s a very big problem for us.” “Gaps and barriers. Well, for Metis, of course, there’s jurisdictional, jurisdictional barriers being primary.”
5. There is a lack of infrastructure at all levels to support data collection.	“The accountability is based on a lot of process measures that don’t really guarantee that we’re going to obtain the objective of reducing mortality or incidence rates from injuries. And in the middle, all the epidemiologic data and the data systems that have to be in place, and the collection systems, That is very underdeveloped and usually unfunded.” “The second one [barrier] would be, as I mentioned, infrastructure and resources and funding.”
6. Inadequate human resources limit the ability to collect, analyze, and respond to Indigenous-specific health indicators.	“My experience is you know, you have a three nurse station and you’re happy to have two…You know, all the time you hear of public health? Well that takes second fiddle to the urgent needs of primary care.” “Most communities, that’s [aggregated data] what they would be interested in because they wouldn’t have the capacity to manage the record stuff anyways.” “All of that used to [go to] FNIHB and they were supposed to give us back the information in a timely fashion. But over the last two years they don’t, they say they don’t have enough staff to input the data so it just kind of sits in boxes.”
7. There is a lack of information being returned to communities to inform public health planning or service delivery.	“The reports are to reflect the needs of the organization, period…. Now the data, any health related data doesn’t go to communities. Communities have no access to data.” “Another challenge in some of the analysis of the national surveys are the sample size…For some of the sub-analysis you have to have a population of 75,000 or greater for that analysis to occur. But we’re bulked in with [another larger community] so it doesn’t provide us really a very valuable tool in the long run.”
8. Tension exists between agencies that collect universal health indicators and Indigenous-driven or owned processes.	“We say we provide the data and remain as objective as possible…The Stats Act [states that] the data collected is the property of the Crown, and that’s, yes it is a subject of controversy between Aboriginal people and Stats Canada.” “One of our big projects over the next while is going to be to sell the idea of the project t those other departments, and, you know, we’ve been trying to get buy-in … but the other departments have not been supportive, so it’s very frustrating. They’d prefer, generally, to go with Stats Can. Stats Can has the benefit of being in the same boys club.”
9. There is a lack of trust for processes that are derived external to the communities, and concern for how data will be used if the principles of ownership, control, access, and possession are not respected.	“And to me, there’s a problem of trust… There’s a problem of trust between governments and First Nations.” “With increased trust in the national process after the first round [of an Indigenous health survey], which like I said was very regional, it’s possible to do bigger national components.”
10. There is a need for culturally appropriate health measurement tools that are not pan-Indigenous.	“If we want to reflect something that’s Indigenous, than we need to look at, well, what are Indigenous ways of living and try to capture that so that we actually can see that…. And currently that doesn’t exist. Everything is a reflection of the colonial system.” “First we need to develop some First Nations health indicators and talk with our communities about what they think key indicators of their health would be.” “There needs to be an investment to bring Metis health infrastructure, programs, and services on par with those that are currently available to First Nations and Inuit.”
11. There is a desire to use a population health approach with a community level focus.	“We attempt as best we can to adopt both a holistic approach and a population health approach within the work that we do … [but] most of the data is, you know, is I would say, the data is more indicative of individual health than it is of population.” “The organization as a whole, in terms of the Center…our mandate is to gather and share Metis population health information.”
12. There is a preference for community initiated and driven processes that adequately address the need to build public health capacity.	“We developed it as a community template that communities can use and our regions can use to report on their health, and then hopefully they can negotiate with the provinces and with FNIHB (First Nations and Inuit Health Branch).” “I think that by gathering the information I think it’s such a wonderful, it’s an education for the communities as well as a wonderful planning tool. And I think that as we gather more information that is community driven I think it becomes much more meaningful for them, and hopefully between all of us we can improve the health status of the communities.” “[Another need] is capacity building at the community level in that lack of infrastructure funding and resources are primary health issues.”

This project was reviewed and approved by the Research Ethics Board at the University of Ottawa. It was funded by an operating grant from the Canadian Institutes of Health Research. Dr. Smylie was supported by a CIHR-Associated Medical Services Inc–Institute of Aboriginal Peoples’ Health Senior Research Fellowship for the duration of the study. The CIHR had no role in the design or conduct of this study.

Results

Inventory of Currently Available Health Measurement Systems

Health measurement systems containing Aboriginal-specific data were identified at the national, provincial, regional, and community levels. National indicator sets are found within Statistics Canada (2001a, 2001b, 2005), Canadian Institute for Health Information (2005a, 2005b), Public Health Agency of Canada (2005), Health Canada (2002), FNIHB (2005a, 2005b, 1999), Indian and Northern Affairs Canada (INAC, 2005a, 2005b, 2005c), First Nations Centre (2005a, 2005b, 2005c), Métis Centre (2005) and national Aboriginal organizations representing First Nations, Inuit, and Métis people (Inuit Tapiriit Kanatami, 2005). Most provinces/territories do not have standardized methods to collect ethnicity data in their vital statistics and health care utilization databases, but some have linked their health care databases to lists of subpopulations of Aboriginal people including the Indian Register maintained by INAC, regional Métis council membership lists, or band council lists to generate Aboriginal-specific data (Martens et al., 2002; British Columbia Provincial Health Officer, 2002; Kliewer et al., 2002). Subprovincial regional health information can be generated by tribal council (Prince Albert Grand Council, 2005), or by geography in areas such as northern Saskatchewan which are known to have high population proportions of Aboriginal people (Irvine et al., 2004). Community health measurement systems tended to be either research project-based (Grafton, 2001; Parlee and Marlowe, 2001), or were drawn from the required reporting for specific programs as determined by program funder. Table 3 describes sources and coverage for select health indicators.

Table 3.

Examples of Indigenous-specific Coverage for Certain Health Indicators (complete compendium available in project background paper, Anderson et al., 2005a).

Indicator	Source	Coverage
Population	Band lists, Census data, Statistics Canada	Registered and nonregistered First Nations (FN) people on and off reserve, Inuit urban and remote, and Métis
Language use — Aboriginal language	Aboriginal Peoples Survey (APS) First Nations and Inuit Regional Health Survey (FNIRHS)	Registered and nonregistered FN (off reserve), Métis, Inuit FN on reserve and some Labrador Inuit communities
Life expectancy	First Nations and Inuit Health Branch (FNIHB) INAC	Registered FN on/off reserve Registered FN on/off reserve
Infant mortality rate	FNIHB in-house data INAC basic departmental data	Registered FN
Crude mortality rate	FNIHB in-house stats	Registered FN
Respondent-reported health of child	FNIRHS	Registered FN on reserve and Labrador Inuit
Disease specific mortality rates for malignancies, acute MI, stroke	Health Canada	Registered FN on and off reserve
Rates of screening e.g. pap smears or mammography	May be available provincially, but not uniformly and require data linkage, e.g., Health and Health Care Use of Manitoba’s First Nations Canadian Community Health Survey*	Registered FN Off reserve Registered and Non-registered FN, Métis and Inuit

^*The CCHS, like most national surveys, excludes sampling from reserves. It asks about the ethnicity of the responder and the options include North American Indian, Métis, and Inuit; however, on the website no Aboriginal-specific data is presented.

Only rarely in the literature was mention made of the involvement of Aboriginal people or communities in defining, collecting, or analyzing health indicators, and few examples were available of how the data collected was used by the local community (First Nations and Inuit Regional Health Survey, 1999; Boyd and Associates, 2002; Abonyi et al., 2005; Parlee and Marlowe, 2001; Grafton 2001; Auer and Andersson, 2001a, 2001b). These were more likely to be found in the grey literature, and were either led by Aboriginal organizations or used a community-based participatory approach. Common elements in these articles included the development of locally and culturally relevant frameworks and/or terms for health measurement; involvement of community members throughout the process; and consideration of community priorities and context. These examples are more fully discussed in the project background paper (Anderson et al., 2005a).

When examining Aboriginal health data it is important to understand which of Canada’s constitutionally defined “Aboriginal groups” the data actually refers to, or covers. As highlighted in Table 3, the most data is available for registered First Nations people living on reserve, with a paucity of data for nonregistered First Nations, Métis, and Inuit people. According to the 2006 Census, registered First Nations living on reserve comprise 24% of the total Aboriginal population, implying the lack of data for over 75% of the Aboriginal population (Statistics Canada, 2008).

Perceived Barriers to Effective Use of Current Health Measurement Systems

Although there are a multitude of Aboriginal health indicators available, they have not been effectively used to improve the health status of Aboriginal people. Reasons for the ineffectiveness were identified from the thematic analysis of the key informant interview transcripts (presented in Table 2). A key problem was that current health indicators have been selected primarily to satisfy fiduciary accountability requirements as opposed to informing public health policy or planning. The utility of currently available health assessment data is limited by its poor quality, which in part is due to the lack of systematic, appropriate, and consistent ethnic identification of Aboriginal peoples in health care data sets (such as national cancer registries or discharge databases used to calculate morbidity rates). Quality is also negatively affected by the multijurisdictional nature of health care delivery for Indigenous people, with a lack of communication across jurisdictions.

There is a lack of both infrastructure and human resources at all levels to support the collection, analysis, and response to Indigenous-specific health indicators. This is related to the next theme, which is that despite submitting much required data, little meaningful and useful information is returned to communities to inform their health planning.

Many Indigenous communities have had negative experiences with academic researchers or government organizations, and this has led to distrust of processes that are developed external to communities. This presents a substantial barrier, as the large majority of existing public health data collection processes have been developed externally. There is concern over how health assessment data will be used if the principles of OCAP² are not respected. Tension exists between agencies that collect universal comparable health indicators and agencies that promote the development and collection of Indigenous-centred and owned health information.

Participants expressed a desire for health measurement systems that would be culturally appropriate, not pan-Indigenous, and incorporate a wellness perspective. They would be based on traditional health frameworks including midwifery, incorporate Indigenous ways of knowing, and focus on community health rather than individual health status. Community initiated and driven processes that build the capacity of communities to define, collect, analyze, and respond to health information are preferred.

Discussion

This study has shown that although Aboriginal-specific health indicators are available, they are concentrated at the national level, and are either pan-Aboriginal or registered First Nations specific. The themes identified above demonstrate why the use of current Indigenous health indicators has not effectively translated into programming and policies to improve the health of Indigenous peoples. The currently available data highlights the disease status and health disparities between registered First Nations people and the general population. It has limited utility to address health status and public health inequities at a community level as the data is more reflective of national priorities, is often either not returned at all to communities or is not available below national or provincial/territorial levels, and fails to incorporate Indigenous conceptualizations of health. Although this did not emerge as a dominant theme in the interviews, it became apparent when a comparison was made between what informants identified as desirable characteristics of Indigenous–specific health measurement systems and what currently exists. Commonly stated elements of an Indigenous-specific health measurement system included broad definitions of health, an increased focus on wellness, traditional elements (e.g., use of traditional medicines, participation in traditional activities, midwifery), and using frameworks familiar to Indigenous people (e.g., Medicine Wheel or Métis Infinity Symbol; Bartlett, 2005). The utility of current systems is further limited by the inclusion of indicators that have been selected largely to satisfy fiduciary accountability requirements with little public health expertise in its design.

There is significant concern regarding the quality of Aboriginal health data. The lack of opportunity to self-identify as First Nations, Métis, or Inuit when encountering the health care system leads to the generation of databases without ethnic flags, making it very difficult and in some cases impossible to generate numerators to calculate health outcome/disease rates (Smylie and Anderson, 2006). The generation of denominators is also difficult, given the reliance on the census for this. There are a significant number of reserves that do not participate in the census or are incompletely enumerated. The phenomenon of ethnic mobility, that is the changing propensity of people to self-identify as Aboriginal, complicates the comparison of rates over time. This phenomenon also affects the accuracy of population estimates, and therefore health assessment data, in Australia (Anderson et al., 2005b). One example of poor quality data collection and calculation occurred in 2005 when the First Nations and Inuit Health Branch released an infant mortality rate of 6.4 per 100,000. This rate was significantly lower than regional rates that had met the scrutiny of peer review, and is considered an undercount because of methods that undercounted deaths more than births. Methods of data collection included vital registry data of variable quality for four western provinces and data collected from nursing stations for the rest of the country. This particular example highlights the substandard data sources as well as substandard methods (FNIHB, 2005c).

In addition to compromising the data quality of available Aboriginal health data, the lack of self-identification protocols contribute to the ongoing oversight of nonregistered First Nations, Métis, and many Inuit people, particularly in urban communities. This allows a tacit acceptance of the disparity in health status since it cannot be accurately quantified, and is therefore not addressed with targeted programs, policies, and appropriate funding for each population. The only source of comprehensive Indigenous-specific health information are periodic surveys, which rely on self-reported health status and only occur every few years with insufficient sampling to provide data at the regional, subregional, or community levels. This is an inadequate basis for regional or community public health planning and surveillance. In the literature review, no other studies were identified which specifically examined the goals of Indigenous-specific health measurement systems in Canada, the level of Indigenous involvement in system development, or the utility of Indigenous-specific health information in health planning at any level.

The main limitation of this study is that our data collection was limited to published and grey literature as well as the eleven key informants that we interviewed. Although we deliberately sampled across geographic, Aboriginal ethnic, and governmental/nongovernmental strata in an attempt to obtain national, provincial, regional, and community perspectives and we did identify saturation in our data collection, it is possible that we have missed some sectors and therefore relevant information that could have either strengthened our thematic analysis or identified other themes of importance. Also, as generally only one person per sector was interviewed, opposing viewpoints within sectors could have been missed. That being said, not a single informant from any of the sectors suggested they considered the currently available Indigenous health information to be “good” or even “adequate.” Health planners and policymakers concerned with health care delivery to Indigenous people in Canada are faced with the challenge of doing this job without the population health assessments needed to identify health priorities, monitor disease, and evaluate the performance of the health system.

Health measurement systems that form the basis for these population health assessments do exist for the non-Indigenous population, and it is a health equity imperative to identify Indigenous people in these systems. Indigenous-specific health measurement systems are an important complement to the ongoing collection of universal comparable health indicators as we progress towards the elimination of current health disparities. These systems will be most useful if there is Indigenous community involvement at all stages of development, implementation, and ongoing use; if they are reflective of local priorities and context; and if they incorporate Indigenous understandings of health. The availability of Indigenous-specific health measurement systems will increase the ability of community and regional level health planners to respond to the needs of First Nations, Inuit, and Métis communities.