Outcomes reported in evaluations of programs designed to improve health in Indigenous people

Abstract

Objective

To assess the outcomes reported and measured in evaluations of complex health interventions in Indigenous communities.

Data Sources

We searched all publications indexed in MEDLINE, PreMEDLINE, EMBASE, PsycINFO, EconLit, and CINAHL until January 2020 and reference lists from included papers were hand‐searched for additional articles.

Study Design

Systematic review.

Data Collection/Extraction Methods

We included all primary studies, published in peer‐reviewed journals, where the main objective was to evaluate a complex health intervention developed specifically for an Indigenous community residing in a high‐income country. Only studies published in English were included. Quantitative and qualitative data were extracted and summarized.

Principal Findings

Of the 3523 publications retrieved, 62 evaluation studies were included from Australia, the United States, Canada, and New Zealand. Most studies involved less than 100 participants and were mainly adults. We identified outcomes across 13 domains: clinical, behavioral, process‐related, economic, quality of life, knowledge/awareness, social, empowerment, access, environmental, attitude, trust, and community. Evaluations using quantitative methods primarily measured outcomes from the clinical and behavioral domains, while the outcomes reported in the qualitative studies were mostly from the process‐related and empowerment domains.

Conclusion

The outcomes from qualitative evaluations, which better reflect the impact of the intervention on participant health, remain different from the outcomes routinely measured in quantitative evaluations. Measuring the outcomes from qualitative evaluations alongside outcomes from quantitative evaluations could result in more relevant evaluations to inform decision making in Indigenous health.

What is known on this topic

• Evaluations of Indigenous health programs are important for program development and decisions regarding resource allocation; however, they often fail to fully demonstrate the value of programs to participants and communities

• Evaluating complex health interventions in Indigenous health is time‐consuming and expensive thereby restricting the scope of the measures assessed

What this study adds

• Defines specific participant and community‐identified outcomes for decision makers to consider when commissioning evaluations of health interventions designed for Indigenous communities

• Provides evidence of a sustained bias toward measuring clinical outcomes in evaluations of complex health interventions implemented in Indigenous communities and little consideration for outcomes reported by participants and their communities

• Describes the quantitative and qualitative methods used to evaluate complex health interventions in Indigenous communities from the United States, Australia, Canada, and New Zealand

1. INTRODUCTION

The number of interventions to improve Indigenous health has increased substantially in recent years and gains in health outcomes have been reported from many communities globally. ¹ Sustaining these gains is challenging especially when intervention outcomes do not always align with the health and well‐being aspirations of the communities for whom the interventions were developed. ² Holistic or “whole picture” ³ , ⁴ , ⁵ evaluations, co‐produced with Indigenous communities and providing data on the broad impact and value of health programs for Indigenous populations, could provide important data to guide decision making in program development and ultimately help sustain improvements in health outcomes.

Achieving outcomes that adequately represent Indigenous conceptualizations of health and well‐being ⁶ usually requires the development of complex health interventions ⁷ implemented within environments plagued by ongoing injustice and social, economic, and political disadvantage. Capturing outcomes that demonstrate the extent to which complex interventions are effective in promoting Indigenous concepts of health ⁶ and address community priorities is difficult and expensive. ⁸ , ⁹ Comprehensive evaluations are also subject to budget constraints which dictate the scope and the reporting of relevant outcomes. However, they are essential for decision making around the viability and development of Indigenous health programs. ⁹

Synthesizing data from evaluations of complex Indigenous health interventions could provide insight into the outcomes commonly reported. Such data could help decision making in the evaluation process by providing guidance around the outcomes to consider when conducting holistic evaluations in Indigenous communities. The authors acknowledge that the term Indigenous does not represent the unique cultures and traditions of the populations described herein. Use of the term in this paper does not infer that the populations discussed are a homogenous group. This review aims to evaluate the outcomes reported and measured in evaluations of complex health interventions in Indigenous communities.

2. METHODS

2.1. Study overview

We conducted a systematic review based on standard methods and reporting in accordance with the Preferred Reporting Items for Systematic Reviews and Meta‐analyses (PRISMA) ¹⁰ (Appendix S1). A protocol for this review was registered with PROSPERO (ID: CRD42017075485).

2.2. Inclusion criteria

All primary studies, published in peer‐reviewed journals, where the main objective was to evaluate a complex health intervention developed specifically for an Indigenous community were included. Complex interventions are a specific intervention approach that is amenable to the processes involved in bringing together multiple systems and stakeholders to achieve program delivery with multiple outcomes. ¹¹ For the purposes of this review, evaluation studies were included if they fell into the following categories as defined by the Centers for Disease Control: process evaluation, outcome evaluation or objectives‐based evaluation, economic evaluation (cost‐effectiveness evaluation, cost‐benefit analysis, cost‐utility analysis), and impact evaluation. ¹² Only evaluation studies of interventions among Indigenous communities from countries classified as “high income” by the World Bank in 2017 ¹³ were included due to the similarities in the health inequities they experience compared with the non‐Indigenous populations within the same countries. Definitions for key terms used in the inclusion criteria, such as Indigenous, evaluation, program or intervention, and complex intervention, are provided in Appendix S2.

2.3. Search strategy

All published literature indexed in the following electronic bibliographic databases were searched from inception to January 2020: MEDLINE, PreMEDLINE, EMBASE, PsycINFO, EconLit, and CINAHL. Due to resource limitations, only literature written in English or with a version translated into English was searched. Reference lists of relevant systematic reviews and included studies were hand‐searched for additional studies. Terms for the search strategy were adapted from published reviews that conducted similar searches. ¹⁴ , ¹⁵ The specific search terms are shown in Appendix S3.

Title and abstract screening were undertaken by SC to identify articles for full‐text screening. An independent title and abstract screen against the inclusion criteria were undertaken by MH on 30% of the initial list of papers and results were reviewed with SC. Disagreements were resolved through discussion and consultation with KH.

2.4. Assessment of reporting of evaluation studies

The full text of studies meeting the inclusion criteria were independently reviewed by SC, MH, and CY to assess the comprehensiveness of reporting using an appraisal adapted from a tool developed by the International Development Research Centre's (IDRC). ¹⁶ An assessment of the comprehensiveness of reporting provides information to demonstrate the rigor with which the studies were conducted. Assessing the rigor of the studies increases confidence in the results reported, particularly the findings regarding the impact of programs as reported by participants and communities. Additional questions relating to whether the research aligned with community priorities, data ownership, and control were included to determine the extent to which activities relating to the conduct of research in Indigenous populations were met. ¹⁷

2.5. Data extraction and synthesis

Data from the included studies were extracted and included the following items: date of publication, first author, country, participant age, sample size, study design, health focus, type of intervention, type of evaluation, data collection methods, and all quantitative and qualitative outcomes. The outcomes were grouped and classified into domains, commonly reported in Indigenous health literature. ¹⁸ , ¹⁹ , ²⁰ , ²¹ Measures of outcomes were extracted from studies that used quantitative methods in the evaluation. The methods used to derive outcomes from intervention participants and program staff where extracted from evaluations employing qualitative methods. We did not conduct a full thematic analysis of the qualitative data, rather we extracted data specific to the summary of outcomes. A complete analysis of the qualitative data was beyond the scope of this study but is the focus for a separate paper currently underway.

3. RESULTS

3.1. Literature search and study characteristics

The search yielded 3523 publications and after screening the titles, abstracts, and full‐text papers, 62 studies met the inclusion criteria. Of the included studies, 29 were impact or outcome evaluations, 14 were process evaluations, five were economic evaluations and 14 included more than one type of evaluation. The search results are shown in Figure 1.

FIGURE 1.

Search results

Twenty‐one (34%) of the studies were conducted in Australia, 20 (32%) in the United States, 17 (27%) in Canada, and four (6%) in New Zealand. The number of participants in the evaluation studies ranged from eight to 1580 with the highest proportion of studies (29 or 47%) having less than 100 participants. Participant ages ranged from seven months to 85 years and 15% of studies focused only on children. The study designs varied with the most common being: descriptive studies (16 or 26%) and mixed‐methods studies (16 or 26%) followed by quasi‐experimental studies (14 or 23%). Experimental study designs were the least used (1 or 2%). Broadly, the interventions covered eight main health areas: chronic disease, child health, dental, healthy lifestyle, injury, maternal health, mental health, and organ and tissue donation. Most studies were evaluations of interventions addressing chronic disease (19 or 31%) or mental health (17 or 27%). A summary of the characteristics of the included studies is provided in Table 1 and additional details in Appendix S4.

TABLE 1.

Characteristics of included studies (n = 62)

Characteristic	N (%)
Country
Australia	21 (34)
United States	20 (32)
Canada	17 (27)
New Zealand	4 (6)
Study year
1970‐1990	4 (6)
1991‐2011	27 (44)
2012‐2018	31 (50)
Sample size
0‐50	18 (29)
51‐100	11 (18)
Greater than 100	23 (37)
Unclear/Not Reported	10 (16)
Population (age range in years)
Children only (0 – less than 18)	9 (15)
Adults only (greater than or equal to 18)	19 (31)
Children and adults	24 (39)
Unclear/Not Reported	10 (16)
Study design
Descriptive	16 (26)
Economic evaluation	5 (8)
Experimental	1 (2)
Quasi‐Experimental	14 (23)
Observational	10 (16)
Mixed methods	16 (26)
Health focus
Chronic disease	19 (31)
Child health	3 (5)
Dental	1 (2)
Healthy lifestyle (nutrition/exercise)	8 (13)
Injury	9 (15)
Maternal health	4 (6)
Mental health	17 (27)
Organ and tissue donation	1 (2)

3.2. Assessment of reporting in included studies

The majority of studies consistently reported on items relating to the feasibility (69%‐98%) and the accuracy (77%‐97%) of the study. The least reported item for feasibility was whether the program outcomes addressed a priority health issue as determined by the community and among the items reported for accuracy, the least reported item was how the evaluation outcomes were derived. There was poor and inconsistent reporting of items regarding the evaluator (18%‐45%), utility (19%‐35%), and propriety (13%‐58%). The least reported items for propriety were data ownership and control and community feedback. Appendix S5 provides a summary of the results.

3.3. Synthesis

We classified outcomes reported in the evaluations into 13 domains: clinical, behavioral, process‐related, economic, quality of life, knowledge/awareness, social, empowerment, access, environmental, attitude, trust, and community. The clinical, process‐related, economic, social, empowerment, and community domains also include descriptive subdomains. Definitions of the domains and corresponding subdomains are provided in Appendix S5, and summaries of the outcome domains and subdomains measured and reported for the quantitative and qualitative evaluations are provided in Tables 2 and 3, respectively. Of the 62 studies, 52 (84%) studies provided a quantitative analysis of outcomes and 31 (50%) studies provided qualitative data including information on the techniques used to identify the outcomes reported.

TABLE 2.

Outcomes measured in the quantitative evaluations (n = 52)

Outcome domain and subdomains a		Number of studies using instrument type to measure changes in outcomes
	Number of studies measuring outcomes in domain	Standard b	Researcher‐developed	Not specified/Not reported
Clinical: anthropometric, physiological health, mental and social well‐being, mortality, childhood development, disease stage/progression, population health measures of the distribution, and determinants of disease frequency 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67	39	33	4	2
Behavioral: change/lack of change in actions that affect wellness 29 , 32 , 54 , 55 , 56 , 59 , 60 , 62 , 65 , 66 , 68 , 69 , 70 , 71 , 72 , 73 , 74	25	12	9	4
Process‐related: patient satisfaction, quality, participation, usefulness, relevance, engagement, dose, reach, fidelity, extent, barriers, and facilitators to participation 31 , 64 , 67 , 69 , 75 , 76 , 77	17		14	3
Economic: health care costs, societal costs, health resource utilization 30 , 61 , 68 , 78	12	11		1
Quality of life: patient's general perception of the effect of illness and treatment on physical, psychological, and social aspects of life 29 , 31 , 34 , 35 , 37 , 41 , 45 , 47 , 48 , 54 , 56 , 62	12	9	2	1
Social: social functioning, social positioning, safety, family/support networks, employment, arrests, and jail terms 30 , 33 , 39 , 41 , 42 , 45 , 47 , 48 , 56 , 61 , 62	12	8	3	1
Empowerment: self‐efficacy, self‐esteem, self‐concept, self‐confidence, ownership, cultural identity, resilience 41 , 47 , 53 , 54 , 62 , 65 , 67 , 71 , 79 , 80	10	7	3
Knowledge/Awareness: change in knowledge about information related to disease or health 37 , 45 , 56 , 59 , 61 , 66 , 71 , 75 , 79	9	2	7
Access: access to health services 37 , 41 , 51 , 59 , 64 , 69 , 70	7	1	5	1
Environmental: changes to external surroundings or condition that affect health 29 , 59 , 60 , 62 , 81	5	1	3	1
Attitude: shift(s) in perceptions, opinion, and views affecting health behaviours 46 , 56 , 72	3	2	1
Community: capacity building, cultural pride/appreciation, re‐engagement with community cultural practice/customs, engagement with services, community acceptance, community collaboration, community culture, community self‐determination 67	1		1

^a Outcome domains and the specific subdomains measured in the included evaluation studies, refer to Appendix S5 for definitions of domains and subdomains.

^b Includes: validated instruments, accepted standard measures, official records, institutional/local/national databases.

TABLE 3.

Outcomes reported by participants in the qualitative evaluations (n = 31)

Outcome domains and subdomains a	Number of studies reporting on outcomes in domain	Number of studies using method to obtain outcomes b
		Interview	Focus group	Open‐ended questionnaire	Observation/Other
Process‐related: patient satisfaction, quality, participation, acceptability, engagement, reach, barriers, and facilitators to participation 31 , 32 , 68 , 74 , 76 , 78 , 80 , 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89	22	17	5	3	1
Empowerment: self‐esteem, self‐concept, self‐confidence, ownership, cultural identity, professional/personal development, cultural security, skill transfer, aspirations 31 , 80 , 82 , 85 , 87 , 88 , 89 , 90	16	10	5	5	1
Community: capacity building, cultural pride/appreciation, re‐engagement with community cultural practice/customs, engagement with services, community acceptance, community collaboration, community culture, community self‐determination 31 , 32 , 47 , 59 , 60 , 62 , 64 , 65 , 67 , 74 , 83 , 87 , 90	15	10	5	4	1
Behavioral: change/lack of change in actions that affect wellness 31 , 47 , 54 , 59 , 60 , 62 , 71 , 74 , 77 , 82 , 87 , 88	12	7	1	5	2
Social: social functioning, social positioning, safety, family/support networks 29 , 37 , 47 , 54 , 59 , 77 , 78 , 80 , 84 , 87 , 89	11	7	3	4
Knowledge/Awareness: change in knowledge about information related to disease or health 47 , 59 , 60 , 62 , 65 , 67 , 77 , 80 , 82 , 83 , 88	11	6	1	4
Clinical: physiological health, mental, and social well‐being 31 , 47 , 65 , 78 , 80 , 84 , 87 , 89	8	4	3	3
Access: Access to health services includes traveling out of the community to obtain needed services 32 , 37 , 59 , 60 , 83 , 84 , 88	7	6	1		1
Trust: trust that developed between health service providers and community members as a result of participation in the health intervention 32 , 37 , 50 , 74 , 88	5	5	1
Economic: healthcare costs, health resource utilization 64 , 74 , 83 , 84	4	3	1
Attitude: shift(s) in perceptions, opinion, and views affecting health behaviors 59 , 65 , 88	3	2	1
Environmental: changes to external surroundings or conditions that affect health 29 , 79	2	2
Quality of life: patient's general perception of the effect of illness and treatment on physical, psychological, and social aspects of life 84	1	1

^a Outcome domains and the specific subdomains reported in the qualitative portions of the included evaluation studies, refer to Appendix S5 for definitions of domains and subdomains

^b Some studies used more than one qualitative technique to identify outcomes.

3.4. Outcomes measured in the quantitative evaluations

Of the 13 outcome domains reported across the included studies, 12 outcome domains were analyzed quantitatively in the evaluations in this review as shown in Table 2. Thirty‐nine (75%) of the 52 quantitative studies assessed outcomes in the clinical domain. Examples of outcomes in this domain included anthropometric measures, measures of changes in indicators of physiological health such as fitness, measures of changes in mental health states such as depression, and population health measures. Twenty‐five (48%) studies measured outcomes in the behavioral domain, such as alcohol use, exercise, and nutritional intake. Seventeen (33%) studies analyzed outcomes in the process‐related domain. These outcomes related to the implementation of interventions and included outcomes such as patient satisfaction, quality of services provided, and participation in the intervention. Other commonly measured outcomes were in the economic, quality of life, and social domains (all 12 studies or 23%).

The instruments used to measure changes in outcomes due to the interventions varied among studies. These were categorized as either standard instruments or researcher‐developed instruments. Standard instruments represented validated tools or measures, accepted standard measures, and official records such as clinical records or institutional/local/national databases. Standard instruments were the most common method of measurement in seven of the 12 outcome domains reported in quantitative evaluations. Researcher‐developed instruments were the main method for measuring outcomes in the process‐related, knowledge/awareness, access, environmental, and community domains as shown in Figure 2.

FIGURE 2.

Outcomes measured in quantitative evaluations of Indigenous health interventions and the instruments used for measurement (n = 52)

3.5. Outcomes reported in the qualitative evaluation data

All of the 13 outcome domains we identified across the included studies were reported as themes in the qualitative data and are shown in Table 3. Twenty‐two (71%) qualitative evaluation studies reported outcomes in the process‐related domain, 16 (52%) studies reported outcomes in the empowerment domain, and 15 (46%) studies reported outcomes in the community domain. Examples of outcomes in the empowerment domain were self‐efficacy, self‐esteem, and self‐confidence, and examples of outcomes in the community domain were cultural pride/appreciation, community self‐determination, and capacity building. The other outcomes that were reported in a number of evaluations that used qualitative methods were in the behavioral, social, and knowledge/awareness domains.

Four qualitative methods were used to obtain data about outcomes from participants and staff: interviews, focus group, open‐ended questionnaire, and observation/other. Other methods included techniques that incorporated Indigenous methods of communication for example “Yarning” which is central to the culture of Indigenous communities in Australia. This form of communication describes how Indigenous people engage in conversations and dialogue to pass on knowledge through a process of cultural connection. ²² The majority of studies used interviews to elicit data from participants and staff. Open‐ended ended questionnaires were also a commonly used method to derive outcomes.

Information about the importance or value of outcomes could be inferred from some of the qualitative data; however, a comprehensive analysis of the qualitative data was beyond the scope of this paper and is further explored in a separate paper. Five qualitative studies reported unintended outcomes (spillovers or externalities, which can be described as unintentional outcomes that represent a benefit to individuals or community beyond participants in the intervention). Specifically, one study reported an outcome in the knowledge domain and four studies reported on skills transfer in the empowerment domain. None of the included studies provided a measure of the importance or value of outcomes to participants.

3.6. Comparison of outcomes from the quantitative versus qualitative evaluation data

The types of outcomes reported in evaluations that included qualitative methods as part of the assessment differed to the outcomes reported in evaluations that used quantitative methods for analysis. Generally, the quantitative evaluations focused on outcomes from the clinical, behavioral, economic, quality of life, social, environmental, and attitude domains when compared to the outcomes reported in the qualitative assessments. Conversely, more qualitative evaluations reported on outcomes from the process‐related, empowerment, trust, and community domains compared with the quantitative evaluations. Notably, outcomes in the trust domain appeared in the qualitative evaluation data but not in any of the quantitative evaluation studies and only one quantitative study measured outcomes in the community domain, whereas 15 studies reported outcomes in the community domains in the qualitative evaluations.

4. DISCUSSION

This review described a range of outcomes from evaluations of complex health interventions implemented in Indigenous communities from four high‐income countries. Quantitative methods were the primary method for evaluation, mostly measuring outcomes related to the clinical and behavioral aspects of health. Few evaluation studies used qualitative methods as the primary method for evaluation, and in most cases, qualitative data were supplementary to a quantitative assessment. Nevertheless, the qualitative studies provided valuable information on the outcomes reported by participants relating to the impact of interventions on their health and quality of life and their experiences. Our findings suggest that such outcomes reported by participants are yet to be routinely included alongside the outcomes prioritized for measurement by those who commission and conduct evaluations.

Community perspectives on what works must be valued and should contribute substantially to the data that is included in health program evaluations. ⁹ However as evidenced by findings from this review, the outcomes reported by participants providing insight into the community perspective on the impact of health programs are generally not measured quantitatively. Given that quantitative evaluation methods continue to be privileged over qualitative methods for decision making around policy and funding, such omissions may be contributing to judgments about programs that do not reflect community perceptions regarding the development and future of programs. ⁹ Arguably, decision makers may already be aware of this disconnect and consequently do not always find evaluations useful. ⁹ Partnership and ongoing dialogue with communities in the design and development of evaluations may aid in ensuring that relevant outcomes are included and measured and that the results get used. ¹⁷

Previous research has highlighted a bias toward the measurement of clinical health outcomes compared with community‐identified outcomes in evaluations of health programs in Indigenous populations. ⁵ Incorporating outcomes that community members value, like trust and empowerment into quantitative evaluations, such as economic evaluations, remains a challenge. ²¹ , ²³ As shown in this review, few studies employ standard objective measures to evaluate the impact of community‐identified outcomes such as community collaboration that encompass broader conceptualizations of health and well‐being in Indigenous populations. Community‐identified outcomes are important to capture when assessing interventions designed for Indigenous people as they provide data on the aspects of programs that reflect the priorities of the individual participant but also community and cultural values. These will have implications for participation and engagement that are unique to the Indigenous communities. However, the fact that few studies measured community‐identified outcomes may reflect the scarcity of appropriate instruments ²¹ and, in particular, those validated in Indigenous populations.

Support for the development of valid instruments to use in evaluations that measure outcomes reported by participants is needed. Moreover, further research is needed to determine how and the extent to which participant reported outcomes contribute to Indigenous visions of improved quality of life. Given the expense of conducting evaluations, there may be a reluctance to include participant reported outcomes in evaluations. Future research could identify some outcome trade‐offs decision makers would be willing to make to advance efforts to improve long‐term health outcomes in Indigenous communities and better focus evaluations on the outcomes important to all stakeholders.

Community‐identified outcomes are increasingly recognized as important for achieving long‐term health outcomes in Indigenous communities. ⁵ However, Indigenous health policy agendas are largely driven by a Western biomedical model which prioritizes the objective measures of clinical outcomes over the outcomes which that model would define as “subjective” measures such as trust, self‐efficacy, and self‐esteem. ⁸ , ²⁴ Outcomes from two domains were consistently identified as major themes from qualitative investigations about individual experiences with the health interventions, namely the community and trust domains. None of the studies employing quantitative methods to evaluate the interventions included measures of outcomes from the trust domain and only one measured outcomes from the community domain, which underscores the notion of a bias toward objective measures.

The little attention given to outcomes from the trust and community domains in the quantitative studies is at odds with the consistent message in the Indigenous health literature regarding the importance of community engagement and trust between health service users and providers and the role of these outcomes in facilitating sustained program participation and engagement. ²⁵ , ²⁶ In particular, the exclusion of trust as an outcome is curious given that there are existing recognized instruments that measure trust as an outcome. ²⁷ Further research is needed to understand why this outcome is excluded from these evaluations particularly when Indigenous communities have historically and continue to experience grave injustices in health service delivery.

Evaluations in this review were mostly impact or outcome evaluations and employed a variety of study designs, with descriptive designs being the most common. The variation of study designs used in health evaluations is well documented and is to be expected. This is largely due to the contextual factors from each community that influences decisions around appropriate research approaches. ²⁸ While experimental designs are generally preferred for obtaining quantitative evidence on the effectiveness of complex interventions, ⁷ experimental designs have traditionally been unpopular in Indigenous populations due to their inflexible nature and may be inappropriate particularly for complex public health interventions. ²⁸

Descriptive designs offer an opportunity to collect data in a manner that is culturally appropriate and acceptable because they are flexible in rapidly changing environments, allow participants to express themselves in ways they are comfortable with, and are easier to adapt to cultural considerations around communication. ²⁵ Quasi‐experimental studies, mostly pre/post designs, were another design broadly used in the included studies which may indicate that there are improvements in researcher and community engagement and collaboration allowing for successful implementation of these methodologies. Use of quasi‐experimental designs in evaluations may offer an alternative to experimental studies and present evaluators with an option to use a design that provides more objective and reliable data around the effectiveness of programs, within the constraints of what is acceptable to the community.

A strength of this review is the comprehensive search criteria which can be replicated to generate data from Indigenous populations from low‐ or medium‐income countries. However, this review had limitations. Firstly, we identified a relatively small number of published studies given the number of programs that have been implemented in Indigenous communities over the years. This points to a lack of peer‐reviewed publications on evaluations of programs in Indigenous health. However, it is noted that evaluations are also likely to be published as reports in the gray literature by agencies conducting and/or funding the evaluations rather than as articles in peer‐reviewed journals. A summary of the implications of findings from this paper is provided in Appendix S7.

We only examined evaluations of interventions designed for and evaluated in Indigenous communities and did not include studies of mixed populations. This approach was used to ensure that all outcomes reported were relevant to Indigenous communities which may not be the case in studies of mixed populations.

5. CONCLUSION

Indigenous health‐related policy and resource allocation decision making can be enhanced by evaluations that include the measurement of outcomes that encompass the holistic notions of health and well‐being prized in Indigenous communities. Defining the outcomes from complex Indigenous interventions globally may guide the selection of outcomes by evaluators and decision makers in their contexts and help to avoid expensive investigations gathering nonrelevant information. Decision makers should be aware that the outcomes routinely measured in quantitative evaluations may not provide the full picture of the impact and experiences of health programs on communities. Such awareness could be instrumental in increasing support for more holistic evaluations to better inform decision making on Indigenous health programs.

CONFLICT OF INTEREST

None to declare.

Supporting information

Author matrix

SupInfo

Chando S, Howell M, Young C, et al. Outcomes reported in evaluations of programs designed to improve health in Indigenous people. Health Serv Res. 2021;56:1114–1125. 10.1111/1475-6773.13653