Defining collective capability in Australian evaluations that are conducted by, for and with Indigenous peoples for health programmes, policies and services: a concept analysis protocol

Abstract

Introduction

Indigenist evaluation is emergent in Australia; the premise of which is that evaluations are undertaken for Indigenous, by Indigenous and with Indigenous people. This provides opportunities to develop new models and approaches. Exploring a collective capability approach could be one way to inform an Indigenist evaluation methodology. Collective capability suggests that a base of skills and knowledges exist, and when these assets come together, empowerment and agency emerge. However, collective capability requires defining as it is not common terminology in population health or evaluation. Our aim is to define the concept of collective capability in Indigenist evaluation in Australia from an Australian Indigenous standpoint.

Methods and analysis

A modified Rodgers’ evolutionary concept analysis will be used to define collective capability in an Australian Indigenous evaluation context, and to systematically review and synthesise the literature. Approximately 20 qualitative interviews with Aboriginal and Torres Strait Islander knowledge holders will clarify the meaning of collective capability and inform appropriate search strategy terms with a consensus process then used to code the literature. We will then systematically collate, synthesise and analyse the literature to identify exemplars or models of collective capability from the literature.

Ethics and dissemination

The protocol has approval from the Australian Institute of Aboriginal and Torres Strait Islander Studies Ethics Committee, approval no. EO239-20210114. All knowledge holders will provide written consent to participate in the research. This protocol provides a process to developing a concept, and will form the basis of a new framework and assessment tool for Indigenist evaluation practice. The concept analysis will establish definitions, characteristics and attributes of collective capability. Findings will be disseminated through a peer-reviewed journal, conference presentations, the project advisory group, the Thiitu Tharrmay reference group and Aboriginal and Torres Strait Islander community partners supporting the project.

Strengths and limitations of this study.

• This paper describes a protocol for defining collective capability within an Indigenist methodological context, and includes the investigator team being driven by Aboriginal researchers.

• In-depth interviews with Indigenous knowledge holders will occur in the first instance to inform the search terms and define collective capability from an Indigenous standpoint.

• The concept may be completely new and not fully established in the literature relating to the Australian evaluation context.

• The concept may not be internationally relevant.

Introduction

Reform in Indigenous evaluation practice is occurring in Australia; there is a call to action to better understand where progress is being made in Aboriginal and Torres Strait Islander health and social outcomes, and the effectiveness of programmes, policies and services in supporting this progress. Evaluation practice is an important vehicle to make judgements on whether programmes, policies or services are working or not working within a particular context.^1–3 Evaluations of programmes, policies and services relevant to Indigenous people and communities are essential to establish an evidence base and an understanding of progress in health and social outcomes.^{2 4–7} There is, however, little evidence through quality programme evaluations to show what is working and what is not working, and how Aboriginal and Torres Strait Islander peoples benefit, implying that policies and programmes are not working as well.^{4 7–10} With limited and poor-quality published evaluation evidence, there is a gap in knowledge and lessons learnt.

The current Indigenous evaluation landscape in Australia has tended to be positioned in settler-colonialism and may be responsible for the current poor state of Indigenous evaluations in Australia.^{7 8 11} Dominant settler-colonial approaches and perspectives applied to evaluation methodologies, engagement, design and methods becomes problematic for evaluations of Indigenous programmes, policies and services within community settings as it often excludes local context, perspectives, experiences and knowledges.^{2 12 13} Additionally, current practice commonly fails to address underlying power imbalances due to external evaluation teams who are often outsiders to communities.^{2 7} Indigenous cultures are highly contextual; context matters.¹⁴ For Aboriginal and Torres Strait Islander people, this considers the diversity in culture, place, knowledge systems, experiences and lifeworlds.¹⁵

As has been shown elsewhere in the world, Indigenist approaches applied to evaluation translates to evaluation that benefits indigenous people; includes indigenous people; and is guided by principles, practice and knowledges that are indigenous.^16–18 Internationally, indigenous evaluation has been defined and operationalised to be ‘by indigenous, for indigenous, with indigenous and as indigenous’ (Wehipeihana, p.370).^{2 16 17 18} In Australia, we see some elements of Indigenist approaches integrated into evaluation practice and often manifest as Indigenous governance,^{7 11} cultural protocols that inform ethical and respectful relationships with communities,^{7 12 19} and processes and strategies for meaningful knowledge translation with communities.3 4 7 20 However, these are not features of standard evaluation practice.

Indigenist research and evaluation methodologies are evolving in Australia.^{7 21–24} Indigenist approaches acknowledge Aboriginal and Torres Strait Islander capabilities in research processes and structures such as participatory research methods; governance; community engagement and cultural protocols, to inform strengths-oriented evaluation application and ensure cultural safety of evaluation practice.^{11 19 24} Creating platforms to support Aboriginal and Torres Strait Islander people to be in the driver’s seat of decision making processes, and valuing their knowledge and community expertise allows for an assets-based practice in Indigenist evaluation. This promotes evaluation as inclusive, safe, respectful and ethically aligned with Aboriginal and Torres Strait Islander ways of knowing, being and doing. Further, Indigenist approaches support leadership capability of Aboriginal and Torres Strait Islander people and communities in evaluation.^{7 12}

Existing frameworks and cultural protocols guiding evaluations of Aboriginal and Torres Strait Islander programmes and policies capture some elements of Indigenist approaches, and are intended to inform cross-cultural practices and commissioning processes. The work of Williams⁷, Wright et al ²⁵ and Rogers et al ¹⁹ draw on codesign processes in evaluations that centres Aboriginal and Torres Strait Islander perspectives, experiences and leadership. These frameworks provide clear guidance of processes to ensure local knowledge and cultural protocols are embedded in the design and processes of evaluations. Further, they ensure that governance through elders and knowledge holders are an essential component of evaluation practice at the community level. Additionally, the Lowitja Institute’s evaluation framework to improve Indigenous health supports evaluations to benefit Aboriginal and Torres Strait Islander people.⁴ The principles-based framework supports cross-cultural process for evaluators and commissioners of evaluations to ensure strength-based approaches, partnerships, capacity building, cultural competence and Aboriginal and Torres Strait leadership are embedded into all stages of evaluations as codesign processes.

These frameworks focus towards non-Indigenous evaluators, researchers, organisations and commissioners of evaluations to being culturally competent and safe. The concept of collective capability aims to (re)centre Indigenous epistemology and to ensure culturally robust evaluation practice through joining together concepts of collectivist and capability that are novel in evaluation.^{26 27}

Indigenous societies are often described as ‘collective’ or ‘collectivist’, because the emphasis is on the group above individuals. The role of collectivism is a shared and relational intent; to determine the solutions of the issues that Indigenous peoples collectively define and identify.²⁸ In part, collectivism is strongly aligned with self-determination for indigenous peoples.²⁹ Capability relates to the tools, skills and resources that enhance the well-being of a person to live and lead a life that they value; thus gaining the freedom to do the things that align with these values.26 27 30 31 Values are context specific, relating to ‘knowing, doing and being’ that are intricately linked to Indigenous ‘lifeworlds’ and integrates knowledge, kinship structures and realities.¹⁵ Therefore, when collective and capability come together, it implies that the collective action of individuals and their capabilities results in decision making and participation processes and structures that benefit the ‘collective’. Our sense is that collective capability suggests that a base of skills and knowledge exist within Aboriginal and Torres Strait Islander people and communities, and when these assets come together, empowerment and agency emerge.

Applying Indigenist approaches to evaluation dismantles the deficit framing of evidence by placing a lifeworld and solutions-based perspective to better reflect Aboriginal and Torres Strait Islander realities, thus improving the quality and usefulness of the evidence base. Further, to enhance Indigenous evaluation practice in Australia and move to Indigenist evaluation, new frameworks and models are required; that support governance, leadership and centre Indigenous knowledge and perspectives.

We aim to establish a collective capability definition and extract operational elements of collective capability within evaluation practice in Australia, and of potential relevance internationally, to inform Indigenist evaluation.

Methods and analysis

Methodology

Concept analysis has been used in nursing science to develop theory and models to inform clinical practice.³² It has since been applied to a number of fields and disciplines including public health, social work and health policy.^33–36 The methodology is a systematic approach drawing from a combination of primary research literature and grey literature, and an analysis of these literatures to determine the characteristics and attributes of a concept that appears vague and ambiguous. A concept analysis can be used to establish meaning and clarity of the concept and has been described as ‘the systematic examination of the attributes or characteristics of a given concept for the purpose of clarifying the meaning of that concept’ (Hughes, p. 1184).³⁵

Methods

A modified Rodgers’ evolutionary method

Rodgers’ evolutionary concept analysis is a systematic and inductive method of analysis to clarify concepts that may be ambiguous and vague to inform knowledge that has mostly been applied in nursing.^{37 38} A five-step process is used to define, describe and explain a concept within the context in which it used. The steps include: (1) Identification of the concept, including the definition, associated terms and relevant use; (2) Selection of the sample and setting of data collection; (3) Collection and analysis of data that identify the attributes, antecedents and consequences of the concept; (4) Exemplars of the concept (if appropriate) and (5) Identify implications for nursing and further research and development.³⁴ Rodgers’ evolutionary method is cyclical in nature and acknowledges that concepts are continually evolving and changing. Therefore, the analysis will not necessarily determine an endpoint for a concept and may require further research to redefine the concept as it develops over time.³⁷

We will be modifying Rodgers’ evolutionary method to include a fieldwork component with knowledge holders to develop the concept from an Indigenous standpoint. Further, the inclusion of inquiry with Indigenous knowledge holders aligns with Indigenist research approaches; to privilege Indigenous voices, experiences and knowledge. Indigenist approaches acknowledge western research knowledge and methods, however, they choose to centre knowledge systems that reflect Indigenous lifeworlds. Knowledge holders refers to Aboriginal and Torres Strait Islander peoples who have expertise in evaluation, Indigenous research methods and community-based research.³⁹ We will start with in-depth qualitative interviews (step 1, see figure 1) with knowledge holders to draw out how collective capability is described; establish a definition and explain how it is operationalised in Indigenist evaluation practice in Australia.⁴⁰ Yarning as a conversational method will be applied to the interviews to enable a two-way exchange between the interviewer and the knowledge holder. Yarning aligns with Indigenous lifeworlds and the relational nature of Indigenist approaches.^{41 42} Associated terms, characteristics and attributes will also be identified in the interviews; these will then be used to inform the literature search strategy.

Figure 1.

Process of modified Rodgers’ evolutionary concept analysis.

Aboriginal and Torres Strait Islander governance

Inline within Indigenist approaches, this development is led and managed by Aboriginal researchers. Two governance structures will provide guidance and advice to the project. We will establish a project advisory group (advisory group) to inform and provide advice on the project approaches. Members will have expertise in Indigenous evaluation, community-based research methods and policy making. Additionally, Thiitu Tharrmay is an Aboriginal and Torres Strait Islander reference group that provides advice to parts of the research programme at the Australian National University. In the Ngiyampaa language, ‘thiitu tharrmay’ means ‘to share knowledge’. Thiitu Tharrmay membership includes experts in Indigenous health and community-based research, research methods and policy.

Recruitment

A purposive sampling strategy of professional networks from the team and advisory group will inform the recruitment of Indigenous knowledge holders. The knowledge holders are relational to the Aboriginal team members and advisory group; we have existing relationships informed by our Indigeneity and work. The criteria for recruitment aligns with our research objective and includes knowledge holders who have contributed to Indigenous evaluation practice; either as evaluators, experts on evaluation panels; commissioning agencies of evaluation; or are an Aboriginal community or community organisation recognised for participation in evaluation. We aim to undertake approximately 20 interviews, or until we start to see thematic and theoretical exhaustion in the interviews.^{43 44} The interviews and analysis to define the concept of collective capability will be complete by February 2022.

Implementing the seven steps of the concept analysis of the literature

1. Undertake qualitative interviews using a conversation method known as yarning,^{41 42} and includes:

   1. Can I ask you why you said yes to this interview?

   2. What words come to mind when you hear the word evaluation?

      1. What do you think Indigenous evaluation is?

      2. Probing questions: Can you tell me a bit more about your experience (term, or the words the participant has mentioned).

   3. What is important in Indigenous evaluation?

   4. What does the word collective mean to you?

      1. Probing question: What other words represent collective?

      2. Probing question: How would you explain collective to other people?

   5. What words come to mind when you hear the word capability?

      1. Probing question: What other words represent capability?

      2. Probing question: How would you explain capability to other people?

   6. If we brought the words collective and capability together, how would you explain collective capability to other people?

      1. Probing question: What other words would you use to explain collective capability?

      2. Probing question: I’ve bought along these images along today, do any of these resonate collective capability, and can you tell me why? (there will be approximately six abstract images. No identifying of people or places)

   7. How would you explain collective capability in Indigenous evaluation?

      1. Probing question: Do you think it happens now in Indigenous evaluation?

      2. Probing question: What would you see if collective capability was applied to Indigenous evaluation?

      3. Probing question: How would things be different if collective capability was happening now, like you described?

An interview guide providing further detail of the yarning process and how they may unfold has been developed as an additional document (online supplemental file 1).

Analysis of interview data will be conducted using conventional content analysis drawing on the knowledge holders lived experience and knowledge of the concept collective capability.⁴⁵ An inductive approach will also be applied to analyse the data to find meaning from the content of text through consolidating and organising the text, and to identify associated terms, characteristics (including the antecedents and consequences of the characteristics) and attributes to describe collective capability.⁴⁶

The content analysis results will be used as the search terms to inform the search strategy. The search strategy will involve an iterative process drawing on the associated terms identified in the qualitative interviews with knowledge holders. We will use the associated terms to inform the search terms. This may include or extend the search terms listed in box 1 which were identified by the research team.

Box 1. Example search terms.

Indigenous OR “First Nation” OR “First People” OR Aboriginal OR

“Torres Strait” OR Maori OR “Native American” OR “American Indian” OR “Native Hawaiian” OR, “Alaska Native” OR metis OR inuit OR sami

Capability, capacity, self-determination, sovereignty, control, ownership, decision-making, Indigenous-led (this may be covered by the other terms)

Community, community-based, co-design

Evaluation, program monitoring, accountability

Boolean terms: Collective AND Capability building

Terms identified from Step 1 content analysis

Additionally, we will draw on the discussions of how collective capability has been described in the interview data to establish the characteristics and attributes of collective capability in indigenist evaluation. This may include patterns or themes from the analysis of the data and will inform the inclusion/exclusion criteria of the search strategy.

Inclusion

Literature that describes collective capability as per findings from the qualitative interviews and/or describes its characteristics will be considered for inclusion. Particular focus will be on Indigenous evaluation practices and how the concept is applied to evaluation design, methodology and methods. We will include Indigenous specific and universal programmes that target Indigenous populations.

We are interested in evaluation design, methodology and methods that include participatory approaches. We will also include study designs that include experimental or clinical trials.

There may be different theories that define the concept of collective capability within other contexts/disciplines, it will be important to include the theoretical context that applies to Indigenous self-determination, agency and participation. Therefore, theoretical materials (commentaries, conceptual writing, think pieces) will be included as well as empirical literature (research studies and review articles). Theoretical material from community development and health promotion that describes theories, frameworks and/or models where the concept is a component will be considered.

Exclusion

Articles in languages other than English will not be included. We will exclude programme evaluations of universal programmes where the focus is not on Indigenous peoples or communities.

The following electronic databases will be searched for relevant literatures: Applied Social Sciences Index and Abstracts; Scopus; Google Scholar; Google; Informit Indigenous collection; ProQuest Dissertations and PubMed. We will also include grey literature, in particular government evaluation reports, commissioned evaluation reports, dissertations and evaluation workshops and conference material.

The search

Two Aboriginal team members will independently assess titles and abstracts of relevant articles and literature, and remove any duplicates. Searching and screening will occur concurrently, and we will use the COVIDENCE software for title/abstract screening, full-text screening, quality assessment and data extraction.⁴⁷ Results from the search strategy will be screened and search terms reviewed based on citations selected for inclusion. A second search will occur with revised terms and repeat the screening. Snowballing of reference lists of included articles will be used to identify additional case examples. Disagreements will be resolved through consensus, potentially a third Aboriginal team member will be included to act as the mediator and make the final decision.

Data extraction and synthesis

The two team members will then independently review the full text of sample literature to identify characteristics of collective capability, as conceptualised and described by the knowledge holders in the in-depth interviews, and any new characteristics that may emerge from the literature. This will include antecedents (events or phenomena prior) and consequences (what happens as a result). We will use the questions below in, table 1 to form the coding framework for the core analysis phase, as outlined by Tofthagen and Fagerstrøm, and to identify patterns in the text.³⁷ Extracted data will include, but not limited to: definitions of collective capability; associated terms; attributes; antecedents; consequences; examples of collective capability being used; setting/context; discipline; theory used (yes/no and summary if yes); year of publication; and country. We will use QSR NVivo V.12 software to store, manage, code and analyse data.⁴⁰ A general inductive approach for analysing qualitative data will be applied.⁴⁶

Table 1.

Questions for the core analysis phase

Criteria	Description
Surrogate terms	Do other words say the same thing as collective capability? Do other words have something in common with collective capability?
Antecedents	Which events or phenomena have been associated with collective capability in the past?
Attributes	What are the characteristics of collective capability?
Examples	Are concrete examples of collective capability described in the data material?
Consequences	What happens after or as a result of collective capability?

The team members will meet to confirm the coding of text and terms that are commonly used in the literature, and the main themes and patterns that emerge during the analysis. We will use inter-rater reliability (IRR) as the method for the coding agreement process.^{48 p.385} IRR ensures trustworthiness of the interpretation and coding of the data by coders using the same coding framework. A percentage crude agreement measurement will determine the two coders reach consensus of the same result.⁴⁹ As a general rule, consensus of approaching or exceeding 85% agreement on 95% of the codes will be applied, as consistent with other studies.^{49 50}

After distillation of key collective capability attributes (including definitions of collective capability) we will develop and then test the tool (table 2) for examples from the literature describing the characteristics of collective capability in Indigenist evaluation. This will provide examples of models for collective capability in evaluations.

Table 2.

Attribute assessment for collective capability in evaluation

	Attribute 1	Attribute 2	Attribute 3	Attribute 4	Attribute 5
Example 1	X	.	X
Example 2		X			X
Example 3	X	X	X	X	X
Example 4		X	X	X	X

The findings will be presented to the advisory group and the Thiitu Tharrmay internal reference group at the Australian National University. This approach further ensures that dissemination of findings and feedback are captured from an Indigenist standpoint and in line with Indigenist approaches. Implications and further research will also be established during this step.

Patient and public involvement

Experiences of evaluations for Indigenous populations in Australia have informed the development of the research question. The research question, project design, recruitment strategy and results are informed by Indigenous Australians and their expertise.

Ethics and dissemination

The protocol has been reviewed and informed by Thiitu Tharrmay. The draft protocol was presented by BLM to the Thiitu Tharrmay for input, discussion and feedback, and the protocol for primary data collection has approval from the Australian Institute of Aboriginal and Torres Strait Islander Studies (AIATSIS) Human Research Ethics Committee, ethics approval no. EO239-20210114.

Knowledge holders will be provided with a participant information sheet outlining that information from the interviews will be used to develop a definition for collective capability and inform how collective capability might look in Indigenous evaluation. Knowledge holders will be informed that their participation in the interview is voluntary and that they may withdraw at any time prior to publication of data, including withdrawal of consent for use of any interview data you have provided. If knowledge holders withdraw from the research, data will be securely destroyed. Knowledge holders will remain anonymous and will be provided with the transcript of their interview to review. Any information that they do not want to be included in the analysis can be removed at this stage. The data text from the interviews will be deidentified prior to analysis. Knowledge holders will provide written consent to participate in the interviews.

We will share the preliminary results with the advisory group for input and sense making of the results and the outcomes. These processes ensure a face validity process is embedded into the methodological approach, aligning with Indigenist approaches. Findings will be disseminated through a peer-reviewed journal; conference presentations and presentations to the advisory group, the Thiitu Tharrmay reference group and Aboriginal and Torres Strait Islander community partners supporting this project. All knowledge holders will be asked if they wish to be invited to forums where the presentation of findings will occur, and for access to publication links related to the research to be provided to them.

In accordance with the Intellectual Property Rights and Moral Rights under the AIATSIS Code of Ethics for Aboriginal and Torres Strait Islander Research, and the Aboriginal and Torres Strait Islander Data Sovereignty Principles,⁵¹ participants have ownership over their data. Team members of the project act as data custodians ensuring data security, data integrity and the ethical sharing of data. A data management plan was developed and approved by the AIATSIS Human Research Ethics Committee, ensuring that data collection, management, storage and ownership processes aligned with the rights of Aboriginal and Torres Strait Islander people to access and control their data.

Discussion

To assist in improving evaluation practice in Australia, a shift in practice is required. An Indigenist collective capability approach may provide a solution. This first requires defining the term and its constituent elements, in context. This requires Indigenous peoples to lead all elements of developing methodologies and methods, decision making and participation as core processes and structures for evaluations in Indigenous contexts.

We are using an existing non-Indigenous framework (Rogers) for this concept analysis because we were unable to identify a suitable Indigenous framework. To overcome any shortcomings of Rogers’ framework, we have modified it to include Indigenous knowledge holders to define collective capability and inform search terms for the review of the literature. This approach to modifying existing frameworks and methods has been used internationally^{52 53} and domestically⁵⁴ to ensure Indigenist elements are incorporated. Further, these modification approaches can then be underpinned by Indigenous worldviews and consistent with the United Nations Declaration of the Rights of Indigenous Peoples (UNDRIP),⁵⁵ creating culturally safe and appropriate processes and structures for the population of interest.^{11 13}

This concept analysis centres Aboriginal and Torres Strait Islander knowledge, experience and expertise to ensure a strengths-oriented methodology that is Indigenous led and aligns with the AIATSIS principles: Indigenous self-determination; Indigenous leadership; Impact and value; and Sustainability and Accountability.⁵⁶ The weaving of Indigenist approaches with a modified Rodgers’ evolutionary method supports a strengths-oriented approach to developing the concept ‘collective capability’. Additionally, engaging with Aboriginal and Torres Strait Islander knowledge holders allows the theoretical development of collective capability from an Indigenous standpoint.

From an ‘Indigenist collective capability’ framework, we expect that the quality of evaluation will be improved, as we move to Indigenous empowerment and leadership in evaluation practices. Including Indigenous methodologies and methods, better cultural understanding of the context in which services and programmes are situated and sense-making of findings of evaluations are ways to enhance the quality of evaluations.2 4 7 8 11 Further, there is a need to create safe spaces for Aboriginal and Torres Strait Islander peoples to actively participate, include an Indigenous lens, and embed such practices as standard practice in evaluation in Indigenous contexts.3 7 13 19

It is possible that knowledge holders in the study may not recognise ‘collective capability’ as being aligned with Indigenous evaluation practice. Collective capability may also be completely new and not present in the literature relating to Indigenous evaluation in Australia and across the world. Although unlikely, the concept may be too abstract to be understood or operationalised. Should the concept not be recognised as ‘collective capability’, other descriptive terms from the interviews and the review of the literature will inform an alternative term.

As systematic shifts are occurring across evaluation practice, Indigenist evaluation practice could become standard practice for evaluations of Indigenous programmes, policies and services that are intended to benefit the Aboriginal and Torres Strait Islander population. However, adoption of new theoretical and methodological frameworks can take time and can also be competing with current western approaches that are already embedded in evaluation practice.²⁴ There is a need to improve the quality and usefulness of programme evaluations to strengthen the evidence base, with a focus on centring Indigenous perspectives, knowledge and priorities in evaluation practice.⁸

This protocol proposes a method for the development of a new model of Indigenist evaluation with the aim of improving Indigenous evaluation practice in Australia. Through the work described, we will define collective capability and establish characteristics, attributes and how collective capability is conceptualised in Indigenous evaluation practice using the concept analysis methodology. Operational definitions could be established through the exploration of the literature and identifying exemplars or models applying collective capability. Lastly, the findings from the concept analysis will help inform the development of a collective capability framework and assessment tool for Indigenist evaluation.

Ethics statements

Patient consent for publication

Not applicable.