Exploring the use of the aboriginal and Torres Strait Islander quality appraisal tool in Indigenous health research

Background

The Aboriginal and Torres Strait Islander Quality Appraisal Tool (QAT) is a 14-item scale designed to assess the quality of Indigenous involvement in Indigenous-related health research [1]. Drawing on four research teams’ experiences with the QAT, we offer insights into implementation, refining it for Indigenous Peoples in Canada.

The QAT focuses on the degree of involvement of Aboriginal and Torres Strait Islander Peoples in research processes, degree of community engagement, the level of community capacity building, and researchers respect for cultural and intellectual property. The QAT provides a four-point criterion scale (Yes, Partially, No, Unclear). To aid users, the QAT authors created a companion document that describes each of the fourteen items and how to assess whether studies meet specified criteria [2].

The QAT is open access and intended for use alongside other suitable critical appraisal tools that prioritize the rigour of study design and methodology [1]. Since publication in 2020, the QAT has gained recognition for its capacity to improve research design and reporting of research findings involving Aboriginal and Torres Strait Islander Peoples as well as research involving Indigenous Peoples in other regions [3–5]. To date, the QAT has not been examined for applicability to Indigenous Peoples beyond Australia. We describe our use of the QAT in four research projects on Indigenous health in Canada (three based in Alberta and one in Ontario). Each team conducted literature reviews (two systematic, two scoping) in a range of topics. One team performed a systematic review and meta-analysis on global Indigenous critical illness incidence and outcomes from inception to 2022 [6]. Another team carried out a systematic review of midwifery outcomes among Indigenous women, based on a rapid review conducted in 2018 [7]. The third team undertook a scoping review to explore existing knowledge on data collection regarding Indigenous identities in healthcare settings and Indigenous healthcare user data globally [8]. The final team conducted a scoping review on social and cultural drivers of brain health, with the publication still pending. Since these projects fall outside the scope of the current paper, readers are encouraged to consult the published papers for more information. Each team used the QAT to appraise the degree and rigour of culturally and community-relevant perspectives in the review findings. Following their separate reviews, each research team provided brief written summaries detailing their research, the application of the QAT, and their experience with using the tool. The teams did not agree in advance to trial the QAT and compare experiences. Instead, we convened after completing each project to discuss the individual experiences and challenges each team’s experienced while using the QAT. We share insights and recommendations from QAT usage to guide other teams considering using the tool.

Insights gained with the QAT

The QAT serves to legitimize and promote Indigenous-led research practices. It can be particularly beneficial for those unfamiliar with evaluating Indigenous specific research studies. Its application highlights its value in assessing research quality from an Indigenous perspective, offering a standardized set of criteria for critically assessing Indigenous engagement, research leadership, governance, and input. We identified several insights to aid other research teams using the QAT, which are described here.

It was important to establish precise definitions of survey items at the outset of the review process to ensure consistency. While the QAT companion document outlines broad definitions for each question and what should be sought in sources [2], practical application demanded further elucidation, especially for questions with subjective components. Challenges arose in defining community inclusion (Q2), Indigenous research leadership, which could range from traditional knowledge holders to elected officials or academics (Q3), and whether the research was beneficial to Indigenous participants and communities (Q12). The four research teams found these questions difficult to answer, especially as such details are often not explicitly discussed in conventional research manuscripts. Additionally, it may be hard to ascertain Indigenous leadership involved in manuscript development or study design from an author list alone or if authors do not deliberately report all aspects of their leadership roles. Each team shared their unique processes to adapt these questions to their own criteria. For instance, one team noted that for Q2 they decided that securing community consent did not meet the “yes” threshold for community consultation and engagement. Instead, they rated the article as “partial.” This reveals divergences in the tool’s applications worth deeper consideration and anticipation by those using it.

Distinguishing between assigning a ‘partial’ rating or opting for ‘unclear’ when appraising questions was not always straightforward. Achieving consensus between multiple raters within each team proved crucial in establishing scoring parameters. Items eliciting clear ‘yes’ or ‘no’ responses, such as whether Indigenous intellectual and cultural property access and ownership agreements were negotiated (Q6 and Q7), were typically easiest to identify. A recommendation from one research team, embraced by the other teams, suggested that raters maintain records of their decision-making process to justify their ranking choices, informing later consensus discussions. The QAT companion document was helpful as it offered practical examples and provided aspects for raters to consider in their evaluation [2].

A significant hurdle acknowledged by the research teams and other studies using the QAT [3, 4] is the omission, passive or deliberate, of pertinent information within research papers that could aid in the evaluation of various questions. For example, Q2 is an assessment of community consent to the research, yet a considerable number of papers do not report on the early engagement or consent processes. Qualitative and multi-method manuscripts tend to need more extensive explanations than quantitative analysis, placing such work at a disadvantage in certain publications with strict wordcount limits [9]. Consequently, these limits and journal priorities may hinder the inclusion of necessary information regarding Indigenous engagement, Indigenous data ownership, control, access, and possession, adherence to community protocols, policy impacts, and benefits, and capacity-building for Indigenous Peoples [3, 10]. Such gaps could signify differing priorities between researchers/academic institutions and Indigenous communities [11], reflecting disparities in ethical and grant committee priorities, as well as challenges associated with incorporating such details within the rigid parameters of journal publications [3]. Notably, authors adherence to the consolidated criteria for strengthening the reporting of health research involving Indigenous Peoples (CONSIDER) statement, which provides a checklist synthesizing the existing national and international statements and guidelines regarding health research involving Indigenous Peoples, could mitigate many concerns if systematically upheld by editors [12].

Despite calls for equitable and beneficial research with Indigenous Peoples and communities, the research teams noted that engagement and inclusion were not always facilitated at a high level. Instead of fostering active partnerships and co-development, efforts tended to focus on more attainable forms of engagement, such as hiring an Indigenous community member to work on the project. A commonly reported form of Indigenous involvement was denoted in Q3 (Did the research have [Indigenous] leadership?), where the four research teams defined leadership as authorship on the manuscript. While academic authorship typically signifies research leadership within a Western academia, it may not align with Indigenous communities’ definition of research leadership. Other reported forms of Indigenous leadership include positions in peer review, policy development, merit review practices, and adjudication of research proposals [13]. However, what constitutes research leadership should be determined collaboratively with the Indigenous partners and communities with whom researchers work.

Lastly, although the QAT serves as a valuable tool for Western-trained researchers to evaluate Indigenous involvement in research, a 14-item questionnaire format constrains Indigenous epistemologies often founded upon relationality [14]. Despite the comprehensiveness of the QAT, we note that it lacks questions promoting researcher self-reflection and exploration of personal and professional growth stemming from relationships, reciprocity, and engagement in the research process. These limitations highlight the need for ongoing dialogue and adaptation to ensure that research practices in Indigenous contexts honor and integrate these essential values and principles.

Recommendations for QAT users

Prior to use, it is essential for research teams to engage in a collaborative consultation with Indigenous Peoples, organizations, communities, Elders, Knowledge Keepers, community members, and/or advisory committees to which the research is accountable, to determine appropriateness and suitability of the questions for the people and communities most affected. As an integral part of this initial process, researchers should outline the definitions and criteria underpinning each question to ensure transparency and consistency. Previous studies utilizing the QAT have often provided a brief outline of how some items were qualified [3, 15].

Another suggestion is to decide whether to include external resources when rating. Since publications may have limited information on Indigenous involvement, some may receive lower scores on the QAT. Research teams should establish beforehand whether they will rely solely on the material reported within the manuscript or include additional information from external sources, such as cohort profile papers, researcher websites and profiles, other publications, or direct communication with research teams. While the QAT evaluates research quality rather than the quality of reporting, these are intrinsically interconnected and researchers should consider reporting standards when determining the level of additional information to include in QAT scoring.

We recommend employing at least two raters on each source to ensure a high degree of agreement and consensus of ratings. All four research teams in our study utilized at least two raters for article reviews. In cases of rating discrepancies, the reviewers convened to discuss and reach a consensus. This approach aligns with the literature’s support for using two raters—and previous applications of the QAT [3, 5]—as it enhances inter-rater reliability and ensures adherence to guidelines [16, 17]. Although the four research teams did not calculate inter-rater reliability, we recognize that doing so could have further strengthened our reviews. By implementing these recommendations, QAT users can enhance the accuracy and reliability of their assessments while promoting transparency in their research.

Finally, while the QAT has demonstrated utility in retrospectively evaluating levels of Indigenous engagement, its effectiveness could be enhanced by adapting it for application during the research design phase. This broader scope could encourage teams to formulate research questions that reflect local concerns, engage in co-designing methodologies, make informed decisions in selecting analytical approaches, interpret findings contextually, and ensure ongoing involvement throughout a study. Embracing this proactive approach aligns with the evolving landscape of collaborative research methodologies, decolonizing research, mitigating the risk of tokenistic engagement of Indigenous peoples, addressing power imbalances, and contributing to more equitable and respectful research practices.

Limitations

The opinions of four Canadian health research teams cannot fully capture the broad spectrum of perspectives or research contexts. It is important to note that certain team members utilizing the tool are non-Indigenous. A promising avenue for future research involves examining the validity of the QAT within local Indigenous contexts. This engagement would allow for a more context-specific assessment of the QAT’s applicability, ensuring that it aligns with the unique needs and perspectives of Indigenous communities. Additionally, a valuable area for investigation is the evolution of reporting practices in Indigenous research and health sciences more broadly over time.

Conclusions

By initiating the first discussion on perspectives of applying the QAT across diverse review topics, we aim to offer practical guidance for researchers interested in this appraisal tool. While we found the QAT to be useful for assessing Indigenous Peoples’ involvement in health research, we recommend that the tool be contextualized by local Indigenous Peoples to determine relevance and suitability. Additional recommendations include establishing clear definitions and criteria among raters, determining whether to include external resources during the rating process, employing at least two raters to ensure guideline adherence, and adapting it for application during the design phase. Consensus regarding decision rules and resource inclusion should be established prior to the application of the appraisal tool.

Abbreviations

CONSIDER

CONSolIDated critERtia for strengthening the reporting of health research involving Indigenous Peoples (CONSIDER) statement

QAT

Aboriginal and Torres Strait Islander Quality Appraisal Tool

Author contributions

SW participated in the design of the work and interpretation of the details from each research team’s experience with the QAT, and drafted the main manuscript. KW, MG, SMB, SLB, SAE, LF, RIH, KL, DAM, AV, JW, and RTO participated in the design of the work, the interpretation of details, and manuscript revision and drafting. All authors read and approved the final manuscript.

Funding

This work was funded by the Indigenous Wellness Core Scientific Office of Alberta Health Services.

Data availability

No datasets were generated or analysed during the current study.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.