Evaluation of evidence-based literature and formulation of recommendations for the clinical preventive guidelines for immigrants and refugees in Canada

Abstract

Background:

This article describes the evidence review and guideline development method developed for the Clinical Preventive Guidelines for Immigrants and Refugees in Canada by the Canadian Collaboration for Immigrant and Refugee Health Guideline Committee.

Methods:

The Appraisal of Guidelines for Research and Evaluation (AGREE) best-practice framework was combined with the recently developed Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to produce evidence-based clinical guidelines for immigrants and refugees in Canada.

Results:

A systematic approach was designed to produce the evidence reviews and apply the GRADE approach, including building on evidence from previous systematic reviews, searching for and comparing evidence between general and specific immigrant populations, and applying the GRADE criteria for making recommendations. This method was used for priority health conditions that had been selected by practitioners caring for immigrants and refugees in Canada.

Interpretation:

This article outlines the 14-step method that was defined to standardize the guideline development process for each priority health condition.

Primary care practitioners who care for recently arrived immigrants and refugees have raised concerns over the lack of evidence-based guidelines for clinical prevention, noting that it is not always clear whether current recommendations made for the general population in Canada can be generalized to this population. In 2006, the Canadian Collaboration for Immigrant and Refugee Health (CCIRH) Guideline Committee was formed to address this issue by first identifying the top-priority health conditions for this population. The group of 20 health conditions identified was very diverse ranging from infectious disease to chronic conditions including depression. The challenge was creating a rigorous interdisciplinary process and then to generate pragmatic recommendations. This document outlines the systematic approach designed to produce the evidence reviews.

A variety of methods is used for developing clinical guidelines and practice recommendations.¹ We used the recently developed approach of moving away from recommendations classified by letters and numbers to the simplified classification system recommended by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group² and applied this to clinical preventive actions. Our guideline development process followed the Appraisal of Guidelines Research & Evaluation (AGREE) instrument (www.agreetrust.org), which is recognized internationally as providing best-practice criteria for evidence-based guideline development.

We developed the recommendations on the basis of a pre-specified process overseen by the CCIRH Guideline Committee. Defining a methods process ensured that each guideline was developed in a systematic, reproducible manner and was based on the best evidence available. This process was based on existing guidelines including the Canadian Medical Association Journal handbook on developing clinical practice guidelines¹ and the ADAPTE framework for adapting existing guidelines.³ Our process emphasized identifying immigrant- and refugee-specific evidence on efficacy and population characteristics from guidelines, systematic reviews and primary studies. When immigrant- and refugee-specific evidence was unavailable, we used specific criteria, adapted from the Cochrane Handbook,⁴ to judge how this evidence applied to our intended target population.

Conditions considered most important by practitioners caring for immigrants and refugees in Canada were assigned to groups of content experts to develop evidence reviews with clinical conclusions for recent immigrants and refugees to Canada using a logic model and following a structured 14-step process. The guidelines focus on clinical care gaps¹ during the “health settlement period,” which we define as the first five years of residence in a new country for an immigrant or refugee. This is the time in which health practitioners are likely to have initial contact with this population and the time during which stressors from one’s country of origin and country of settlement are most likely to manifest. Immigrants and refugees are thus grouped together by this organizing period of resettlement; however, the heterogeneity, complexities, and differences between and within these groups were recognized throughout the process.

In our process, we emphasized making clinically relevant recommendations and establishing an extension to existing guidelines rather than a replacement or revision.

Methods

The AGREE checklist was used to guide the overall development process: a panel of experts and a guideline committee set the scope and purpose of the guidelines; stakeholders were engaged to select priority conditions and to merge recommendations. To ensure rigour and applicability, we developed 14 standardized steps (Box 1). The guideline committee and other guideline experts and practitioners provided feedback to improve clarity of presentation, and we accepted funding only from university and government sources to ensure editorial independence. Here we describe our standardized evidence review.

Box 1: 14-step methods process of the Canadian Collaboration for Immigration and Refugee Health.

1. Develop clinician summary table

2. Develop logic model and logic model key questions

3. Set the stage for admissible evidence (search strategy)

4. Assess eligibility of systematic reviews

5. Search for immigrant and refugee population-specific data

6. Refocus on key clinical preventive actions and logic model key questions

7. Assess quality of systematic reviews

8. Update reference systematic reviews

9. Assess eligibility of new studies

10. Integrate data from updated search

11. Synthesize final evidence bank and draft two key clinical actions

12. Develop summary of findings table

13. Identify gaps in the evidence and needs for future research

14. Develop clinical preventive recommendations using GRADE

Step 1: Develop clinician summary table

A standardized clinician summary template (Table 1) was used to guide setting the framework for each selected condition. During the subsequent steps, this clinician summary table was used to focus development of the preventive guidelines, on the basis of the condition’s prevalence in the population of interest, population-specific clinical considerations (e.g., stigma and awareness of screening and treatment options), clinical actions upon migration, screening tests, screening interval or timing, and treatment.

Table 1:

Clinician summary table: template for review teams

Population	Immigrant/refugee
	Adults	Children
Clinical preventive action	A: service X is recommended	D: service X is not recommended
Population-specific clinical considerations	Equity considerations: Baseline risk (incidence/prevalence) v. the Canadian general population Rate of clinically important beneficial and harmful outcomes (mortality and morbidity) Genetic and cultural factors (e.g., knowledge, attitudes, practice, cultural preferences, diet preferences) Adherence variation: physicians’ or patients’ adherence to recommendations
Clinical actions during migration	During migration, refugees and immigrants are/are not screened/treated for condition X (based on the Citizenship and Immigrant Canada Immigration Medical Examination, and the International Organization for Migration predeparture screening/treatment)
Screening tests	Consideration of performance of tests and instruments
Screening interval	When relevant
Treatment	Existence of treatments with effectiveness to be determined in evidence reviews
Other recommendations	The Canadian Task Force on Preventive Health Care also recommends screening for this condition; their recommendation states that ...
Implementation issues and cost reference	Cost of treatment Barriers to provision of services The technical document for this evidence review can be found at www.ccirh.uottawa.ca

Step 2: Develop logic model and key questions

The logic model presented in Figure 1 illustrates a causal pathway for the guideline; our logic model was adapted from the US Preventive Services Task Force⁵ with the addition of a box to consider patient perspectives. The logic model outlines the population of interest (immigrants/refugees), the intervention (i.e., screening), the target condition, adverse effects of screening/diagnosis and treatment, treatment options and outcomes, and the link between treatment and reduced morbidity and mortality. The model illustrates how one expects identification of the condition to lead to treatment and reduced morbidity and mortality in the population of interest. This logic model identified the need to consider whether intermediate outcomes would be accepted as the basis for the recommendations, and if so, the strength of association between intermediate and clinical outcomes. For example, high-risk behaviour is an intermediate outcome in reducing morbidity and mortality from HIV (Figure 1).⁶ Review group leaders were asked to use this logic model to define the PICO (Population, Intervention, Comparison, Outcome) format for each clinical action. These elements guided the search for evidence and are outlined in Table 1.

Figure 1:

Sample logic model for HIV (adapted from US Preventive Services Task Force).⁵ Open rectangles designate the potential screening population and patient factors to be considered; shaded rectangles designate interventions and related outcomes; and circles and numbers provide points in the evidence chain that were used to develop the search questions. Note: ARV = antiretroviral, EIA = enzyme immunoassay.

Step 3: Set the stage for admissible evidence

We followed the process used by the US Task Force and the Canadian Task Force on Preventive Health Care to focus on evidence most critical to making a recommendation.¹ We began with a search of specific guidelines and systematic reviews for the target population of immigrants and refugees to document the current state of direct evidence. We extended these searches to capture evidence from the general population. The search strategy was modelled on that used in the Cochrane Collaboration⁷ and was conducted by one of two clinical librarians. The following databases were searched: MEDLINE, PreMEDLINE, Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effectiveness, EMBASE, CINAHL, National Guideline Clearing House, and the CMA Infobase. We also searched the databases and publications of the Canadian Task Force, the US Task Force, the Centers for Disease Control and Prevention, and the World Health Organization. We asked authors to create flow charts of their searches, using the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA)⁸ framework as a template (Appendix 1, available at www.cmaj.ca/cgi/content/full/cmaj.090289/DC1).

Step 4: Assess eligibility of systematic reviews

Search strategies, abstracts, and relevant full-text articles were reviewed independently by two members of the review group on the basis on the inclusion criteria and specified outcomes of interest.

Data from each eligible systematic review were extracted and documented in a table with the following headings: author and year, objective, number and types of studies included, setting, participants, intervention and findings (see Appendix 2 for example, available at www.cmaj.ca/cgi/content/full/cmaj.090289/DC1). If no eligible systematic review was found, then the review group team searched for the next best available study (i.e., randomized controlled trials, observational studies) that addressed the question.

Step 5: Search for immigrant and refugee population-specific data

A tailored search process was used to gather information on population-specific considerations relevant to immigrants and refugees in the following areas:

1. Baseline risk (prevalence) versus the Canadian general population;

2. Rate of clinically important beneficial and harmful outcomes (e.g., mortality, morbidity);

3. Genetic and cultural factors (e.g., knowledge, attitudes, practice, cultural preferences, diet preferences); and

4. Compliance variation (e.g., physician and patient adherence to recommendations).

Step 6: Refocus on key clinical preventive actions and key questions

After reviewing the literature and available evidence, review group teams were asked to focus on the most relevant clinical action(s) and immigrant and refugee subpopulation(s) and to select three or fewer candidate recommendations with added value over and above existing guidelines.

Step 7: Assess quality of systematic reviews

For each recommendation, all relevant systematic reviews were compared to ensure consistency among findings. If the conclusions of the systematic reviews were consistent, the most recent review was selected. Any inconsistencies in reviews were explicitly addressed: reasons for inconsistencies including the evidential base or the interpretation were explored, and the most appropriate systematic review was selected, considering the purposes of these guidelines.

The most relevant systematic reviews were then assessed for quality to ensure they met the four criteria assessed in the National Institute for Health and Clinical Excellence (formerly the Health Development Agency) critical appraisal tool for evidence-based briefings or reviews of reviews:⁹ systematicity (the review must apply a consistent and comprehensive approach); transparency (the review must be clear about the processes involved); quality (the review must have appropriate methods and analysis); and relevance (the review must be relevant in terms of focus — i.e., populations, interventions, outcomes and settings).

Step 8: Update reference systematic reviews

To find new primary studies published since the selected systematic review, a search using the same approach as in Step 3 was conducted.

Step 9: Assess eligibility of new studies

As in Step 4, two reviewers independently screened for relevant studies, then assessed each study for eligibility. Each relevant study was summarized to describe study design, description of the clinical intervention, details about length of intervention and follow-up, outcomes, population characteristics, and data analysis.

For studies evaluating the effectiveness or safety of treatment or screening, the Cochrane Effective Practice and Organisation of Care Review Group’s data collection checklist¹⁰ and the Newcastle-Ottawa Scale¹¹ for assessing the quality of nonrandomized studies in meta-analyses were used to assess study limitations.

Step 10: Integrate data from updated search

Any new relevant and eligible studies that could modify or substantially strengthen the conclusions of the “reference” systematic review were assessed and added to the worksheet.

Step 11: Synthesize final evidence bank and draft two key clinical actions

Review group teams synthesized the evidence from the updated systematic reviews explicitly incorporating immigrant and refugee population-specific clinical considerations and value judgments in order to draft preferably no more than two key clinical actions, targeting (where necessary) specific populations or regions.

Step 12: Develop summary of findings table

Both desirable and undesirable effects of the intervention were summarized in both absolute and relative terms for each patient-important outcome using the summary of findings table format adopted by the Cochrane Collaboration¹² (Table 2).¹³ The quality for each outcome was assessed using the items specified by the GRADE Working Group (indirectness, consistency, precision, reporting bias and study limitations). Observational studies that met these five criteria were upgraded if they also met one of three additional criteria (dose-response, influence of confounding variables, large effect).² A separate table was developed for each clinical action or question. For dichotomous outcomes, relative risk or odds ratios were extracted from the reference systematic review (or next best available study). Number needed to treat for one person to benefit (NNT) was calculated as: 1/(control event rate*[1-relative risk]. The control event rate was taken from the control group of the reference systematic review or best available study.

Table 2:

Sample summary of findings table on treatment with antiretrovirals for HIV and AIDS

Patient or population: Patients with HIV/ AIDS

Setting: Outpatient clinics in Australia, Europe, North America

Intervention: Treatment with three antiretrovirals

Comparison: Treatment with two antiretrovirals

Source: Enanoria WTA, Ng C, Saha SR, et al. Treatment outcomes after highly active antiretroviral therapy: a meta-analysis of randomised controlled trials. Lancet Infect Dis 2004;4:414-25.13

Outcomes	Absolute effect*		Relative effect (95% CI)	No. of participants (studies)	GRADE quality of evidence	Comments (95% CI)
	Risk for group treated with 2 antiretrovirals	Difference with treatment with 3 antiretrovirals (95% CI)
Death	20 per 1000	8 fewer per 1000 (1 fewer to 9 fewer per 1000)	RR 0.62 (0.45–0.86)	3979 (15)	Moderate†	NNT 132 (91–357)
Treatment efficacy (number of people achieving cut-off value of HIV RNA)	220 per 1000	750 more per 1000 (119 more to 780 more per 1000)	RR 4.41 (1.54–12.62)	1932 (6)	Low‡§	NNT 1 (1–8)
Withdrawals due to adverse effects	100 per 1000	81 more per 1000 (38 more to 198 more per 1000)	RR 1.81 (1.19–2.79)	6380 (11)	Low†§	NNH 12 (6–53)

Note: CI = confidence interval; GRADE = Grading of Recommendations Assessment, Development and Evaluation; NNH = number needed to harm; NNT = number needed to treat; RR = risk ratio.

^*The basis for the assumed risk (e.g., the median control group risk across studies) is provided in footnotes. The corresponding risk (and its 95% CI) is based on the assumed risk in the comparison group and the relative effect of the intervention (and its 95% CI).

^†Consistency downgraded because test for heterogeneity was statistically significant.

^‡Indirect because viral load is indirect outcome for mortality and morbidity.

^§Adverse events are indirect outcome for severe side effects.

Step 13: Identify gaps in evidence and needs for future research

Review group teams identified gaps in the literature and outlined recommendations for future research on such topics as implementation, inequalities and vulnerable groups, cost-effectiveness, and implications of applying the recommendations in health care settings.

Step 14: Develop clinical preventive recommendations

For each condition, the guideline committee reviewed the clinician summary table, the logic model, and the summary of findings tables, and met with the review group leader to clarify details. Then, for each key clinical action, the guideline committee discussed each of the issues in the GRADE system^14,15 (Table 3): the balance between desirable and undesirable effects — the relative importance of burden, benefits, harms; values and preferences; and quality of the available evidence. We explicitly decided not to use cost and feasibility in judging the basis of the recommendation because we did not have sufficient confidence in the data. Rather than report the strength of the recommendation as weak or strong, the guideline committee chose to make the recommendation only in the event of net benefits and report the basis for the recommendation to provide clinicians with key information to consider when selecting or discussing the preventive recommendation with a patient. The guideline committee took votes if the agreement was not unanimous, and majority prevailed.

Table 3:

Basis of recommendations (adapted from GRADE^14,15)

Issue	Process considerations
Balance between desirable and undesirable effects	Those with net benefits or trade-offs between benefits and harms were eligible for a positive recommendation
Quality of evidence	Quality of evidence was classified as “high,” “moderate,” “low,” or “very low” on the basis of methodologic characteristics of available evidence for a specific clinical action
Values and preferences	Values and preferences refer to the relative worth or importance of health state or consequences of following a particular clinical action

Discussion

This 14-step process was useful for ensuring sufficient uniformity among the transdisciplinary teams for each condition. Specifically, this systematic approach enabled the review group teams to meet the requirements of the GRADE quality-assessment process and the steering group to apply the GRADE recommendation process. These steps were also designed to conform with AGREE, the current quality standard for guidelines. We worked with each review group leader and team to assure we met the 23 AGREE criteria in the six domains: scope and purpose, stakeholder involvement, rigour of development, clarity and presentation, applicability, and editorial independence (Box 2).

Box 2: Six domains of AGREE criteria as applied to the clinical preventive recommendations for immigrants and refugees to Canada.

1. Scope and purpose: The logic model was helpful in guiding and focusing discussion on the most important clinical actions.

2. Stakeholder involvement: A group of 40 primary care physicians with experience in working with refugees and immigrants from across Canada participated in the Delphi survey and in guideline development. Several Delphi participants have pretested the recommendations in their clinical practices across the country. Public involvement is especially challenging given the global origins of immigrants and refugees. So far, we have consulted with one community group, Edmonton Multicultural Health Brokers Cooperative, representing 18 ethnic communities.

3. Rigour of development: Despite the proliferation of high-quality systematic reviews in the Cochrane library and elsewhere in the literature, few articles offered a systematic review or existing evidence-based guideline that addressed the relevant question in immigrants and refugees. Thus, for most topics, we had to conduct our own systematic literature review. In most cases, no systematic reviews were available even for nonimmigrants (i.e., the general population). We recognized the importance of explicitly stating how we decided on the best evidence; details are contained in the evidence review for each condition. We considered values and preferences, but found limited data and methods to guide systematically synthesizing and incorporating values and preferences. Cost and feasibility issues emerged in all evidence reviews and in discussions of implementing and disseminating recommendations; however, given the differences between provinces and territories, we were unprepared to introduce these issues formally when judging the basis of a recommendation. In certain cases (e.g., dental and vision screening), evidence pointed to the need for system changes to improve the feasibility and cost-effectiveness of basic preventive actions for immigrants and refugees. To incorporate new evidence, we will be exploring options for regularly updating these guidelines.

4. Clarity and presentation: The guidelines will be available in several formats tailored to the user in English and French. Consumer or patient summaries will need to be modified to take into consideration the various languages of patients receiving health services. It will be important to consider different communication tools that fit the needs of the target audience. For example, some health care providers could want a simple checklist, others a single sheet or poster to paste on the wall. Reviews summarizing the evidence for each condition will be available on the CMAJ website (cmaj.ca) and technical documents describing the 14-step process will be available on the CCIRH website (www.ccirh.uottawa.ca).

5. Applicability: In preparing these guidelines, we have assumed that most primary care physicians will not have access to additional resources to support their care of immigrants and refugees from developing countries. Recommendations are supplemented by evidence reviews that provide clinical considerations and practice-level suggestions. Methods for measuring adherence and clinical outcomes related to these guidelines will be needed to assess the utility of these guidelines.

6. Editorial independence: These guidelines have been developed with university and government funding only. The views or interests of these funding bodies have not influenced the final recommendations. All contributors have provided explicit statements declaring whether they have any conflict of interest.

Conclusion

This systematic approach enabled the review group teams to meet the requirements of the GRADE quality assessment process and the guideline committee to apply the GRADE recommendations. This process also fulfilled the AGREE criteria, the current quality standard for guidelines.

Key points

• We combined the AGREE best-practice framework with the recently developed GRADE approach to develop evidence-based clinical preventive guidelines for immigrants and refugees to Canada.

• This methods paper documents the systematic approach used to produce the evidence reviews and apply the GRADE approach.

• The 14-step approach included building on evidence from previous systematic reviews, searching for and comparing evidence between general and specific immigrant populations, and applying the GRADE criteria for making recommendations.

• For each recommendation, the basis (balance of benefit and harms, quality of evidence, and values) is stated explicitly to ensure transparency.