Key Ingredients of Anti-Stigma Programs for Health Care Providers: A Data Synthesis of Evaluative Studies

Abstract

Objective:

As part of its ongoing effort to combat stigma against mental illness among health care providers, the Mental Health Commission of Canada partnered with organizations conducting anti-stigma interventions. Our objective was to evaluate program effectiveness and to better understand what makes some programs more effective than others. Our paper reports the elements of these programs found to be most strongly associated with favourable outcomes.

Methods:

Our study employed a multi-phased, mixed-methods design. First, a grounded theory qualitative study was undertaken to identify key program elements. Next, each program (n = 22) was coded according to the presence or absence of the identified key program ingredients. Then, random-effects, meta-regression modelling was used to examine the association between program outcomes and the key ingredients.

Results:

The qualitative analysis led to a 6-ingredient model of key program elements. Results of the quantitative analysis showed that programs that included all 6 of these ingredients performed significantly better than those that did not. Individual analyses of each of the 6 ingredients showed that including multiple forms of social contact and emphasizing recovery were characteristics of the most effective programs.

Conclusions:

The results provide a validation of a 6-ingredient model of key program elements for anti-stigma programming for health care providers. Emphasizing recovery and including multiple types of social contact are of particular importance for maximizing the effectiveness of anti-stigma programs for health care providers.

As part of its OM Anti-Stigma Initiative, the MHCC partnered with organizations and investigators conducting anti-stigma interventions targeting various health care provider groups in Canada, with the purpose of evaluating program outcomes.¹ Using existing evidence on the value of social contact^2–5 as an initial point of departure, OM partnered with programs using some form of social contact or contact-based education in the delivery of their program. Typically, social contact-based approaches emphasize the inclusion of planned exchanges between people with lived experience of mental illness and the target audience as a part of the program curriculum.⁵ In many cases, target audiences hear personal stories from, and (or) interact with, people who have recovered or are successfully managing a mental illness.

While all programs evaluated by OM included some form of social contact, the extent and nature of the contact varied from program to program, as did many other characteristics, including program length, educational emphasis, program context and delivery features, and target audience (for example, practicing professionals, compared with students). Online eTable 1 contains a description of the various partner programs, their targeted audiences, and their main program elements. To enhance the comparability of the various studies, OM developed and adopted a common outcome scale, the OMS-HC,^6,7 and had data-sharing arrangements with its partners.

Two RCTs were first conducted to confirm the general effectiveness of the contact-based approach. Both trials returned positive results.^8,9 Subsequently, with efficacy confirmed, the goal became the identification of characteristics associated with maximal effectiveness. Most of the evaluative studies used a before-and-after study comparison to evaluate effectiveness, and data collected in this way became the main source of data for assessing program characteristics or key ingredients associated with the best outcomes.

With 22 total pre–post data sets from a diverse set of studies (but all using the OMS-HC), it became necessary to identify a systematic approach to quantifying the outcomes associated with each potential key ingredient. Analysis of individual study results had not identified individual characteristics (such as age, sex, or whether a person had a friend or close relative with a mental illness) as being significant determinants of outcome.^1,8,9 For this reason, we chose a strategy based on contrasting study-level characteristics using methods commonly employed in meta-analysis, including meta-regression. These techniques can accommodate heterogeneity across studies and provide a method of weighting the contributions of larger and smaller studies when generating pooled effect estimates. Implementation of the overall strategy required a multi-phased, mixed-methods approach. First, a qualitative study was required to identify potentially important program characteristics and to accurately classify each intervention according to those characteristics. Next, the aforementioned quantitative strategies were used to evaluate the impact of these characteristics on outcomes.

In our paper, we report the comparative evaluation of anti-stigma interventions affiliated with OM, including the elements of these programs found to be associated with the most favourable outcomes.

Clinical Implications

• Anti-stigma interventions incorporating social contact are effective in a broad range of health care providers and trainees.

• Programs that include a recovery emphasis, personal testimony from a trained speaker who has lived experience of mental illness, that employ multiple forms of social contact, that teach skills involving what to say and what to do, that employ myth-busting, and that use an enthusiastic facilitator perform significantly better than programs that include only some of these ingredients.

• A recovery emphasis and having multiple forms of social contact are especially critical for maximizing outcomes.

Limitations

• The studies evaluated here consisted of before-and-after comparisons and were usually uncontrolled.

• Considerable heterogeneity was observed even after modelling for intervention ingredients. Other important determinants of outcomes remain to be identified.

• Most of these evaluations were short-term, leaving unanswered questions about the long-term effects of anti-stigma interventions.

Methods

Our objective was to identify and validate program characteristics most predictive of positive outcomes. To accomplish this task, a multi-phase, mixed-methods approach was undertaken.¹⁰ First, qualitative methods were used to identify key intervention elements believed to be integral to reducing stigmatizing among health care providers. Then, quantitative techniques were used to examine the association between program outcomes and the key ingredients. Methods are described in more detail below.

Phase 1: Qualitative—Identifying Key Intervention Elements

The first phase was a qualitative examination of partner programs using a grounded theory methodology.^11–13 The purpose of this investigation was to explicate the process for designing and delivering successful anti-stigma programs for health care providers, which included the identification of program elements believed to be important for maximizing stigma-reduction outcomes.¹⁴ Data collection took place between January 2013 and November 2013 and proceeded through the method of theoretical sampling.¹³ Triangulation was ensured by collecting data from multiple sources and using multiple methods,¹⁵ including: in-depth interviews with program facilitators and instructors, people with lived experience of mental illness involved in program delivery (for example, as first voice speakers, client educators, or program co-facilitators), and other program stakeholders (n = 23); direct observation of programs (n = 8); analysis of qualitative feedback from program participants (n = 1812); supplementary follow-up or clarification interviews with program stakeholders to glean additional program details or information needed for saturation of emerging categories and themes (n = 12)16; and, a review of available program documents (for example, facilitators’ manuals, program syllabi, participant hand-outs or supplementary resources, marketing materials, and program reports) (n = 48).

Data analysis proceeded through grounded theory’s constant comparison method, using the open-axial-selective-theoretical procedure for coding.^11–13 A single coder was used. Open (line-by-line) coding was first undertaken to identify themes and key ideas in the data. Axial coding was then employed to specify the thematic categories and to describe each theme to the point of theoretical saturation.^17,18 Then, selective and theoretical coding was used to identify the relations among the categories and themes and to generate the final theoretical model. Follow-up interviews with key stakeholders were used as a verification or member-checking exercise and for confirmation of internal validity.¹⁶ Theoretical memoing was ongoing throughout the process of data collection and analysis.^11–13,18

Phase 2: Quantitative—Validating Key Intervention Elements

Data from 18 before-and-after evaluation studies and RCTs were included in the quantitative analysis. Both arms of the RCT studies provided largely independent assessments of (before-and-after) outcomes because the control groups in these RCTs also received the interventions (for ethical reasons) at a later date than the active treatment groups.^8,9 Results from each RCT were thus included in the quantitative analysis as 2 separate estimates. This resulted in a total of 22 separate pre–post data sets included in the analysis.

Each program (n = 22) was coded according to the presence or absence of 6 key program ingredients identified through the qualitative investigation. These 6 key ingredients were analyzed as binary variables. A single coder, with in-depth knowledge of the various programs and their content, coded the programs according to the presence or absence of identified ingredients. Coding guidelines were derived from the detailed descriptions of the 6 program elements generated through the qualitative research. In the event of a coding uncertainty, program materials were reviewed and (or) programmers were contacted to clarify program details.

The main outcome measure was the pre-to-post mean change in total OMS-HC scores. In addition, 12 programs had 3-month follow-up data available. For these programs, we also analyzed pre-to-follow-up mean score changes against identified key ingredients. Among the 22 total programs evaluated, all but 3 used the full 20-item OMSHC scale. Three programs used a shorter, 12-item version of the scale.⁶ Scores from the 12-item scale were rescaled to the 20-item version for inclusion in the analysis.

The data synthesis used a random effects model for meta-analysis of aggregate data to investigate the association between the mean changes and each key ingredient. Weighting of the studies in the meta-analysis was based on the inverse of the variance of the study’s estimated effect. Forest plots were used to visually represent study outcomes. The random effect model was chosen because it accounts for both random variability and the variability in effects among the studies. We calculated the I² value to measure heterogeneity.¹⁹ This statistic represents the proportion of variance between studies that is due to heterogeneity. Data were analyzed using the metan commands in Stata, version 12.²⁰

Results

Phase 1: Key Intervention Elements

The grounded theory analysis resulted in the generation of a theoretical model describing the process for designing and delivering successful anti-stigma programs for health care providers.¹⁴ The model included 6 intervention elements viewed as being particularly important for reducing stigma by improving attitudes and behavioural intentions. These elements are as follows: that the program should include social contact in the form of a personal testimony from a trained speaker who has lived experience of mental illness; that the program should employ multiple forms or points of social contact (for example, a presentation from a live speaker and a video presentation, multiple first-voice speakers, multiple points of social contact between program participants, and people with lived experience of mental illness); that the program should focus on behaviour change by teaching skills that help health care providers know what to say and what to do; that the program should engage in myth-busting; that the program should use an enthusiastic facilitator or instructor who models a person-centred approach (that is, a person-first perspective as opposed to a pathology-first perspective) to set the tone and guide program messaging; and that the program should emphasize and demonstrate recovery as a key part of its messaging. The theme of recovery was articulated to mean that programs should emphasize that recovery from mental illness is both real and probable, and show what recovery looks like by demonstrating competence and successful living of people with lived experience of mental illness. Knaak and Patten¹⁴ provide more complete descriptions of identified key intervention elements and other results of the qualitative research.

An additional ingredient was also identified in phase 1—providing a refresher or booster sessions as a way to sustain positive effects over time.^1,14,21–23 As only 3 of the programs for which we have outcome data available included booster and refresher sessions as part of their program curriculum, this ingredient could not be fully evaluated in subsequent phases of the study.

Phase 2: Validation

Online eTable 1 presents the characteristics of the 22 studies, which included a total of 1812 before-and-after OMS-HC assessments. In each case, a participant’s postintervention score was subtracted from their pre-intervention score, and the mean change seen in a particular program was used to quantify the impact of that program. Programs marked with an asterisk are those for which 3-month follow-up data were available.

All programs showed a trend toward lower stigma scores postintervention, although individual study effects were not always statistically significant at the 5% level of significance (Figure 1). The vertical line represents no effect, and all studies whose 95% confidence interval crossed this null value failed to achieve statistical significance. The overall mean change in OMS-HC was 2.5 points, with a 95% confidence interval arising from a random-effects pooling of 1.8 to 3.1. The tau-squared value (the between-study variance) for this model was 1.6. Analysis of mean score changes from baseline to follow-up for the 12 programs with available data showed that these changes were largely sustained at 3-months’ postintervention (overall mean change, −2.4 points; 95% CI −3.3 to −1.6).

Figure 1.

Forest plot of mean difference in Opening Minds Scale for Health Care Providers score, all studies included

The joint test of the full meta-regression model, including all 6 theorized key ingredients, showed evidence for association of these covariates with the size of the treatment effect (F = 3.13, df = 6/15, P = 0.03) and an I² of 58.5%, which indicates that a moderate proportion of the residual variation was attributable to heterogeneity.

Programs that contained all 6 ingredients performed significantly better than those that did not (Table 2, Figure 2). Individual analyses of each of the ingredients showed that programs that included multiple forms or points of social contact had significantly larger score improvements than programs that did not include this program ingredient (Table 2, Figure 3). Programs that emphasized recovery also performed significantly better than those that did not (Table 2, Figure 4). There was a trend toward programs that more effectively set the tone, having better outcomes, but this did not achieve statistical significance (P < 0.10). The other individual ingredients were not significantly associated with better outcomes (Table 2).

Table 2.

Effects of individual key ingredients on Opening Minds Scale for Health Care Providers mean score change from meta-regression models

Key characteristic	Studies, n	Score change Mean (95% CI)	Univariate analysisa (95% CI)	P
Overall (crude)	22	−2.5 (−3.2 to −1.8)
Recovery			−1.7 (−2.9 to −0.5)	0.008
No	7	−1.3 (−2.3 to −0.3)
Yes	15	−3.0 (−3.8 to −2.3)
Multiple contact			−1.6 (−2.8 to −0.5)	0.006
No	12	−1.6 (−2.5 to −0.6)
Yes	10	−3.2 (−4.0 to −2.5)
Set tone			−1.3 (−2.8 to 0.3)	<0.10
No	5	−1.6 (−2.5 to −0.7)
Yes	17	−2.8 (−3.6 to −2.0)
Personal testimony			−1.1 (−2.4 to 0.3)	0.11
No	8	−1.8 (−2.9 to −0.6)
Yes	14	−2.9 (−3.7 to −2.0)
Skills			−0.7 (−2.0 to 0.7)	0.32
No	12	−2.2 (−3.4 to −0.9)
Yes	10	−2.8 (−3.7 to −1.9)
Myth busting			−0.6 (−1.9 to 0.8)	0.37
No	13	−2.3 (−3.3 to −1.2)
Yes	9	−2.9 (−3.8 to −1.9)
All 6 ingredients			−1.5 (−0.1 to −2.9)	0.04
No	17	−2.1 (−2.9 to −1.4)
Yes	5	−3.7 (−4.8 to −2.5)

^aUnivariate model, all 6 ingredients, I² = 68.4%

Figure 2.

Forest plot of mean difference in Opening Minds Scale for Health Care Providers score, stratified according to whether studies included all 6 evaluated ingredients

Figure 3.

Forest plot of mean difference in Opening Minds Scale for Health Care Providers score, stratified according to whether studies included multiple forms of social contact

Figure 4.

Forest plot of mean difference in Opening Minds Scale for Health Care Providers score, stratified according to whether studies emphasized and demonstrated recovery

In an exploratory analysis, we also examined the role of length of social contact. Results showed that length of social contact was not a significant predictor of program outcomes, suggesting that the quality of the social contact is more important than the length of that contact. We completed the same analysis for overall program length. Results showed that program length was not significantly related to program outcomes.

Discussion

The results of the meta-regression provide a validation of the 6-ingredient model of key program elements, as programs that included all 6 ingredients performed significantly better than those that did not. Two ingredients emerged as being most predictive of positive outcomes—including an emphasis on, and a demonstration of, recovery; and including multiple forms and (or) points of social contact. Setting the tone may also be an independently important ingredient, although its effects could not be confirmed in our analysis.

While some ingredients did not emerge as independently significant in our analysis, this should not discourage programmers from including them in their program curricula, nor should it lead programmers to assume that these elements are not important to program success. It is important to point out that the absence of evidence is not tantamount to evidence of absence.

For example, while the inclusion of a personal testimony component was not an independently significant ingredient, a main way by which many of the evaluated programs demonstrated and emphasized recovery was through personal testimony. The personal testimony components of programs that successfully modelled recovery had several main features: a hopeful and inspiring message; an onthe-way-up story^5,24; details about current achievements and successes; and delivered in a manner consistent with recovery. It is thus possible that the impact of personal testimonies on stigma reduction are, at least in part, dependent on the extent to which that testimony effectively communicates a message of, and belief in, recovery. Also, most programs that used multiple contact features included a personal testimony as one of its core program elements.

As well, it is possible that the identified ingredient of focusing on behaviour change by teaching health care providers skills about what to do and what to say may emerge more strongly if the window of measurement is extended, especially if such skill-building initiates persistent improvements in communication.^23,25

The significance of the recovery theme as a key ingredient for anti-stigma programming is consistent with existing theories about health care provider stigma, especially that of prognostic negativity and how this is believed to influence stigmatization among health care professionals.^4,26,27

Offering multiple forms or points of social contact is, as far as we know, a new finding about a potentially valuable moderator of effective social contact.² In the qualitative study, this theme centred on the idea that different people learn in different ways, and people will therefore connect to different people’s stories and experiences in different ways. In this respect, including multiple activities (for example, a presentation from a live speaker and a video presentation, multiple first-voice speakers, multiple points of social contact between program participants, and people with lived experience of mental illness) was discussed by program stakeholders as a strategy for maximizing the effectiveness of the social contact approach.

While our study was not a direct attempt to establish and validate fidelity criteria for designing and delivering successful anti-stigma programs for health care providers, it may be considered as contributing to this process.²⁸ In keeping with the work of others interested in understanding the critical components for stigma reduction,^5,24 the key ingredients for maximizing program outcomes identified and validated in our study represent a central and critical step in the development of comprehensive models for successful anti-stigma programming.

Our study had several limitations. All of the studies were before-and-after comparisons. As they were uncontrolled, some of the effects observed may be nonspecific. Therefore, the absolute effects observed should not be causally linked in their entirety to the intervention activities themselves. However, the relative effectiveness of the various programs can still provide a mechanism for comparing the programs with one another. A considerable degree of heterogeneity remained evident in our meta-regression models even after inclusion of key ingredients in those models. Clearly, there are other determinants of program effectiveness that our models did not quantify. Although we included both arms in RCTs as independent interventions, they were not entirely independent as the data were collected in the same setting within the context of a single protocol. However, the respondents themselves were independent. As we examined 6 potential key ingredients, there is a risk of type I error. The use of a single coder may have increased the risk of systematic observer bias, although the establishment of coding criteria and the completion of clarification or member-checking interviews may help to minimize this risk. However, as the criteria were established before the data analysis was conducted, their selection was not influenced by preconceived ideas deriving from the outcome data. Also, our study may have lacked power to identify the effects of some of the potential key ingredients.

Conclusions

The results provide a validation of the 6-ingredient model of key program elements for anti-stigma programming for health care providers. Emphasizing recovery and including multiple types of social contact are of particular importance for designing and delivering effective anti-stigma programs to health care providers.

Abbreviations

MHCC

Mental Health Commission of Canada

OM

Opening Minds

OMS-HC

OM Scale for Health Care Providers

RCT

randomized control trial