A multilevel modeling approach to examining the implementation-effectiveness relationship of a behavior change intervention for health care professional trainees

Abstract

Changing Minds, Changing Lives, a seminar-mediated behavior change intervention, aims to enhance health care professionals’ (HCPs’) social cognitions for discussing leisure-time physical activity (LTPA) with patients with physical disabilities. This study examines which seminar implementation variables (presenter characteristics, delivery components) predict effectiveness using multilevel modeling. HCP trainees (n = 564) attended 24 seminars and completed Theory of Planned Behavior-based measures for discussing LTPA at pre-, post-, 1-month post-, and 6-months post-seminar. Implementation variables were extracted from presenter-completed questionnaires/checklists. Seminars presented by a HCP predicted positive changes in all cognitions pre-post but negative changes in attitudes and perceived behavioral control (PBC) over follow-up (ps < .05). The number of seminars the presenter had delivered predicted negative changes in attitudes and PBC during follow-up (ps < .001). Inclusion of audiovisual components predicted positive changes in attitudes pre-post (p < .001). Presenter characteristics may be “key ingredients” to educational interventions for HCPs; however, future studies should examine additional implementation variables.

Electronic supplementary material

The online version of this article (10.1007/s13142-017-0526-9) contains supplementary material, which is available to authorized users.

BACKGROUND

Leisure-time physical activity (LTPA) behavior is infrequently discussed in patient-provider interactions [1], due to health care professionals’ (HCPs’) lack of knowledge, resources, and confidence [2–4]. Among people with a physical disability, a perceived lack of accessible LTPA information may contribute to the low rates of LTPA participation [5, 6]. Given persons with disabilities consider HCPs (including physicians, nurses, rehabilitation therapists, and kinesiologists) to be credible and preferred sources of LTPA information [7, 8], HCPs could be influential messengers for promoting LTPA if this knowledge-practice gap can be closed.

Recognizing that educational seminars are a preferred means of continuing education for HCPs [9] and are effective at increasing HCPs’ knowledge [10, 11], the Canadian Paralympic Committee offers a nationwide, seminar-mediated intervention called “Changing Minds, Changing Lives” (CMCL). CMCL aims to increase current and future HCPs’ intentions to discuss LTPA with patients with a physical disability [12]. The CMCL seminar curriculum is embedded within a standard PowerPoint presentation that was developed using a participatory process [13]. The curriculum is designed to be delivered orally in 50 min by a current HCP and/or a physically active person with a physical disability. The curriculum is evidence-based in that it includes up-to-date research evidence and Canadian statistics about LTPA participation for, and how to promote LTPA during clinical interactions with, individuals with physical disabilities. The curriculum is also theory-based in that it has a foundation in the Theory of Planned Behavior (TPB) in order to increase the likelihood of enhancing attendees’ social cognitions (e.g., thoughts and feelings, such as attitudes, subjective norms, perceived behavioral control (PBC)) for discussing LTPA with their current and/or future patients with a physical disability [13]. The TPB posits that one’s attitudes (beliefs about the consequences of one’s behavior), subjective norms (perceptions of what significant others want one to do, and the value attached to these perceptions), and PBC (confidence in one’s abilities to perform the behavior regardless of present barriers) predict intentions to perform the target behavior [14]. In turn, the target behavior is directly predicted by one’s intention (the effort and planning one is willing to exert for performing the behavior), as well as PBC. The TPB has been suggested to be an ideal behavioral theory for promoting the uptake and use of research knowledge among HCPs [15–17], and TPB social cognitions have been shown to account for 41% of the variance in HCPs’ intentions to discuss LTPA [18]. Because of the emerging use of theories in translational science [19] and the empirical support already received from TPB [15–17], it was deemed this would be the best theoretical approach when developing the CMCL curriculum.

When requested, CMCL seminars are delivered to current and future HCPs across Canada. Although presenters vary between seminars, they are trained to use the standardized curriculum regardless of their location. However, because CMCL seminars are implemented under real-world conditions, presenters are encouraged to adapt seminars to the local context and resources. For example, certain seminar settings may allow for the addition of an audiovisual (AV) component (e.g., a video) or for the demonstration of adapted sport equipment, and certain regions may have resources about local LTPA options for people with a physical disability. (For more information about the implementation of the curriculum and how presenters are trained, please see [13]). The use of local interventionists and an adaptable curriculum was implemented so that each presentation could be tailored to the audience and local resources, thereby enhancing curriculum sustainability [13, 20–22]. However, the variability in presenter characteristics (e.g., profession, experience delivering intervention) and intervention delivery (e.g., seminar duration, number of attendees) across Canada makes it challenging to determine the key implementation components that increase CMCL’s effectiveness (i.e., enhance attendees’ TPB social cognitions for discussing LTPA with patients with a physical disability). Presenter characteristics and intervention delivery represent different implementation domains that may influence intervention effectiveness: characteristics of individuals and intervention characteristics, respectively [23].

Indeed, researchers [20–22] have suggested that presenter characteristics and intervention delivery can influence an intervention’s real-world effectiveness. The influence of these implementation variables on effectiveness requires process evaluation data to be collected. In line with this perspective, an evaluation of the impact of CMCL seminars on HCPs’ social cognitions for discussing LTPA with their patients with a physical disability revealed that after attending a CMCL seminar, HCPs reported significant increases in their TPB social cognitions; however, these changes were not maintained at 1- and 6-month follow-up [18]. Hierarchical regression models revealed that the number of seminars a presenter had delivered was a negative predictor of pre- to post-seminar changes in PBC [18], suggesting that the presenter’s experience may be the most influential ingredient impacting change in attendees’ social cognitions for discussing LTPA with patients.

Although providing HCPs with the knowledge, resources, and confidence for discussing LTPA will close an existing knowledge-practice gap, more proximal education of HCP trainees may prevent this gap altogether. Fostering trainees’ positive intentions to discuss LTPA may result in a normative culture of LTPA discussion in patient-provider interactions (c.f. Diffusion of Innovations [24]). To foster this culture, CMCL seminars are delivered to both current and future HCPs. However, compared to current HCPs, trainees likely have limited practical knowledge and experience working with patients and colleagues; thus, changes in trainees’ cognitions may be influenced by different implementation variables. Thus, the current study sought to compliment the previously published CMCL evaluation [18] by examining the implementation-effectiveness relationship among trainees so that researchers and practitioners would better understand how to tailor future educational seminars for current and future HCPs.

The examination of methods for determining the optimal delivery of interventions and the inclusion of theory in intervention evaluations is needed [25, 26]. As behavioral researchers become increasingly concerned with achieving intervention effectiveness under “real-world” conditions [27], the use of theory for identifying the modifiable and non-modifiable intervention components that predict intervention effectiveness is becoming more important [11, 20]. Given that most studies examining the implementation-effectiveness relationship has been correlational or comparative [20, 21, 28], the analysis conducted by [18] was one of the first studies to use hierarchical regression techniques to use implementation variables to predict theory-based outcomes. However, the analyses were limited by sample size (n = 97 HCPs), precluding more advanced statistical techniques. Since the methods for the CMCL process evaluation involved four repeated measures for each participant (i.e., longitudinal data nested within individuals) and participants who attend a given seminar are, presumably, more similar to one another (i.e., they have the same presenter, as well as similar geographic, demographic, and health care training variables) than participants who attend different seminars (i.e., individuals nested according to seminar; refer to Electronic Supplementary Material 1 for a figure demonstrating the nested structure of the data), the use of multilevel modeling (MLM) would be advantageous and feasible for analyzing data from the large sample of HCP trainees who attended CMCL seminars during the study period (n = 618; [29]). A multilevel analysis could determine whether implementation variables (i.e., presenter characteristics and intervention delivery components) predict changes in attendees’ social cognitions over time. To our knowledge, this study is the first to use MLM to examine the relationship between the implementation and effectiveness of a real-world, theory-based behavior change intervention for HCPs.

The purpose of the current study was to examine which presenter characteristics and intervention delivery components predict changes in HCP trainees’ social cognitions for discussing LTPA with their future patients with a physical disability. In order to account for the nested characteristic of the data, multilevel analyses were conducted to determine whether meaningful between-seminar differences predict changes in trainees’ social cognitions over time. In line with findings from the HCP evaluation [18], we hypothesized that the number of seminars that the CMCL presenter had delivered would be a significant predictor of changes in trainees’ PBC from pre- to post-seminar. Given the presenters’ role in persuading attendees to discuss LTPA, we hypothesized that other presenter characteristics (e.g., whether the presenter was a HCP) would predict changes in trainees’ subjective norms. Given that CMCL was designed to transmit information and enhance attendees’ confidence for discussing LTPA, we also hypothesized that intervention delivery components would predict changes in trainees’ attitudes and PBC for discussing LTPA with their future patients with physical disabilities.

METHODS

Participants

Twenty-six CMCL seminars were delivered to 970 HCP trainees across Canada during the study period (November 2011 to August 2012), with 618 trainees agreeing to participate in the current study (63.7% participation rate). The majority of participants were female (73.0%) and Caucasian (80.4%). A large percentage of participants were from Ontario (37.7%) or Saskatchewan (24.0%), and were enrolled in a rehabilitation therapy program (e.g., physical therapy, occupational therapy, recreation therapy; 35.6%) or a physical education/kinesiology program (33.0%). Participants’ average age was 21.95 years (SD = 4.53) and had a mean of 2.14 years (SD = 1.43) before they transitioned from a trainee to a practicing HCP. Participants engaged in LTPA regularly (M ± SD = 4.13 ± 1.79 days/week); however, only 2.6% of participants were involved in parasport (parallel sport opportunities for people with physical disabilities, such as wheelchair basketball and sit-skiing; emphasized in the CMCL seminars as a means to achieve LTPA because the Canadian Paralympic Committee is committed to providing sport opportunities for Canadians with a disability). Prior to the CMCL seminar, the majority of participants reported working with patients with a physical disability “sometimes” or less (80.4%), and when working with these patients, most participants reported “never” or “rarely” discussing LTPA (29.8% and 31.6%, respectively) and “never” discussing parasport (59.1%). Complete demographic characteristics for participants are presented in Table 1.

Table 1.

Demographic characteristics of trainees who participated in the CMCL evaluation study

Characteristic	Trainees in study N = 618
Gender
Male	157 (25.4)
Female	451 (73.0)
Ethnicity
Caucasian	497 (80.4)
Asian	46 (7.4)
Black	16 (2.6)
Native Canadian	11 (1.8)
Other	29 (4.7)
Age (years)	21.95 ± 4.53
LTPA (days/week)	4.13 ± 1.79
Involved in parasport	16 (2.6)
Seminars delivered to trainees per province (#)
British Columbia (1)	18 (2.9)
Alberta (1)	63 (10.2)
Saskatchewan (8)	148 (24.0)
Ontario (6)	233 (37.7)
Newfoundland/Labrador (5)	87 (14.1)
New Brunswick (3)	15 (2.4)
Prince Edward Island/Nova Scotia (2)a	54 (8.7)
Program of Study
Medicine/nursing	3 (0.5)
Physical therapy	29 (4.7)
Occupational therapy	62 (10.0)
Recreational therapy	129 (20.9)
Recreation/leisure studies or management	54 (8.7)
Physical education/kinesiology	204 (33.0)
Fitness/health studies	37 (6.0)
Education	57 (9.2)
General arts and science/not yet specified/other	43 (7.0)
Years to profession (years)	2.14 ± 1.43
Frequency of working with patientsb
Never	151 (24.4)
Rarely	173 (28.0)
Sometimes	173 (28.0)
Frequently	73 (11.8)
All the time	26 (4.2)
Frequency of discussing LTPA with patientsb
Never	184 (29.7)
Rarely	195 (31.6)
Sometimes	135 (21.8)
Frequently	56 (9.1)
All the time	22 (3.6)
Frequency of discussing parasport with patientsb
Never	365 (59.1)
Rarely	150 (24.3)
Sometimes	58 (9.4)
Frequently	11 (1.8)
All the time	2 (0.3)

LTPA leisure-time physical activity. All values are n (%) except for age, days of LTPA per week, and years to profession, which are M ± SD

Some participants declined to respond to certain questions. Hence, n < 618 for some variables

^aPresenter checklists were not completed for the presentations in Prince Edward Island and Nova Scotia; thus, the 54 trainees from these provinces/presentations were not included in multilevel analyses (hence n = 564 in the multilevel models)

^bFrequency specific to patients with physical disabilities

Procedure

The study protocol received institutional ethical approval. In general, CMCL seminars were delivered as a guest presentation during regular classroom time. Upon their arrival at a CMCL seminar, trainees were invited to participate in the current study. Attendees who provided informed consent completed a demographic questionnaire, as well as hardcopy measures assessing their social cognitions for discussing LTPA with their future patients with a physical disability prior to and immediately following the seminar. To assess maintenance of change in social cognitions, participants were emailed a link to an online questionnaire at 1 and 6 months following their attendance. Following each seminar, presenters completed a presenter checklist to provide information about intervention delivery components.

Measures

Social cognitions for discussing LTPA with future patients

Items for attitudes, subjective norms, PBC, and intentions were derived from Ajzen [30], with “discussing LTPA with future patients” inserted as the target behavior. The stems for the PBC items were adapted to target the perceived control of the measure rather than motivation, as argued by Rhodes and Courneya [31]. All social cognition items were rated on a seven-point Likert scale. For TPB construct scales with more than two items, item scores were averaged to give an overall construct score. Table 2 lists the items, response scale, and the internal reliability or correlation of the items for each scale included in the social cognition questionnaire.

Table 2.

Questionnaire items and descriptive statistics for trainees’ social cognitions for discussing leisure-time physical activity with their future patients

Theory of Planned Behaviour construct (# items) Items included in scale	Response scale	Internal reliability score (α) or correlation (r)	Descriptive Statistics Time				Paired t-tests** t-value (df)
			Pre	Post	1-month	6-month	Pre-post	Post-6-month
Attitudes (5 items)
Instrumental attitudes  1. Attending this CMCL presentation will help me discuss physical activity and parasport to my future patients with a physical disability. Affective attitudes  2. Complete the statement, “Discussing physical activity and parasport to my future patients with a physical disability would be__________”  a. harmful/beneficial  b. worthless/valuable  c. difficult/easy  d. unpleasant/pleasant	1 = Strongly disagree, 7 = Strongly agree Anchors represent extremes (1/7) on 7-point Likert scale	αs ≥ .81	5.62 ± 0.85	6.20 ± 0.70	5.66 ± 1.11	5.59 ± 0.99	−19.556 (535)	8.637 (157)
Subjective norm (1 item)
1. Other professionals in my field think I should discuss physical activity and parasport with my future patients with a physical disability.	1 = Strongly disagree, 7 = Strongly agree	N/A	5.40 ± 1.29	5.96 ± 1.12	5.31 ± 1.24	5.09 ± 1.26	−12.294 (510)	7.405 (150)
Perceived behavioral control (2 items)
If you were really motivated and had all the resources that you needed, how confident are you in your ability to…   1. …discuss physical activity and parasport with your future patients with a physical disability?   2. …persuade your future patients with a physical disability to participate in physical activity and parasport?	1 = Not at all confident, 7 = Completely confident	rs ≥ .71 ps < .001	4.91 ±1.38	5.84 ±0.90	5.35 ± 1.00	5.25 ± 1.00	−19.814 (534)	5.636 (158)
Intention (2 items)
1. In the next four weeks, I intend to seek out additional information about physical activity and parasport for my future patients with a physical disability.  2. In the next four weeks, I intend to seek out additional information to use to persuade my future patients with a physical disability to engage in physical activity and parasport.	1 = Strongly disagree, 7 = Strongly agree	rs ≥ .84 ps < .001	4.32 ± 1.40	5.21 ± 1.32	3.63 ± 1.67	3.61 ± 1.59	−19.053 (521)	9.182 (153)

The column indicating scale internal reliability (Cronbach alpha) scores for the items on the scale and Pearson correlations between the items on the scale represent the lowest value across the four time points (pre-CMCL, post-CMCL, 1-month follow-up, 6-month follow-up). All internal reliability scores were acceptable. Descriptive statistics (M ± SD). All TPB items were guided by Ajzen’s [14] approach to constructing a TPB questionnaire. However, we adapted the stem for the PBC question based on the recommendations of Rhodes and Courneya [31]

CMCL Changing Minds, Changing Lives

**All t-tests were statistically significant at p < .001

Implementation variables

Ten different implementation variables were considered based on the nature and objectives of the presentations, as well as the feasibility of collecting data from multiple intervention sites across Canada [22]. Data for three presenter characteristics (e.g., whether the presenter was a HCP themselves) were obtained from a demographic questionnaire that all presenters completed during their CMCL training session [13]. Data for the seven intervention delivery components (e.g., whether an AV component was added) were obtained from presenter checklists which served as both a “roadmap” for consistent delivery of the CMCL curriculum and as an implementation data collection tool [22]. Table 3 provides the operationalization and summary of the 10 implementation variables. Presenter checklists were completed for 24 of the 26 CMCL seminars delivered by nine different presenters to 564 HCP trainees during the study period. As a reliability check, the first author attended and completed a presenter checklist for two CMCL seminars delivered by two different presenters. For the seven checklist items included in the current study, agreement between the researcher and presenters was high (86 and 100%).

Table 3.

Operationalization and summary of the ten implementation variables considered in predicting change in health care professional trainees’ social cognitions

Implementation variables	Abbreviation	Continuous variable	Dichotomous variablea
		Range	Yes	No
Presenter characteristics
Seminar number using the new CMCL curriculum	CMCL#	1–4
Years the presenter has been part of CMCL staff	CMCLyears	0–5
Whether the presenter is a HCP themselves	HCPpresenter		19	5
Intervention delivery components
Number of attendees present	attendees	10–115
Duration (minutes)	duration	50–150
Parasport athlete present at seminar to share his/her experience with the role his/her HCP played in his/her LTPA success	athlete		22	2b
Parasport equipment available for viewing and use by attendees	equipment		17	7
Educational resources about LTPA for people with a physical disability distributed to attendees	resources		17	7
Inclusion of audiovisual component (e.g., photos, videos) not part of standard CMCL curriculum	AVadded		7	17
Partner with a community organization for a parasport demonstration and/or activity	demonstration		1	23b

Data for the presenter characteristics were extracted from presenter demographic questionnaires completed prior to interventionist training. Data for the intervention delivery components were extracted from the presenter checklists that were completed for 24 of the 26 seminars delivered to HCP trainees during the study period

AV audiovisual; CMCL Changing Minds, Changing Lives; HCP health care professional; LTPA leisure-time physical activity.

^aNumber of seminars with each implementation variable over the 24 seminars for which Presenter Checklists are available

^bDue to highly unequal distribution of both the “athlete” variable (22 seminars with 545 participants had an athlete presenter vs. 2 seminars with 19 participants did not) and the “demonstration” variable (1 seminar with 15 participants included a parasport demonstration and/or activity with a partner organization vs. 23 seminars with 549 participants did not), these implementation variables were not included in subsequent multilevel analyses (Durlak and DuPre, 2008)

Data analyses

Preliminary analyses

Using SPSS v. 22, descriptive and normality (i.e., Normality plots with Kolmogorov-Smirnov test and skewness/kurtosis values) statistics were calculated for all TPB variables. Paired samples t-tests were conducted to examine the pattern of changes in TPB social cognitions from pre-post and post-6-month follow-up. Ranges and frequencies for the 10 implementation variables were calculated across seminars (see Table 3).

All data were screened for outliers and normality in SPSS using established guidelines [32]. Complete social cognitive data were available for 93, 93, 30, and 28% of the 564 trainees at pre-, post-, 1-, and 6-months, respectively; imputation techniques were not used to replace missing data as MLM analyses incorporate all participants who have at least one data point (e.g., pre-CMCL). We examined missing data patterns using Little’s MCAR test to determine if the data were missing at least at random, and, if so, missingness would not bias the estimates [29]. Prior to analysis, participants’ TPB questionnaire data were matched with presenter characteristics and intervention delivery components for the seminar attended.

Multilevel approach

Multilevel analyses using Hierarchical Linear Modeling 6 software [33] were conducted to determine whether meaningful between-seminar differences might have affected changes in HCP trainees’ social cognitions over time. Our data was represented by a three-level model: level-1 represents time variables (repeated measures at participant level), level-2 represents individual trainees, and level-3 represents seminars (implementation variables, i.e., presenter characteristics and intervention delivery components; refer to Electronic Supplementary Material 1 for a figure demonstrating the nested structure of the data). The objective was to test for changes (i.e., a linear growth model) in the four social cognitive outcomes (i.e., attitudes, subjective norms, PBC, and intentions). As such, four separate multilevel models were tested to determine trainees’ changes in each cognition over time.

Testing the effect of time on changes in social cognitions

For each outcome, a model with no predictors (i.e., null model) was first tested to determine if there was enough variation to warrant MLM. The intraclass correlation (ICC) was calculated to determine the ratio between the trainee and seminar variance (i.e., levels-2 and 3, combined) over the total variance. The closer the ICC is to + 1.00, the more appropriate a multilevel analysis is for the dataset. Second, a change only model (i.e., unconditional growth model) was tested to determine the change in outcome over time. In line with our hypotheses, we created two time variables (i.e., piecewise analysis [34]) to capture changes (a) pre-post-seminar and (b) post-seminar to 6-month follow-up and because the implementation factors related to change during the intervention (i.e., pre-post-seminar) are likely different from those related to change during follow-up (i.e., post-seminar and 6-month follow-up). To assess whether the addition of the linear trends (i.e., pre-post, post-6-month follow-up) contributed to the model, a pseudo-R ² was calculated, representing the amount of within-trainee variance explained by each variable (i.e., time trends) entered at level-1 [35]. For more details about the piecewise analysis approach and the calculation of the pseudo-R ² values, please refer to Electronic Supplementary Material 2.

Testing the effect of implementation variables on changes in social cognitions over time

Significant univariate effects of individual implementation variables on changes in social cognitions were included in subsequent multivariate models. Any continuous implementation variables (e.g., seminar duration) were entered as “centered” predictors. A pseudo-R ² was calculated for each multivariate model. Examples of the univariate and multivariate models are presented in Electronic Supplementary Material 3.

RESULTS

Preliminary analyses

Data were missing at random and normally distributed, and therefore satisfied the assumptions of MLM [29]. Descriptive statistics for trainees’ TPB social cognitions at the four time points are presented in Table 2. Paired samples t-tests confirmed that from pre- to post-seminar, trainees reported significant increases in all TPB social cognitions to discuss LTPA with their future patients with physical disabilities. Between post-seminar and 6-month follow-up, trainees reported significant decreases in all four TPB social cognitions (see Table 2). These findings confirmed the appropriateness of our piecewise analysis approach for including two linear trends (pre-post increase, and post-6-month decrease) in level-1 of our multilevel models.

Ranges and frequencies for the 10 implementation variables are presented in Table 3. Due to highly unequal distributions of both the “athlete” variable (22 seminars with 545 participants had an athlete presenter vs. 2 seminars with 19 participants did not) and the “demonstration” variable (1 seminar with 15 participants included a parasport demonstration and/or activity with a partner organization vs. 23 seminars with 549 participants did not), these implementation variables were not included in subsequent multilevel analyses [20]. When levels of implementation are all very high or very low across groups (i.e., across CMCL seminars), the lack of variability does not provide much power for detecting between-group differences [20].

Effect of time on change in HCP trainees’ social cognitions

The null models tested for attitudes, subjective norms, PBC, and intentions had significant variance components (all ps < .001) and ICCs indicated that MLM was appropriate for analyzing change in each social cognition (all ICCs ≤ 0.39; see Table 4). Next, unconditional growth models were tested by including two linear change variables (pre-post and post-6-month), centered at post-seminar, into the level-1 model. The linear trends were significant (ps < .001), confirming that there was a significant increase in all four social cognitions from pre-post-seminar and a significant decrease in all four social cognitions from post-seminar to 6-month follow-up. For example, concerning trainees’ attitudes, a significant linear increase was seen from pre-post-seminar (π_1jk = 0.59, p < .001) and a significant linear decrease was seen from post-6-month (π_2jk = −0.33, p < .001); results for the other three social cognitions can be interpreted similarly. Calculation of pseudo-R ² revealed that 15–29% of the within-subject variance was explained by the level-1 linear trends. Using the Optimal Design software [36] to calculate post-hoc power with a moderate effect to reflect the range of pseudo-R ², approximately 24 participants per training session, a p-level of .05 and with a cluster size of 26 training session, our study had a power of .79. Therefore, the power analysis confirms our ability to detect significant differences. See Table 4 for a summary of the results for the null and unconditional growth models.

Table 4.

Null model, unconditional growth model, and multivariate results

		TPB construct
		Attitude	Subjective norm	PBC	Intention
Null model
ICC		0.33	0.39	0.30	0.30
Unconditional growth model
Pre-post: π1jk **		0.59	0.56	0.95	0.82
Post-6-month: π2jk **		− 0.33	− 0.43	− 0.27	− 0.85
Pseudo-R 2		0.23	0.15	0.29	0.25
Multivariate models
Linear model	Implementation variable
Pre-post	CMCL#	0.018 (0.029)	− 0.007 (0.038)	0.018 (0.073)	− 0.051 (0.067)
	HCPpresenter	0.223** (0.052)	0.174* (0.055)	0.305* (0.118)	0.201* (0.102)
	Duration	− 0.002 (0.002)	− 0.002 (0.003)	− 0.002 (0.004)	− 0.002 (0.004)
	Equipment	− 0.044 (0.148)	0.203 (0.172)	0.385 (0.243)	0.060 (0.239)
	AVadded	0.308** (0.082)	− 0.035 (0.106)	0.153 (0.133)	0.266 (0.171)
Post-6-mo	CMCL#	− 0.089** (0.023)	0.038 (0.047)	− 0.085** (0.032)	0.068 (0.082)
	HCPpresenter	− 0.139** (0.043)	− 0.187 (0.100)	− 0.182* (0.064)	− 0.475** (0.140)
	Duration	0.002 (0.001)	− 0.004 (0.003)	0.003 (0.002)	0.003 (0.005)
	Equipment	− 0.141 (0.109)	0.293 (0.174)	− 0.156 (0.161)	− 0.492 (0.294)
	AVadded	− 0.068 (0.079)	− 0.259 (0.150)	− 0.068 (0.120)	0.116 (0.237)
	Resources	− 0.016 (0.060)	− 0.037 (0.111)	− 0.101 (0.098)	0.079 (0.141)
Pseudo-R 2	Null-multivariate	0.26	0.16	0.30	0.29
	Growth-multivariate	0.04	0.01	0.02	0.05

Values corresponding with π are unstandardized growth coefficients. Values for multivariate models indicate coefficients and (SE)

PBC perceived behavioral control; TPB theory of planned behavior

*p ≤ .05, **p ≤ .001

Predicting changes in HCP trainees’ social cognitions over time from the implementation variables

When individual implementation variables were entered as level-3 predictors in a series of models for each social cognition, significant univariate effects were seen for: (1) number of seminars delivered by the presenter, (2) seminar duration, as well as whether (3) the presenter was a HCP, (4) parasport equipment was present, (5) an audiovisual component was added, and (6) educational resources were distributed (all ps < .04). The number of years the presenter had been a CMCL presenter and the number of attendees at the session did not emerge as a significant univariate predictor of changes in any social cognition over time (ps > 0.05). All significant univariate implementation variables were entered into level-3 as potential predictors of both pre-post and post-6-month changes in each social cognition. The intervention delivery component “resources” was only added as a predictor of post-6-month changes in cognitions because educational resources were distributed to trainees at the end of the CMCL seminars once post-seminar data collection was complete.

Table 4 presents the multivariate coefficients and standard errors for each implementation variable on changes in each social cognition over time. Between pre- and post-seminar, having a presenter who was a HCP led to significantly positive changes in all four social cognitions; however; during follow-up (post-6-month), this implementation variable was a significant negative predictor of changes in attitudes and PBC (ps < .05). Adding an AV component was a significant positive predictor of change in attitudes from pre-post-seminar (p < .001). The number of seminars the presenter had delivered was a significant negative predictor of change in attitudes and PBC from post-6-month follow-up (ps < .001). For a visualization of how having a presenter who is a HCP predicts changes in trainees’ attitudes between pre-post and post-6-month, please refer to Electronic Supplementary Material 4; the association of other significant predictors can be interpreted similarly. The complete multivariate model accounted for an additional 1–5% and 16–30% of the variance in changes in social cognitions when compared to the unconditional growth model and the null model, respectively.

Discussion

The purpose of the current study was to use three-level multilevel models to examine which CMCL seminar implementation variables (presenter characteristics and intervention delivery components) predict CMCL effectiveness (changes in HCP trainees’ social cognitions for discussing LTPA with their future patients with a physical disability). For all four TPB social cognitions, HCP trainees’ reported significant increases between pre- and post-seminar and significant decreases between post-seminar and 6-month follow-up. While a number of implementation variables were significant univariate predictors of changes in trainees’ cognitions, only two presenter characteristics (the number of seminars delivered and whether the presenter was a HCP) and one intervention delivery component (whether an AV component was added) emerged as significant predictors of changes in cognitions in the final multivariate model.

Predicting effectiveness from presenter characteristics

In line with our hypothesis and previous findings [18], the number of presentations the presenter had delivered emerged as a significant negative predictor of changes in trainees’ PBC. This variable also emerged as a significant negative predictor of changes in trainees’ attitudes. Previous analyses examined this implementation variable between pre- and post-CMCL seminar only [18]; this study adds to these findings and suggests that the number of presentations given by a presenter is a predictor of effectiveness across the 6-month follow-up period. It is possible that CMCL presenters’ “quality of delivery,” including the presenter’s enthusiasm and clarity [21], may diminish with successive delivery of seminars. This phenomenon can be conceptualized as a component of “program drift” within the Diffusion of Innovations theory [24], whereby experienced interventionists may stray from delivering the program as initially intended with repeated implementation [22]. The few studies that have examined the relationship between implementation “quality of delivery” and program effectiveness have shown positive associations [21]. To minimize “program drift,” interventionists should receive ongoing training to maintain enthusiasm and ensure consistent quality with successive seminar delivery [20]. Measuring presenters’ enthusiasm for delivery prior to each seminar and “quality of delivery,” both with and without additional training sessions, is an avenue for future research that may generate a better understanding of how providers’ experience impacts program drift and attendees’ outcomes.

Between pre- and post-seminar, having a presenter who was a HCP predicted positive increases in all four social cognitions, suggesting that the messenger’s profession may play an important role in “changing minds” of HCP trainees in the short-term. However, in the long-term, having a presenter who was a HCP predicted a negative change (larger decrease) in trainees’ attitudes, PBC, and intentions. For trainees, CMCL seminars were generally delivered as a guest presentation during regular classroom time. Trainees may have found the novelty of the presenter enjoyable [37], particularly if they aspired to be a HCP themselves. Trainees likely perceived the presenter to be an opinion leader, or an individual in a position to influence their thoughts and behaviors through the provision of information [38], which may have bolstered their attention during the seminar. However, following the seminar, HCP trainees were no longer exposed to the presenter, perhaps explaining why having a HCP presenter failed to predict social cognitions during the follow-up period. Taken together, these findings suggest that presenter experience and profession predict effectiveness, but that other presenter characteristics (e.g., enthusiasm for program delivery) may be important for understanding maintenance of attendees’ cognitions.

Predicting effectiveness from intervention delivery components

The addition of an AV component was a significant predictor of increases in HCP trainees’ attitudes from pre- to post-seminar. Analysis of the presenter checklists indicated that the added AV components included pictures or videos about the presenters and/or people with physical disabilities participating in LTPA activities. AV components serve as captivating media to communicate personal experiences and knowledge, and in this case, in a format that is familiar and accessible to trainees [39]. It is possible that these AV components brought the curriculum “to life” and including an additional delivery format more fully engaged the trainees [40]. However, the positive impact of adding an AV component did not predict changes in cognitions during the follow-up period. Future iterations of the CMCL seminars should include pictures and videos in the standard curriculum and explore the use of AV components in follow-up correspondence with attendees to sustain their positive attitudes for discussing LTPA.

Although “duration,” “equipment,” and “resources” were related to changes in trainees’ cognitions in the univariate models, these variables did not emerge as significant predictors in the multivariate models. Intervention delivery components lost their significance for predicting effectiveness when presenter characteristics were also included in the model, suggesting that the presenter may be the “key ingredient” to “changing minds.” That is, regardless of what local adaptations are made to intervention delivery components, presenter characteristics ultimately predict attendees’ outcomes. Research has demonstrated that educational lectures/seminars for HCPs are most effective when the presenter is able to identify key issues, engage the audience, and present material clearly and with enthusiasm [41]. Systematic manipulation of presenter characteristics across seminars would provide stronger support for the factors that are theoretically meaningful for “changing minds”; such experimental studies are an avenue for future research examining the implementation-effectiveness relationship.

Study strengths and limitations

This is the first study we are aware of to use MLM to examine the relationship between the implementation and effectiveness of a real-world, theory-based behavior change intervention for current and future HCPs. The large sample of HCP trainees and the nested structure of the data allowed for the use of MLM, a robust statistical technique for determining the influential factors on multiple levels while accounting for grouping effects on each level. Thus, this study builds on our previous evaluation methods and results [18]. By using MLM, this study has yielded new knowledge on factors that predict effectiveness in trainees, and how these factors may differ from what is important for HCPs. The differences in influential factors between the two audiences may be due to trainees’ limited opportunities to apply the knowledge and skills learned from the intervention during the follow-up period (e.g., limited clinical interactions with persons with physical disability during subsequent classroom time). Although the implementation variables used in this study provided only preliminary indicators, the advanced statistical methods and theoretically informed approach to intervention evaluation and analysis represent a novel approach [25]. Our approach can inform the methods and analyses used by translational scientists who aim to explore the implementation-effectiveness relationship beyond correlational and comparative studies.

Despite the robust analysis approach, the study is limited by the number of implementation variables captured for inclusion into our multilevel models. Examination of the pseudo-R2 values for the null- and growth-multivariate models (Table 4) suggest that while several implementation variables were significant predictors of change in trainees’ social cognitions, their inclusion in the multilevel models explained minimal variance over and above the effect of time. These findings provide impetus for studies to explore the influence of additional implementation variables on effectiveness, such as the value the presenter sees in the program [20], presenter “quality of delivery” (including enthusiasm; [21]), and other intervention delivery components that were not measured in the study. The logistics of the nationwide study precluded the use of audio or video recording of the seminars in order to quantify additional intervention delivery components (e.g., duration spent on each theory-based component in the curriculum). Although [42] suggests that a universal psychometric system for measuring every aspect of a program’s implementation would be impossible in adaptable programs, such as CMCL, further refinement and testing of presenter characteristics and the presenter checklist would allow the measurement of a greater number of implementation variables. Moreover, researchers [21, 22] have suggested that participant responsiveness, including participants’ level of enthusiasm for and participation in an intervention, may explain the relationship between presenter “quality of delivery” and program outcomes. Future real-world program evaluations that include measures of attendees’ active participation and perceptions of presenter characteristics during implementation, as well as attendees’ attempts at applying the knowledge learned following implementation, would allow for a more nuanced examination of the relationship between these variables and seminar effectiveness. Additional implementation measures may also be incorporated to assess additional implementation levels (e.g., structural, organizational) that may influence effectiveness [23, 43].

Another limitation was that there was a large dropout rate over the follow-up period, with only 30 and 26% of participants responding at the 1- and 6-month time points. Many of these follow-up assessments took place during the trainees’ summer term when classes are not in session; thus, they may not have received the survey information if they were not accessing their email regularly. The response rate for trainees was similar to that seen in our previous study with HCPs [18]; however, the MLM approach used in the current study accounts for missing data and results can be interpreted as though no missing data were present [29]. In addition, participants reported changes in TPB-related cognitions following CMCL, but TPB remains one theoretical approach in implementation science. Future studies could utilize other theories (e.g., COM-B framework) to develop and test implementation interventions. Also, the TPB questionnaire was shortened to render data collection feasible pre-post-seminar, and we acknowledge the limitation of not testing a full questionnaire. Future studies need to look at testing the validity of shorter theory-based questionnaires so they can be used in such real-world settings. In addition, due to the logistical challenges of collecting follow-up data from a nationwide sample of trainees, and the high attrition rate over the 6-month period, it was not feasible to collect a behavioral measure to understand the influence of the seminar on trainees’ actual practice years into the future (i.e., once they were practicing in their profession). Finally, the prospective study design precluded the ability to make causal inferences about effect of the implementation variables on participants’ social cognitions. Future experimental studies that manipulate presenter characteristics between seminars would help establish the cause-effect relationship between these variables and attendee outcomes.

CONCLUSIONS

The seminars increased HCP trainees’ TPB social cognitions for discussing LTPA with future patients with physical disabilities in the short-term, but this increase was not maintained during the 6-month follow-up period. Using MLM to explore the implementation-effectiveness relationship, it was found that having a presenter who was a HCP themselves was a significant positive predictor of changes in HCP trainees’ social cognitions in the short term but a negative predictor of change over the follow-up period. The number of seminars the presenter had delivered emerged as a significant negative predictor of change in HCP trainees’ attitudes and PBC over the follow-up period. The only intervention delivery component that predicted changes in trainees’ cognitions was whether an AV component was added to the standard CMCL curriculum; this variable positively influenced trainees’ attitudes in the short-term only. This paper provides a unique example of a robust method for examining the implementation-effectiveness relationship of real-world interventions. However, future research is necessary to examine whether additional presenter characteristics (i.e., enthusiasm for and quality of delivery) and participant responsiveness (e.g., active participation during implementation) predict program effectiveness.

Compliance with ethical standards

Disclosure of potential conflict of interest

This research was partially supported by an Ontario Neurotrauma Foundation and Rick Hansen Institute Award for Capacity Building in Knowledge Mobilization awarded to the JRT and KMG, and a Community-University Research Alliance from the Social Sciences and Humanities Research Council of Canada (SSHRC) awarded to KMG. Preparation of this manuscript was supported by a Knowledge Translation (KT) Canada Fellowship awarded to JRT.

Ethical approval

All procedures followed were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki Declaration and its later amendments. No animals were harmed in the conduct of the study.

Informed consent

Informed consent was obtained from all individual participants included in the study.