Using mixed methods to assess fidelity of delivery and its influencing factors in a complex self-management intervention for people with osteoarthritis and low back pain

Abstract

Objectives and design

Despite an increasing awareness of the importance of fidelity of delivery within complex behaviour change interventions, it is often poorly assessed. This mixed methods study aimed to establish the fidelity of delivery of a complex self-management intervention and explore the reasons for these findings using a convergent/triangulation design.

Setting

Feasibility trial of the Self-management of Osteoarthritis and Low back pain through Activity and Skills (SOLAS) intervention (ISRCTN49875385), delivered in primary care physiotherapy.

Methods and outcomes

60 SOLAS sessions were delivered across seven sites by nine physiotherapists. Fidelity of delivery of prespecified intervention components was evaluated using (1) audio-recordings (n=60), direct observations (n=24) and self-report checklists (n=60) and (2) individual interviews with physiotherapists (n=9). Quantitatively, fidelity scores were calculated using percentage means and SD of components delivered. Associations between fidelity scores and physiotherapist variables were analysed using Spearman’s correlations. Interviews were analysed using thematic analysis to explore potential reasons for fidelity scores. Integration of quantitative and qualitative data occurred at an interpretation level using triangulation.

Results

Quantitatively, fidelity scores were high for all assessment methods; with self-report (92.7%) consistently higher than direct observations (82.7%) or audio-recordings (81.7%). There was significant variation between physiotherapists’ individual scores (69.8% - 100%). Both qualitative and quantitative data (from physiotherapist variables) found that physiotherapists’ knowledge (Spearman’s association at p=0.003) and previous experience (p=0.008) were factors that influenced their fidelity. The qualitative data also postulated participant-level (eg, individual needs) and programme-level factors (eg, resources) as additional elements that influenced fidelity.

Conclusion

The intervention was delivered with high fidelity. This study contributes to the limited evidence regarding fidelity assessment methods within complex behaviour change interventions. The findings suggest a combination of quantitative methods is suitable for the assessment of fidelity of delivery. A mixed methods approach provided a more insightful understanding of fidelity and its influencing factors.

Trial registration number

ISRCTN49875385; Pre-results.

Strengths and limitations of this study.

• This mixed methods investigation of fidelity of delivery and its influencing factors provides valuable information on fidelity assessment methods and factors to be considered in developing and evaluating complex behaviour change interventions.

• The novel use of mixed methods to assess fidelity in this study enabled increased certainty in findings where qualitative data corroborated the quantitative results.

• This study does not explore the fidelity of the quality of delivery (eg, therapist competence) or specific behaviour change techniques which will be reported in a separate publication.

Introduction

Fidelity of delivery is an important, yet often overlooked aspect of behaviour change interventions and has been defined as the degree to which an intervention, treatment or programme is delivered as intended by the intervention developers.^{1 2} Without adequately addressing fidelity in behaviour change research, it is uncertain that changes observed in study outcomes are due to the influence of the independent variable (the intervention being investigated) and not due to variability in its implementation, for example, extraneous elements that may have been added (either accidentally or purposely), or essential elements of the intervention that were omitted.³ In particular, as behaviour change interventions are often complex interventions that typically involve several components with the potential to affect or influence outcomes separately, it is especially important to incorporate adequate fidelity planning and assessment into the development of interventions of this nature.⁴

Despite a recent increased emphasis on improved assessment and reporting of what happens within complex behaviour change interventions,^{5 6} fidelity is still poorly addressed within this context, with few examples of fidelity being assessed comprehensively or reported adequately.^7–9 Where studies have assessed fidelity within a behaviour change healthcare context, there is often limited exploration of the factors that might have influenced that fidelity.^{10 11} Previous work that has specifically examined influencing factors in areas of public health, obesity and stroke research found that provider-level variables, such as experience, knowledge or skills, may influence fidelity of delivery.^12–15 Although the use of both quantitative and qualitative methods has been previously recommended to comprehensively assess fidelity,^16–18 this guidance is not consistently followed. Consequently, the use of quantitative methods in isolation may not allow for exploration of the factors influencing fidelity, knowledge of which could improve how fidelity is enhanced and assessed in future similar interventions.¹⁹ For example, French et al¹¹ used audio-recordings and self-report methods to assess the fidelity of delivery of an educational intervention for general practitioners. The authors acknowledged that the quantitative study design did not allow them to explore the reasons for variations in fidelity scores found.

According to existing guidelines for addressing fidelity within behaviour change research developed by the National Institutes of Health Behaviour Change Consortium,^{18 20} the fidelity of delivery of the intervention by providers is one particularly important aspect of fidelity. This aspect considers strategies that enhance the fidelity of delivery (eg, using treatment manuals or intervention protocols) and methods that assess this delivery (eg, provider self-report, audio or video-recorded observations and direct in vivo observations). However, although previous research has advocated a combination of these strategies in order to assess fidelity in-depth,^{21 22} limited examples exist within the literature. Additionally, few studies have explored the relationship between these methods, and the accuracy of potentially more feasible methods against the ‘gold standard’ of direct observations using prespecified criteria²³ has been poorly investigated.^{18 19 24} As a result, there is little evidence to justify the selection of one method over another or to inform the use of multiple methods simultaneously.

The current study is set within the context of the Self-Management of Osteoarthritis and Low back pain through Activities and Skills (SOLAS) feasibility trial (ISRCTN49875385).²⁵ The trial aims to evaluate the feasibility of providing the SOLAS intervention (experimental group) to promote self-management for patients with osteoarthritis of the hip/ knee and/or chronic low back pain compared with usual physiotherapy, which will serve as the pragmatic control group in order to determine the feasibility of moving to a full scale trial by following the Medical Research Council (MRC) guidelines. The intervention consists of six weekly sessions of 90 min to be delivered by a primary care physiotherapist to a group of 6–8 people. Each session is divided into education and exercise sections (each approximately 45 min in duration). During the exercise section, participants are provided with an opportunity to attempt and practice prespecified exercises, while the education section is further split into four categories: Materials—participants are provided with materials intended to supplement and enhance understanding and uptake of skills; Introduction and review—at the start of each session, the physiotherapist introduces session aims and reviews goals and action plans with participants; Education—the physiotherapist facilitates a group discussion on targeted self-management skill or behaviour of the session using Powerpoint slides and projector; Review and planning—before the session concludes, the physiotherapist recaps participants’ planned activity levels and action plans. For each category, a number of components (ie, specific elements or activities which varied according to the session) were intended to be delivered or addressed during the session by the physiotherapist (eg, provision of pedometers, provision of information on balanced weight and healthy eating), as detailed in an intervention manual^{25 26} (summarised in table 1). Prior to participation, physiotherapists were provided with the manual during a 2-day training course where background variables were collected, in addition to a post-training assessment.²⁶ Details of the development and testing of the fidelity protocol used in this study are described elsewhere.²⁷ This study also does not explore the fidelity of the quality of delivery (eg, therapist competence) or specific behaviour change techniques (BCTs) which will be reported elsewhere.

Table 1.

Structure, content and samples of Self-Management of Osteoarthritis and Low back pain through Activities and Skills (SOLAS) intervention

Session number	Self-management behaviours/skills targeted		Session section (intended duration)		Session category and component example*
1	Physical activity Goal-setting		Education (45 min)		Materials Provide participant handbook Introduction and review Introduce programme aims Education Provide information on benefits of exercise/physical activity Review and planning Review of SOLAS programme and activity levels
			Exercise (45 min)		Exercise Demonstrate protocol exercises
2	Pacing (balancing activity/rest) Physical activity Goal-setting		Education (45 min)		Materials Offer pedometers Introduction and review Review of previous week activities Education Provide information on activity–rest cycle and pacing Review and planning Review of session and goal-setting
			Exercise (45 min)		Exercise Provide opportunity for participants to practice exercises
3	Balanced weight/ healthy diet Physical activity Goal-setting		Education (45 min)		Materials Offer tape measures Introduction and review Review of previous session action plan Education Provide information on balanced weight and healthy eating Review and planning Review of session and goal-setting
			Exercise (45 min)		Exercise Provide opportunity for participants to practice exercises
4	Managing pain with pain relief Physical activity Goal-setting		Education (45 min)		Materials N/A Introduction and review Mid-way review Education Provide information on pain management with ice/heat Review and planning Review of session and goal-setting
			Exercise (45 min)		Exercise Provide opportunity for participants to practice exercises
5	Managing mood (with relaxation) Physical activity Goal-setting		Education (45 min)		Materials Offer relaxation CD-ROM Introduction and review Review of previous session action plan Education Provide information on recognising and managing flare-ups Review and planning Review of session and goal-setting
			Exercise (45 min)		Exercise Provide opportunity for participants to practice exercises
6	Long-term management Physical activity Goal-setting		Education (45 min)		Materials Offer ‘local resources to support physical activity’ leaflet Introduction and review Review of previous session action plan Education Provide information on local resources and supports Review and planning Long-term goal setting and action planning
			Exercise (45 min)		Exercise Provide opportunity for participants to practice exercises

Samples involved
	Wave 1				Wave 2						Total n=
Physiotherapy Site code	A	B	C	D	A	B	C	E	F	G	10 (7)†
Physiotherapist code	A1	B1	C1	D1	A1	B2	C2	E1	F1	G1	9‡
Number of participants recruited	4	6	4	2	4	4	5	4	7	5	45
Sessions delivered	6	6	6	6	6	6	6	6	6	6	60
Direct observations	3	3	3	3	2	2	3	2	1	2	24 (40%)
Self-report	6	6	6	6	6	6	6	6	6	6	60 (100%)
Audio-recordings (rater 1)	6	6	6	6	6	6	6	6	6	6	60 (100%)
Audio-recordings (rater 2)	3	3	3	3	0	0	0	0	0	0	12 (20%)

*A complete list of category components is provided in the online supplementary files.

†Seven sites were involved in total, but three of these delivered the intervention in both waves.

‡One physiotherapist delivered the intervention in both waves.

N/A = not applicable

The aim of this study was to establish the fidelity of delivery of a complex behaviour change intervention and the reasons for these findings using a mixed methods approach. Specifically, the study objectives were

1. to evaluate the agreement of multiple methods for assessing fidelity of delivery

2. to establish the fidelity of delivery of the SOLAS complex behaviour change intervention

3. to explore the potential factors that may have influenced these fidelity results.

Methods

Design

This observational study was a convergent/triangulation mixed methods design.²⁸ This mixed methods approach was chosen as it was felt that thorough integration of findings from both quantitative and qualitative methods would achieve a more comprehensive answer to the study questions by enabling the methods to be ‘greater than just the sum of their parts’.²⁹ Figure 1 graphically illustrates the study design, outlining the sequence of research activities, the priority of the methods and the stage at which integration occurred.³⁰

Figure 1.

Diagrammatic representation of mixed methods convergent/triangulation design.

Quantitative phase

Study sample and procedure

Data were collected during two of the three waves of the SOLAS feasibility trial (table 1), representing 71% of the overall trial data. Fidelity of delivery in this study refers to the assessment of both the delivery of session content, that is, providers deliver the session categories and components as intended (summarised in table 1) (fidelity of content) and session duration, that is, providers deliver the session as long as intended (fidelity of duration). Following pilot testing, it was decided to conduct 24 (40% of sessions) randomly selected direct observations (rated by ET), 60 (100%) self-report (rated by the physiotherapists) and 60 (100%) audio-recordings (rated by ET) to assess fidelity of delivery using a priori checklists (see online supplementary file 1) that had been previously found to be feasible for use.²⁷ To assess inter-rater reliability, 12 sessions (20%) were rated by a second independent rater (AK).¹⁰ Checklists consisted of approximately 25 components for each session, structured according to the SOLAS categories. Components for each session were chosen to address each element specified in the SOLAS intervention manual (summarised in table 1)^{25 26} to be delivered during that session. Each component was rated as ‘yes/present’ equating to a score of two points, ‘no/absent’ (zero points) or ‘attempted’ (one point). Session duration was documented by all methods, and attendance was recorded by self-report.

Data analysis

Fidelity data analysis was consistent with standard procedures^{19 31 32} using SPSS V.20. Specifically, levels of agreement between methods and inter-rater reliability of audio-recorded data were assessed using percentage concordance. Overall mean fidelity of content scores (ie, percentage of manual-specified components delivered as intended) and fidelity scores according to physiotherapy site, physiotherapist, session and session category were obtained by calculating total actual scores as a percentage of the total possible score. Means data were compared using analysis of variances and Kruskal-Wallis tests. Fidelity of duration was established by calculating the difference between the actual and the intended session duration using a one-sample Wilcoxon test. Levels of fidelity were interpreted as previously reported in the literature, with 80%–100% adherence interpreted as ‘high’ fidelity, 51%–79% as ‘moderate’ and 0%–50% as ‘low’ fidelity.^{3 33 34} Finally, the relationship between fidelity scores and (1) the number of participants present (group size) and (2) physiotherapist variables, that is, experience (years qualified), group experience (years delivering group physiotherapy), knowledge of intervention (post-training evaluation score) and previous relevant training (see online supplementary file 2), were calculated using Spearman’s correlation coefficient and Mann-Whitney U test. These physiotherapist variables were chosen for reasons described in the introduction.

Qualitative phase

The aim of the qualitative phase was to explore physiotherapists’ opinions of fidelity of intervention delivery and the factors that they felt may have influenced their fidelity. Individual semistructured telephone interviews were conducted by an experienced qualitative researcher (SG) with each physiotherapist (n=9) within 2 weeks of intervention delivery completion. A topic guide with specific questions and probes related to fidelity was developed by the corresponding author (ET) (see online supplementary file 3). Interviews were audio-recorded and transcribed verbatim. Deductive thematic analysis was used to analyse the interviews as it is a flexible method that works with a range of research questions, including understanding people’s experiences of programmes and healthcare interventions.³⁵

Meaningful units of text were highlighted within each interview, then summarised and coded. Codes dealing with similar issues were grouped across all interviews and refined into themes. The reliability of themes was established by a second reviewer (AK), who independently analysed a randomly selected sample of 50% of the transcript extracts using the coding framework. Percentage agreement was determined between the reviewers’ respective coding of extracts. If agreement was <70%, consensus on conflicting decisions was obtained through discussion.³⁶

Integration

Integration of quantitative and qualitative data occurred at an interpretation level using triangulation methodology. Specifically, a meta-matrix was created to facilitate comparison of the findings.³⁷ This involved presenting the quantitative data in a tabular format alongside summarised qualitative themes, which enabled a transparent approach to determining convergence, discrepancy or silence across the findings of the data sets.³⁸ Convergence was defined as general agreement between the data sets on the element of comparison (eg, overall quantitative fidelity score compared with the majority of physiotherapist opinions of their fidelity levels), while discrepancy was defined as general disagreement between the data sets on the element of comparison.³⁸ Silence was defined as where one set of results addressed a theme or example, but the other set of results did not yield any relevant data.³⁸

Results

Quantitative findings

Agreement

Agreement between direct observations and self-report was 74.6%, 75.4% between self-report and audio-recordings (rater 1) and 86.6% between direct observations and audio-recordings (rater 1). Inter-rater reliability of audio-recordings (rater 1 vs rater 2) was 81.3%. Further detail is provided in online supplementary file 4.

Fidelity of content

Fidelity was found to be high in all assessment methods, with a mean score of 81.7% (range of 61.1%–95.8%) for the audio-recordings, 92.7% (85.2%–96%) for the self-report and 82.7% (72.1%–100%) for the direct observations. Table 2 details the fidelity results for each method with scores <80% (cut-off for ‘high’ fidelity) in bold. Significant differences between physiotherapists’ individual fidelity scores were found. Fidelity scores were also found to differ significantly according to the session category (eg, the category ‘Materials’ was delivered with significantly less fidelity than the ‘Education’ category).

Table 2.

Fidelity results calculated from % totals

	Direct observations (DO) % (SD)	Self-report (SR) % (SD)	Audio 1 (AO) % (SD)
Total % mean fidelity score (SD)	82.7% (10)	92.7% (6.4)	81.7% (12)
% mean fidelity score per session (SD)*
1	88.8% (5.24)	95% (4.5)	91.6% (4.5)
2	82.8% (5.7)	92% (6.9)	86.8% (10.5)
3	85.6% (12.9)	96% (4.2)	81.4% (10.6)
4	83.3% (14.4)	90.9% (8.3)	75% (14)
5	74.1% (11.9)	89.4% (8.1)	78.7% (11)
6	82% (9.6)	92.7% (3.78)	74.9% (12)
% mean fidelity score per site (SD)*
A (delivered twice, same physiotherapist)	78.7% (7.6)	95% (5.4)	81.3% (11.5)
B (delivered twice, two physiotherapists)	76.7% (5.6)	92.8% (5.3)	71.1% (10)
C (delivered twice, two physiotherapists)	84.8% (11.8)	91.8% (7.7)	84.9% (8.1)
D	87.2% (4)	93.2% (2.9)	87.1% (4.4)
E	83.1% (13)	94.3% (3.8)	88.3% (8.7)
F	72.5%	85.2% (9.6)	72.9% (15)
G	100% (0)	94.7% (4.4)	92.8% (5.4)
% mean fidelity score per category (SD)†
Materials	72.1% (19.4)	86% (17)	61.1% (29.6)
Introduction and review	82.6% (16.3)	92.9% (12.8)	76.2% (24.5)
Education	93.3% (8.6)	97.1% (6.6)	95.4% (6.9)
Exercise	80.4% (14)	95.4% (7.1)	82.4% (13)
Review and planning	77.1% (33)	90.8% (21.6)	69.8% (39.6)
% mean fidelity score per physio (SD)‡
A1	78.8% (7.6)	95.1% (5.4)	81.3% (11.5)
B1	76.1% (7.9)	92.2% (7.2)	72.3% (9.3)
B2	77.5% (0.4)	93.4% (3)	72.6% (12.5)
C1	93.4% (2)	85.2% (4.6)	91% (3.5)
C2	76.2% (11)	98.5% (1.9)	78.8% (6.6)
D1	87.2% (4)	93.2% (2.9)	87.1% (4.4)
E1	83.1% (13)	94.3% (3.8)	88.3% (8.7)
F1	72.5%	85.2% (9.6)	69.8% (14.7)
G1	100% (0)	94.7% (4.4)	95.8% (5.4)

Bold values = scores<80%.

*Significant differences between sites (p<0.001) and between sessions (p=0.007) according to AO, not significantly different according to DO and SR.

†Significant differences between categories according to DO (p=0.007), SR (p<0.001) and AO (p<0.001).

‡Significant differences between physiotherapists according to DO (p=0.019), SR (p=0.004) and AO (p<0.001).

Fidelity of duration

All methods found a statistically significant difference between the actual duration of the exercise component and its intended duration of 45 min (see online supplementary file 5). When this was analysed for each individual session for all methods combined, a statistically significant difference (p<0.001) was only found for session 1 between the actual and intended duration. The difference between the actual and intended duration for all other sessions was not found to be statistically significant (p>0.05).

Group size: participants in attendance

The average number of participants present for all groups across all sessions was 3, with an SD of 1.3 (see online supplementary file 6). Overall, group attendance ranged from one to six participants for any session across both waves. The total number of participants recruited for each group was significantly different between physiotherapists (eg, seven participants were recruited for the group delivered by physiotherapist F1 compared with only two recruited for the group delivered by D1), as were the numbers of participants present (average group size). The size of groups did not differ significantly between sessions.

Factors associated with fidelity

Both direct observation and audio-recorded data showed a significant correlation between fidelity scores and the physiotherapists’ post-training evaluation scores. Direct observation data also found a significant negative correlation between group sizes and the fidelity scores. Physiotherapist years qualified and experience of delivering groups were found to have significant, negative correlations with fidelity scores for the audio-recorded and self-report data respectively (table 3).

Table 3.

Association between fidelity scores and variables

Variable	Direct observations	Self-report	Audio-recordings	Statistical test
Group size (number present)	−0.434 (p=0.034)*	−0.215 (p=0.98)	−0.193 (p=0.151)	Spearman’s r (p value)
Physiotherapist experience (years qualified)	−0.09 (p=0.676)	−0.186 (p=0.154)	−0.346 (p=0.008)*
Physiotherapist group experience (years delivering group physiotherapy)	0.171 (p=0.424)	−0.364 (p=0.004)**	0.136 (p=0.312)
Physiotherapist post-training evaluation score (%)	0.581 (p=0.003)**	−0.152 (p=0.245)	0.314 (p=0.018)*
Physiotherapist previous relevant training (yes/no)	U=33 (p=0.302)	U=201 (p=0.107)	U=243 (p=0.840)	Mann-Whitney U (p value)

*p is significant at p<0.05, **p<0.005.

Qualitative findings

Inter-rater reliability of coding achieved 81.6% agreement. Overall, physiotherapists felt that they had delivered the programme with good fidelity. All physiotherapists discussed some deviations from the protocol or adaptations made during delivery, for example, goal-setting was found to be challenging to complete as intended. Other adaptations either concerned difficulties with use of programme materials (eg, using the projector) as intended or providing additional information during the education content. Five physiotherapists also discussed deviation from protocol in relation to duration, mostly during the first session, with one stating that her ‘time management around the education wasn’t always exactly what it should have been’ (B1, transcript line 278–286). In terms of factors that influenced fidelity (ie, reasons discussed for the aforementioned adaptations and deviations), six themes were found that were structured into three levels of factors—physiotherapist, participant and programme level (figure 2).

Figure 2.

Visual representation of themes of qualitative interviews—factors influencing fidelity of Self-management of Osteoarthritis and Low back pain through Activity and Skills (SOLAS) delivery.

At the physiotherapist level, eight of the nine physiotherapists felt that knowledge of the content of the SOLAS programme facilitated their delivery of the education session and made it easier. Conversely, deviations from protocol within the education content discussed by six physiotherapists were due to the provision of additional information which was influenced by their previous experiences of delivering similar groups (eg, talking more about pain pathophysiology because of previous classes delivered on this topic). This formed the theme ‘Physiotherapist knowledge and experience influenced delivery of SOLAS—education content’.

At the participant level, five physiotherapists felt that participants’ individual needs such as their understanding of content or language literacy levels influenced the delivery of education and exercise components and that adaptation sometimes occurred in response to these needs, creating the theme of ‘Individual needs influenced delivery of SOLAS—education, exercise, goal-setting’. The number of participants present was discussed by seven physiotherapists as another participant-level factor that influenced fidelity of delivery and formed the participant-level theme of ‘Group size influenced delivery of SOLAS—goal-setting, use of materials’. A further participant-level theme was ‘Group dynamics influenced delivery of SOLAS—goal-setting’ as four physiotherapists felt that groups with good dynamics and interaction between participants led to better group discussions and better facilitation of goal setting.

The ‘Amount of education content influenced delivery of SOLAS—duration’ was a programme-level factor discussed by six physiotherapists, who felt that the amount of education content that was involved in the first session led to more time spent on the education aspect than intended as per protocol. Finally, all nine physiotherapists believed that the good resources (eg, booklets and handouts, venue space) enhanced and facilitated the delivery of the programme as intended and that occasionally poor or problematic resources (eg, lack of venue security) negatively influenced the delivery of the programme as intended. This created the theme ‘Resources/materials influenced delivery of SOLAS—education and exercise content’. Exemplary quotes are provided in table 4.

Table 4.

Qualitative findings of factors influencing fidelity results

Factor level	Theme	Exemplary quote (physiotherapist code, transcript line)
Physiotherapist	Physiotherapist knowledge and experience influenced delivery of SOLAS—education content	‘In my previous experience I would have done a lot more actually on the pain side of things … so in my previous class I would have had, you know, maybe one full class on maybe pain perception and, kind of, the influence of emotion and feelings … so I think I would have probably maybe talked a lot more around that pain section than maybe somebody else would have’. (C1, 75–99)
Participant	Individual needs influenced delivery of SOLAS—education, exercise, goal-setting	‘People don’t like writing them [action plans] there and then you know with pencils given and whatever—yes it’s very hard to get people to write down things like that …. Where I work there is a lot of people health literacy is very low … so therefore that’s a challenge for them…so I tend to be very careful about pushing it out really’. (F1, 141–187)
Participant	Group size influenced delivery of SOLAS—goal-setting, use of materials	‘The only thing I might find a little bit hard would be the goal setting. I suppose you’d—that would be a bit more challenging because you’d have more numbers in the group’. (G1, 118–132)
Participant	Group dynamics influenced delivery of SOLAS—goal-setting	‘People were willing to engage you know as a group, in their goals … so that made it very easy that we didn’t actually have any clients that weren’t willing to talk in the group, so it was very much an interactive group’. (E1, 225–231)
Programme	Amount of education content influenced delivery of SOLAS—duration	‘I found the content in week one was nearly too much … by the time I finished talking and ran through the exercises, the hour and a half was finished. And so nobody actually practiced any of the exercises on the first day’. (B2, 96–106)
Programme	Resources/materials influenced delivery of SOLAS—education and exercise content	‘The slides didn’t work for me this time …. You can’t lock that room …. once or twice I didn’t bring the laptop at all and I just had to print it out, all of the slides on A4 laminate and so we talked all the slides ….’ (F1, 207–240) ‘I think I only left out maybe three [exercise] stations, something like that. Because we didn’t have a bouncer and … we didn’t have a bed’. (C1, 113–121) ‘Nothing but positive feedback for all the content and the-the resources … I just think they complimented the—the education fantastically, I just thought they added much more to the programme than not having these resources.’ (E1, 414–415)

Integrating qualitative and quantitative findings

Each theme was further analysed according to the quantitative data comparing physiotherapists who scored ‘high’ (ie, ≥80%; physiotherapists D1, C1, E1 and G1) to those who scored ‘moderate’ (ie, ≥50%–79%; B1, B2, C2 and F1). Physiotherapist A1 was not included in this analysis as her score was categorised as ‘moderate’ by direct observations and ‘high’ by audio-recorded data at 78.7% and 81.3%, respectively. A difference between these physiotherapist groups was found in only one theme, ‘Group size influenced delivery of SOLAS’. Physiotherapists who scored higher on the fidelity assessments (average group size of 2.5 participants) believed it was easier to deliver goal-setting as intended with smaller groups. Conversely, physiotherapists with moderate fidelity scores (average group size of 3.7 participants) felt it was harder to facilitate goal-setting as intended with less numbers and believed it would be easier with bigger groups due to better engagement and group discussion. Further details of the triangulation are provided in table 5 where a meta-matrix was used to compare between findings from both data sets. For the most part, convergence was found between the qualitative and quantitative data, though four qualitative themes relating to influencing factors had no corresponding quantitative data (silence). No areas of discrepancy were found.

Table 5.

Triangulation of quantitative and qualitative results: convergence and discrepancy of findings

Outcome of interest	Quantitative fidelity findings		Qualitative interview findings	Convergence/ discrepancy/ silence
Fidelity findings:	Overall fidelity of content scores	High fidelity (>80%) Direct observations: 82.7% Self-report: 92.7% Audio-recordings: 81.7%	Overall physiotherapists felt that their fidelity was good. Some adaptations and deviations were discussed as occurring in the delivery of the following aspects of the programme: Goal-setting (Introduction and review, Review and planning categories) Education content Exercise content Use of programme materials Duration of session 1	Convergence
	SOLAS categories scoring <80% fidelity
	Materials	Moderate fidelity (50%–79%) Direct observations: 72.1% Audio-recordings: 61.1%
	Introduction and review	Audio-recordings: 76.2%
	Review and planning	Direct observations: 77.1% Audio-recordings: 69.8%
	Fidelity of duration—sessions significantly different to intended duration
	Session 1	Education duration: 58.9 min* Exercise duration: 31.4 min*
Factors influencing fidelity findings:	Correlation between quantitative variables and fidelity scores		Theme 1: physiotherapist knowledge and experience influenced delivery of SOLAS— education content	Convergence
	Physiotherapist experience (years qualified)	Audio-recordings: –0.333 (p=0.011)***
	Physiotherapist group experience (years delivering group physiotherapy)	Self-report: –0.430 (p=0.018)***
	Physiotherapist post-training evaluation score (%)	Direct observations: 0.581 (p=0.003)** Audio-recordings: 0.314 (p=0.018)***	Theme 1: physiotherapist knowledge and experience influenced delivery of SOLAS— education content	Convergence
	Group size (average numbers of participants present)	Direct observations: –0.434 (p=0.034)***	Theme 3: group size influenced delivery of SOLAS—goal-setting, use of materials	Convergence
	No corresponding quantitative data		Theme 2: individual needs influenced delivery of SOLAS—education, exercise, goal-setting	Silence
			Theme 4: group dynamics influenced delivery of SOLAS—goal-setting
			Theme 5: amount of education content influenced delivery of SOLAS—duration
			Theme 6: resources/materials influenced delivery of SOLAS—education and exercise content

SOLAS, Self-management of Osteoarthritis and Low back pain through Activity and Skills.

*p<0.001, **p<0.005, ***p<0.05.

Discussion

The aim of this mixed methods study was to explore and evaluate fidelity of delivery within a feasibility trial of a complex behaviour change intervention using multiple assessment methods. The study found good agreement between researcher-delivered direct observation and audio-recorded fidelity assessment methods, with lower agreement found between provider self-report and researcher-delivered methods. The intervention content was delivered overall with high fidelity, with some variation between physiotherapists and between certain intervention categories. The intervention duration was found to have deviated significantly from intended during the first session only. Subsequently, qualitative interviews with physiotherapists confirmed these fidelity findings. Finally, both qualitative and quantitative data showed that physiotherapists’ knowledge and previous experience, as well as the group size, were factors that influenced their fidelity of intervention delivery. The qualitative data contributed further, and postulated additional participant and programme-level factors as aspects that also influenced the overall fidelity results.

Agreement between direct observations and audio-recordings for assessing the fidelity of delivery was found to be excellent.³⁹ Agreement between both of these methods and provider self-report assessment was lower, as providers consistently rated themselves higher than the independent raters. These findings are perhaps unsurprising as both direct observations and audio-recordings were rated by the same researcher, and numerous previous studies have shown that providers’ subjective assessments of fidelity are often rated higher than independent assessments.^{31 32 40} Taking direct observations as the commonly cited ‘gold standard’,^{11 18 20} these findings reinforce that self-report methods may not be the most accurate method for assessing fidelity in a complex behaviour change study. However, they may still have their place for recording data and also for enhancing fidelity to the protocol by serving as an aide memoire for providers.⁴¹ Although direct observations and audio-recordings have their own limitations,^{24 42} previous piloting of these assessment methods found that they were feasible and acceptable to physiotherapists.²⁷ Additionally, the good agreement between audio-recordings and direct observations found in this study suggests that audio-recordings may be a viable alternative with limited resources, as has been done in similar interventions.⁴³ However, where resources allow, a combination of multiple quantitative methods may provide the most in-depth assessment of fidelity.

One of the key study findings of this study was reinforcing the value of using mixed methods research for the assessment of fidelity. This approach was emphasised in the recent MRC guidelines for conducting process evaluations of complex interventions⁶ and is becoming increasingly used in the widespread implementation of evidence-based interventions⁴⁴ but does not yet appear to be common practice within fidelity assessments of behaviour change interventions.^{10 21} The integration of quantitative and qualitative results enabled the triangulation of findings to provide a better overall picture of the fidelity of the SOLAS intervention and its influencing factors. The importance of the qualitative contribution to answering the ‘why’ question is evident in the fact that the physiotherapist interviews unearthed strong participant and programme-level factors associated with fidelity results that were not apparent from the quantitative data alone. While this may be predominantly due to the focus of the quantitative analysis on physiotherapist-level variables which were chosen based on existing literature, the participant and programme-level factors identified by this analysis such as group dynamics or amount of programme content may have been difficult to quantitatively analyse to demonstrate association with fidelity results.

This study found that the factors that may influence the fidelity of an interventions’ delivery can occur on three levels : provider, participant and programme. Where previous studies have explored factors that have influenced fidelity of intervention delivery, many have focused solely on provider-level factors, demonstrating associations between fidelity and factors such as provider training or skills.13 19 21 45 The findings of this study have valuable implications for future studies that aim to assess and enhance fidelity of similar interventions as they indicate that planning for fidelity should include considering potential influencing factors at each of these three levels. These results are consistent with recent conclusions by Masterson-Algar et al in a stroke rehabilitation setting,¹⁴ who found that investigating fidelity within clinical trials should also take the individual needs of patients into account, and also concur with the findings of an education-based intervention that found the most common reason for adaptation within intervention delivery was insufficient time.⁴⁶

On the physiotherapist-level, better knowledge of the intervention content and structure was found to positively correlate with quantitative fidelity scores, with a causative link established via the qualitative investigation. This echoes previous findings by Huijg et al, who showed that physiotherapist skill level was one of the most important predictors of fidelity.²¹ A more targeted approach to enhancing fidelity in future interventions may therefore be warranted, such as identifying physiotherapists at higher risk of lower fidelity using post-training evaluation scores and employing more focused fidelity assessment of delivery or further training for these providers,¹³ as has been previously employed in similar interventions.^{10 43} The results of the study also showed that physiotherapists with more experience of certain aspects tended to emphasise these at the expense of delivering other components as comprehensively as intended in the protocol. These experience-based adaptations invoke the well-established issue of adaptation versus fidelity. For years, research has debated the concept of fidelity versus adaptation, with the case made for both strict fidelity and for modifying interventions.⁴⁷ A third view is that both fidelity and adaptation are essential, and achieving an appropriate balance between both can allow an intervention to maximise its effectiveness, while being generalisable and flexible enough to be implementable.^{48 49} To achieve this, our fidelity checklists included components that encouraged elements of treatment individualisation (eg, individualised feedback regarding exercises). However, it may be that these checklists still did not allow for enough individualisation within delivery, an aspect that should be considered by other researchers seeking to conduct similar fidelity assessments.

A limitation of this study was the timing of the interviews, which did not allow a ‘pure’ convergent/triangulation design. Typically, the qualitative and quantitative methods occur concurrently in this design;²⁸ however, they were scheduled to take place after physiotherapists had experienced delivery of an entire 6-week SOLAS intervention. Although a sequential explanatory design³⁰ where quantitative results were analysed prior to completing the interviews might have enabled further probing of the factors influencing fidelity, interviews were conducted within 2 weeks of the intervention completion to minimise recall bias. Due to time constraints, it was not possible to have the quantitative data collected and analysed beforehand. Finally, this study mostly focuses on the adherence of delivery (eg, intervention content and duration) and does not address the quality or competence of delivery of SOLAS (eg, interpersonal or communication style of the physiotherapist), or use of specific BCTs, which is being addressed in a separate publication. This study also does not examine the broader aspects of fidelity such as provider training or participant receipt as these were beyond the scope of this publication and will be addressed in a future paper.

Conclusions

In process evaluations and fidelity assessments of large-scale complex interventions, it is often recommended to complete and report the results of the fidelity assessment before the trial outcomes so as not to bias reporting.⁵⁰ Future work will investigate the relationship between this evaluation of fidelity of delivery, the SOLAS feasibility trial outcomes (analysis currently under way) and the evaluation of fidelity of BCT delivery, enabling a potentially more insightful and accurate interpretation of findings. This study also has valuable implications for further research and the overall science of fidelity as it contributes much needed information to the limited current evidence for the application of fidelity assessment methods within the area of complex behaviour change. The findings have demonstrated how multiple quantitative methods can be used to assess the fidelity of delivery of a complex behaviour change intervention, and that a combination of methods may be most suitable, depending on their acceptability and available resources. We have also shown how the use of a mixed methods approach, integrating both quantitative and qualitative data, provides a more insightful understanding of the factors influencing fidelity.