Ensuring treatment fidelity in intervention studies: Developing a checklist and scoring system within a behaviour change paradigm

Abstract

Background

Treatment fidelity refers to the degree to which an intervention is implemented as intended. Promoting treatment fidelity is important to achieve a valid comparison in intervention research. However, it is often underreported: few studies detail the use and development of fidelity measures. This study aims to promote the treatment fidelity of a modified version of the Derbyshire Language Scheme (M‐DLS), a manualised intervention for children with language difficulties, by exploring participants’ opinions on training and intervention delivery. Results inform development of a checklist and scoring system to monitor and promote treatment fidelity in a comparison trial.

Method

Ten student speech and language therapists (SLTs) and two research assistants (RAs) participated in the study. All received training on the M‐DLS, and 10 were video‐recorded completing role‐plays of an M‐DLS session in small groups. Feedback was gathered after training and role‐plays in focus groups and interviews. Feedback was interpreted using the constructs of the Theoretical Domains Framework (TDF). A treatment fidelity checklist was then developed using the feedback. The first author and two RAs rated role‐play videos using the checklist to trial it to inform amendments and to promote interrater reliability. Interrater agreement was calculated using Spearman's test of correlation.

Results

Participants discussed the importance of having clear materials and time to practise sessions. They suggested amendments to the materials and training to promote treatment fidelity. The checklist and scoring system accounted for participants’ suggestions, with amendments detailed in a log. Spearman's correlation results suggested agreement between the raters was strong.

Conclusions

Results emphasise the importance of training quality, practice and reflective opportunities and clear materials to promote treatment fidelity. The construction of the checklist and scoring system was described in detail, informing the development of future checklists. After further trialling, the checklist can be used to ensure the M‐DLS is delivered with high treatment fidelity in the comparison trial.

What this paper adds

What is already known on this subject

• Treatment fidelity is an essential component of intervention effectiveness and efficacy studies, ensuring the intervention is delivered as intended. It is also an essential component of evidence-based clinical practice. However, few research studies report the treatment fidelity process or publish the checklists used, depriving clinicians of useful information for implementation.

What this study adds

• This study describes in detail the iterative process of treatment fidelity checklist development, engaging those implementing the intervention in development. This ensured clarity and interrater reliability of the checklist. Furthermore, a novel scoring system was developed so that accuracy of implementation can be easily compared across users and across practice attempts.

What are the clinical implications of this work?

• The importance of treatment fidelity when implementing effective and efficacious interventions cannot be overstated. The treatment fidelity checklist developed for research can be easily adopted to support accurate implementation in clinical practice through an audit process.

INTRODUCTION

Between 5 and 19 percent of preschool children have language disorders that require support from speech and language therapists (SLTs) (Law et al., 2017; Tomblin et al., 1997). A subgroup of these children will face additional behavioural, emotional, and social difficulties (Lindsay & Dockrell, 2012) that can persist into adulthood (Clegg et al., 2005). These negative consequences of language disorder highlight the need for early intervention before children enter formal education. For very young children, parent–child interaction (Klatte & Roulstone, 2016) and programmes such as the Hanen Parent Programme (Manolson, 1992) have been shown to be effective (Roberts & Kaiser, 2011). However, for those preschool children who have more persistent difficulties, a more structured and specialist approach is required targeting specific language domains such as syntax and vocabulary. In the United Kingdom, the Derbyshire Language Scheme (DLS) (Knowles & Masidlover, 1982) is possibly the most commonly adopted approach for this group of children. Since its publication as a manualised intervention in 1982, originally for children with learning disability, the DLS has been widely used in clinical practice. Despite limited research evidence, SLTs clearly subjectively believe it to be effective as evidenced by its continued use and adaptation into clinical practice (Morgan et al., 2013). Anecdotal evidence suggests that few SLTs implement the intervention as described in the manual, but instead tend to adopt the information carrying words concept as an approach. Law et al. (2012) report the evidence base for the DLS as ‘indicative’, stating it is a beneficial approach to consider based on its significant use in practice by SLTs. One randomised control trial investigated the effectiveness of SLT for children with speech and/or language impairment at a service‐wide level. Children with comprehension difficulties received the DLS (Broomfield & Dodd, 2011). The results suggest that the DLS was significantly more effective than no treatment for children with comprehension difficulties.

Since the development of the DLS there have been few manualised interventions targeting language disorder in preschool children addressing vocabulary and syntax. Building Early Sentences Therapy (BEST) (McKean et al., 2012) is a usage theory based intervention (Tomasello, 2005) targeting syntax and vocabulary development. Preliminary investigations have demonstrated its effectiveness with preschool populations (Trebacz, 2021). Research into the effectiveness of BEST continues with the LIVELY (Language Intervention in the Early Years) project (McKean et al., 2020), an intervention evaluation study comparing a modified DLS (M‐DLS) (McKean & Masidlover, 2019) with BEST. For changes to be attributed to a specific intervention such as BEST or M‐DLS, considerations must be made to ensure the intervention has been delivered with high treatment fidelity. Treatment fidelity is the degree to which an intervention is administered as intended (Kaderavek & Justice, 2010). Whilst the DLS is widely used, anecdotal evidence suggests that it is not always implemented as intended. Changes made when implementing an intervention, could obscure the evidence or theoretical base supporting it. The degree to which treatment fidelity is achieved may influence intervention results (Kaderavek & Justice, 2010; Schlosser, 2002). Therefore, promoting treatment fidelity is a key step in producing a valid comparison in all intervention research.

The heterogeneity of SLTs’ caseloads frequently leads to the adaptation of therapy programmes when implemented in practice (Rousseau et al., 2002). Therefore, treatment fidelity is violated, and the most effective components of an intervention may not be implemented. Whilst treatment fidelity is recognised as an important concept across research, systematic evaluation of treatment fidelity in SLT is often limited (Hayden et al., 2015) and it is not consistently reported in studies (Kaderavek & Justice, 2010). Treatment fidelity monitoring is required at all levels of intervention research, from design of intervention delivery through to implementation in practice. Borrelli et al. (2005) describe a five‐phase model to promote treatment fidelity across the stages of developing and administering intervention to ensure internal validity of the study. These comprise study design, training providers, delivery of intervention, receipt of intervention and enactment of intervention skills.

An important aspect of treatment fidelity is ensuring that clinicians are trained to carry out an intervention as intended, and that the quality of training be evaluated and monitored. All intervention providers should be trained in the same manner, using a clear and detailed manual, to increase the likelihood of systematic delivery. (Bellg et al., 2004; Gearing et al., 2011; Kaderavek & Justice, 2010). However, a treatment manual does not always prevent ‘therapist drift’, that is, ad hoc changes made when implementing an intervention (Gearing et al., 2011; Moncher & Prinz, 1991). In research, treatment fidelity measures are utilised to minimise therapist drift (Bellg et al., 2004). These include manualising an intervention, completing fidelity checks using a fidelity measure and/or providing training update sessions to support intervention delivery.

Studies indicate that video recording of intervention sessions and rating them with treatment fidelity checklists or tools, can be a useful method of systematically promoting treatment fidelity (e.g., Hinckley & Douglas, 2013; Yamada et al., 2015). Checklists are coded to align with the key components of an intervention and can be used to evaluate the treatment fidelity of intervention delivery (Kaderavek & Justice, 2010). This is considered the reference standard for measuring the treatment fidelity of an intervention (Gearing et al., 2011; Kaderavek & Justice, 2010). However, the process of designing and validating treatment fidelity checklists is rarely described in any detail (Borrelli et al., 2005; Conlon et al., 2019; Ory et al., 2002). Detailing the development of the checklist can support judgement of the checklist's validity and evaluation of the psychometric rigour of a fidelity checklist (Hayden et al., 2015). Yamada et al. (2010) is one of few studies describing the development of a checklist in detail. The researchers generated items for each intervention activity, using content analysis to determine which elements of the intervention were relevant.

The Theoretical Domains Framework (TDF) (Atkins et al., 2017) is widely used to assess barriers and facilitators for implementation of behavioural interventions (Mazza et al., 2013; Murphy et al., 2014). The TDF has 14 theoretical domains, each formed through the synthesis of 128 constructs found in 33 behaviour change theories. Therefore, it enables consideration of potential influences on a given behaviour, such as the influences on implementing an intervention with high treatment fidelity (Atkins et al., 2017). The COM‐B (Capability, Opportunity, Motivation‐Behaviour) model (Michie et al., 2011), developed from behaviour change theories, is a useful model for considering the barriers and enablers to implementing interventions that require new skills and knowledge to be employed by practitioners, effectively changing the practitioners’ behaviour (e.g., Alexander et al., 2014). It is used in this study as a framework for developing training content to ensure that capability, opportunity and motivation enablers were all appropriately considered (see Supporting information 1).

This paper describes the development of an objectively scored treatment fidelity checklist for the modified version of the DLS (M‐DLS), providing a template that can be adapted for use in other efficacy or effectiveness studies. Treatment fidelity of the M‐DLS is monitored at two of five phases for promoting treatment fidelity (Ory et al., 2002): clinician training and intervention delivery. Participant feedback on training and M‐DLS materials was analysed, and the results informed the development of a scored checklist used to monitor the treatment fidelity of the M‐DLS in the LIVELY project. A parallel checklist was developed for use with BEST that is not reported here.

RESEARCH QUESTIONS

The aim of this study is to develop a treatment fidelity checklist to ensure that the M‐DLS is delivered with high treatment fidelity in the LIVELY project. It is explored through the following research questions:

1. Which aspects of the M‐DLS training and role‐play support participants’ understanding of the M‐DLS, and therefore promote treatment fidelity?

2. After training and role‐playing the M‐DLS, which aspects of the intervention materials do participants find compromise treatment fidelity?

3. Following training and role‐play what amendments should be made to the M‐DLS treatment fidelity checklist to promote treatment fidelity?

METHODOLOGY

Study design

An exploratory sequential design was adopted to evaluate the training and implementation of the M‐DLS. This study was completed in three phases, and treatment fidelity was considered at two levels: clinician training and intervention delivery (Ory et al., 2002).

Participants

Purposive sampling was used to recruit participants with appropriate training in speech and language therapy to reflect the typical users of the DLS and to ensure that discussion would be appropriate to the research questions (Ritchie & Lewis, 2003). Twenty‐two final year students studying for a degree entitling them to legally register in the United Kingdom as SLTs, were contacted by email. Additionally, two research assistants (RAs), both qualified SLTs employed for the LIVELY project, were recruited. Twelve participants agreed to take part in the study: 10 student SLTs and two RAs. No participants had any previous formal training on the DLS. However, the RAs had previously received some informal training, in the form of information sharing, on the M‐DLS. Subsequently, all 12 participants received training on the DLS from the first author. Ten participants took part in the focus groups, two participants (both RAs) were unable to attend. The same 10 participants took part in role‐play sessions in four small groups (see Figure 1). Participants were organised into groups according to their availability.

FIGURE 1.

A flow diagram illustrating the number of participants at each stage of the study. [Colour figure can be viewed at wileyonlinelibrary.com]

Procedure

Phase one: DLS intervention training

The first author attended informal M‐DLS training led by the third author. From this, a training plan based on one session of the M‐DLS was developed. The COM‐B model (Michie et al., 2011) was used as a framework (see Supporting information 1) to ensure different influencing factors on behaviour change were considered, that is, the participant's capabilities, motivation and opportunities for carrying out the intervention. The behaviour goal (B) was ‘for participants to be confident and capable of carrying out the DLS intervention’. Participants received a 1‐h training session on the M‐DLS (McKean & Masidlover, 2019), presented by the first author in October 2019. The first author presented the training with a PowerPoint presentation, handouts and interactive activities. Two video recordings were used to demonstrate the intervention task to participants. Video one was a ‘correct’ demonstration of the intervention and video two demonstrated ‘things not to do’. Participants then practised the session using provided materials.

Focus groups

Following training, feedback was gathered from participants in two focus groups lasting 32 and 51 min, respectively. They were recorded using two high‐quality digital voice recorders. Both focus groups were structured using predetermined questions and prompts in a topic guide. Participants were given M‐DLS materials, session plans and manuals to support discussion. The first author led the groups, putting across alternative views to encourage discussion (Ritchie & Lewis, 2003).

Phase two: Role‐playing the M‐DLS

In groups of two or three, participants were asked to role‐play a shortened M‐DLS intervention session using session plan A1: 2–3 word level. Child and SLT roles were assigned to participants. Each participant completed one session as the SLT, with two participants acting as children. The children were randomly assigned one of four ’behaviour profiles’ (e.g., poor attention, fiddles with toys, at two‐word level) and wore stickers to illustrate their word level to the SLT. The SLT was given 5 min to read the session plan and prepare for the session, whilst the ‘children’ familiarised themselves with their behaviour profiles. Participants rotated until each participant had acted as the SLT. It was suggested that participants give each child six turns in each session; three for the Comprehension task and three for the Expression task, to ensure role‐plays were of an appropriate length. To explore the perceived facilitators and barriers when scoring the session, some participants were asked to score their role‐play sessions using a score sheet consisting of a list of sentences used in the task. Where groups had only two participants, the first author acted as a child during the role‐play session. Role‐play sessions were video recorded using a digital camera to obtain observational data for later analysis. Nine role‐play sessions were video‐recorded; one session was not video‐recorded due to technical difficulties.

Group interviews

Group interviews by the first author were semi‐structured using a topic guide immediately after both role‐play sessions. Interviews varied in length from 20 to 40 min. Semi‐structured interviews were used to allow in‐depth exploration of the participants’ opinions and experiences of the M‐DLS manual and role‐play. Group interviews were audio‐recorded and participants were given materials to support discussion (Ritchie & Lewis, 2003).

Phase three: Developing the checklist

A M‐DLS treatment fidelity checklist (Version 1) had been developed by the LIVELY project, based on the BEST treatment fidelity checklist. Feedback obtained from focus group sessions and from group interviews was used to amend this, resulting in Version 2. Amendments and alterations to the checklist were kept in a log (see Supporting information 2). The first author used checklist Version 2 to rate all nine role‐plays to inform further amendments to the checklist and to calculate total treatment fidelity scores for each role‐play. The two RAs were each asked to rate two videos independently to promote interrater reliability and inform amendments. Each RA was randomly assigned two videos (videos 4, 6, and 7, 9) to rate using checklist Version 2. Rating methods were discussed with the RAs to ensure consistency, for example, how to rate whether a participant had completed an action ‘Sometimes’ rather than ‘Almost never’. The first author and the RAs agreed to keep a tally of actions taken by the participant to determine the reported results. After all videos had been rated, the RAs met with the first author to discuss their outcomes. To promote interrater reliability, when the agreement between the raters was low for an item on the checklist, the item was amended to increase clarity. An iterative process of revision and feedback was implemented to gain feedback from the RAs and student participants, for three further versions of the checklist. Subsequently the RAs rated the same two videos using Version 5 of the checklist to determine whether amendments had led to changes in total scores and had increased interrater agreement. Further suggestions from the first and second authors and the RAs were considered to produce the final checklist.

Materials

The M‐DLS and toys

The adapted DLS manual (M‐DLS) was developed by McKean and Masidlover (2019) to ensure the manual stayed true to the original intentions of the DLS. The training focused on one session plan of the M‐DLS (A1: 2‐3‐word level). The session plan comprises a comprehension task followed by an expression task, both centred on a teddy bear picnic. M‐DLS cue cards created by Masidlover were used in role‐play sessions. The required toys were provided for the participants.

Training materials

The first author created PowerPoint slides and handouts for use in the M‐DLS training session, with input from the second and third authors. These were designed in line with the M‐DLS manual and were made to encourage participation in the session.

Interview topic guides

Topic guides were constructed by the first author for use in the focus groups and group interviews. Use of the TDF (Michie et al., 2013) ensured numerous influences on behaviour were considered and to facilitate theoretical assessment and organisation of interview results. Before the discussion, the concept of treatment fidelity, and the purpose for the research was explained to the participants (Ritchie & Lewis, 2003). Participants were asked to make suggestions to improve the topic guide for future sessions. One participant suggested it would be helpful to reflect on each other's role‐play sessions.

Child behaviour profiles

Four child behaviour profiles were produced for use in role‐play sessions. These were used to reflect differences seen in child behaviour, to mimic a naturalistic scenario and to help understand how fidelity might be impacted in typical preschool settings (Harn et al., 2013).

Data analysis

All focus groups and group interviews were transcribed verbatim and checked for errors against the recordings by the first author. Data was analysed using a simplified method of content analysis; the data were inspected for recurrent information and grouped under the domains of the TDF. Sub‐themes emerged where data were replicated across interviews. Use of the TDF enabled systematic organisation of the data, keeping data analysis focussed on the research questions for future consideration when developing the fidelity checklist. Data that were not relevant to the study were not included in the analysis. Field notes were used throughout interview sessions to record non‐audio data, such as an individual pointing to an item in the manual (Michie et al., 2011). To promote the accuracy of the feedback from group interviews, member checking was used: interview summaries were provided to participants to confirm whether accurate conclusions had been made (Creswell, 2018).

The interview results informed amendments to the M‐DLS treatment fidelity checklist and were kept in an Amendment log (see Supporting information 2). Total fidelity scores were calculated and reported for each video using checklist Versions 2 and 5. This was to determine whether the amendments made to the checklist caused the total scores to change.

Interrater agreement was considered across the whole checklist; the first author's checklist ratings were compared to the RA's ratings for each variable per video. Spearman's rank‐order correlation was used to analyse the data in SPSS Statistics. Correlation coefficients were established twice for each video, once for ratings using checklist Version 2, and once for ratings using checklist Version 5. This test was appropriate for the non‐parametric and ordinal data. Correlation coefficient values of 0.50 to 0.69 were considered a strong correlation and values from 0.69 to 1 were considered a very strong correlation (Frey, 2018). The p‐values were reported to conclude whether agreement was significant.

Ethical considerations

This study was approved by the Speech and Language Sciences Ethics Committee at Newcastle University. All participants were given information sheets and consent forms which they signed. They were informed that all data would be kept confidential and that they could withdraw at any time.

RESULTS

Research question 1: Which aspects of the M‐DLS training and role‐play support participants’ understanding of the M‐DLS, and therefore promote treatment fidelity?

Focus groups and group interviews involved discussion about the M‐DLS training and role‐play, regarding ‘training providers’ (Ory et al., 2002). Four TDF domains were predominant in the discussion:

Knowledge

Participants thought they understood the purpose of the DLS from the training and that the presentation gave them sufficient background knowledge of the M‐DLS.

Environmental context

Participants discussed features of the training environment and the materials used. Comments included:

• The duration of the training session was appropriate.

• More time would allow participants to ask questions and practise in smaller groups.

• The layout of the room could have been more inclusive, suggesting grouped tables rather than rows to encourage discussion and enable individuals to see the role‐play more clearly.

• The resources in the presentation (role‐play task, video and handouts) supported their understanding of the M‐DLS.

• It would be beneficial to watch a video of an SLT with a group of children carrying out the session.

• One participant commented that the training video did not show the tray of toys.

Participants made comments on the M‐DLS session plan:

• There were incomplete sentences and minor typing errors.

• The description of the goals was stated as confusing due to the ‘prose’.

• Abbreviations were used which were not clarified in the session plan (although were elsewhere in the manual).

• Font use and highlighting was inconsistent, highlighting was distracting

• Use of ellipsis was described as ‘annoying’.

• Different terminology was used in the plan from that in the manual and materials, such as the use of ‘cup’ rather than ‘beaker’.

Participants found some sections of the plan hard to understand, raising questions as to:

• How many turns each child would receive

• Whether to repeat the same cards or use different cards for each turn

• Whether the storyline was meant to be said to the child or was only for the SLT

• Whether the toys are provided with the manual. Some participants suggested that different toys could alter the word levels and the difficulty of instructions

• Whether ‘Information Carrying Words’ is synonymous with the term ‘key words’

Reinforcement of knowledge

Participants thought training reinforced their knowledge. They commented on how the trainer (first author) defined all stages of the DLS so that “everyone had the same base of knowledge”; how defining each information carrying word level was beneficial, as it led to further questions and answers regarding ambiguities. Participants reported that the presentation was detailed yet they “did not feel they were being talked down to”, describing the trainer as very “approachable” during the training.

Beliefs about capabilities

There was discussion surrounding the participants’ capability to carry out the intervention. They stated that the training was helpful but that more practice and experience would be needed before implementing the intervention “fully”. Participants stated that having the session plan would make them feel more capable to complete a session and observing another session in its entirety would be advantageous. They suggested that it would be important to see the more complicated session plans role‐played in a group setting. One participant reported concern over controlling the equipment during the session.

Research question 2: After training and role‐playing the M‐DLS, which aspects of the intervention materials do participants find compromise treatment fidelity?

During discussion of the role‐play and the session plan, nine of the 14 TDF domains were predominant (see Figure 2) with headings emerging under each domain.

FIGURE 2.

A figure to show the predominant TDF domains and headings which emerged through discussion. [Colour figure can be viewed at wileyonlinelibrary.com]

Abbreviation: TDF, Theoretical Domains Framework.

Knowledge

Procedural knowledge: The following issues emerged from the whole group regarding the M‐DLS session plan:

• Uncertainty about which word level to start at.

• Uncertainty as to how to complete actions differently according to word levels.

• Uncertainty if they were to reset equipment after each turn.

Issues raised by individual participants included:

• Using the cue cards to follow the procedure rather than the instructions in the session plan.

• Forgetting to follow the hierarchy of prompts.

• Not sure whether they used the hierarchy of prompts accurately.

• Not knowing they had to shuffle the M‐DLS picture stimulus cards rather than choosing which ones to use (e.g., choosing the more ‘basic’ cards, perceiving the ‘pouring in’ cards as a higher word level due to the presence of a preposition).

• Forgetting to show the cards to the children after the instruction.

• Not saying “look you got it right” when showing the cards.

• Not taking out both sets of equipment because they did not read the plan properly.

• Unsure whether to introduce all the items on the tray to the children.

• Unsure whether to give children second turns at an instruction.

Skills

Competence: Overall, participants stated that they would prefer to complete the session without simultaneously reading the plan (i.e., they wanted more preparation time). Most responded that they would not look at the session plan “all of the time” during a session. Participants discussed their awareness of making mistakes, explaining that they felt they could not change their actions in the session without confusing the children.

Skills in practice: The participants used the session plan to inform their actions. Participants discussed how more observations of model M‐DLS sessions and more practice time, would support their intervention delivery, skill development and confidence.

Adaptations for children: Participants were unsure how to adapt the session when a child was non‐verbal, not noticing that the plan states that you can work on comprehension only. Participants commented that they adapted the prompts sometimes because they felt that they had to be flexible when working with a ‘tricky’ child.

Beliefs about capabilities

Perceived competence: Discussion regarding competence produced a mixture of results. Participants reported feeling concerned that they had not said an instruction correctly and that they would have been more prepared in a “real” session. All reported that they did not feel confident that they were completing the plan as intended. One participant reported that completing the role‐play after observing another participant was advantageous. The same participant described feeling confident doing the session “their way”.

Managing children: Participants felt that without stickers on the role‐play children (and potentially children in a real scenario), it would be difficult to keep track of different word levels, especially when children have unpredictable behaviour. Most participants stated that they gave good encouragement to the children.

Beliefs about consequences

Participants thought their incorrect actions could influence the session. For example, not giving another child the opportunity to give instructions in the Expression task might compromise their participation.

Motivation and goals

Participants were largely unsure what the goal of the session was or if it had been achieved. This was due to not reading the goals or not understanding the goal due to the wording in the plan. They were not sure how many turns or activities were necessary to complete the goal. Some thought they generally achieved the goal despite not carrying out certain parts of the plan.

Social influences

Child influences: Discussion revealed that managing children made it difficult to read the plan during the session. Moreover, participants explained that the children were “unpredictable” and continually moved the toys. Participants explained that it would be difficult to balance scoring, prompting and managing the child's behaviour. One participant reportedly felt unable to complete the task if there were more than three children.

Environment and resources

Resources: Participants made suggestions to rectify difficulties found while using some of the equipment. Participants stated the picture stimulus cue cards were hard to handle, and that it would be beneficial to have them connected or in a flipbook. Additionally, they suggested that a numbering system would link cue cards to score sheets. They stated that the equipment tray was too small. Participants stated that an equipment list would be helpful; one participant became confused over how many of each toy was required. Participants stated that terminology lacked clarity and should be the same across materials, such as ‘beaker’ and ‘cup’.

Room layout: Participants considered ways to organise the room to optimise a M‐DLS session. Most participants stated that they would carry out the session on the floor; one participant described a specific set‐up around a table. Issues considered were:

• Space available for seating children and SLT.

• Possibility of grouping children according to word level ability.

• The SLT's ability to observe all the children.

• Potential to hide the tray of toys.

• Ease of keeping the teaching space tidy.

• If consistency across sessions would be better achieved at a table or on the floor.

Session plan: Overall, participants reported that the session plan was hard to read while the session was in progress, contained too much information and that overall, they felt that they “couldn't look at it” during the session.

Participants noted explicit problems with the session plan:

• They thought the goal at the top of the page was an instruction.

• The use of different fonts made instructions unclear.

• It was not useful to have the section on homework within the plan.

• Prompts went over to the second side of the page.

• It was unclear what to say to the children.

• The structure was unclear.

• Participants did not notice that you could work on comprehension only with a non‐verbal child.

• The description of recasting was confusing; they were unsure when to recast.

Participants made suggestions for the session plan, such as:

• Having “SLT says” in bold to draw their attention.

• Having a clear step‐up and step‐down for activities.

• Clarifying the difference between prompts for expression tasks and prompts for comprehension tasks.

Scoring: Participants recommended including the following on M‐DLS scoring forms:

• An ordered list of the instructions used within the session.

• Numbered instructions so you would not have to read the sentence whilst scoring.

• Underlined information carrying words, so that you know which words are the target words for in each instruction.

• A scoring table with columns for different prompts.

• A scoring table with columns or space for each child's initials to identify them.

• A scoring sheet per child. However, one participant felt that this might be confusing and suggested having a scoring table per child.

• Recognition in the scoring form of the context around the child's score, such as if a child needed a repetition or how many errors were made.

• A scoring table split into two sections for expression and comprehension tasks.

Prompting: Results suggest participants were unsure of how to use the prompt hierarchy. For example, participants were not sure how to prompt the child if they interrupted an instruction. Participants found the language used to describe the prompts confusing. Some participants did not realise the prompt hierarchy was describing the process for dealing with errors.

Memory, attention and decision processes

Decision making: Some participants chose not to shuffle the picture stimulus cue cards, whilst some shuffled them at the start to put them in a random order. One participant decided to vary the storyline script because they thought it repeated itself. Participants reported that they used the toys to help count the number of turns the children had received.

Attention: Participants explained that children can become distracted, making the session challenging.

Behavioural regulation

Feedback: Participants stated that they used appropriate feedback such as “well done”. One participant was unsure of whether to use explicit negative feedback such as “you got that wrong”. One participant stated that they did not say, “look we got it right” as stated in the plan.

Self‐monitoring: Participants acknowledged they would have done things differently if they had read the plan “more closely”. When asked to reflect on the session, participants commented that they gradually forgot to say “look we got it right” as well as “can you say it” during the expression task. One respondent reported accidentally giving clues through eye gaze. One participant expressed feeling that they put pressure on the child to imitate the recast, and upon reflection, realised they should not have.

Research question 3: Following training and role‐play what amendments should be made to the M‐DLS treatment fidelity checklist to promote treatment fidelity?

Interview and focus group results were used to make amendments to the treatment fidelity checklist. The treatment fidelity checklist (Version 1) drafted by the LIVELY project acted as a starting point. Amendments were recorded in an amendment log (see Supporting information 2 for detail). All amendments were discussed and agreed with the LIVELY project. The nature of the amendments included changes of content to more accurately reflect the intervention activities, increased clarity of wording and formatting for ease of use.

Development of a scoring system

A scoring system was developed to quantify the Likert scale and to enable comparison of treatment fidelity scores between role‐play sessions. An ordinal scale from 0 to 4 was assigned to the Likert scale: never (0), almost never (1), sometimes (2), almost every time (3) and every time (4).

As a result, sessions executing variables ‘almost every time’ or ‘every time’ would receive scores of three and/or four, totalling a high overall score of fidelity. In contrast, sessions that do not adhere to the manual, and so, execute variables ‘never’ or ‘almost never’, receive scores of zero and/or one, totalling a low overall score.

Negative wording was removed from the checklist, and variables were added which addressed actions discussed in interviews that went against the intentions of the manual and negated treatment fidelity. Negative marking was introduced to account for these variables in Version 4 (see Supporting information 2). For these variables the Likert scale is numbered in the same way but the scores are subtracted from the final tally. For example, if a participant completed an action which negated treatment fidelity ‘Always’, they would be scored four. The score of four would then be subtracted from the total, reducing the score of treatment fidelity. Variables were colour coded as to whether they promoted treatment fidelity (yellow) or negated treatment fidelity (orange) (see Figure 3). The total is calculated by adding the yellow scores and then subtracting the orange scores.

FIGURE 3.

Two variables from Version 4 of the checklist: one yellow, promoting treatment fidelity, and one orange, negating treatment fidelity. [Colour figure can be viewed at wileyonlinelibrary.com]

Total treatment fidelity scores for the role‐play sessions

The total treatment fidelity scores for all recorded role‐play sessions (n = 9) were calculated using checklists Versions 2 and 5 (see Table 1). Participants scored zero for ‘Recording’ and ‘Partial activity’ variables because these were deliberately omitted from the role‐play due to resource constraints. The total possible score for Version 2 is 88 for Version 5 is 68. Overall, total scores using Version 5 were lower than for Version 2 (see percentage scores in Table 1), likely due to the addition of more rigorous checklist variables based on participant feedback, and amendments after trialling. A post hoc paired two‐tailed t test indicates that this difference is statistically significant (p = 0.0307).

TABLE 1.

The total treatment fidelity scores for nine recorded role‐play sessions calculated using Version 2 and Version 5 of the checklist.

Total treatment fidelity scores
Video	1	2	3	4	5	6	7	8	9
Scores using Version 2
Researcher/88	60	62	57	46	42	53	43	33	48
	68.2%	70.5%	64.8%	52.3%	47.7%	60.2%	48.9%	37.5%	54.5%
RA/88				48		59	44		53
				54.5%		67.0%	50.0%		60.2%
Scores using Version 5
Researcher/68	43	46	40	20	36	34	28	21	35
	63.3%	67.6%	58.8%	29.5%	52.9%	50.0%	41.2%	30.9%	51.6%
RA/68				23		38	25		29
				33.8%		55.9%	36.8%		42.6%

Version 2 is scored out of 88. Version 5 is scored out of 68.

Abbreviation: RA, research assistant.

Interrater reliability results

Videos of the role‐play sessions were rated by the first author and one of the RAs in the LIVELY project team. The results from Spearman's rank‐order correlation illustrate correlation coefficients greater than 0.6 for every video, showing strong agreement between the first author and each RA (see Table 2). Ratings for all but videos 4 and 5 using Version 2, have correlation coefficients greater than 0.69, suggesting agreement is very strong. As p < 0.05 in all cases, the results are significant, and it is concluded that there is strong positive correlation between the first author's and the RA's checklist ratings. Therefore, both raters coded checklist variables with the same or similar scores. Correlation coefficients calculated for Version 5 range from 0.776 to 0.928 and are higher than for Version 2, which range from 0.662 to 0.855. This suggests that interrater agreement using Version 5 of the checklist is stronger than with Version 2.

TABLE 2.

Spearman rank‐order correlation coefficients and associated p values calculated to show interrater reliability between the first author and the RAs for four videos using checklist Versions 2 and 5.

	Version 2		Version 5
Video	Correlation coefficient	p value	Correlation coefficient	p value
4	0.686	<0.001	0.824	<0.001
6	0.662	0.001	0.776	<0.001
7	0.714	<0.001	0.928	<0.001
9	0.855	<0.001	0.884	<0.001

Abbreviation: RA, research assistant.

DISCUSSION

The findings from this study support those of existing literature regarding the consideration of numerous variables when promoting treatment fidelity (e.g., Bellg et al., 2004; Kaderavek & Justice, 2010). Using a behaviour change paradigm to structure the collection and analysis of participant feedback ensured that a range of barriers and facilitators to intervention implementation were considered. Findings from this study will be discussed with reference to the three research questions and previous literature. The development of the treatment fidelity checklist and scoring system will be explored, as will clinical implications and future directions for research.

Which aspects of the M‐DLS training session support participants understanding of the M‐DLS, and therefore promote treatment fidelity?

Post‐training discussion highlighted the importance of having appropriate resources and an interactive environment to support participants’ understanding of the M‐DLS intervention. Training is an important activity to increase an individual's ability to carry out an intervention correctly (Gearing et al., 2011; Ory et al., 2002). The importance of training quality, the materials used and their link to treatment fidelity are mentioned less often in the literature (Bellg et al., 2004). Findings from this study indicate that training supported participants’ knowledge and understanding of the M‐DLS. Training quality and materials should contain specific elements to maximise participants’ understanding of the intervention. These include clarification of the manual and session plans, with provision of adequate time to practise the intervention. Although the video observation and role‐play were reported as beneficial, participants highlighted that multiple practice sessions would have improved their ability to deliver the intervention. Participants emphasised the need for observations of more complicated sessions, for example, with groups of children. These adjustments would promote increased treatment fidelity by supporting participants’ knowledge and understanding of the intervention.

Although results from this study highlight that considering the quality of training is important for promoting treatment fidelity, they also reinforce the idea that treatment fidelity must be considered beyond training and into implementation. Participants all received the same training from the same trainer with the same materials, as recommended by Bellg et al. (2004). However, this was not enough to ensure participants would carry out the M‐DLS in the same way, or as intended as set out in the manual. Participant feedback and differing treatment fidelity scores illustrate that participants completed the session in different ways. This could be due to different prior experiences of the participants, alongside different interpretations of explanations and materials. Despite training being an important step in promoting treatment fidelity, training alone is not enough to ensure treatment fidelity (Bellg et al., 2004) but regular checking (audit) is required to ensure clinicians follow the intervention as intended.

After role‐playing the M‐DLS, which aspects of the intervention materials do participants find compromise treatment fidelity?

The way the intervention manual is structured and formatted has an impact on how the participants engage with it. Issues raised in the focus groups after training were discussed in more detail after role‐play sessions. Participants linked unclear text and formatting in the manual to their misunderstandings and different actions. For example, participants did not work on comprehension only when a child was non‐verbal because, for some reason, they did not notice this instruction in the session plan. Whilst it may seem obvious that materials need to be clear and easy to use, participants’ feedback exemplifies how critical this is when promoting treatment fidelity. Some of these issues relate to the time available to read session plans and practice the content during the training and the role‐play sessions. Other issues would have been resolved with greater opportunity for discussion and some guided reading of the manual, for example, drawing participants’ attention to crucial instructions and the use of formatting to differentiate types of instruction. These outcomes informed the broader design of the M‐DLS including the development of a session‐by‐session scoring sheet with simpler, clearer instructions for use in the LIVELY study (McKean et al., 2020).

What amendments should be made to the M‐DLS treatment fidelity checklist to promote treatment fidelity?

Group discussion highlighted key points that caused participants to vary from the intended session plan. For example, participants misinterpreted (or forgot) to use the prompt hierarchy, which is an integral part of the M‐DLS. It was therefore necessary to code for this action in the treatment fidelity checklist to ensure clinicians and researchers would monitor use of the prompt hierarchy in future implementation. Similarly, participants stated that toy arrangement was unclear within the session plan; for example, one participant did not use all the toys stated in the plan, therefore compromising the intended difficulty of the instructions. Consequently, a variable was added to the treatment fidelity check list to account for this. The final checklist (see Supporting information 3) includes variables that code for essential criteria of the M‐DLS and additional variables found to affect its implementation in role‐play, aligning to the reference standard for promoting treatment fidelity (Bellg et al., 2004).

In order to judge the most effective treatment fidelity checklist of all the versions created, treatment fidelity scores were calculated using Version 2 and Version 5 of the checklist for each video. Typically, scores of 50% refer to low fidelity and scores of 80% to 100% represent high treatment fidelity (Noell et al., 2002). Scores using both versions of the checklist were calculated at 70.5% or below, suggesting that the role‐play sessions were delivered with relatively low treatment fidelity. This is supported by participant feedback, which outlines numerous misinterpretations and ambiguities of the way they implemented the M‐DLS plan and used the materials. This suggests that both checklists appropriately measured fidelity of the role‐plays and would therefore be useful in research or clinical evaluation of treatment fidelity. However, although Version 5 had fewer items (a score of 68 available compared to 88 for Version 2), scores with Version 5 were lower by an average of 6.5% compared to Version 2. It is hypothesised that such amendments account for a more comprehensive and rigorous measure of treatment fidelity using Version 5 of the checklist.

Statistical analysis suggests that the amendments made to the checklist improved interrater reliability. Correlation coefficients using Version 5 of the checklist were higher, suggesting that agreement between the raters was stronger when using Version 5 of the checklist compared to Version 2. Whilst this may be due to increased familiarity with the checklist and videos, it also suggests that amendments in Version 5 may have removed some ambiguities experienced when using earlier checklist versions, discussed in the Amendment log (Supporting information 2).

Participant feedback highlighted the need for clear manuals and session plans. These findings informed amendments to the checklist to improve its clarity and to promote ease of use by clinicians within and outside the research team. Moreover, differences interpreting instructions in the manual could lead a researcher or clinician to interpret variables of the treatment fidelity checklist differently. It was, therefore, vital for explicit instructions to accompany the checklist, and for page numbers referencing the M‐DLS manual to be included. This provides users with direct access to clarification in the M‐DLS manual, ensuring their interpretations are in line with the intentions of the M‐DLS.

Reflections on the development of the checklist

This paper describes the construction of the M‐DLS treatment fidelity checklist in detail, considering participants’ (N = 10) feedback, combined with the first author's and associated RA's experiences of trialling the checklist. The development of treatment fidelity checklists in research is rarely documented in replicable detail. For example, to improve a fidelity measure for a motor speech treatment, Hayden et al. (2015) evaluated a checklist against strategies recommended in current literature (Schlosser, 2002). However, their study does not give detail about how and why components of the checklist were selected. Similarly, Conlon et al. (2019) did not expand on the development of the checklist for the Verb Network Strengthening Treatment (VNeST) intervention, stating only that procedures aligned with the intervention and were according to best practice recommendations (Bellg et al., 2004). Yamada et al. (2010) provide a more detailed account; they used content analysis to rate checklist items according to how relevant they were in representing the intervention. However, the method they used to select the initial items lacks detail. This absence of replicable detail leaves it difficult to compare the current study's constructive process to those processes used in other studies. The process reported in this paper was advantageous to checklist development. Combining participant feedback with suggestions from users of the pilot versions of the checklist meant that numerous influences on treatment fidelity were considered. This contributes to improvement of the checklist's accuracy when accounting for the treatment fidelity of the M‐DLS. Moreover, previous studies (e.g., Bellg et al., 2004; Yamada et al., 2010) state that their checklists were devised by the team involved in the study. It is recognised that this is often unavoidable but results from this study suggest that exposing the intervention materials to individuals outside the research team can strengthen the rigour of the checklist by bringing a more impartial overview.

The present study details the development of a novel scoring system that utilised participant feedback at several stages of checklist development. Participants made changes when implementing the session plan that, on their reflection, would negate treatment fidelity. It became clear that whilst it is vital to code for the components of the intervention that are correctly implemented (Hayden et al., 2015), it is also useful to code for variables that are sometimes added by clinicians and that are not in the manual. Therefore, variables were amended in the checklist to account for two types of actions: actions that must be completed to achieve good treatment fidelity and actions that should not be present, yet were sometimes added by participants. In early iterations of the checklist negative wording was used to account for these differences. However, when the checklist was trialled, the negative wording was evidently confusing. As a result, variables that negate treatment fidelity were scored differently to those that promote it and were colour coded to account for this change.

The M‐DLS checklist is scored using a Likert scale similar to other checklists (e.g., Hayden et al., 2015). However, the novel aspect of this scoring system is that it accounts for variables beyond the components of the intervention. This differs from previous research where only intervention components are scored (e.g., Conlon et al., 2019; Hayden et al., 2015). This difference is considered a strength of this study, as by coding for actions that can be added by a clinician, the checklist comprehensively accounts for numerous ways treatment fidelity can be affected. Moreover, the novel scoring system provides information about the reliability of the clinician's implementation and their tendence for therapist drift (Bellg et al., 2004).

Clinical implications

The research–practice gap is well acknowledged in healthcare (e.g., Olswang & Prelock, 2015) and this study goes some way to address it. Training in the content of intervention, practising implementation and reflecting on both are highlighted in this study as ways to support clinicians to implement an intervention as intended. Moreover, results suggest that researchers and practitioners alike must ensure training is in depth and consistent with an intervention's aims. Provision of treatment fidelity checklists by implementation developers as part of a manualised or reported intervention would go some way to support accurate implementation by clinicians through regular audit of practice.

The benefits of reflective practice were particularly highlighted in participant feedback. Encouraging practice sessions, role‐plays and reflection on a routine audit basis within clinical services could be beneficial in promoting the knowledge and competence of clinicians ensuring that, even for familiar interventions, the protocols are adhered to, and the “active ingredients” are delivered accurately. Some studies have already implemented similar recommendations; for example, King‐Sears et al. (2018) include reflection in their five‐step process when designing treatment fidelity procedures, and Conlon et al. (2019), held team discussions after role‐plays of VNeST intervention to support therapy delivery. Incorporating this practice into routine supervision or audit sessions would have the benefit of increasing quality and effectiveness across SLT services (McFaul et al., 2022).

Anecdotally, it is widely accepted that elements of the original DLS intervention (Knowles & Masidlover, 1982) are adapted by clinicians to fit their clients’ needs. This results in significant drift from the original intervention to the extent that only minor elements of DLS are used by some clinicians, for example, the information carrying words concept (Morgan et al., 2013). The use of an updated modified manual, such as the one developed by McKean and Masidlover (2019) for the Lively project (McKean et al., 2020), along with a treatment fidelity checklist would be a valuable tool for clinicians to use as a reminder of DLS specific factors and ‘active ingredients’.

When considering implementation of new, effective interventions, provision by researchers of a treatment fidelity checklist that incorporates the core content and recommended techniques as well as the actions that could negate the effective elements is recommended.

Critical evaluation

Strengths and limitations of the study design were considered. All but two participants were student SLTs. The student SLTs lacked clinical experience which may have contributed to their discussions in interviews and focus groups, particularly surrounding their confidence and need for practice. It would be beneficial to seek input from more experienced clinicians. Participants knew each other prior to their involvement, producing a potential familiarity effect which may have made them more comfortable to submit responses with confidence in interviews, adding to the depth of discussion. Alternatively, this familiarity may have led participants to agree or share assumptions regarding issues in order to feel accepted within the group (Connor, 1994), limiting the extent to which participants challenged each other's viewpoints (Hofmeyer & Scott, 2007).

One limitation to the study was the potential for researcher bias, as the first author was involved in every stage of the study. The researcher discussed checklist amendments and interview results with other members of the team to account for this.

Role‐plays involving three people (playing one SLT and two children) gave participants essential experience to base feedback on. However, the role‐plays did not reflect the size of a typical M‐DLS session, which can involve up to six children. The short time allowed for planning the role‐play session did not reflect time that would be allowed in clinical practice. This will have impacted on implementation. However, the limited exposure to the materials revealed numerous interpretations and ambiguities that were advantageous for checklist development.

Use of the TDF to analyse the participant feedback enabled multiple influences on behaviour to be considered (Mazza et al., 2013; Murphy et al., 2014) maintaining the focus of the study on the research questions. It enabled feedback to be organised in a consistent way; grouped under the TDF domains enabled easy access when developing the checklist and making recommendations.

Future research

The checklist developed in this study would benefit from further assessment of internal and external validity if it were to be included in the intervention manual as an audit of treatment fidelity for use by clinicians. Strong interrater agreement suggests that the checklist can be used in future research to monitor the treatment fidelity of M‐DLS intervention sessions. It would be advantageous to trial the checklist with larger groups of children to account for any differences in the consistency of intervention delivery. The process is easily adaptable for the development of checklists for other interventions. It is recommended that treatment fidelity procedures be considered more widely and in more depth in future evaluation studies.

Conclusions

Treatment fidelity is not always promoted in research; many studies do not account for it or do not discuss it in depth, weakening the outcomes of experimental studies. This study highlights how subtle changes made by clinicians, in addition to the quality and clarity of training and session materials, can affect the implementation of an intervention. It emphasises the importance of amending and clarifying materials to promote treatment fidelity, as well as measuring treatment fidelity in intervention research and clinical practice. In line with previous research, the findings of this study support the need for robust measures to promote treatment fidelity beyond a clear intervention manual. These include the use of treatment fidelity checklists, training, practise (e.g., role‐play) and reflective discussion.

The participant feedback informed the development of a treatment fidelity checklist and a novel scoring system specific to a modified version of the widely used DLS intervention. Unlike previous studies, the checklist accounts for variables beyond the components of the intervention by coding for common implementation errors within the scoring system. The construction of the checklist was reported in detail; consequently, this process can be adjusted to develop treatment fidelity checklists for other interventions.

It is suggested that intervention developers and researchers support clinicians’ effective implementation of their interventions by including treatment fidelity checklists in reports of efficacy and effectiveness studies.

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

Supporting information

Supporting Information

Supporting Information

Supporting Information

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are not available due to privacy restrictions.