The case for treatment fidelity in active music interventions: why and how

Abstract

As the volume of studies testing the benefits of active music-making interventions increases exponentially, it is important to document what exactly is happening during music treatment sessions in order to provide evidence for the mechanisms through which music training affects other domains. Thus, to complement systematic and rigorous attention to outcomes of the treatment, we outline four vital components of treatment fidelity and discuss their implementation in non-music– and music-based interventions. We then describe the design of Music Impacting Language Expertise (MILEStone), a new intervention that aims to improve grammar skills in children with specific language impairment by increasing sensitivity to rhythmic structure, which may enhance general temporal processing and sensitivity to syntactic structure. We describe the approach to addressing treatment fidelity in MILEStone adapted from intervention research from other fields, including a behavioral coding system to track instructional episodes and child participation, a treatment manual, activity checklists, provider training and monitoring, a home practice log, and teacher ratings of participant engagement. This approach takes an important first step in modeling a formalized procedure for assessing treatment fidelity in active music-making intervention research, as a means of increasing methodological rigor in support of evidence-based practice in clinical and educational settings.

Music treatment for language outcomes

There is increasing interest in the use of music training and interventions to promote language outcomes in populations of children who have disabilities and who are typically developing. Music therapy approaches in which treatment is implemented by a board-certified music therapist have been explored with the goal of improving language in children with disabilities.^1–3 Researchers in the field of music therapy argue that music affects language as a result of the client–therapist relationship and that music can be understood as an adaptive means of communication.^1,3 This approach has a certain degree of contrast with approaches from cognitive neuroscience, in which neural plasticity and transfer effects of musical skills directly to language through shared neural resources are theorized to drive transfer from music training to language skill. These studies include several pseudo-randomized control trials in children with typical development, where greater improvement in speech/language-related outcomes was demonstrated among children who received musical training (from a music instructor) than among children who received an alternate training, such as art classes or sports.^4–8

Reporting fidelity of interventions

The great diversity in theories of why and how music treatment could facilitate language skills increases the importance of measuring and reporting treatment fidelity, which is crucial in facilitating theory testing.⁹ Treatment fidelity refers to the degree to which implementation of the intervention is controlled, including administration of the treatment’s “active ingredients,” dosage, procedure, and quality.¹⁰ Assessment of treatment fidelity is an often overlooked yet crucial study design element needed to uphold the rigor and reproducibility of each treatment study. Importantly, a high level of treatment fidelity allows the researchers to make conclusions about the mediating variables that are hypothesized to affect study outcomes⁹ and can specifically shed light on how music training is transferred to language skills, whether through the client–therapist relationship, neuroplasticity, or other mechanisms. The literature surrounding more comprehensive methods of measuring treatment fidelity in non-music therapeutic contexts is extensive. Borelli⁹ defined five key fidelity components that should be measured and reported. In this section, we elaborate on four of Borelli’s components of fidelity (see Table 1) and give examples of how these methods can be adopted from language intervention studies in speech-language pathology and clinical psychology.^11,12. While we focus here on the importance of treatment fidelity for assessing the effects of music intervention in children with clinical language disorders, all of the components of treatment fidelity are applicable to research studies on music training conducted in educational settings in children with typical development.

Table 1.

Four components of fidelity

Elements of fidelity	Methods of measuring
Design Key components of treatment	Treatment manual Detailed description of protocol or reference to previously used protocol
Provider training	Report credentials Report training of specific treatment protocol and competency (defined a priori) Report how providers were monitored and received feedback during intervention
Administration Quantity (dosage) of components and quality of adherence to protocol	Protocol checklists Provider self-report Video/audio coding Report reliability
Treatment receipt	Report attendance Report home practice Engagement rating from parents or treatment providers or researchers Exit interview

Element 1: treatment design

Treatment design refers to the practice of defining the active ingredients of the treatment and describing their implementation. A comprehensive treatment manual is a standard of practice, as Yoder and Stone made available in their randomized comparison of communication interventions for children with autism.¹² The treatment protocol should also be reported in detail, as Van Horne et al. described in their recent randomized control trial of grammar therapy.¹¹ Providing a flowchart to describe treatment activities lends itself to interventions in which the subsequent administration of the treatment is dependent on the actions or success of the participants. This can be seen in Van Horne’s and colleagues’ flowchart of prompts for the treatment activities that the treatment providers use during administration of treatment. It is also common to reference a previously published treatment protocol, such as how Yoder and Stone¹² referred to a previous paper for a description of the Responsive Education and Prelinguistic Milieu Teaching intervention design.

Element 2: provider training

Provider training can be fulfilled through reporting who the treatment providers are and how they have been trained on the specific treatment design. Van Horne described the providers’ credentials and elaborated on the types of coaching and resources given to the providers. Similarly, Yoder and Stone gave the credentials and described the training of the treatment providers.¹² If applicable, researchers should also report the competency level the providers achieved in administering the treatment, which requires an assessment of provider skill before the treatment, as well as monitoring and providing feedback to the provider over the course of the intervention.

Element 3: delivery or administration of treatment

Delivery or administration of the treatment can be addressed by answering the following questions. Did the providers adhere to treatment protocol (often referred to as quality of adherence)? What was the dosage of administration of the treatment components (the frequency or intensity with which the active ingredients were administered, which is measured in the average rate of teaching episodes per unit time, the length of the intervention session, and the distribution of episodes over the session)?¹³ Was the dosage consistent across participants and across sessions? To what extent did the treatment include peripheral components not described in treatment design? If there are additional components that are thought to contribute to outcomes, such as amount of practice time at home, then those factors should be quantified and reported.

Methods of measuring administration of treatment can include protocol checklists and provider self-reporting. A frequently used objective method for verification of treatment delivery is video or audio recording the treatment and using behavioral coding to quantify dosage of administration of treatment components and to give feedback to treatment providers on the quality of adherence to the protocol.¹⁰ Van Horne et al.¹¹ monitored administration through videotapes and coded data that was collected on a reported subset of taped sessions. Provider adherence was reported, and dosages of stimuli and recast models were calculated per minute.¹¹ While methods of coding treatment sessions can be time-consuming in a large treatment study, the methodological rigor and information to be gained regarding administration of treatment make it a worthy investment. The coding process can be streamlined by establishing the minimal number of variables (number of sessions rated and raters) needed to obtain sufficiently reliable data through a generalizability study and decision study¹⁴ and by using students interested in interdisciplinary research methods as raters. Yoder and Stone¹² reported that 20% of the sessions were videotaped and coded for quality of adherence to the listed components on a 3-point fidelity rating scale. They also reported the interobserver reliability for the coding of adherence, which is a key step for any researcher utilizing observational coding data.¹⁵ However, videotapes were not used to measure quantity or dosage of treatment administration.

Element 4: treatment receipt

Treatment receipt is an important and often overlooked element of treatment fidelity and refers to how the participants received the treatment.^9,10 It is reasonable to assume that high participant engagement and skill might lead to better outcomes and that not all participants will be equally engaged and competent, even if the treatment was administered in an equal manner to all participants. Van Horne et al.¹¹ addressed treatment receipt by reporting group differences with regard to how many visits each child took to complete the treatment and elaborated on individual cases of children who did not advance during treatment. Yoder and Stone¹² reported parental responsivity as a measure of treatment receipt, because parent involvement was thought to be a key component of the treatment.

Reporting fidelity of music treatment

In this section, we highlight music treatment designs that have addressed one or more of these fidelity elements and discuss areas and methods of fidelity that are particularly vital for providing evidence to support their hypotheses for how music-based interventions improve language outcomes (through the client–therapist relationship, neuroplasticity, or other mechanisms).

The first element is improved consistency in reporting treatment design. As we reported above on the wide variety of underlying theories of change, there are many activities that occur during a music intervention, and researchers should define which components they believe are the active ingredients for improved language skills. Describing the components of treatment within the framework of their hypotheses will strengthen the evidence supporting their theory regarding the mechanisms through which music interventions drive change in language skills. A meta-analysis of the impact of music interventions on reading skills¹⁶ reported commonly used components of music training (for literacy outcomes), such as rhythm, visual representations, and phonology in a music context. Future music interventions could adopt a standardized framework and clarify which components are included in their treatments. In a similar manner, Geretsagger¹⁷ identified common elements of improvisational music therapy (e.g., “scaffolding interaction musically” and “following the child’s lead”) and developed treatment guidelines to address fidelity in future music therapy studies. Habib and colleagues¹⁸ utilized a previously published intervention called “Cognitive-Musical training” as a dyslexia treatment and specifically described key components hypothesized to drive change (e.g., transcoding processes from one motor modality to another). Other ongoing work in the Program for Music, Mind & Society at Vanderbilt University is focused on developing a detailed treatment manual for a group music program for preschool-aged child–parent dyads.

Provider training, the second element of treatment fidelity, is also critical to report in music intervention studies. The provider’s credential or degree and a statement that they have sufficient clinical experience is frequently reported,^{1,4,6,18–20} and it is also often mentioned that they have been trained for the specific intervention protocol.^3,5 Yet for provider training to be fully addressed, training needs to be described in greater detail, such as in the following two interventions. In a study on the effects of music therapy on depression, Erkkila and colleagues²¹ reported that the therapists had 15 months of training and rehearsal sessions and reported the quantity of supervision and feedback that the providers received throughout the study. In a recent randomized control trial by Flaugnacco and colleagues,²² each provider attended the same workshop on the treatment method, the activities and materials were “continuously updated and shared” between providers, and providers were supervised by a neuropsychologist throughout the study. Such descriptions of provider training are crucial for treatment fidelity.

While many studies have defined the treatment components (key ingredients) and discussed the broad format of the intervention (session length and number of sessions per week (e.g., Ref. 18)), there is still inconsistency in reporting the dosage of treatment components and the quality of adherence to treatment design. Careful measurement of the administration of treatment components is required for hypothesis testing of the mechanisms through which music treatment drives change in language skills (i.e., more broadly via the client–therapist relationship or changes driven by elements of music training that directly increase neural plasticity via shared brain networks for music language). Fujioka²³ identified why treatment administration can be so difficult to capture in music interventions: there is an advantage to an ecologically valid method of engaging with music (like improvisational music therapy or Suzuki violin lessons) over a rigid lab-based treatment in which dosage might be easier to capture. However, there are still available methods to quantify the administration of the components of a more naturalistic treatment, such that the dosage of key ingredients (hypothesized to be drivers of change) can be measured for each individual receiving treatment. For instance, Gerry and colleagues¹⁹ suggested that parent-facilitated music listening at home was a key ingredient of change, and thus parent logs were used to measure the dosage of home listening. Habib¹⁸ also addressed the issue of dosage by comparing two levels of general intensity (lessons concentrated over 3 days versus 6 weeks) but did not go further to describe the quantity and distribution of teaching episodes throughout the session. Approaches from music education to determine instructional effectiveness are also relevant, such as observational coding methods developed to measure teaching implementation and student behavior during Suzuki music lessons.^24,25 Through videotapes and a coding schema, Duke and Colprit quantified the proportions of teacher verbalizations, teacher performance, physical positioning, student performance, and student verbalizations.

Erkkila and colleagues²¹ addressed dosage by reporting the average number and standard deviation of improvisations per session. They also addressed quality of administration by calculating the outcome effect across therapists to control for possible differences in administration. Moessler³ utilized videotapes to code behavior using a standardized assessment of the therapeutic relationship. To assess quality of adherence to design, Geretsagger¹⁷ created a 6-point fidelity scale to rate the administration of each common element of improvisational music therapy. However, this method does not measure specific dosage elements received by each participant during sessions. The Music Therapy Rating Scale,²⁶ a video coding scheme to evaluate changes in the client–therapist relationship throughout each session and throughout treatment, is among the resources suggested for measuring the quantity of treatment components (sonorous-musical relationship, nonverbal relationship).^27,28

Treatment receipt is less frequently reported in music treatment literature but is vital in this field owing to the nature of music treatment: how do we measure whether people are engaged in the music activities? Kraus and colleagues²⁹ measured engagement in music classes through a metric capturing both the percentage of attendance and a subjective teacher rating of participation. Results showing that the level of student participation predicted the degree of gains in neural speech processing suggest that the benefit of music interventions might be particularly sensitive to participant receipt of treatment. In a feasibility case study by Tan and Shoemark,³⁰ “acceptability of treatment” included the willingness and enthusiasm of participants to engage in the music-based language treatment. Francois and colleagues⁶ reported that “care was taken to ascertain that both groups were similarly motivated and stimulated,” although no quantitative or qualitative measures were described in these two studies.

Although these methods to address the four components of treatment fidelity have been sparsely used in music treatment studies to date, they are feasible and can provide a wealth of information that will allow clinical–translational scientists and healthcare providers to determine the value of these new interventional techniques and the mechanisms through which the intervention drives change. Here, we report a new system of tracking treatment fidelity in a novel music intervention for children with language impairment as an example of how this approach can be employed.

MILEStone treatment design

Specific language impairment (SLI) is characterized by deficits in receptive and expressive language in the absence of intellectual disability, hearing loss, neurological damage, or developmental disability. Some scholars have recently preferred the term “developmental language disorder” instead of “specific language impairment,”³¹ but the merits of the relative terminology are beyond the scope of this paper. Deficits in grammar are prominent in the areas of syntax and morphology. While there is a strong body of evidence demonstrating effective methods for improving the language skills of preschool-aged children,³² there are currently limited evidence-based options for specific treatment for school-aged children with SLI. Cirrin’s and Gillam’s³³ systematic review found only three treatments that focused on grammar, which is the target of our intervention and the primary impairment in children with SLI. A more recent review found remaining gaps in the evidence for treatments of grammar in school-aged children with language impairments.³⁴

Recent evidence specifically points to relationships between syntactic processing or grammatical skill and rhythm/timing abilities in typically developing children.^35–37 Studies profiling the musical abilities of children with SLI have shown that children with SLI perform more poorly than their typically developing peers on measures of rhythm sensitivity, melodic perception, melodic production, pitch matching, beat perception, and beat production.^37–41 Although it is not yet known whether musical deficits are merely co-morbid to grammatical deficits in SLI or whether they actually cause inefficient language acquisition in SLI, a therapeutic approach that addresses the rhythm deficits in SLI could potentially improve spoken language ability by capitalizing on common underlying mechanisms between the grammar and rhythm domains. Indeed, many groups have proposed the use of musical cues or music training to improve language outcomes in children with SLI.^30,42–44 The theory of change for music affecting language includes the hypothesis that rhythm, social engagement, and other aspects of music training stimulate shared brain networks that can be recruited during language rehabilitation.^45–47 Increasing sensitivity to rhythmic structure may have transfer effects to children’s general temporal processing and sensitivity to syntactic structure. In addition to deficits in language, children with SLI also appear to demonstrate deficits in executive functioning, social–emotional functioning, procedural learning, working memory, and theory of mind.^48–50 It is still undetermined whether the language deficits cause, are caused by, or are comorbid with other cognitive and social–emotional deficits. Thus, treatments for the SLI population may provide maximal benefit by adopting an interactionist approach such as a music training program, stimulating multiple dimensions of language, cognition, and social engagement in an ecological and engaging context.

The approach described here takes a first step toward improved reporting of treatment design and fidelity in music treatment studies that seek to affect language outcomes. Two children with specific language impairment (ages 5;10 and 6;10) participated in the pilot study, which was a part of a larger study examining links between rhythm and grammar skills in children. The study was approved by Vanderbilt’s institutional review board, and all families gave informed consent to participate. The intervention was a musical training program called MILEStone (Music Impacting Language Expertise), a 20-week Suzuki violin lesson program offered to families of children with SLI. The primary theory of change is that instructional episodes to increase rhythm processing skills and sensitivity to rhythmic structure in children will improve language outcomes. See the Appendix (online only) for treatment description.

MILEStone and the four elements of treatment fidelity

Treatment design was addressed in several ways. The intervention generally followed Suzuki Book 1 protocol.⁵¹ A manual was additionally developed that outlined the components of the program and described the key elements of the intervention that are believed to affect outcomes, specifically the administration of activities that may increase rhythm-processing skills.

The intervention also addressed provider training. The instructor had certification to teach Suzuki violin through the Suzuki Association of the Americas. Before the intervention, the instructor met with the researchers to discuss the nature of SLI, potential language barriers, modifications for music instruction, and allowable reinforcement techniques. A researcher who had reviewed videotapes of the sessions met with the instructor three times during the treatment period to provide feedback regarding treatment administration consistency and behavior management strategies, which promoted equal focus, participation, and treatment receipt between participants. To avoid biasing treatment outcomes or modifying the intervention, feedback should be provided based on reviews of the quality and quantity of treatment administration, not based on participant progress in treatment activities or outcomes.

Administration of the treatment was quantified in several ways. Each session throughout the intervention was video recorded from two angles and was utilized for a coding schema to capture teacher and student behavior. The frequency of teaching events was calculated, and implementation across participants and sessions was compared in order to quantify dosage and active ingredients of treatment. An event-based, continuous-recording strategy, which identified the onset of events (instructional episodes) in order, was implemented. Raters used a contingent code for the type of student response to code each episode as an event. Teaching episodes were split into two types: general musical instruction (relating to concepts of rhythm, pitch, and timbre) and postural instruction (relating to how the child holds his/her body or violin). For the purpose of determining the feasibility of the behavioral coding schema for this music treatment, the music teaching episodes were defined and measured broadly; music teaching episodes were not limited to concepts of rhythm (our primary theorized key ingredient) but also included the additional concepts of pitch and timbre. Now that feasibility has been established, future iterations of this study will explore categorization of the codes into rhythm-based teaching events. Postural teaching episodes were also measured, because learning to stand and hold the violin properly is a major component of beginning Suzuki training.

An example of a musical teaching event is as follows: the teacher plays four quarter notes, simultaneously saying the word “alligator,” and asks, "How many notes does 'alligator' have?" An example of an interaction that would not be coded as a musical teaching event is: the teacher says, "Tell me why the violin needs to be held carefully." An example of a postural teaching event is: the teacher says, “Fix your bow hand. Bend your thumb and put it on the bump.” See the Appendix for additional details of the coding schema.

Data for the first six individual violin lessons for each participant are reported in Table 2. Importantly, high stability was found in the number of teaching episodes across participants as well as the proportion of musical to postural teaching episodes. Interobserver reliability was calculated using both point-by-point percent agreement with the primary coder (reported in Table 3) and Pearson’s correlation coefficients. The goals of the initial phase of developing a coding schema are determining feasibility of the schema, training the coders, and establishing reliability. For long-term feasibility of these observational methods with larger numbers of participants, a generalizability study and a decision study may be conducted to determine the minimal number of variables (number of sessions rated and raters) needed to reach sufficiently reliable measurements.

Table 2.

Frequency and rates of events from primary coder

		Type of teaching episode		Child response type
Lessons Participant 1	Total teaching episodes	Music episode (rhythm, pitch, timbre) (out of total)	Posture episode (out of total)	Active response (rate)	Passive response (rate)	Nonparticipatory response (rate)
1	26	19%	81%	85%	15%	0%
2	101	59%	41%	70%	29%	1%
3	48	21%	79%	48%	52%	0%
4	87	33%	67%	67%	33%	0%
5	116	56%	44%	50%	48%	2%
6	115	48%	52%	68%	29%	3%
Average	82.1	39%	61%	65%	34%	1%
Participant 2
1	59	22%	78%	83%	14%	3%
2	122	69%	31%	78%	20%	2%
3	42	48%	52%	62%	26%	12%
4	105	30%	69%	55%	41%	4%
5	98	32%	68%	58%	31%	11%
6	50	36%	64%	54%	36%	10%
Average	79.3	41%	59%	65%	28%	6%

Table 3.

Interobserver reliability: percent agreement with rater 1

	Rater 2	Rater 3	Rater 4
Teaching episode	72%	67%	92%
Teaching type	96%	97%	99%
Response type	95%	96%	97%
False positives	13	11	8

A secondary component of the Suzuki treatment is violin practice at home with the parents. A home practice log was developed as a resource for parents to structure practice time effectively, quantify the amount of time spent practicing at home, comment on the quality of the practice time, and give a numeric rating of child engagement during practice sessions.

The pilot stages of this intervention did not explicitly address quality of adherence to the treatment protocol as a measure of treatment administration, but, in ongoing work, the research team is developing checklists of activities based on the pilot intervention that can be used in the future to rate the quality of administration of each activity.

Treatment receipt was addressed in several different ways. The coding scheme described above also contained an additional type of code: for each teaching episode, raters also created a contingent code for the type of student response to the teaching event. The student’s response to each musical and postural instructional episode was coded as active, passive, or nonparticipatory. An example of an active student response would be if the student verbally responded "four" to “How many notes does ‘alligator’ have?” An example of a passive response would be the student listening to a musical example played by the teacher. An example of a nonparticipatory response is if the student vigorously sawed on the violin with the bow when the teacher asked for a demonstration of rest position. Data of student response can also be seen in Table 2. Both participants had similar rates of active responses, although participant 2 had a higher nonparticipatory rate. After each session, the teacher provided a subjective rating of student participation and attention during the activities. Subjective statements about the quality of engagement during the home practice were also solicited from the parents.

It is not only important to measure dosage of the suspected active ingredients; there are many other components in a naturalistic music treatment that may affect outcomes and should be included in measures of treatment administration. For example, language outcomes in SLI may be affected by components of treatment, including activities that stimulate executive functioning, social–emotional abilities, working memory, and theory of mind. In the case of MILEStone, verbal input from the instructor and verbal output from the student are variables that would be useful to capture during the intervention during future iterations of the study. It is known that incidental learning (gaining knowledge unintentionally) has a powerful effect on children’s language abilities.^52,53 Verbal exchanges between student and instructor during the lessons could be instances of incidental learning of vocabulary and other aspects of language. Estimation of frequency of input is important to capturing incidental learning; thus, it would be useful to develop a coding scheme that captured the quantity and quality of teacher verbalizations or language transactions during music intervention.

Conclusions

To sufficiently address treatment fidelity, four elements should be reported: treatment design, provider training, treatment administration, and treatment receipt. Many traditional measures of fidelity can and should be included in music treatment studies to track implementation of active ingredients. Taking steps to enhance reporting of crucial treatment design facets will increase cross-disciplinary confidence in music treatment as a valid method to improve non-musical outcomes in both clinical and educational settings. It is the authors’ hope that this study will stimulate others who are developing music interventions to provide evidence for the effectiveness of music treatment programs and shed light on the mechanisms through which music treatment programs drive change in language outcomes (i.e., more broadly via client–therapist relationship or elements that directly increase neural plasticity via shared brain networks for music language). When the intensity of administration of treatment components are described and used to show that the treatment is effective, rigorously designed studies can proceed to differential testing based on changes to intensity.^11,12

Supplementary Material

Supp AppendixS1

Appendix. MILEStone treatment description and coding schema