Staff Perceptions and Implementation Fidelity of an Autism Spectrum Disorder Care Pathway on a Child/Adolescent General Psychiatric Inpatient Service

Abstract

While youth with autism spectrum disorder (ASD) are psychiatrically hospitalized at high rates, general psychiatric settings are not designed to meet their unique needs. Previous evaluations of an ASD-Care Pathway (ASD-CP) on a general psychiatric unit revealed sustained reductions in crisis interventions (intramuscular medication use, holds/restraints; Cervantes et al. in J Autism Dev Disord 49(8):3173–3180, https://doi.org/10.1007/s1080-3-019-04029-6, 2019; Kuriakose et al. in J Autism Dev Disord 48(12):4082–4089, https://doi.org/10.1007/s1080-3-018-3666-y, 2018). The current study investigated staff perceptions of the ASD-CP (N = 30), and examined rates of ASD-CP implementation fidelity in relation to patient outcomes (N = 28). Staff identified visual communication aids and reward strategies as most helpful. The number of days of reward identification early in the inpatient stay was associated with fewer crisis interventions later in a patient’s stay.

Children with autism spectrum disorder (ASD) are psychiatrically hospitalized at higher rates and for longer lengths of time than children with other psychiatric disorders (Croen et al. 2006; Kalb et al. 2012; Siegel and Gabriels 2014). Due to limited availability of specialized units for youth with ASD and intellectual disability (ID), these children and adolescents are typically admitted to general psychiatric inpatient services (Siegel 2018). Caring for youth with ASD in a general psychiatric inpatient setting is challenging because the inpatient milieu and brief interventions are often inappropriate for patients with significant communication and/ or social challenges. Further, staff members rarely receive the specialized trainings needed to effectively work with this population (Siegel and Gabriels 2014).

To improve care of youth with ASD on a general psychiatric inpatient service, we developed and evaluated an ASD Care Pathway (ASD-CP). Consistent with guidelines established through expert consensus (McGuire et al. 2015), the ASD-CP consists of a four-module staff training, evidence-informed behavioral strategies for use with patients with ASD, and a toolkit containing intervention stimuli to aid ASD-CP implementation (described in detail in the Methods section). Preliminary evaluations in a public hospital in the Northeastern United States indicated that the ASD-CP was associated with initial (18 months, July 2015—December 2016; Kuriakose et al. 2018) and sustained (18 months, January 2017—June 2018; Cervantes et al. 2019) reductions in use of crisis interventions [i.e., intramuscular (IM) medications and holds/restraints] for 4–17 year old youth with ASD. However, these investigations suggested inconsistent fidelity to strategies within the ASD-CP.

Uptake and sustained implementation of evidence-based practices (EBPs) in real-world settings is notoriously complex. The success of integrating EBPs into routine clinical care depends not only on EBP characteristics but also on factors such as staff knowledge/experience, intervention acceptability, treatment adherence, and characteristics of the clinical setting (Aarons et al. 2009). Further complicating integration, these factors are interrelated. For example, treatment adherence is positively associated with staff attitudes toward treatment (Allinder and Oats 1997; Peters-Scheffer et al. 2013; Corrigan et al. 1998; Verschuur et al. 2020). Service setting, as well as training in and experience with the intervention are associated with staff attitudes. Staff at private facilities have more positive attitudes about EBP implementation compared to staff at public agencies (Aarons et al. 2009). Further, particularly relevant to the ASD-CP strategies, Corrigan et al. (1998) found that staff with more experience with and training in behavior therapy had better attitudes regarding implementation of behavioral treatments. Similarly, research suggests that therapist’s reported allegiance to applied behavior analysis (ABA) therapy impacts adaptive behavior outcomes (Klintwall et al. 2012), as do previous knowledge of EBPs for ASD and knowledge of a specific intervention (Suhrheinrich et al. 2019).

Understanding how these factors are related is essential to improving care because continual assessment of fidelity, ongoing supervision, therapist experience, and adherence to treatment intervention have all been linked to treatment outcomes in child and adolescent mental health services (Lange et al. 2017; Novins et al. 2013). Continued efforts to improve the implementation of EBPs must target factors associated with specific EBPs as well as the organizational setting because complications delivering research-based interventions in clinical care settings can lead to ineffectual care and services (Hoagwood et al. 2001; Novins et al. 2013). Information on relevant factors is particularly lacking in the ASD field where there is limited research on the fidelity and acceptability of ASD-specific interventions across the lifespan and existing work tends to focus on treatments delivered through one-on-one or group formats, to younger children, and outside of inpatient hospital settings (i.e., home, school, and outpatient clinical settings; for examples see Chang and Locke 2016; Mandell et al. 2013; Rogers and Vismara 2008). Notably, while many of the strategies used within the ASD-CP are classified as established interventions according to the National Standards Project (National Autism Center 2015), the effectiveness of interventions within inpatient psychiatric settings is not well known.

Much of the research on treatment fidelity in ASD services focuses on evidence-based interventions that are based on ABA principles and have traditionally been designed to address core symptoms. Overall, this literature suggests that with appropriate training and supports, therapists and parents can deliver interventions with adequate integrity and that higher treatment fidelity is associated with improved patient outcomes (e.g., decreased ASD symptoms, increased scores on developmental measures; Lawton and Kasari 2012; Rogers et al. 2012; Strauss et al. 2012). These findings, though were largely based on efficacy studies rather than real-world effectiveness.

Focusing on the treatment of challenging behaviors, an ASD-associated symptom profile often responsible for psychiatric hospitalization, Strauss et al. (2012) found that consistent parent training led to increased parent fidelity to treatment and reduced child behavior problems. Research examining the fidelity of treatments for disruptive behaviors in community settings, however, is lacking (Doehring et al. 2014). Fidelity research in school settings is more abundant. For example, the Strategies for Teaching based on Autism Research (STAR) intervention that was implemented in a classroom setting similarly resourced to the ASD-CP observed low fidelity rates across teachers. However, group outcomes were moderated by treatment fidelity (Mandell et al. 2013). Across the broader child mental health literature fidelity in implementation appears crucial for intervention success in ASD (Lawton and Kasari 2012; Mandell et al. 2013; Rogers et al. 2012; Strauss et al. 2012).

However, data from the evaluations of implementation for interventions used with the most vulnerable ASD samples are scarce, including older children and adolescents with comorbid ID and those receiving care in inpatient settings. Not only do youth served on inpatient units represent those of highest need, inpatient care is extremely resource intensive. Recognizing the essential ingredients of successful inpatient interventions is crucial to improve cost-effectiveness and optimize care for these youth. Therefore, to better understand the factors associated with initiating and sustaining the ASD-CP in a public hospital, as well as to examine the relationship between these factors and patient outcomes, the current study investigated staff perceptions of the tools and strategies within the ASD-CP and rates of ASD-CP implementation fidelity documented in the medical records. The aims of this research were to: (1) identify the ASD-CP components viewed as most helpful by the staff implementing the program; and (2) identify which components are associated with improved patient outcomes.

Methods

Participants

Study participants consisted of two groups of individuals in a psychiatric care system at a public hospital in the Northeastern United States where approximately 75% of patients receive care through Medicaid. The groups were (1) staff from the pediatric psychiatric emergency service and inpatient units implementing the ASD-CP, and (2) youth who received the ASD-CP during first-time inpatient admissions.

Staff

For analyses on documented fidelity to the ASD-CP, staff were primarily full time nursing technicians, the inpatient unit’s direct care staff, because as the patient’s assigned 1:1 aide they complete most of the ASD-CP documentation. The exact identity of the staff is unknown, though, because the ASD-CP schedules do not record who completed the forms. Data on staff attitudes toward ASD-CP components were obtained from interviews with a range of professionals including seven registered nurses (23.33%), six physicians (20%), two psychologists (6.67%), 11 nursing technicians (direct care staff; 36.67%), and four social workers (13.33%). All staff verbally agreed to participate and there were no consequences for refusal to participate in the interviews. To protect confidentiality, no staff sociodemographic data were collected.

Youth Participants

Youth participants for this study were selected from patients hospitalized on a child/adolescent inpatient unit of a public hospital in a metropolitan city from July 2015 to June 2018. Study inclusion criteria were as follows: (1) Discharge diagnosis of ASD made by a child and adolescent psychiatrist; (2) Expressive communication level was defined as nonverbal (i.e., not using words to communicate) or minimally verbal (i.e., exclusively using single words or short phrases); (3) 1:1 staffing deemed necessary by the admitting physician due to high needs (i.e., low levels of adaptive functioning and/or high levels of disruptive behavior); (4) First-time inclusion in the ASD-CP (we excluded patients who had previously received the ASD-CP); (5) Length of stay of seven or more days to allow time for the strategies implemented to produce any effects. Upon assessment by a child and adolescent psychiatrist, youth were admitted through the pediatric psychiatric emergency service [the Children’s Comprehensive Psychiatric Emergency Program (CCPEP)] to the inpatient unit. For the purposes of this intervention, youth are identified for inclusion in the ASD-CP upon admission.

The patient sample included in this study consisted of 28 youth with ASD receiving services through the ASD-CP, ranging from 4–17 years old (M = 11.21 years; SD = 3.54 years); all were male. Approximately 71% had at least one comorbid psychiatric diagnosis, in addition to diagnoses of ASD/ID (Table 1).

Table 1.

Description of sample

Total sample (N=28)
Sex [N (%)]
Male	28 (100%)
Female	0 (0%)
Age in years
Mean (SD)	11.21 (3.54)
Range	4–17
Comorbid diagnoses [N (%)]
0	8 (28.57%)
1	15 (53.57%)
2+	5 (17.86%)

ASD-CP: Training and Program Components

The ASD-CP training consists of four, 45-min staff training modules as well as an implementation toolkit and supply of developmentally appropriate activities for ASD-CP patients. Module 1 consists of psychoeducation on ASD/ ID and reviews foundational best practice principals (e.g., how to gather basic communication and behavioral information about patients using the tip sheet (described in detail in Table 2); how to withdraw attention from negative behaviors and give attention to positive and appropriate behaviors). Modules 2 through 4 include evidence-informed strategies for working with patients with ASD within inpatient settings (McGuire et al. 2015). Strategies are explained by using the acronym PATHWay (i.e., Module 2: Predictability, Activity; Module 3: Total communication, High reward; Module 4: WAY to cope). In addition to strategies provided during the training, staff receive a toolkit of resources in the form of a binder; toolkit components are described in detail in Table 2. Staff utilize the toolkit throughout the training while engaging in interactive activities and role-plays.

Table 2.

Components of the toolkit Adapted from Cervantes et al. (2019)

Component	Description
Tip sheet	One-page assessment completed at admission with parents/guardians
	Gathers information about:
	How the child communicates and understands language
	The types of challenging behaviors the child demonstrates
	Early warning signs that the child may begin to engage in challenging behaviors
	Triggers for challenging behaviors
	Activities that help the child calm down
	Preferred foods, leisure activities, and rewards
	The back of the page is reserved for staff to add information gathered during the child’s stay
Visual supports
Visual schedule	A list and the order of activities that are left “to do” and a list of activities that are “all done” throughout the day
	Activities are those typically present during hospitalization (e.g. breakfast, meds, talk to doctor) and are presented in the form of 1.5″ laminated picture cards
First-then card	Visual support for transitions that indicate what object or activity will follow the previous, typically less preferred activity using the laminated picture cards (e.g., first medication, then motor break)
Coping card	A card showing pictures of numerous calming activities used to prompt the child to engage in a coping strategy when showing early signs of emotional dysregulation
Staff supports
Staff schedule	A breakdown of the day with activity choices and activities of daily living embedded
	Lists the individualized safety goal chosen by the staff, the reward identified by the staff and patient, and the schedule of reinforcement for meeting the safety goal
Activity ideas	A list of developmentally appropriate activities individualized for each patient (e.g., massage mat, Play-Doh)

Five months prior to the implementation of the ASD-CP (i.e., February—June 2015), initial trainings were lead by one of the developers of the ASD-CP. Following implementation, all new staff were trained in the ASD-CP. Approximately four trainings were held each year by one of the developers of the ASD-CP and/or the nurse educator; no additional outside supports were provided. At the time of this study, 367 staff were trained in the ASD-CP.

Procedures and Measures

Staff Perception

A convenience sample of thirty staff members were interviewed using a brief instrument to determine which components of the ASD-CP were preferred, perceived as most helpful, and how useful the tools and strategies of the ASD-CP are overall, as research suggests staff attitudes and perceptions are associated with implementation rates (Allinder and Oats 1997; Corrigan et al. 1998). The interview addressed both ASD-CP tools, which included the visual schedule, first-then card, picture bank for visual schedules, motor break picture bank, staff schedule, activity ideas, coping card, and tip sheet, as well as ASD-CP strategies, which included the following: withdraw attention, establish predictability (e.g., prime youth for upcoming events), activity (e.g., engage youth, offer motor breaks), total communication (e.g., use multiple means for communicating), high reward (e.g., provide preferred rewards contingent on behavior), address the basics (e.g., monitor for pain, hunger, medication side effects), and child choice. Interview responses were restricted to close-ended questions (i.e., “Which three tools/strategies are the most used for patients on the ASD-CP?”) and a 5-point Likert scale for overall utility of the components, where 1 represents not useful and 5 represents extremely useful, to increase feasibility and efficiency in administration of the survey. Following the close-ended questions, staff could provide open-ended feedback about the ASD-CP.

Staff completing the interview included those personnel identified as potentially willing to provide their input on the ASD-CP by the Director of Nursing, Director of Child and Adolescent Psychiatry, and two of the ASD-CP developers. Of the 31 employees identified, 30 completed the interview (96.77%); one identified staff member was no longer working on the unit at the time interviews took place. Interviews were conducted by one of the authors, a staff member who is involved in the oversight of and data collection for the ASD-CP (EO). Importantly, these interviews collected information on strategies that could not be obtained by record review because they did not require documentation (e.g., using simpler language, coping strategies, motor activity). Of note, 13.33% of interview data was re-entered to ensure accuracy and 100% accuracy was achieved.

Documented Fidelity

A retrospective review of 28 patient toolkits (i.e., ASD-CP binders) was conducted to identify the frequency of documented use of various components of the ASD-CP. Of note, 14.29% of binders were pulled and reviewed again to ensure accuracy in data abstraction and 100% accuracy was achieved. The toolkit houses the tip sheet, the daily staff schedule, and the stimuli used for undocumented strategies, such as the first-then board and visual schedule. As mentioned, the staff schedule includes a majority of the documented ASD-CP components. These components are: identification of a reward, identification of an individualized goal, schedule of reinforcement for reward delivery if the goal is met, and indication of whether the youth had earned the reward at each time interval throughout the day. The tip sheet is independent from the staff schedule and serves as a brief assessment tool (nine questions) to enable better care of the child. With these documents, we recorded the following variables for analysis: percent of tip sheet complete, presence of an individualized goal on any staff schedule, presence of an identified reward, number of rewards earned throughout the day (if any), and percentage of days during hospitalization a staff schedule was present.

Impact on Patient Care

To determine which, if any, documented strategies were associated with improved patient outcomes, data including length of stay and number of crisis interventions (i.e., holds, restraints, IM medications) experienced during hospitalization were also abstracted from the medical record. Given the nature of these behavioral strategies and the fact that patients must first come in contact with the contingencies of the intervention program before we can expect their behavior to change, we were particularly interested in the implementation of strategies during the initial phase of hospitalization (days two through six) and the occurrence of crisis interventions in the later phase of hospitalization (day seven and later). These time frames were used to both standardize evaluation across participants and to allow time for the strategies implemented during the initial phase of hospitalization to produce any effects. In order to allow for consistent comparison, day one was not considered part of the initial phase as patients were admitted at differing times on their first day, and therefore, was not a full day for some patients.

Statistical Analyses

Staff Perception

Descriptive statistics were conducted to describe responses on staff interviews regarding their perception of the utility of both the documented and undocumented strategies and tools of the ASD-CP.

Documented Fidelity

Based on preliminary observations from the record review, and as observed previously (Cervantes et al. 2019), the ASD-CP toolkit is implemented inconsistently. Therefore, to further elucidate this trend, descriptive statistics were first calculated to estimate use of primary toolkit components, including completion rates of the tip sheet and proportion of days a staff schedule was present during patient stay. Then, frequency of documented use of strategies within the staff schedule (i.e., reward identification, individualized goal identification, reinforcement schedule identified, receipt of reward per reinforcement schedule) was examined. Notably, because the staff schedule contains a majority of the documented care pathway intervention strategies, all fidelity estimates of staff schedule subcomponents rely heavily on the staff schedule being present. When a staff schedule was not present, it was assumed that none of the subcomponents were implemented.

Impact on Patient Care

Due to the shared variance among the proportion of days the staff schedule was present and the fidelity estimates for strategies within the staff schedule, only the subcomponents of the staff schedule were used in further analyses exploring their impact on patient care. The record review demonstrated that some components were being documented more often and more consistently than others. For example, reporting reward receipt per the schedule of reinforcement was inconsistent in both the frequency of documentation and the manner of documentation. Therefore, this variable was considered invalid and excluded from the analyses. Schedule of reinforcement was also excluded from analyses as it was viewed as clinically unimportant without information regarding receipt of reward, thus leaving the proportion of days a reward and an individualized goal were identified as the two fidelity variables gathered from the staff schedule. Functioning independently from the staff schedule, completion percentage of the tip sheet was also added as a fidelity variable.

To determine if these variables were related to patient care, and given the small sample size and the violation of the assumption of normality, a series of nonparametric Mann—Whitney tests were conducted.

Impact on Crisis Interventions in the Later Phase of Hospitalization

First, to explore the relationship between fidelity and crisis interventions, participants were grouped by presence of crisis interventions in the later phase of hospitalization (i.e., no crisis interventions [N = 23]; one or more crisis interventions [N = 5]). Potential discrepancies between groups were examined in the proportion of days an individualized goal was identified in initial phase of hospitalization, the proportion of days a reward was identified in the initial phase of hospitalization, and the percent of tip sheet completion. Of note, the first crisis intervention in the later phase of hospitalization happened within a five day range for all participants (N = 5).

Impact on Length of Stay

Participants were then grouped by length of stay, dichotomized due to sample size limitations as short and long groups. Short length of stay was defined as under two weeks (< 14 days; N = 11) and long length of stay was defined as two weeks or more (≥14 days; N = 17). Again, differences in the proportion of days an individualized goal was identified in the initial phase of hospitalization, the proportion of days a reward was identified in initial phase of hospitalization, and the percent of tip sheet completion were examined.

The relationships between the available sociodemographics and documented fidelity were also examined through Mann—Whitney (age group) and Kruskal Wallis (number of comorbid conditions) tests and between sociodemographics and outcome indicators (length of stay and presence of crisis interventions in the later phase of hospitalization) through Chi-square tests to account for any confounding effects of age and number of comorbid psychiatric conditions.

Results

Staff Perceptions

The utility of the ASD-CP tools overall received an average rating of 4.17 (SD = 0.75) out of 5 on the Likert scale. Scores for the strategies overall were somewhat higher, with an average rating of 4.33 (SD = 0.66). No staff provided negative appraisals of tool or strategy utility overall; ratings for both ranged from 3 (somewhat useful) to 5 (very useful).

The visual schedule and first-then card were most frequently rated as the most useful, obtaining more than 20% of endorsements each. The staff schedule and coping card were most frequently rated as the least useful tools. For ASD-CP strategies, staff most frequently listed high reward and activity as most useful. Withdraw attention, child choice, and total communication were indicated as the least useful strategies (21.11% each; Table 3).

Table 3.

Top three staff picks of ASD-CP tools and ASD-CP strategies

ASD-CP components ranked most useful
ASD-CP tool	% (N)	ASD-CP strategy	% (N)
Visual schedule	25.56 (23)	High reward	25.56 (23)
First-then card	22.22 (20)	Activity	21.11 (19)
Picture bank	17.78 (16)	Predictability	16.67 (15)
Tip sheet	14.44 (13)	Withdraw attention	11.11 (10)
Activity ideas	7.78 (7)	Child choice	10.00 (9)
Motor break picture bank	6.67 (6)	Total communication	7.78 (7)
Staff schedule	4.44 (4)	Address the basics	7.78 (7)
Coping card	1.11 (1)
ASD-CP components ranked least useful
ASD-CP tool	% (N)	ASD-CP strategy	% (N)
Staff schedule	24.44 (22)	Withdraw attention	21.11 (19)
Coping card	18.89 (17)	Child choice	21.11 (19)
Motor break picture bank	17.78 (16)	Total communication	21.11 (19)
Activity ideas	16.67 (15)	Address the basics	14.44 (13)
Tip sheet	7.78 (7)	Predictability	8.89 (8)
First-then card	7.78 (7)	Activity	6.67 (6)
Visual schedule	3.33 (3)	High reward	6.67 (6)
Picture bank	3.33 (3)

Documented Fidelity

Corresponding to anecdotal observations, the staff schedule was inconsistently present throughout patient stay and inconsistently completed when present. On average, staff schedules were present for just over 60% of a patient’s stay on the ASD-CP, with a median of 73.86%. However, the standard deviation was large (SD = 32.21%) and estimates ranged from 0–100%. Within the staff schedule, an individualized goal was present more frequently than a preferred reward. The average proportion of patient stay where an individualized goal was present was 63%. Again, this varied widely across patients (SD = 29.69%); the median was 75% and the range was 0–95%. When interested in only the initial phase of hospitalization the average was 73.57% (SD = 37.34%; range 0–100%) with a median proportion of days a goal was identified at 100%. Rewards were identified less frequently and its presence averaged approximately 39% of the patient stay. The variability was again wide (SD = 28.55%) and ranged from 0–90%; but, the median remained low (34.83%). When examining the identification of a reward within the initial phase of hospitalization only, the average rose to 47.86% with a higher median (60%). Variability remained large (SD = 33.70%) and ranged from 0 to 100%.

For patients with three or more staff schedules, which denote a 15 min or 30 min schedule of reinforcement, available (i.e., N = 24), approximately 46% were placed on a 15 min fixed interval schedule and remained on that schedule throughout their stay. About 17% were placed on a 30 min fixed interval schedule and remained on that schedule throughout their stay. However, only about 17% of youth reinforcement schedules were thinned from 15 to 30 min intervals during the course of their hospital stay. Schedule of reinforcement changed inconsistently for five patients (20.8%).

Completion rates of the tip sheet differed from those of the staff schedule. While the range of completion remained the same (0–100%), the tip sheets were typically complete or largely complete (Median = 100%; M = 92.46%; SD = 20.07%). Further, as described, the tip sheet leaves an allocated but optional space on the back for staff to write information obtained throughout a patient’s stay. Across the sample, this space was utilized on approximately 64% of patient’s tip sheets. Finally, for those tip sheets with documented dates of completion (N = 20, 71.43%), most were completed within one day of admission (Median = 1; M = 1.60; SD = 1.60) with a range of 0–5. See Table 4.

Table 4.

Descriptives for documented fidelity

(N = 28)			M (SD)	Median	Range
Proportion of days Staff Schedule present across hospital stay (%)			61.33 (32.21)	73.86	0.00–100
Goal Identification	Percentage of individualized goal identified ac ross hospital stay (%)		63.12 (29.69)	75.00	0.00–95.00
	Percentage of individualized goal identified during initial phase (%)		73.57 (37.34)	100.00	0.00–100
Reward Identification	Percentage of reward identified across hospital stay (%)		39.35 (28.55)	34.83	0.00–90.00
	Percentage of reward identified during initial phase (%)		47.86 (33.70)	60.00	0.00–100
(N = 24)			% (N)
Schedule of reinforcement (N = 24)	Held at 15 min intervals throughout stay		45.83 (11)
	Held at 30 min intervals throughout stay		16.67 (4)
	Increased from 15 to 30 min intervals throughout stay		16.67 (4)
	Decreased from 30 to 15 min intervals throughout stay		0.00 (0)
	Inconsistently changed throughout stay		20.83 (5)
(N = 28)	M (SD)	Median			Range
Percent of tip sheet complete (%)	92.46 (20.07)	100			0.00–100
(N = 20)	M (SD)	Median			Range
Days elapsed between admission and tip sheet completion (#)	1.60 (1.60)	1.00			0–5

Impact on Patient Care

Results of the Mann—Whitney and Kruskal—Wallis tests indicated that most relationships between patient sociodemographics (i.e., age group, number of comorbid psychiatric conditions) and fidelity estimates (i.e., reward, tip sheet) were not statistically significant (p > 0.05). However, number of comorbid diagnoses was associated with frequency of goal identification (H[2] = 8.60, p = 0.014). Follow-up Mann—Whitney tests indicated that patients with two or more comorbid diagnoses were less likely to have goals identified in the initial phase of hospitalization than patients with one comorbid diagnosis (U = 7.00, p = 0.005) and patients with no comorbid diagnoses (U = 6.50, p = 0.045), though the latter did not reach statistical significance following Bonferroni correction, where the adjusted significance level was 0.017. Sociodemographics did not differ by length of stay or crisis intervention during the later phase of hospitalization (all p > 0.05).

Impact on Crisis Interventions in the Later Phase of Hospitalization

Examining the association between documented ASD-CP fidelity and the presence of crisis interventions, proportion of days a reward was identified on the staff schedule in the initial phase of hospitalization was significantly higher for youth who did not experience a crisis intervention in the later phase of hospitalization (U = 14.50, p = 0.007, r = − 0.50). However, no significant results were found regarding the proportion of days an individualized goal was identified and tip sheet completion percentage (all p > 0.05; Table 5).

Table 5.

Differences in documented ASD-CP fidelity among youth with and without crisis interventions in the later phase of hospitalization

Total sample (N = 28)	No crisis intervention (N = 23)	At least one crisis intervention (N = 5)	p value	Effect size (r)
Percentage of individualized goal identified during initial phase
M rank	15.33	10.70	.264	–
M	78.26	52.00	–	–
SD	33.53	50.20	–	–
Med	100.00	60.00	–	–
Percentage of reward identified during initial phase
M rank	16.37	5.90	.007*	−0.50
M	55.65	12.00	–	–
SD	30.13	26.83	–	–
Med	60.00	0.00	–	–
Percent of tip sheet complete
M rank	15.07	11.90	.447	–
M	93.24	88.89	–	–
SD	21.12	15.71	–	–
Med	100.00	100.00	–	–

^*p < 0.05

M Rank mean rank, M mean, SD standard deviation, Med. median

Impact on Length of Stay

No significant findings were found between documented fidelity and patient length of stay. Specifically, findings were not significant when examining the relationship between length of stay and proportion of days a reward was identified in the initial phase of hospitalization, percentage of days an individualized goal was identified in the initial phase of hospitalization, or percentage of tip sheet completion (all p > 0.05; Table 6).

Table 6.

Differences in documented ASD-CP fidelity among youth with short and long lengths of hospital stays

Total sample (N = 28)	Short length of stay (N = 11)	Long length of stay (N = 17)	p value
Percentage of individualized goal identified during initial phase
M rank	12.32	15.91	.264
M	67.27	77.65	–
SD	37.17	38.00	–
Med	80.00	100.00	–
Percentage of reward identified during initial phase
M rank	14.23	14.68	.890
M	47.27	48.24	–
SD	32.59	35.40	–
Med	40.00	60.00	–
Percent of tip sheet complete
M rank	14.05	14.79	.817
M	87.88	95.42	–
SD	30.00	9.67	–
Med	100.00	100.00	–

M Rank mean rank, M mean, SD standard deviation, Med. median

Discussion

Overall, staff perceived the ASD-CP tools as useful, giving high average ratings for the utility of tools (4.17, SD = 0.75) and strategies (4.33, SD = 0.66). Staff reported that the visual communication aids (i.e., visual schedule and first-then card) were useful. Interestingly, staff did not find the total communication strategy similarly effective; however, the predictability strategy was rated as somewhat effective (both strategies indicate the need to utilize visual communication tools). One possible explanation of the differing ratings is that staff viewed the ASD-CP tools and strategies as separate entities, and staff may have been utilizing the visual communication tools without their corresponding strategy. Staff may have found the ASD-CP tools more accessible and easier to implement than the ASD-CP strategies. It is also possible that staff did not accurately recall that visual tools are one of the total communication strategies, and may have only considered them one of the predictability strategies. If this assumption is accurate, the need for continued support and booster trainings in the ASD-CP is especially relevant.

In terms of useful strategies, staff rated high reward and activity as most useful. This is promising because both strategies include developmentally appropriate interactions with patients. Specifically, staff were given the means (i.e., tools, strategies, materials) to interact with ASD-CP patients in a developmentally appropriate manner. Previously, the unit did not have specific training or tools for patients with ASD/ID. From additional clinical contact with and observations of the staff following the implementation of the ASD-CP, staff appeared to become more comfortable and knowledgeable about how to work with minimally verbal patients with high adaptive needs and challenging behaviors, which we hypothesize resulted in a positive milieu shift.

Regarding usefulness of the high reward strategy, staff are trained to provide verbal praise, access to a preferred activity, and/or tangible rewards when patients are behaving in an appropriate and safe manner, ideally increasing the likelihood that this behavior continues through positive reinforcement. Of note, in theory, this strategy inherently includes use of the staff schedule (i.e., the schedule identifies patient goals and reinforcement schedule, and allows for documentation of receipt of reward); however, based on staff ratings, the staff schedule was not used as frequently as other tools. As a result, it is likely that staff gave frequent rewards and specific praise, but may have not used the staff schedule to record this activity. This hypothesis is supported by the inconsistent completion of the staff schedule (median = 73.86%, range 0–100%). However, it is promising that staff identified this behavioral strategy as a useful approach, as the use of rewards has been shown to be an effective strategy in reducing problem behaviors (Safran and Oswald 2003). Further, results indicated that only about 17% of patient reinforcement schedules were thinned from 15 to 30 min intervals during the course of their stay. While a failure to thin the frequency of reinforcement would generally indicate that a patient’s behavior had not improved, additional patient-specific (e.g., symptom severity and diagnostic comorbidities) and environmental factors may have contributed to decisions about reinforcement schedules. This also holds implications for generalization of behavioral gains made during the admission, as feasibility to continue a short duration differential reinforcement system in less restrictive settings is low. On the other hand, it is possible that patients’ goals were altered based on success of initial goals or based on clinical need, resulting in the use of the same reinforcement interval. Conversely, staff may have opted to keep the same reinforcement interval in order to avoid altering the status quo of patient management, which may have occurred if reinforcement schedules were thinned. Alternatively, monitoring and assessment to determine if thinning was indicated may not have occurred consistently. While direct care staff have training in behavioral strategies, these individuals are not specifically trained in ABA. Future research should examine the utility and feasibility of more structured supervision, oversight, and analysis of data collected on the ASD-CP.

In addition, results of staff interviews indicated that withdrawing attention and child choice strategies were less useful. These strategies are typically recommended for the ASD population during 1:1 interactions in home, school, and outpatient settings, however they may be more difficult to implement on an inpatient setting in the presence of other patients or, in regard to withdrawing attention, may be discouraged due to the severity of challenging behavior and related concerns about patient and staff safety. The utility of these strategies may also be linked to the amount of time dedicated to teaching and practicing these strategies during the ASD-CP training; it is possible that additional rehearsal of these strategies is needed in order for staff to feel confident with implementation.

While the identification of a goal on the staff schedule and the completion of the tip sheet were not associated with crisis interventions, the frequency of reward identification in the initial phase of hospitalization was associated with fewer crisis interventions in the later phase of hospitalization, with a large effect size (U = 14.50, p = 0.007, r = − 0.50). Specifically, when rewards were more frequently identified in the initial phase of hospitalization, patients were less likely to require a crisis intervention later in their stay. Thus, the identification of rewards appears to be a marker of quality care. In identifying a reward for a patient, staff would need to understand and have knowledge of a patient’s desires and interests, in order to determine what is most rewarding for an individual patient. Identification of rewards, which is information typically gathered on the tip sheet, may lead to the delivery of more personalized care. Similar suggestions, recommending individualization of care, have been made within pediatric settings for patients with ASD (Koski et al. 2016). Taken together, we may interpret that that this behavioral strategy may have reduced the severity of challenging behaviors, leading to a decreased need for crisis interventions. Importantly, this behavior change may have occurred through either antecedent management or the provision of consequences. Rewards are typically tied to positive reinforcement, a consequence strategy, and if implemented consistently after adaptive behavior, this may underlie our findings. However, the identification of a reward may also signal that the staff providing care were more attuned to patient needs and preferences and therefore may have anticipated and avoided environmental conditions likely to evoke challenging behavior more frequently, thus representing antecedent control. Further research is needed to understand the mechanism underlying this effect.

Unsurprisingly, the identification of goals, identification of rewards, or completion of the tip sheet was not associated with length of stay. Consistent with previous findings demonstrating that ASD-CP implementation was not associated with reduced length of stay (Cervantes et al. 2019), the identification of goals and rewards or completion of the tip sheet was also not associated with length of stay. Thus, factors beyond the ASD-CP must be investigated to understand the drivers of efficient and timely discharge of youth with ASD.

This study is not without limitations. This is a small sample of staff and patients. Interviewed staff were selected by administrators and ASD-CP developers based on willingness to complete the survey and, therefore, may have more positive attitudes towards the ASD-CP than staff who were not identified. Additionally, the number of children with ASD on the pathway who interviewed staff had interacted with was not collected so actual experience with the pathway is unknown. Also, all interviews were conducted by only one of the authors (EO). This individual was specifically chosen to conduct interviews due to his familiarity to staff members to increase the likelihood that staff members would respond honestly about their use of ASD-CP tools and strategies. Further, without data on participant functioning (e.g., intelligence quotient [IQ], adaptive functioning, operational definitions of intensity and frequency of presenting problems), it is impossible to fully understand how youth respond on the ASD-CP or how symptom presentation differences influence fidelity rates. This is particularly important in light of the finding that individualized goals were less frequently identified for patients with multiple comorbid diagnoses. While number of diagnoses is a crude measure of psychiatric complexity, it is possible that certain ASD-CP components may be more challenging to carry out with more challenging patients. This study is also limited by the lack of systemized fidelity checks because there is no capacity on the units to capture the frequency of and adherence to ASD-CP strategies. In the absence of this information, staff interviews and ASD-CP permanent product tools (i.e., tip sheet and staff schedule) were used to document fidelity to the treatment protocol and documentation of the use of these tools may be incomplete.

These results suggest that the use of the high reward strategy and identification of a reward early in an inpatient stay may reduce crisis interventions later in a patient’s stay. In the absence of specialized ASD/ID inpatient units, it is essential that research examine the fidelity of real-world inpatient interventions to better serve this vulnerable population.