A mixed-methods process evaluation of Family Navigation implementation for autism spectrum disorder

Abstract

There is growing interest in Family Navigation as an approach to improving access to care for children with autism spectrum disorder, yet little data exist on the implementation of Family Navigation. The aim of this study was to identify potential failures in implementing Family Navigation for children with autism spectrum disorder, using a failure modes and effects analysis. This mixed-methods study was set within a randomized controlled trial testing the effectiveness of Family Navigation in reducing the time from screening to diagnosis and treatment for autism spectrum disorder across three states. Using standard failure modes and effects analysis methodology, experts in Family Navigation for autism spectrum disorder (n = 9) rated potential failures in implementation on a 10-point scale in three categories: likelihood of the failure occurring, likelihood of not detecting the failure, and severity of failure. Ratings were then used to create a risk priority number for each failure. The failure modes and effects analysis detected five areas for potential “high priority” failures in implementation: (1) setting up community-based services, (2) initial family meeting, (3) training, (4) fidelity monitoring, and (5) attending testing appointments. Reasons for failure included families not receptive, scheduling, and insufficient training time. The process with the highest risk profile was “setting up community-based services.” Failure in “attending testing appointment” was rated as the most severe potential failure. A number of potential failures in Family Navigation implementation—along with strategies for mitigation—were identified. These data can guide those working to implement Family Navigation for children with autism spectrum disorder.

Background

Patient Navigation (PN) is an evidence-based approach to improving access to health services by eliminating barriers to care. PN is an intervention rooted in the Chronic Care Model and is designed as a means to reduce racial, ethnic, and socioeconomic disparities in time to diagnosis by shortening the interval between a positive screen (e.g. a mammogram for breast cancer) and definitive diagnosis. PN has been shown to reduce disparities by providing culturally tailored patient education, linkage to community resources, and facilitating appointment scheduling and reminders (Ali-Faisal et al., 2017; Blakely and Dziadosz, 2008; Braun et al., 2012; Freeman, 2006; Marshall et al., 2016; Myers et al., 2015; Rohan et al., 2016). PN has proven effectiveness for a variety of disorders including cancer, HIV, asthma, and mental health disorders (Freund et al., 2014; Gleason et al., 2016; Koester et al., 2014; Myers et al., 2015). Numerous studies and PN programs have recently arisen within a variety of healthcare settings (Association of Maternal & Child Health Programs, 2018; Boston Medical Center, 2016; Family Navigation Project, 2018; Family Navigator Program, 2018; Family Voices Colorado, 2013; Markoulakis et al., 2016, 2018; National Institutes of Health, 2017). While specific standards of care for PN (e.g. what are the core components of the intervention, how is fidelity defined and measured) are currently being developed (Parker et al., 2010), the general principles of PN are consistent across settings. These include the following: navigators share key attributes (culture, language, lived experience) with the population served, navigators are trained in motivational interviewing and/or problem-solving strategies, and navigators work to support engagement in care (Clark et al., 2014; Freund et al., 2014; Gunn et al., 2014; Rohan et al., 2016).

Family Navigation (FN) is a version of PN that involves a focus on the family unit as opposed to an individual patient (Feinberg et al., 2016). FN is a particularly promising intervention for improving access to care for children with autism spectrum disorder (ASD) who experience significant challenges in obtaining diagnoses and accessing both ASD and non-ASD health services (Benevides et al., 2016; Chiri and Warfield, 2012; Mandell et al., 2009; Shattuck and Grosse, 2007). Racial, ethnic, and socioeconomic disparities are also pronounced among children with ASD (Acharya et al., 2017; Angell and Solomon, 2017; Parikh et al., 2018; St Amant et al., 2018; Yingling et al., 2018). Given that FN is an intervention designed to both improve access to care and reduce disparities, it is no wonder that there is growing interest from both the research and clinical ASD community in this intervention (Association of Maternal & Child Health Programs, 2018; Boston Medical Center, 2016; Family Navigation Project, 2018; Family Navigator Program, 2018; Family Voices Colorado, 2013; Markoulakis et al., 2018; National Institutes of Health, 2017).

Despite this growing interest in spreading FN, few data exist on how it can be successfully implemented in clinical practice, and data-driven strategies for dissemination are virtually non-existent (Paskett et al., 2011; Valaitis et al., 2017). Moreover, the definition of FN as a whole, its individual components, and the differentiation between the intervention and strategies for implementation remain ill-defined. Implementation of health systems interventions can be complex, and understanding the details of implementation can be critical to an intervention’s future success (MacDonald et al., 2016). Multiple studies of FN in cancer and HIV demonstrate varying success or diminution of impact of navigation upon implementation in real-world practice (Donelan et al., 2011; Hedlund et al., 2014; Ramachandran et al., 2015). Details of and reasons for implementation challenges are generally unknown, but the lack of concrete data on barriers to implementation is a likely contributor (Hedlund et al., 2014).

Process mapping and analysis is a method of systems evaluation utilized frequently in engineering and business (Damelio, 2011), and utilization of such methods in health and medical services is growing (Abujudeh et al., 2016; Carter et al., 2015; Damelio, 2011; Lins et al., 2018). The goal of process analysis is to improve health outcomes by identifying and reducing system inefficiencies in complex, multi-step interventions. Previous research has found that engaging in process analysis can lead to systems that are more appropriate, effective, and sustainable, with improved health service delivery (Abujudeh et al., 2016; Carter et al., 2015; Lins et al., 2018; Pojasek, 2013).

Therefore, we undertook a process analysis of the implementation of FN in the context of a large, randomized effectiveness trial of FN to reduce time to diagnosis and treatment for children with ASD, using a novel method—a Failure Modes and Effects Analysis (FMEA)—derived from the fields of engineering and operations. Our version of FN was developed to reduce the time from initial screen to definitive diagnosis and access to treatment for children with ASD. While prior FN studies have focused on improving access to services, the FN study evaluated here targets both reduced time to diagnosis and improved access to services. The intervention was tested over multiple pilot phases in multiple settings (Broder-Fingert et al., 2018).

The current FMEA study is the first in a larger, mixed-methods approach based on the Consolidated Framework for Implementation Research (CFIR) to understanding FN implementation (Damschroder et al., 2009). Specifically, this study was designed to explore the “process” domain of CFIR. CFIR was chosen because it provides a comprehensive framework to systematically identify factors from multi-level contexts and is useful in studying processes prospectively (Keith et al., 2017). We focused on the “Reflecting and Evaluating” construct of the process domain, defined as “group and personal reflection … reflecting or debriefing before, during, and after implementation” (CFIR Research Team, 2018). The aims of this study were to (1) delineate steps in the process of FN implementation and (2) understand potential failures in the process and the relative impact of such failures. The goal of this work is to guide investigators, clinical leaders, and policymakers on implementing FN for ASD and ultimately ensure future dissemination to vulnerable populations.

Methods

Study design

We employed a sequential mixed-methods design, using qualitative semi-structured interviews to inform an FMEA that includes both quantitative and qualitative analyses (Johnson and Onwuegbuzie, 2004). Data from the quantitative analysis were given more prominence since it allows for comparison and prioritization of potential failures. The study was approved by the institutional review board at Boston University.

The intervention

In this study, we evaluated a specific version of PN, FN. FN is rooted in the principles of PN and differs only in that it focuses on the entire family unit, as opposed to the specific patient. FN was implemented in this study as a means to reduce the time from an initial positive ASD screen in primary care to diagnostic resolution (completion of a diagnostic assessment) and deployment of services. The intervention is being tested as part of an ongoing National Institute of Mental Health–funded multi-site randomized controlled trial to improve access to diagnostic and treatment services for children at risk of ASD (Broder-Fingert et al., 2018). The study is being carried out in 14 primary care clinics and community health centers across three academic developmental and behavioral clinics in three states: Massachusetts, Connecticut, and Pennsylvania. In this study, children with a confirmed risk for ASD—as determined by a positive ASD screen at an 18- or 24-month primary care well child visit—are recruited from primary care and randomized either to conventional care management or to the FN intervention. The study population includes 350 families from primarily racial and ethnic minority populations (52% Black, 32% Hispanic, 81% publicly insured). Families receiving FN work one-on-one with a navigator who provides off-site support (e.g. home visits, accompanied family to appointments) to overcome logistic barriers related to the diagnostic visit and set-up services following the diagnostic evaluation. Navigators undergo extensive training in motivational interviewing, collaborative decision-making, child development, and social service provision. In addition, they participate in weekly supervisory meetings and have sessions with families recorded and reviewed by a supervisor to monitor fidelity to the intervention. More details about the study population and FN intervention are published elsewhere (Broder-Fingert et al., 2018).

Process map development

Interview participants

We interviewed 12 key stakeholders in FN implementation to develop our process map. Our methods were based on those used in other healthcare settings (Yarmohammadian et al., 2014). Interview participants included Family Navigators (n=3), study coordinators (n=3), research assistants (n = 1), developmental and behavioral pediatricians (DBPs; n=3), and Principals or Co-investigators (n = 2). Participants were all members of the above-noted multi-site FN study funded by the National Institute of Mental Health.

Semi-structured interviews informing process map development

We employed an inductive approach to interviews, using a set of open-ended questions about implementation in general, stages of implementation, and key participants in implementation.

Convenience sampling was used to recruit interview participants; incentives were not provided. All interviews were carried out by either a single research assistant (S.Q.) or the Principal Investigator (S.B-F.) and ranged from 20 to 60 min in length. Questions focused on systems and individual-level factors involved in implementation. Extensive field notes were taken during each interview and were entered into a semi-structured memo template immediately after the interview to allow for the synthesis of key aspects of the data; qualitative software was not used. The research assistant and Principal Investigator subsequently met to review notes and update a draft of the process map. Interviewees were then given an opportunity to comment on the process map. This process was repeated until no new process steps were identified and the process map was finalized (Figure 1).

Figure 1.

Process of implementing Patient Navigation.

FMEA

FMEA is a quality assurance process that is commonly used to address potential flaws and errors during the design stage of a product’s life (Willis, 1992). FMEA is frequently used by manufacturing and medical device investigators and is only now emerging as a tool to evaluate behavioral health interventions (Hosoya et al., 2015; Kawai et al., 1989; Manger et al., 2015; Rienzi et al., 2015; Sorrentino, 2016).

In this study, the FMEA was performed using a process map of FN implementation (methods for creation of the process map are described above) via the following seven steps:

1. The process map was presented at a meeting of nine investigators responsible for FN implementation across 14 clinics in three states. Investigators included health services researchers (n=2), DBPs (n=3), primary care pediatricians (n=2), and study coordinators (n = 2).

2. A single investigator (S.B-F.) presented the process map and described each step identified in the map. Participants in the FMEA were then given 15 min to review the map in detail on their own. The group reconvened and was then given the opportunity to comment or ask questions.

3. FMEA methodology was explained in detail, and examples of other FMEA studies were presented (Ashley et al., 2017; Yousefinezhadi et al., 2016). Investigators were then given three hypothetical examples to practice the FMEA method.

4. Because the process map included 66 steps, it was not feasible (due to time limitations) to perform the FMEA on every step. Specifically, because the FMEA requires rating three elements of each step, if each of the 66 steps had been rated, it would have required each person to perform over 200 ratings across the entire map. Therefore, after the training, each expert was asked to identify potential “high-risk” failures in the process map. Experts were instructed that high-risk failures should be defined as those that would be most consequential to implementation if the failure occurred. It was further explained that consequential failures were those that were most likely to occur, would have the greatest negative impact on implementation if the failure occurred, or would be least likely to detect (i.e. go unnoticed and therefore difficult to find and address). Experts were given a copy of the process map and allowed to indicate confidentially every step in the process they deemed as “high risk.” The experts were allowed to identify as many steps as they wanted as “high risk.” Of the 66 steps in the process, 7 steps emerged as high risk based on the frequency of selection by experts during the identification process.

5. The investigators were then asked to complete a confidential online survey in which they assigned a numerical score for each of the seven steps identified as high risk. For each step, a space for free text was provided to allow investigators to describe and justify each rating. Three indexes were used for each high-risk failure mode per the FMEA protocol: the occurrence rating, the severity rating, and the detectability rating (Institute for Healthcare Improvement, 2017). The occurrence rating corresponded to the likelihood of the failure occurring, the severity rating to the likelihood of severe consequences were the failure to occur, and the detectability rating to the likelihood of not detecting the failure after it occurred. A 10-point scale was used to score each category, with 10 representing an extremely high likelihood.

6. Investigators were also given the opportunity to comment on potential causes and actions to reduce occurrence through free text.

7. Finally, the risk priority number (RPN) was calculated as the product of the three scores: RPN = Occurrence × Severity × Detection and subsequently used to rank the various steps in order of importance. The RPN represents the overall risk assigned to a step, and steps with a higher RPN were ranked above those with lower RPNs. These values are reported in Table 1.

Table 1.

Failure modes, descriptions, mean risk priority numbers, and mitigation actions.

Core process (n = number of stakeholder ratings)a	Description	Potential failure	Occurrence (1–10) Mean (SE)	Detection (1–10) Mean (SE)	Severity (1–10) Mean (SE)	Risk priority number Mean (SE; Range)	Mitigation actions
Setting up community-based services, (n = 9)	Navigators work with community agencies including early intervention, public schools, insurance, and social security administration	Navigator not familiar with resources available Family does not follow-up on recommendations Difficult to access (wait lists, insurance restrictions, limited options)	6.5 (1.1)	5.4 (0.9)	7.6 (0.7)	279.1 (74.1; 45.0, 630.0)	Supervisor develops list of options Navigator counsels family on value of services Develop relationships with community partners
Initial meeting with family (n = 9)	Navigators meet with families after initial positive ASD screen, usually within 7days of initial referral to navigation	Family no-shows, cancels, or never schedules meeting Family does not provide accurate contact information Child(ren) interrupt visit	6.3 (0.7)	3.3 (1.2)	7.9 (0.7)	180.3 (75.1; 20.0, 576.0)	Obtain multiple forms of contact information Call/text/email reminder Demonstrate value (e.g. solving a problem) Navigator presents barriers faced by others
Initial navigator training (n = 18)	3-day intensive training in motivational interviewing, problem-solving, community resources, and child development	Initial training not comprehensive Does not attend training Practice session ineffective model Navigator not qualified	4.2 (0.8)	5.2 (1.0)	7.4 (0.7)	164.4 (50.3; 8.0, 405.0)	Clear expectations Senior navigators mentor Interview candidates with critical thinking skills, home visiting experience, and cultural concordance, as well as relevant experience
Navigator fidelity monitoring and refresher training (n=18)	Navigators have weekly meetings with their supervisor to review cases. Navigator/family sessions are audio-recorded and reviewed by supervisor	Data not collected Training rushed Navigators not committed Inaccurate records Supervision meetings not prioritized	5.1 (0.5)	4.2 (0.7)	5.8 (0.5)	146.1 (35.1; 6.0, 480.0)	Mandatory refresher trainings Teach back More time for training Online trainings Emphasize value of data Dedicated meeting
Attending intake and/or testing appointments with families (n = 9)	Navigators accompany families to their developmental and behavioral pediatrics appointments	Family forgets appointment Transportation fails Childcare falls through for family	6.2 (0.8)	1.7 (0.3)	8.1 (0.6)	86.4 (20.8; 16.0, 200.0)	Reminder calls and texts Assists family with transportation and childcare

ASD: autism spectrum disorder.

^aBecause two groups of categories were collapsed for the final analyses, some processes had more ratings than others.

Quantitative data analysis of FMEA data

Data were analyzed using descriptive analysis in Microsoft Excel. For each failure mode, means and standard errors (SEs) were calculated for the RPN and for each of the three indexes. SAS® 9.4 software was used to perform a one-way analysis of variance, comparing mean RPNs across the core processes.

Qualitative data analysis of FMEA data

Two investigators read through all comments that raters provided explaining reasons for potential failures in the free text portion of the FMEA (S.Q., S.B-F.). Emergent themes were identified through a content analysis approach (Vaismoradi et al., 2013). Because the number of comments was limited, it was unnecessary to formally develop code books. Instead, the team systematically reviewed responses together and discussed key concepts. All comments were then organized into themes by both investigators.

Results

Process map

We identified 66 total steps in the process of implementing FN (Figures 1 to 3). Five categories of individuals were identified as key to implementing FN: patients/families, navigators, DBPs, early intervention providers, and the research staff. The two elements of the process that encompassed the most steps were (1) navigator training (14 steps) (Figure 2) and (2) diagnostic assessment (25 steps) (Figure 3).

Figure 3.

Diagnostic assessment.

Figure 2.

Navigator training.

Quantitative analysis: failure ratings

Completion of the ratings of all seven steps took an average of 26 min (range 14–47 min). The majority of raters completed ratings in a single session (one login into the survey), although two raters completed the survey over multiple days/sessions.

Initially, our expert panel identified seven steps in the process of FN implementation as potential “high-risk” failures, which were then included in the full FMEA analysis (Table 1). Of these steps, four were considered to be conceptually overlapping and were therefore collapsed such that the final FMEA included five steps. These included the following: “setting up community-based services,” “attending intake and/or testing appointment with the family,” “initial meeting with family,” “initial navigator training,” and “navigator fidelity monitoring and refresher training.” Common reasons for failure included FN not being aware of appropriate services, families not being receptive to services, difficulties associated with setting up services, family scheduling conflicts, and insufficient time for training. The step in the process with the highest RPN, representing the overall highest risk assigned to a step, was “setting up community-based services” (mean, SE: 279.1, 74.1). Severity, occurrence, and detection were each rated using a 10-point scale, with 10 representing an extremely high likelihood of severe consequences, a failure occurring, and a failure not being detected, respectively. The step in the process with the highest severity rating was “attending intake and/or testing appointment with the family” (8.1, 0.6). The step in the process with the highest occurrence rating was “initial meeting with family” (6.3, 0.7). The step rated with the highest detection rating was “setting up community-based services” (5.4, 0.9). Differences were significant between the mean detection and severity ratings (ps = 0.02 and 0.02, respectively) across processes, but not the occurrence or total RPN ratings (ps = 0.16 and 0.13).

Qualitative analysis: themes related to failure causes and mitigation

Two specific themes emerged in stakeholder descriptions of potential causes for and actions to reduce the occurrence of each failure (Table 1). These included (1) insufficient monitoring and training and (2) challenges to engaging families.

Theme 1: monitoring and training.

Regarding monitoring and training, stakeholders suggested more training and monitoring would be beneficial. For example, one stake-holder noted, “[Fidelity] recordings are great, but sites could also schedule observed navigation interactions, follow up calls with family by non-navigator staff, etc., to assess fidelity.” Another stakeholder suggested, “Mandatory refresher trainings at least yearly that are scheduled with enough advance scheduling to ensure full participation.”

Theme 2: family engagement.

Regarding family engagement, many stakeholders noted challenges in engaging families with FN, as well as the need for support in engaging some families. One stakeholder noted, “Helping families understand the value of the navigator prior to introduction” would be useful in decreasing this failure. Another suggested, “Navigators could compile and update lists of barriers that many families in the community have experienced in the past and use it as a talking point for their initial interview.”

Discussion

This is the first study to use a novel tool—the FMEA—to evaluate the process of implementing FN for children with ASD. We found the process of implementation to be complex (i.e. 66 steps, five key categories of participants). When assessing potential failures in implementation, we found “setting up/recommending community-based services,” “initial meeting with the patient/family,” “initial FN training,” “fidelity monitoring and refresher training,” and “attending intake and/or testing appointment with the family” to be the most high risk for failure. The failure rated as most severe if it occurred was “attending intake and/or testing appointment with the family.” The failure rated most likely to happen was “initial meeting with the patient/family and the navigator.” The failure rated least likely to be detected was “setting up community-based services.”

The first important finding of this study is the proof of principle. The importance of creating process maps to increase efficiency and reduce error has long been established in industry and manufacturing (Martin and Osterling, 2013). The technique has recently started to gain prominence in the healthcare sector, although primarily in the medical device, procedural, and quality and safety setting (Faerber et al., 2015; Huynh et al., 2016; Terezakis et al., 2011; Walsh et al., 2013). Given the complexity of many implementation strategies, methods for rigorously analyzing each element of an implementation strategy are critical for both investigation and dissemination purposes. Understanding each step in the process of implementation, how they interact, and where they might fail can inform two critical elements of implementation. First, it is integral to understanding what the “active ingredients” of any implementation strategy are. As is true for intervention development, implementation strategies can often be separated into “core components” and an adaptable periphery (Demby et al., 2014). Given the rapid growth of the field of implementation science, new strategies for implementation are continuously being developed and tested (Powell et al., 2015). Establishing methodologies for evaluation are a high priority for the implementation science research community (Chambers and Norton, 2016; Neta et al., 2015; Proctor et al., 2012). This study takes on additional significance when considering that strategies for rigorously measuring or evaluating the “Process” domain of CFIR—the domain we are addressing in this study—are yet to be described (Lewis et al., 2015; Society for Implementation Research Collaboration (SIRC), 2018). Second, the FMEA can be useful in identifying potential failures in implementation. The method used in this study, in which an FMEA was performing in the context of an ongoing trial, is a particularly appealing technique for those looking to speed the pace of implementation from effectiveness trial to “real-world” practice.

The specific findings of our FMEA are also notable. First, implementation of FN is highly complex. Understanding implementation complexity is critical to developing formal implementation strategies for dissemination (May et al., 2016; Pfadenhauer et al., 2017). Given the growing interest in FN—for ASD and other disorders—understanding the complexity of the implementation process is critical (Esparza, 2013; Feinberg et al., 2013, 2014; Freund et al., 2014; Markoulakis et al., 2018; Perkins, 2015). It is likely that implementing FN for other disorders would be equally complex, and understanding the many stages in the process would be valuable for the non-ASD community as well. Using FMEA, we have identified a set of risks to successful FN that may be helpful in the planning and implementation of FN for other disorders or in other settings.

Setting up community-based services was the step with the largest RPN. A well-established challenge in obtaining services for children with ASD is the complexity of the service systems that provide care (Bryson et al., 2008; Klin et al., 2015; Locke et al., 2016; Stahmer et al., 2015; Thomas et al., 2007). Families and children with ASD are required to navigate the health, educational, and community-based service sector. Clearly, the challenges families face in navigating such services also impact implementation of interventions that must also cross several systems (Locke et al., 2016; Wood et al., 2015). Understanding which systems are involved, and how one can best interact with them, may be valuable information for anyone planning to implement FN in a new setting.

Another important finding of this study is that failure of “attending intake and/or testing appointment with the family” was rated the most severe if occurred (mean, SE: 8.1, 0.6) and had a high likelihood of occurring (6.2, 0.8). On the other hand, the likelihood of detection was also rated as higher than other failures (1.7, 0.3), rendering the overall severity less than many other failure modes. It is particularly notable that within the process map as a whole, the diagnostic assessment was the process that included the most steps (25 in total). Whether the core processes (e.g. the number of steps in an intervention) predict the significance, severity, and/or likelihood of detecting failure in implementation is unclear but may be worth further investigation (Chamberlain et al., 2011; Saldana et al., 2012).

Finally, fidelity monitoring and training is a major challenge for many ASDs and behavioral health interventions, and new and improved methods to measure and maintain fidelity are currently under development (Peavy et al., 2014; Sineath et al., 2017; Stahmer et al., 2015; Yates et al., 2013). The results of this study reiterate the value of such efforts, as a failure in fidelity monitoring was rated critical. When analyzing suggestions to ensure that fidelity monitoring and refresher training did not fail, we identified a predominant theme: more continual monitoring and training was preferable. This is a very important finding for anyone proposing to implement FN or for those working to develop implementation strategies (Broder-Fingert et al., 2018).

Limitations

This study has several limitations. Given that this is a new application of the FMEA methodology, there are certain caveats to its interpretation. First, the processes being evaluated at each step are very different, and the upper and lower limits of the RPN vary widely. Although this is expected, it can make interpretation challenging, particularly because there is no published cut-off value for interpreting RPN. According to the Institute for Healthcare Improvement (2017) who developed the FMEA methodology for the healthcare setting, comparisons are made between different steps/core processes only for the purpose of prioritizing which steps are most important to address. Steps with higher RPN scores should be prioritized above processes with lower scores, but mean scores are not meant to be compared between different interventions or across different studies. Therefore, the RPN scores in this study may only be useful to FN and not generalizable to other implementation or intervention efforts.

Another limitation is a lack of data on the validity of the scoring system. Although no data exist on reliability or validity of the FMEA for implementation science, one prior study of medication administration found face validity to be high, but other forms of validity (content validity, criterion validity, construct validity) were low (Shebl et al., 2012). Therefore, we feel this tool is best used within the context of larger, mixed-methods studies, which is how we are using it. Moreover, we are working on a methods paper to further explore the strengths and weaknesses of this methodology.

Finally, FN is a highly complex process, and while this study proposes 66 steps, this may not accurately reflect the full scope of what is necessary for completing FN. However, with increased detail, the usefulness of the process map as an overall picture of FN decreases (Madison, 2005). Similarly, with 66 steps, it was not feasible to analyze each one through FMEA. It is, therefore, possible that there are steps that would have higher RPNs had we been able to fully analyze every step. Future work could delve more fully into specific pieces of the process. In addition, we conducted our FMEA through an online survey, rather than through in-person discussions. This could decrease the consensus around how and why specific steps are likely to fail. Future work could address this limitation by conducting FMEA in-person discussion as a group with all of the key stakeholders present. Another challenge is that temporality (timing of each step) is challenging to depict but may be important in identifying failures that are particularly costly with respect to time.

Finally, while navigators, clinicians, and research staff were involved in our process evaluation, families receiving the intervention were not included. This was because our analysis was focused on the implementation of the intervention rather than how it was received by families. In the future, we hope to work with families on a similar analysis to understand their perspective and assess for differences across the two evaluations.

Future directions

The FMEA identified five high-risk steps in the FN intervention that can be modified to prevent future failures. “Setting up community-based services” could be improved by developing inventories of community resources and agencies. Establishing contacts at key agencies to keep navigators updated on wait lists and availability would provide navigators with helpful information to support families as they navigate service systems. Also, tools (i.e. novel care coordination technology platforms) may be useful. With respect to reducing failures in “attending intake and/or testing appointment with the family,” it may be beneficial to establish a robust reminder system as well as have navigators help create backup transportation and childcare plans for families. Asking parents for at least two alternative contacts (which is now being implemented in our current work) may help in attendance of appointments as well as establishing an initial meeting with the navigator. With respect to navigation, navigator training could be improved by having more frequent training sessions or by employing a mentored navigation approach in which new navigators are paired with experienced navigators to attend visits with families as part of their training. Fidelity monitoring of navigation could be improved by increasing monitoring and incorporating in-person observations or other methods besides recording. Moving forward, the high-risk steps identified in this FMEA will be addressed to improve the FN intervention that was evaluated. In addition, it would be beneficial to consider how failures at these high-risk steps impact clinical, service, and implementation outcomes to inform future strategies for mitigation. Failures associated with poor outcomes may require additional investigation and process improvement.

Conclusion

As the evidence for FN and other evidence-based patient engagement programs continue to grow, delineating and evaluating the process of implementation is critical for future successful dissemination (Perkins, 2015). Moreover, understanding the process, as well as potential failures, is critical for investigators looking to study—and hospital systems looking to implement—this evidence-based intervention. This study provides a new method for evaluating implementation, with a focus on early detection and mitigation of failures in the process. We are currently using these data to refine our implementation strategy and adapt the intervention to meet the needs of the population as well as the context in which it is being carried out.