Implementation Methodology from a Social Systems Informatics and Engineering perspective applied to a parenting training program

Abstract

For implementation of an evidence-based program to be effective, efficient, and equitable across diverse populations, we propose that researchers adopt a systems approach that is often absent in efficacy studies. To this end, we describe how a computer-based monitoring system can support the delivery of New Beginnings Program (NBP) a parent-focused evidence-based prevention program for divorcing parents. We present NBP from a novel systems approach that incorporates social system informatics and engineering, both necessary when utilizing feedback loops, ubiquitous in implementation research and practice. Examples of two methodological challenges are presented: how to monitor implementation, and how to provide feedback by evaluating system-level changes due to implementation. We introduce and relate systems concepts to these two methodologic issues that are at the center of implementation methods. We explore how these system-level feedback loops address effectiveness, efficiency, and equity principles. These key principles are provided for designing an automated, low-burden, low-intrusive measurement system to aid fidelity monitoring and feedback that can be used in practice. As the COVID-19 pandemic now demands fewer face-to-face delivery systems, their replacement with more virtual systems for parent training interventions requires constructing new implementation measurement systems based on social system informatics approaches. These approaches include the automatic monitoring of quality and fidelity in parent training interventions. Finally, we present parallels of producing generalizable and local knowledge bridging systems science and engineering method. This comparison improves our understanding of system-level changes, facilitate a program’s implementation, and produce knowledge for the field.

Introduction

To improve population’s health requires a directed, systematic, and expedited approach to ensure that evidence-based programs (EBPs) are delivered with a high fidelity at scale. Scientists have built most of our research methods around clinical and group-based trials [e.g., the CONSORT Statement (Begg et al., 1996; Brown et al., 2008; Gottfredson et al., 2015; Schulz et al., 2010)], but neither efficacy nor effectiveness trials have focused systematically on issues that are critical for implementation of EBPs in real-world settings. The methods of Implementation Science, which addresses issues concerning adaptation (Chambers & Norton, 2016), scaling up and scaling out (Aarons et al., 2017), and sustaining (Palinkas et al., 2020; Spear et al., 2018) efficacious/effective programs, differ dramatically by explicitly or implicitly addressing complex systems (Berkel et al., 2019; Brown et al., 2017; Czaja et al., 2016; Landsverk et al., 2018; Lich et al., 2013; Palinkas et al., 2011). However, Implementation Science as a field remains in its infancy, and theories and frameworks require further development (Damschroder et al., 2009; Gottfredson et al., 2015; Institute of Medicine & National Research Council, 2009; Nilsen, 2015; Sales, 2018; Skolarus et al., 2017; Smith, 2017; Spoth et al., 2013).

Our main premise is that “social systems informatics” (SSI) can help deliver, monitor, evaluate, and ultimately improve the large-scale implementation of EBPs in community-based organizations. Defining the components of social systems informatics (Brown et al., 2013), the word “social” places an emphasis on interactions with others (e.g., parents and children), “systems” comprise distinct, but interacting components or subsystems as well as loosely connected systems (e.g., service setting and researchers), and “informatics” refers to the science of gathering, storing, processing and using implementation data (e.g., fidelity, quality, attendance, homework). Thus, SSI pertains not only to measurement of interacting components, but also to feedback among them for decision making at several stages of implementation. SSI facilitates the collection of data through low burden, automated, or unobtrusive measures (Webb et al., 1999) involving machine coding of digitized information (Gallo, Pantin, et al., 2015) and analyses from machine learning algorithms (Bickman, 2020; Brown, 2020). At a minimum, the system includes the program itself, its delivery system, and the broader environmental context, each of which affect one another (Chambers et al., 2013; Mabry & Kaplan, 2013). Studying each component separately produces an incomplete picture. By bringing a systems perspective of interacting components into implementation science, it becomes possible to identify how components can work together to achieve successful EBP implementation.

This paper calls for a redesign of how we implement, monitor, and evaluate EBPs as they are delivered at scale. Two SSI innovations are introduced in this paper; the first is a systems approach using feedback loops between Inputs, Processes, Outputs and Outcomes; the second is the use of automated and low burden measurement systems. These approaches are illustrated using the New Beginnings Program (NBP) (Sandler et al., 2016; Wolchik et al., 2007; Wolchik et al., 2002; Wolchik et al., 2013) as well as other prevention programs.

New Beginnings Program

NBP is a manualized, group-format program that targets parenting skills to improve child adjustment following parental divorce (Wolchik et al., 2007). Two efficacy trials conducted in the context of a university-based research center have establised the long-term efficacy of NBP in improving parenting and reducing rates of mental health and substance use diagnoses for children (Wolchik et al., 2013; Wolchik et al., 2000). NBP is cost-effective based on reductions in contact with the criminal justice system and mental health services fifteen years after exposure to the program (Herman et al., 2015).

Each of the 10 program sessions teaches new parenting skills in one of four domains: positive parent-child relationships, communication, positive discipline, and shielding children from interparental conflict. Facilitators coach parents on the use of skills in each session, and parents are given homework to practice the skills with their children between sessions. Previous studies have demonstrated support for NBP’s process theory that home practice is the primary active ingredient leading to change (Berkel, Sandler, et al., 2018), and that fidelity to the content of the program and the quality of facilitators’ process skills are important for ensuring competent home practice (Berkel, Mauricio, et al., 2018). We have also previously described the variability in implementation when NBP was delivered by community agencies as well as challenges involved in feasible and valid methods to monitor and support implementation in community settings (Berkel et al., 2019).

Based on positive results of the efficacy trials, we developed a partnership with four county-level family courts and community-based mental health service agencies to conduct an effectiveness trial of NBP (Sandler et al., 2016). From a public health perspective, the courts are an ideal collaborating agency because they have access to the full population of parents who were married and are divorcing. Because courts do not have the mission or infrastructure to deliver NBP, family-focused agencies represent a second organizational partner. The role of the researchers in this service system is to train providers to implement the program with a high degree of fidelity and quality. Significant challenges in achieving this goal involve developing an effective and efficient system of training and supervision that is feasible in real-world settings and developing measures of implementation that are feasible to collect and predictive of program impact.

Inputs, Processes, Outputs, Outcomes and their Feedback Loops

A system’s view of implementation includes a focus on inputs, processes, outputs, and outcomes that permits a complete start-to-finish perspective on how best to structure, sequence, and measure activities to reach outcomes (Davenport et al., 1996). Figure 1 displays these four elements and provides examples of each that are pertinent to NBP. Implementation inputs include resources available (e.g., Facilitators, Space Allocation, and Funding Stream). Processes include training and supervision as well as program delivery (e.g., NBP session, and NBP group discussion). Outputs include homework activities and parenting skills. Finally, Outcomes include long-term mental health of children (e.g., internalizing and externalizing problems). Feedback loops are used to monitor and direct where system change is needed. For example, if the outcome of participant absenteeism is high, one way to increase the number of clients receiving a full dose of NBP is to revise the process; e.g., expand make-up opportunities for missed NBP sessions. This change could invoke another feedback loop from process to input if make-up sessions require more staffing. Below we describe each feedback loop and provide examples from NBP and other prevention systems in use today.

Figure 1.

Four Feedback Mechanisms involving Inputs, Processes, Outputs, and Outcomes

Types of Feedback Loops and their Functioning

Here we present six types of feedback loops that vary in their focus on effectiveness, efficacy, and equity. The Traditional Feedback Loop in a system involves modifying the Input based on Output, as shown in the topmost loop in the Input-Process-Output diagram in Figure 1 (F1). An example of this feedback loop is using the number of participants reached as Output, and the number of people hired to deliver the EBP as Input. If this Output is too small, we may consider increasing reach by hiring more people to deliver the program. Similarly, equity in delivering health services may be increased by adding resources to recruit in underserved communities.

A second common feedback loop in systems ties the Outcomes back to Process, which we have labeled F2. For example, if attendance toward the end of an EBP is low, we can consider redesigning Processes for client engagement, tracking, reminders, and/or efficiency in the delivery of the intervention.

The feedback loop of F3 Process to Input would lead to immediate revision of resources when the Process is not functioning as expected. For example, delays in analyzing the fidelity of the NBP delivery may be corrected by providing more data system infrastructure. The feedback loop F4 involving self-feedback of Process involves a self-corrective system. This would include recoding of reports so that they are more informative. A Process can be monitored to become more efficient, e.g., faster or more economical in converting Input into Output. As an example, the Community Anti-Drug Coalition of America’s (CADCA) seven strategies for community change (Paine-Andrews et al., 2002) centers on coalition building as a key change process by monitoring and improving the coalition (Yang et al., 2012).

The bottom most loops connect Outcome (e.g., a population prevalence of a condition -- rather than a traditional Output measure) to both Input and Process (F5 and F6). Such feedback loops are inherent in the Substance Abuse and Mental Health Service (SAMHSA) Center for Substance Abuse Prevention’s (CSAP) five-stage, Strategic Prevention Framework (SPF) (Center for Substance Abuse Prevention, 2009) which proposes that decision making about prevention is based on Outcomes, e.g., levels of youth substance abuse in local communities. In this approach, the ultimate test is whether substance abuse decreases in a community. If not, then the entire strategy is examined for improvement, including its Inputs and Processes. Such a system is designed to improve health effectiveness, and SAMHSA’s explicit focus on cultural competence throughout the SPF process emphasizes health equity as well.

These six feedback loops illustrate possible variations in evaluating implementation. Some feedback systems are designed primarily for efficiency, such as the self-monitoring and correcting loop, while others naturally emphasize effectiveness over efficiency such as feedback on low quality of NBP homework leading to increased supervision. It is quite possible to monitor any one or a combination of these feedback loops in practice with a non-randomized design, to test how efficient, effective, and equitable it performs within a specified context.

Innovative Measurement Systems for Implementation

This re-engineered approach to improving output depends on our ability to feasibly and accurately collect data on inputs, processes, and outputs. Standard approaches to collecting data in community settings have been limited to simple process or output measures (e.g., number of participants attending). We will discuss technology-based innovations (e.g., machine learning, computerized adaptive testing) that can support comprehensive data collection necessary to improve delivery and outcomes (Gibbons & deGruy, 2019).

Implementation measurement can be classified in terms of system inputs (e.g., financial and workforce resources provided), processes (e.g., the training sub-system), and output (e.g., quality of program delivery, how well parents completed homework), which are conceptually linked to outcomes. All are important to monitor, but only with feedback loops can the system provide opportunities for improvement. We focus on measuring two dimensions of fidelity to the program, the amount of program content covered and quality or skill with which the content is delivered (Durlak & DuPre, 2008). A provider who reads a manual word-for-word demonstrates high fidelity to content, but low skill (Berkel et al., 2019).

Declines in implementation fidelity are considered a major cause of the observed drop off in participant outcomes when programs go to scale (Chambers et al., 2013; Gallo, Pantin, et al., 2015; Schoenwald et al., 2008). Thus, efficient monitoring and feedback of fidelity is critical for improving implementation. Currently, the fidelity feedback system used in nearly all EBPs has four important limitations: traditional feedback is costly, delayed, static, and based on a limited number of sessions. It is costly because it requires highly trained observers, there is a delay between when a session is delivered and when it is rated, and it is static since it cannot provide real-time feedback. These limitations often make providing fidelity feedback unfeasible for community agencies.

SSI provides methods for conducting implementation assessments, improving the reliability of assessments, and enabling rapid feedback to support sustained, high quality implementation. Recent work on machine learning demonstrates the ability to improve mental health services (Bickman, 2020) and to predict program sustainability with high levels of accuracy relative to human coding (Gallo et al., 2020). Other examples of innovative, automated methods include the recognition of stages of implementation through text mining (Wang et al., 2016), recognition of verbal discordance through image processing of gestures (Inoue et al., 2010; Inoue et al., 2012), measurement of therapeutic alliance through computational linguistics (Gallo, Pantin, et al., 2015), and identification of motivational interviewing through topic modeling (Atkins et al., 2014). These computational methods will become even more essential as we move towards virtual delivery of programs due to COVID-19 pandemic. The recording of materials, dedicated and individualized camera and microphone recordings for each participant will create more digital data exploitable for computational methods.

Moving from human rater to machine rater system to assess fidelity and quality of implementation.

The development of reliable and valid indices of implementation provides the foundation for developing a monitoring and feedback automated system. Methods from computational linguistics can be used for efficient monitoring. For instance, Educational Testing Services (ETS) use advanced computational linguistic methods to automatically score student essays for the GRE, GMAT, and SAT (Burstein et al., 2013; Kakkonen et al., 2005). Linguistic indicators of quality are mined from a database of essays previously scored by human experts. For monitoring implementation, a number of researchers have focused their attention on the assessment of fidelity and quality (Atkins et al., 2014; Brown et al., 2013; Gallo, Pantin, et al., 2015). Our own research group has provided a proof of concept for automatic fidelity coding for a preventive parent training intervention in a bilingual context that targeted drug abuse and sexual risk behavior (Brown et al., 2013; Gallo, Pantin, et al., 2015). Another research team (Atkins et al., 2014) focused on treatment of mental health problems using motivational interviewing. Both approaches extracted linguistic indicators of fidelity at the sentence level (Gallo, Pantin, et al., 2015), and topic identification at the word level (Atkins et al., 2014). Figure 2 shows a conceptual model on how a machine rater can monitor program delivery. There are two types of cues that can be monitored through computational methods: linguistic and non-linguistic cues as shown in Figure 2. As an example of linguistic cues, in Familias Unidas -- another parent-focused intervention (Pantin et al., 2007)-- a computational linguistic method estimates the parent’s level of engagement by tracking the type and number of questions asked during the sessions (Gallo, Pantin, et al., 2015). In that study, the machine learning algorithm achieved high Kappa scores (.83) when compared with human recognition. Non-linguistic cues include the recognition of tone of voice or gestures (Bozkurt et al., 2009; Gallo, Berkel, et al., 2015; Inoue et al., 2012). This less explored source of information offers the opportunity to expand the feedback and monitoring regardless of the language being used to deliver the program.

Figure 2.

Conceptual model on how a machine rater can monitor program delivery

To illustrate an element of the automated system for NBP proposed by Gallo et al. (2016), consider identifying linguistic patterns that are coded as “indicated belief,” -- one of the indicators of quality. Indicated Belief in the parent’s ability of learning the material includes such statements as, “You guys learned this material very well” or “I’m confident you will master this program.” We mined these relevant linguistic cues based on the qualitative coding of the transcripts to automatically assess normalizing in a session. Using positive and negative examples of “Indicated Belief” we trained two machine learning algorithms (Support Vector Machine (Tong & Koller, 2001) and K-nearest neighbor (Sebastiani, 2002)) and achieved a high accuracy of 94% compared to human raters (Gallo et al., 2016). This pilot data suggests that aspects of quality can be tracked by text mining and machine-based methods, in 100% of the sessions, cheaply, and faster than human observation.

Next Generation Implementation of the New Beginnings Program Viewed from a Systems Perspective

We now apply our SSI perspectives to the NBP in three specific areas, (Wolchik et al., 2007). NBP was designed based on a theory that the program will strengthen parenting, which in turn will lead to improved child outcomes. In delivering the program, providers can deliver the content true to the manual (i.e., program fidelity) or leave out manualized material. They may connect with and motivate participants and deliver the program in a clear and coherent way or in a less engaging way. Participants can attend the sessions, engage with providers and other parents, and incorporate what they learn in the program into their daily lives or not (i.e., participant responsiveness). Parents are taught “Home Practice is the Program” and provided parenting skills each week in a group setting. Thus, home practice is an element of responsiveness that is of primary theoretical importance (Wolchik et al., 2007). With fidelity, quality, and participant responsiveness being directly linked to targeted program outcomes (Durlak & DuPre, 2008), this becomes our mediational model (Berkel et al., 2011).

The predictive validity of different measures of implementation was tested with the proximal outcomes. Measures of responsiveness, parent ratings of home practice efficacy and provider ratings of parents’ competence in completing the home practice assignments, predicted improvement in parenting above and beyond attendance (Berkel, Sandler, et al., 2018). Therefore, these two Output measures would be useful measures for monitoring implementation in real-world settings. In addition, independent observers’ ratings of provider’s quality of delivery at a given session predicted each of these responsiveness indicators at the following session, making these ratings important for Process monitoring (Berkel et al., 2015; Berkel et al., 2016; Berkel, Sandler, et al., 2018).

A full implementation monitoring and feedback system would have two types of inputs: in-session information collected during a session and web-based data entry for those that occur outside of program sessions or that are not audio-based (see Figure 3). Providers’ audio signals can be captured and scored for their fidelity to the manual and quality of delivery, then provided as feedback for supervision (Atkins et al., 2014; Gallo, Pantin, et al., 2015). Participants’ audio signals would be captured from the session and scored for their active participation in the session. The data from these sources would provide feedback to both participants and providers to strengthen their effective use of the program.

Figure 3.

Dynamic Implementation Monitoring and Feedback System Using Empirically Supported Indicators of Implementation

Discussion

This paper presents the benefit of using a social system informatics perspective for monitoring and feedback at different stages of implementation, as applied to a parent-focused prevention program for divorced parents. An SSI perspective can improve the delivery of evidence-based programs in communities and service delivery organizations (Atkins et al., 2014; Gallo, Pantin, et al., 2015). At its core, this involves the development and integration of a low-burden measurement system that is actionable and sustainable by the service delivery organization or community.

Social system informatics subsumes principles from systems engineering which aims to provide rapid feedback for problem solving (Figure 4). Systems engineering deviates from the rigid scientific method which currently guides typical effectiveness and efficacy trials. A single feedback loop in the scientific method uses the conclusions (output) to inform the hypothesis being testing (input) in subsequent trials. This delays the system response to new contexts that can make parts of the trial obsolete, say when COVID-19 necessitates virtual sessions. The engineering method, on the other hand, uses multiple feedback loops at several stages of an implementation trial which are used to create a solution, develop a prototype, and test it an interactive fashion.

Figure 4.

Comparison between the scientific and engineering problem solving

In this paper, we proposed a redesign of fundamental components of the NBP to illustrate how this social systems informatics approach would work. We suggested innovative methods for measuring fidelity, quality, and adaptation of an intervention’s delivery that are built on automatic, computer-based, low cost, strategies. For automated measures of responsiveness this included attendance and homework completion. Feedback offers rapid and efficient measurements around key implementation phases. We also noted distinct types of feedback systems involving short and long-term loops. Formal testing of full implementation strategies or smaller implementation components such as feedback subsystems may be accomplished by using randomized implementation trials or non-randomized evaluations.

An automatic computer-based fidelity assessment can potentially be a more reliable than human observers. Computer-based assessments avoid “halo” effects that may occur when a provider does well in quality but not so well on content delivery. Yet, more research is needed to assess the validity of computer-based fidelity methods when compared with human-based assessments. To assess fidelity and quality, we are currently extending computational linguistic methods developed for other parent-training intervention programs (Berkel et al., 2019; Gallo et al., 2016).

In general, the proposed system level and computational methods should be integrated with traditional methods. In this way, we utilize the best skills from human effort and the best skills from automated methods. For instance, a measuring method can be improved when a fast computer-based instrument is used at the first stage to assess the overall implementation fidelity, then followed by a high-quality human-based instrument for stages of implementation that are inherently unable to be operationalized by computer. This two-stage measuring method can lower the burden of local agencies when implementing EBPs. The implementation of parent training programs as well as other EBPs can benefit from these methods to enable efficient, effective and equitable delivery of these programs as they are rolled out and sustained over time.