Applying User Input to the Design and Testing of an Electronic Behavioral Health Information System for Wraparound Care Coordination

Abstract

Health information technology (HIT) and care coordination for individuals with complex needs are high priorities for quality improvement in health care. However, there is little empirical guidance about how best to design electronic health record systems and related technologies to facilitate implementation of care coordination models in behavioral health, or how best to apply user input to the design and testing process. In this paper, we describe an iterative development process that incorporated user/stakeholder perspectives at multiple points and resulted in an electronic behavioral health information system (EBHIS) specific to the wraparound care coordination model for youth with serious emotional and behavioral disorders. First, we review foundational HIT research on how EBHIS can enhance efficiency and outcomes of wraparound that was used to inform development. After describing the rationale for and functions of a prototype EBHIS for wraparound, we describe methods and results for a series of six small studies that informed system development across four phases of effort – predevelopment, development, initial user testing, and commercialization – and discuss how these results informed system design and refinement. Finally, we present next steps, challenges to dissemination, and guidance for others aiming to develop specialized behavioral health HIT. The research team's experiences reinforce the opportunity presented by EBHIS to improve care coordination for populations with complex needs, while also pointing to a litany of barriers and challenges to be overcome to implement such technologies.

Among the approximately 15 million young people with mental health conditions, children and adolescents with serious emotional and behavioral disorders (SEBD) are at the greatest risk for negative health and functioning outcomes, including school dropout, drug and alcohol abuse, incarceration, and unemployment (Perou et al., 2013). These youth also consume a disproportionate share of the nation's overall children's mental health care resource. Of $10 billion expended annually by Medicaid on mental health services for children and adolescents, $6 billion goes to treating the 10% of youth with the most serious and complex needs (Pires, Grimes, Allen, Gilmer, & Mahadevan, 2013). A large proportion of these expenses are accounted for by inpatient, residential, and other out-of-home treatment options that may be unnecessary when effective and intensive service options are available in the community (Cooper et al., 2008; Stroul & Friedman, 1996)Tolan & Dodge, 2005)

As early as 2003, the Institute of Medicine formally recognized care coordination as a priority area for quality improvement in health care (Institute of Medicine, 2003). Today, effective, community-based care coordination has emerged as a top health care priority for populations with complex behavioral health needs (Au et al., 2011; McDonald et al., 2007). Because they often receive services from multiple public systems and helpers, children and adolescents with SEBD are a key population for which greater integration and coordination of care is needed (Burns, 2002; Tolan & Dodge, 2005; U.S. Public Health Services, 2001).

Wraparound Care Coordination

In children's behavioral health, the term “wraparound” has historically been used to refer to many concepts and services. For example, using a “wraparound approach” may refer to general philosophy of holistic, family-directed care, while the term “wraparound services” may be used to refer to an array of non-professional community services such as transportation, recreation, and mentoring (Burchard, Bruns, & Burchard, 2002). In recent years, however, Wraparound has been increasingly used to refer to a defined care planning and management process for youth with the most serious and complex behavioral health needs. In the past 15 years, substantial efforts have been undertaken to better specify the wraparound practice model (Burns & Goldman, 1999; Walker & Bruns, 2006), provide more consistent training and implementation supports (Matarese & Walker, 2011), and develop and deploy fidelity measures (Bruns, Burchard, Suter, Leverentz-Brady, & Force, 2004; Bruns, Leverentz-Brady, & Suter, 2008; Bruns, Suter, Force, & Burchard, 2005; Walker & Sanders, 2011). Having undergone this series of specification efforts, the wraparound process is now generally considered the best-defined and most widely implemented care coordination model for youths with SEBD, estimated to serve over 100,000 youths and their families (Bruns, Sather, Pullmann, & Stambaugh, 2011).

Wraparound is an intensive, structured, team-based care coordination process that prioritizes the preferences and perspectives of family and youth throughout the design and implementation of the plan of care (Bruns et al., 2010; Burchard, Bruns, & Burchard, 2002; VanDenBerg & Grealish, 1996; Walker, Bruns, & Penn, 2008). When implemented with adherence to the defined model, wraparound is implemented over the course of four phases. During the engagement phase (several meetings over two weeks), wraparound facilitators use active listening skills to identify family strengths and needs, conduct an initial functional assessment, and develop a crisis and safety plan. During the planning phase (1–2 meetings over two weeks), facilitators identify relevant team members, use a team process to elicit and prioritize family needs, develop creative strategies to meet needs, and relevant indicators of progress. During the implementation phase, facilitators meet regularly (at least once per month) to check in on completion of strategies and services, review and celebrate accomplishments, track progress per identified indicators, make collateral contacts on behalf of the youth and family, and ensure connection to and engagement of natural supports. During the transition phase, facilitators develop an effective transition plan, rehearse responses to potential future crises, and identify future sources of natural support for the family.

Wraparound incorporates a number of basic features of effective care coordination as identified in research and reviews (e.g., Au et al., 2011), including low caseloads that allow care coordinators to engage families and continually identify priorities. Wraparound is also typically facilitated by a unique individual so that roles of helpers are clear and effects of treatment are not diluted. As described in the remainder of this paper, model-adherent wraparound care coordination also incorporates a range of activities that would benefit greatly from the functionality of health information technologies such as personal health records (PHR) and measurement feedback systems (MFS), in that it requires development and implementation of an action plan that spans helpers and systems, and tracking of client progress with modifications as necessary over time. (For more details, see Bruns et al., 2013; Walker & Bruns, 2006; Walker et al., 2008).

Research base

Wraparound care coordination has now been the focus of ten controlled, peer-reviewed studies (see Bruns & Suter, 2010, for a review). A meta-analysis of seven of these found significant effects of wraparound across all five domains examined, including residential placements, mental health outcomes, overall youth functioning, school functioning, and juvenile justice outcomes (Suter & Bruns, 2009). In addition, multiple studies of wraparound care coordination for youths with SEBD have found reductions in overall expenditures, with reductions driven by lower utilization of inpatient, residential, and group care options (e.g., Grimes et al., 2011; Urdapilleta et al., 2011; Yoe, Ryan, & Bruns, 2011). While it is important to note that many of the controlled studies lacked “gold standard” methodological rigor (only three employed random assignment), the wraparound research base is adequately robust for it to be included on an increasing number of inventories of research-based models (e.g., Washington State Institute for Public Policy, 2012).

Similar to many research-based practices, however, wraparound outcomes have been found to be contingent on implementation quality and adherence to the practice model (Bruns, Suter, Force, & Burchard, 2005; Bruns, Suter, & Leverentz-Brady, 2008; Cox, Baker, & Wong, 2010; Effland, Walton, & McIntyre, 2011) and the presence of organizational and systems supports (Bruns, Suter, & Leverentz-Brady, 2006). Research on large-scale wraparound initiatives is now accumulating showing poor outcomes when implementation efforts are characterized by poor adherence to practice standards and lack of program and system supports (Browne, Puente-Duran, Shlonsky, Thabane, & Verticchio, 2014; Bruns et al., 2013). Given the complexity of wraparound implementation, public systems and managed care organizations will benefit greatly from the availability of workforce development and implementation support technologies that can, among other things, streamline the workflow of practitioners (e.g., care coordinators), enhance skillful practice, support data-informed supervision and coaching, and reinforce adherence to the wraparound practice model (Bruns, Pullmann, Sather, Denby Brinson, & Ramey, 2014).

The Current Paper

Federal entities such as the Office of the National Coordinator for Health Information Technology (ONCHIT, 2011), Center for Medicare and Medicaid Services (CMS, 2010), and many others have all stressed the importance of Health Information Technologies (HIT) to modernize and enhance healthcare and maximize outcomes for populations, including children with complex needs. A report from the Agency for Healthcare Research & Quality (McDonald et al., 2007) specifically references the need for implementation supports such as HIT in care coordination. Federal guidance for mechanisms such as Medicaid Health Homes in the Patient Protection and Affordable Care Act (PPACA) also include provisions to encourage appropriate application of HIT (Cohen, 2014). Such guidelines and encouragement are based on the premise that effective HIT can improve client outcomes by managing data about treatment target(s), client progress, and practices employed, and providing feedback to support appropriate clinical decision making (Bickman, 2008; de Beurs et al., 2011; Holzner et al., 2012). Effective HIT is also proposed to improve efficiency of provider staff and reduce duplication of effort when multiple providers are involved in service delivery, such as is the case in care coordination models (McDonald et al., 2007).

Despite this emphasis on HIT, recent studies suggest only about 30% of behavioral health providers have implemented any kind of electronic health record (EHR) technology (Cohen, 2014). Furthermore, although studies of the impact of HIT have yielded predominantly positive results, dissatisfaction with existing EHRs remains a substantial barrier to their uptake (Buntin, Burke, Hoaglin, & Blumenthal, 2011). Given the complexity of providing care coordination models for youth, well-designed HIT could provide a helpful assist to wraparound implementation. However, with the exception of a small number of systems that have been developed for specific public programs (e.g., Hale, 2008), HIT specific to the wraparound practice model is not readily available to the field.

The current paper presents preliminary results of an ongoing effort to mobilize the potential of HIT to support implementation and outcomes of wraparound care coordination initiatives for youth with SEBD and their families and to overcome barriers to HIT use for this widely attempted practice model. First, we will summarize the potential for specific types of HIT, such as electronic behavioral health information systems (EBHIS), to enhance efficiency and outcomes of wraparound care coordination. We go on to describe an effort to develop an EBHIS specific to wraparound, including the functionality of a prototype system, its proposed outcomes, and the links between these. Next, we describe methods and results for a series of six small development studies across four phases of effort – (1) predevelopment, (2) development, (3) initial user testing, and (4) commercialization – and discuss how these results informed development and refinement. Driven by best practices in digital technology design that emphasize continued user input during iterative system development (e.g., Courage & Baxter, 2005), these steps were designed to incorporate user/stakeholder perspectives at multiple points. Finally, we situate our results within the larger HIT literature and present next steps as well as challenges and recommendations for the new system. Our overall goals for the paper are to: (1) present one research team's approach to EHR development and testing that may serve as a model for the broader behavioral health field; (2) provide a progress report regarding a potential important HIT development project specific to youth care coordination; and (3) highlight challenges and potential solutions to development, user testing, and commercialization that may be faced by others engaged in similar projects.

Functions, Outcomes, and Challenges of Behavioral Health Information Technology

Electronic Health Records

An EHR is broadly defined as an electronic collection of comprehensive documentation regarding healthcare provided to an individual (Hoyt & Adler, 2013; Katehakis & Tsiknakis, 2006). EHR systems aim to promote more efficient transfer and sharing of information between providers, staff, and patients/consumers with a range of proposed benefits around the quality and efficiency of documentation (Ford, Menachemi, & Phillips, 2006). While specific functionality of EHR systems may vary, common functions include: (1) entry and management of health information and data; (2) efficient retrieval of clinical data; (3) order management to reduce illegibility errors; (4) decision support using alerts and reminders; (5) electronic communication; (6) patient education; (7) administrative processes to enhance scheduling, claims submission and eligibility verification; and (8) reporting and population health support (Hoyt & Adler, 2013).

Personal health records

The personal health record (PHR) is a subtype of EHR that allows an individual to manage and update his or her own health record. These systems attempt to promote a more proactive role in healthcare maintenance by providing individuals access to view their records stored in one or more providers' EHRs (Tang, Ash, Bates, Overhage, & Sands, 2006). While EHR systems are typically developed by and located within a single provider organization, electronic PHRs are intended to bring health information across multiple providers together to help clients with complex needs to better engage in care, facilitate communication across multiple providers, and create a single record that follows the client across multiple settings. In a randomized study, Druss and colleagues found that individuals with SMI and a comorbid medical condition assigned to the PHR group increased their use of preventive health services and reported improved quality of medical care and engagement in services compared to the usual care group (Druss, Ji, Glick, & Esenwien, 2014).

Measurement feedback systems

Measurement feedback systems (MFS) are another subtype of HIT specifically aimed at improving clinical care through outcome monitoring and feedback of results of monitoring into practice. As described by Bickman (2008), a MFS consists of (1) one or more measures administered regularly throughout treatment to collect information on the process and progress of treatment, and (2) presentation of the information to provide timely and clinically useful feedback to clinicians. The goal of information presentation is to ensure that the practitioner (therapist, care coordinator, team member) can be more responsive to the needs of the client or family by continuing or revising the plan of care. Research on MFS shows they can enhance communication and transparency (Carlier et al., 2010), strengthen client engagement (Shimokawa, Lambert, & Smart, 2010), and improve adult and youth outcomes independent of the specific treatment approach (Bickman, Kelly, Breda, & Andrade, 2011; Lambert et al., 2003). Many proponents of measurement and feedback suggest that considerable variance in behavioral health treatment outcome may be accounted for by the degree to which goals are set and progress monitored and used in the treatment process (Scott & Lewis, 2015; Shimokawa et al., 2010), provided that implementation is of high quality (de Jong, van Sluis, Nugter, Heiser, & Spinhoven, 2012).

The Promise and Challenges of Using EHR to Improve Care Coordination

As described above, the functions offered by EHR, PHR, and MFS hold promise for supporting realization of several attributes of high quality health care, including coordination of care among providers, clinical decision support, shared decision making among the client or patient and a care team, and measurement of outcomes to support accountability and continual improvement (Bipartisan Policy Center, 2012). Research also points to the potential of EHR to support coordination of care for populations with complex needs and multiple helpers who often experience fragmented care, by improving communication and information sharing among disparate providers (Druss et al., 2014; Morrow, 2013).

At the same time, the development of EHR technology intended to support quality of care for any specific population – such as youth with SEBD – will need to overcome a range of commonly identified barriers (McGinn et al., 2011). First, lack of in-house expertise and general knowledge about EHR can result in reluctance to invest in or build an EHR that might serve the needs of the provider(s) and client population. Second, privacy and data security concerns make providers even more reluctant to invest in EHR. Third, despite the intentions of EHR systems, actually sharing information electronically across providers is quite challenging. The ever growing number of HIT products and information formats, and lack of lack of data integration and/or interoperability protocols, only makes this more difficult (Kellermann & Jones, 2013). Fourth, even with the exponentially growing number of HIT products available on the market, most provide a relatively narrow range of functions (i.e., clinical record keeping; outcomes monitoring; enrollment, eligibility determination and billing). Few (if any) are aligned with the specific research-based care coordination or integrated care model being implemented, such as Assertive Community Treatment (ACT; Bond, Drake, Mueser, & Latimer, 2001) for adults with SMI or wraparound for youth with SEBD.

Finally, perhaps the largest barrier to EHR adoption is the up-front costs of investing in these systems (Fleming, Culler, McCorkle, Becker, & Ballard, 2011). In addition to financial investment, provider organizations must also devote substantial management and staff time in training and preparation for uptake, to the point that the financial reward for implementing an EHR may not be clear even with the prospect of improved quality and efficiency. The exclusion of most behavioral health providers from the benefits of the Health Information Technology for Economic and Clinical Health (HITECH) act of 2009, which was intended to help overcome the above barriers and provide offsets to up-front HIT investment costs, further exacerbates this problem (Glasgow, Kaplan, Ockene, Fisher, & Emmons, 2012). For the EHR to live up to its promise of improved coordination, quality, efficiency, and outcomes of care for populations with complex behavioral health needs, provider organizations, managed care organizations, care management entities, and other potential users will not only need to find a system that can execute a wide array of relevant supportive functions, but also be able to overcome this array of obstacles to EHR use.

Predevelopment Phase: Planning an EHR Specific to Wraparound Care Coordination

Recognizing the potential for a well-designed EHR to improve coordination of behavioral health care–particularly for clients with complex needs such as youth with SEBD and their families– and the lack of an existing publically available system – our team, in conjunction with a small HIT organization (Social TecKnowledgy, Inc.) with a long history of behavioral health, social services and education technology development, began predevelopment work on an EBHIS specific to wraparound care coordination in 2011. Initial research was supported by a Phase I Small Business Technology Transfer (STTR) award from the National Institute for Mental Health (R41 MH095516). During this phase of effort, the research and development team (1) reviewed the literature on necessary functions of effective EHR; (2) obtained input from potential users through a formal survey of wraparound providers and experts; and (3) obtained feedback on the functionality and commercialization potential of a prototype of the system via a national webinar. Based on these inputs, the research team developed a theory of proposed action for implementation of a wraparound-specific EBHIS, to inform system development and subsequent research and evaluation. Below we review each of these sources of information.

Literature Review: Core EHR Functions

Having received input that a comprehensive EBHIS was needed, the development team sought to ensure that core EHR functionality would be reflected in the system. From sources such as the Institute of Medicine's (2003) report on EHR and leading researchers' summaries of desirable EHR functionality (e.g., Ambinder, 2005; Hoyt & Adler, 2013), the following functions were prioritized:

Communication and teamwork

First and foremost, an EBHIS to support coordination of care must include methods for disparate types of helpers (the care coordinator; his or her supervisor; providers; representatives from schools, child welfare, and other involved systems; natural supports on the wraparound team) and the youth/family to communicate. This means the EBHIS must be capable of providing all team members access to the wraparound plan of care, and its component strategies, services, and action steps. It must also facilitate communication via functions such as secure messaging, upload of relevant documents, and scheduling support.

Workflow and documentation

EHR holds the potential for improving efficiency, reducing errors, and improving documentation completion and compliance by key practitioners, such as care coordinators. To do so, the wraparound-specific EBHIS must be capable of replacing paper-based record-keeping of core wraparound documentation, such as basic youth and family information, strengths inventories, standardized assessment data, team member contact information, the plan of care and crisis plan (including all relevant strategies in each), team meeting notes, and youth-specific goals and outcomes. Providing electronic structures for entering and maintaining such documentation also supports workflow efficiency, such as when information from one domain (e.g., youth and family information) can “auto populate” required documents such as treatment plans.

Billing and expenses

In addition to recording strategies and services in the plan of care, a truly comprehensive EBHIS for care coordination will have the capacity to maintain an up-to-date inventory of all service providers (e.g., therapists, respite providers, family support organizations, behavioral support specialists) and rates for their relevant billable services. The system will then have the capacity to allow care coordinators to authorize (or obtain authorization for) such services, and for providers to bill for services provided. The system will also be able to provide reports on expenditures by youth, by care coordinator, and the program as a whole.

Measurement and feedback

Despite being a core component of the practice model, collection and feedback of objective data is rarely achieved in wraparound practice (Bruns, Burchard, Suter, Leverentz-Brady, & Force, 2004; Bruns et al., 2008; U.S. Surgeon General, 2001). Thus, core functionality is needed that promotes collection and management of data on services and progress, and feedback of information via facilitator, supervisor, and manager dashboards as well as aggregate reporting.

Intervention support

In addition to storing and managing key information, an EBHIS for wraparound care coordination ideally will have the capacity to support effective intervention, such as through reminders and clinical alerts based on standardized assessment data. Other methods for promoting effective treatment include information in the provider inventory that facilitates selection of services most likely to be effective, such as family satisfaction data and whether the provider uses specific evidence-based practices. In addition, the EBHIS could support the key health care goal of client (youth and family) choice and personalization by including links to additional information on the treatments and strategies included in the family's plan of care. The EBHIS should also have the capacity to generate reports on care coordinator progress completing necessary steps of the process in timely fashion. Ideally, it will also regularly issue requests for feedback from the family on satisfaction and fidelity.

Standards for EHR

Even if all the above functions are present, to be effective the EBHIS must have basic features such as secure remote access, robust backup systems, and security and privacy compliance with HIPAA standards.

Input from the Field: National Stakeholder Survey

Methods

Eighty-one advisors of the National Wraparound Initiative (NWI) were sent a survey inquiring about the type of data elements to include in a MFS specific to wraparound. NWI advisors included representatives of wraparound provider organizations (n=32), national, state, and local behavioral health officials (n=14), researchers and evaluators (n=11), family and youth advocates (n=10), national trainers and technical assistance providers (n=9), and others (n=5). Respondents were asked to rate the level of priority of 11 potential types of process (e.g., wraparound fidelity, stage of implementation) and outcome (e.g., school attendance, behavior) data elements on a scale ranging from 0 (lowest priority) to 4 (highest priority).

Results

The survey was completed by 46 advisors, who were representative of the composition of the overall NWI advisory membership. Results are summarized in Table 1. Among the 11 potential process and outcome domains, six data elements received ratings of 3 (“high” priority) or 4 (“highest” priority) from over 50% of respondents. The elements prioritized by potential users were observed to align with the wraparound theory of change and prior implementation research and were prioritized for inclusion in the system.

Table 1.

Results of user survey on process and outcome data elements to include in an electronic behavioral health system for wraparound care coordination

	Level of priority
	High	Highest	High or Highest
Family Support & Connectedness	40%	37%	77%
Progress toward Priority Needs	41%	27%	68%
Specific Priority Needs	46%	21%	67%
Functioning outcomes	31%	32%	63%
Team fidelity assessment	39%	22%	61%
Family satisfaction	37%	22%	59%
Strengths	37%	13%	50%
Plan of care components	41%	9%	50%
Emotional/behavioral functioning	28%	13%	41%
Risk indicators	21%	19%	40%
Stage of implementation	27%	9%	36%

Input from the Field: Feedback on Webinar

Having received federal development funding and partnered with a small software development company in development, the team sought basic input on the functionality and commercialization potential for a new, wraparound-specific EBHIS. On October 22, 2012, we previewed a prototype variant of Social TecKnowledgy's web-based Team Management System (TMS) software, hereafter referred to as TMS-WrapLogic, via a gotomeeting webinar that included within-presentation surveying capacity.

Method

The webinar had 175 registered attendees. In an initial survey question, attendees indicated whether they were wraparound providers (52%), public system representatives (20%), evaluators or IT specialists (n=16%), trainers or technical assistance providers (9%), or youth or family members/advocates (2%). Three surveys were launched during the webinar. The first asked “what data elements does your wraparound or system of care initiative's IT system manage?” (check all that apply). The second asked, “What aspects of the system seem most appealing or important?” (check all that apply). Finally, a third item asked “What do you think would be your initiative's interest in the system?” (choose the best answer).

Results

Although gotomeeting does not provide response rates on survey items, it is assumed the majority of webinar attendees participated. On the item focused on functions of existing IT systems, a majority of respondents (52%) reported that their IT system manages “service elements such as plans, strategies, services, and costs.” Nearly as many reported that their existing system managed “outcomes data such as placements and school functioning” (48%) and “process data such as fidelity and satisfaction” (45%). Perhaps most interestingly, 38% chose the response option “we don't have an IT system that manages elements such as these.” Regarding the most important/appealing functions of the TMS-WrapLogic system, the most frequently endorsed response option was “Produce high-level reports on outcomes” (71%). Sixty-one percent endorsed the options “facilitate data-driven supervision” and “flexibility and customization to my organization/system.” Relatively fewer endorsed “interoperate with existing data systems” (40%) and “serve as an electronic health record” (39%).

Finally, regarding attendees' level of interest in the system, 82% reported it would be “High,” with 48% reporting “we do not have a wraparound-specific IT system,” 25% reporting “looks like an improvement on our current system,” and 9% reporting “TMS could add needed functions to our current system.” Eighteen percent reported low interest, because “we are not in a position to consider a new IT system” (11%) or “we have an IT system that meets our needs” (7%).

Development Phase: Designing TMS-WrapLogic

From 2012–2014, the university-based research team and software developer worked with a range of internal and external testers nationwide to iteratively develop and refine an EBHIS that included the core functions of an EBHIS as described in the literature, as well as prioritized by potential users. Below and in screen shots, we provide a summary of how TMS-WrapLogic was designed to align with the defined wraparound practice model, as well as how it aims to achieve each of the five core EHR functions reviewed above.

Alignment with the Practice Model and Intervention Support

TMS-WrapLogic maintains information on all elements of the wraparound team and plan of care (POC) in formats that align with the research-based practice model and theory of change (Walker & Matarese, 2011). The POC Section of TMS-WrapLogic (See Figure 2) is organized via tabs that correspond to the sequence of activities that engage the family and build a plan that will be the focus of coordinated wraparound teamwork. Within the “Needs, Outcomes, and Strategies” tab, TMS-WrapLogic helps guide the care coordinator through a process of working with the team to implement elements of the wraparound process that have been found to be associated with strengthened engagement and positive outcomes (Bruns et al., 2008), such as developing a team mission statement and focusing on needs statements in the family's own words. Each needs statement is connected to specific strategies, as well as one or more outcomes statements, data on which must be entered over time. If a Strategy is a Billable Service, the care coordinator can enter the service, service provider information, and units authorized, via a link from this page as well (See Figure 3).

Figure 2.

Entering and editing billable service information within a youth's Plan of Care in TMS-WrapLogic.

Figure 3.

TMS-WrapLogic Workflow Pane.

Communication and Teamwork

TMS-WrapLogic allows administrators to set user permission levels for their wraparound initiative that facilitate secure and appropriate sharing of information, including youth and families, providers and other team members, care coordinators, and upward further to supervisors and managers and higher level administrators (who can access individual or aggregate information on services, expenditures, and outcomes across multiple providers). A “Report and Form Builder” facilitates building of dashboard and report structures via an intuitive process of “joining” requisite variables into a data view. Data summaries can be presented in “real time,” or to run in the “background” of TMS-WrapLogic, producing reports at specified times (e.g., quarterly). Reports can also be set to automatically be emailed to relevant stakeholders on specific dates.

To ensure that data are actually used to promote decision-making, custom Dashboards are displayed at system launch, presenting information that is most critical and actionable to the user's role. For example, a youth or family's dashboard might present information about the next team meeting, upcoming appointments, plan of care elements, and the team's ratings of progress over time. A care coordinator's dashboard may present Reminders, current residential placement of youth on her/his caseload, and/or most recent outcomes assessments for all youth. A supervisor's dashboard may present mean length of wraparound enrollment and expenditures per youth by care coordinator.

Managing Workflow and Documentation

TMS-WrapLogic includes a Work Flow function (see Figure 4) that presents a customizable Task List that tracks the completion of necessary care coordination steps and tasks as well as completion of the Youth Record and POC. The Work Flow window is accessible from any page and lists Tasks and relevant information for a specific youth. Each Task has a hyperlink to the location within TMS-WrapLogic where the care coordinator can complete the Task. TMS-WrapLogic also displays upcoming meetings or deadlines (e.g., updated assessment or POC) on the user's home page. Reminders can also be sent through email to users. TMS-WrapLogic also promotes efficiency for practitioners via system-generated emails that obtain electronic signatures. This allows all team members to acknowledge receipt of, and agreement with, documents such as the POC or crisis plan. TMS-WrapLogic also has functionality allowing secure uploading of documents such as mental health assessments or Individual Education Plans, as is expected of EHR systems that support coordination of care across multiple helpers and systems.

Figure 4.

Example of CANS data report for a wraparound-enrolled youth.

Billing, Expenses, and Service and Provider Resources

Linking practice to billing is a central administrative function of entities that house wraparound initiatives such as care management entities (CMEs, Center for Health Care Strategies, 2011), managed care organizations, and Medicaid Health Homes. Within TMS-WrapLogic, Administrators can prepopulate services by funding source, service categories, service codes, and authorized and credentialed service providers. When a billable strategy is identified, the user can get authorization for a set number of units and even send a contract for services to the selected vendor. Service providers can track their units of service, view relevant information as team members, update assessments, upload service notes, and adjudicate a billable service claim from within the system. At administrative levels, dashboard reports provide data on service units and costs, reported quality of care, and other relevant variables.

Assessment, Measurement, and Feedback

As described above, outcomes monitoring and feedback can promote positive outcomes independent of treatment model. As such, the wraparound practice model states that teams will identify one or more outcomes for each priority need in a youth's POC, collect data on this outcome as well as progress toward meeting the need, and feed information back into the team process. Because such assessment, data management, and reporting is often not achieved, TMS-WrapLogic includes a range of brief process and outcome measures prioritized based on results of surveys of NWI advisors (Table 1). Examples include progress towards needs, status of achieving the family's vision, family satisfaction, and family support and connectedness. Results from team check-ins are presented on a printable youth dashboard, some via pictographs, to facilitate use in team meetings by team members with varying levels of data fluency, including youth and families. TMS-WrapLogic also includes a range of standardized assessment tools such as the Child and Adolescent Needs and Strengths (CANS) measure (Lyons, Griffin, Fazio, & Lyons, 1999). As shown in Figure 5, a common configuration for a youth or supervisor includes depiction of changes in CANS scores over time, to facilitate focused strategizing, mid-course correction, or determine when transition out of formal wraparound may be warranted.

Figure 5.

Theory of impact for a wraparound-specific electronic behavioral health system.

User Testing Phase: Refining TMS-WrapLogic

Formative Usability Testing

Usability is defined as “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (ISO, 1998). As part of the NIMH-funded research strategy, the research and development team tested usability of a prototype TMS-WrapLogic system that included a limited set of features such as entry and management of youth, team, POC information and MFS functionality.

Method

Test users included 10 care coordinators and 8 supervisors from five diverse wraparound sites in four states (MD, IN, CA, WA). The number of staff participating from each site ranged from 1–5. Half (50%) had been in their current position for 1–2 years. Thirty-nine percent (39%) had been in their current position for over 3 years, and 12% for less than one year. Test users were trained on the system via interactive webinars and instructed to enter and manage relevant data for up to three enrolled youths on their caseloads. Test users completed detailed surveys on the ease of use of different TMS-WrapLogic functions. Users also participated in focus groups aimed at providing more detailed feedback on strengths and needs for improvement. Qualitative data from the online survey, as well as feedback given during the focus groups, were organized by salient themes, and then coded by minor and major response categories. Only an overall summary of results is presented here.

Results

As shown in Table 2, over 80% of users found functions related to entry and management of (1) youth/family/team data, (2) assessment and progress monitoring data, and (3) satisfaction and fidelity data to be “fairly” or “very easy.” However, only 62% of users found the process of managing data and information for the wraparound POC to be easy, indicating a need for improvement in the system around these functions. Ninety percent of users agreed that the dashboard reports were “somewhat” or “very understandable.” The vast majority (88%) rated the system as likely to be “somewhat” or “very feasible” for use in everyday wraparound implementation, and 76% reported that they felt their site or program would benefit “a good deal” or “very much” from the use of TMS-WrapLogic. The remaining 24% said they thought their program would benefit “a little bit.”

Table 2.

Reported usability of prototype TMS-WrapLogic functions.

	Enrolling youth in the system	Entering youth and family plan elements	Entering progress in wraparound	Entering youth and family perception of quality and satisfaction
Very difficult	0 (0%)	0 (0%)	0 (0%)	0 (0%)
Somewhat difficult	1 (7%)	5 (38%)	3 (20%)	2 (13%)
Fairly easy	9 (64%)	7 (54%)	9 (60%)	9 (56%)
Very easy	4 (29%)	1 (8%)	3 (20%)	5(31%)

Results of focus groups with 16 of 18 users revealed consistently positive feedback about the system's overall ease of use, including the assessment elements, document uploading capabilities, and dashboards. Many users reported dashboards and reminder features, in particular, would help them organize their implementation of the wraparound process. At the same time, users made a range of recommendations about needed improvements to the usability of the system. Users recommended a system that was more visually “pared-down” and required fewer mouse clicks per operation. Responding to this input was viewed as critical, since fewer mouse clicks are often identified as a basic metric of system usability (Krall & Sittig, 2002). Some users found the rating of youth and family support to be confusing, which was addressed via embedded definitions. Feedback on placement of certain buttons and the calendar system by which dates are assigned to elements were addressed in TMS-WrapLogic revisions, some of which required fairly extensive “rewiring” of the overall logic of the system architecture and links between back-end databases.

Some users expressed more general concern about integration of TMS-WrapLogic with other required HIT, leading to development of an interoperability frameworking module (discussed below) to be used during system installation. Finally, several sites noted that their staff do not always have access to an internet connection while in the field, and would therefore be unable to access TMS without going into the office, pointing to the need for a version of TMS-WrapLogic that can run on tablets or smartphones, allowing connectivity via secure cellular networks.

Postdevelopment Phase: Summative Usability Testing

Based on user feedback, such as from the testing described above, and experts in the field, particularly trainers and technical assistance providers affiliated with the NWI, a comprehensive TMS-WrapLogic EBHIS that included all functions described was completed in 2014. Final steps in the NIMH-funded research project include laboratory-based usability testing (described below), a new round of field testing (now underway), and a randomized trial comparing outcomes for practitioners using TMS-WrapLogic and the families they serve to practitioners and families not using the system. Below we describe results of initial usability testing for the fully developed system.

Method

Lab-based “in-vitro” testing was performed on site at a local Wraparound initiative in Washington State. Test users were 6 staff members (4 Wraparound Facilitators and 2 Supervisors) from the agency, all female. Five of the staff had been in their current positions over 6 years; the sixth had been in her position one year.

Testing occurred over two days. On day one, a trainer from the small business partner, Social Tecknowledgy trained on various elements of TMS-WrapLogic. Test users were then given de-identified case files and hypothetical scenarios which required testers to enter and review/interpret data for the given youth and family. Members of the research team led each tester through the scenarios, and took notes as each tester used a “think aloud” method (Lewis, 1982), a method used to gather data in usability testing in product design and development, to describe what was occurring throughout the process. Test users were asked to verbalize what they were looking at, thinking, doing, and/or feeling as they went about the specified task. This enabled the research team to record the user experience of task completion (rather than only its final product).

Test users completed three scenarios and participated in a focus group at the end of Day 2. Prior to completing each scenario, testers were asked on a scale of 1 to 5 (1=not very easy; 5=very easy), “How easily do you expect to perform this task?” Once the scenario was completed, test users were asked, “How easily were you able to perform this task?” Once all three scenarios were finished, the testers completed the System Usability Scale (SUS; Brooke, 1996), a 10- item measure that yields a composite score that reflects the overall usability of the system. Scores range from 0–100; scores above 68 are considered “above average” in terms of usability.

Results

As shown in Table 4, test users found tasks in all three scenarios to be easily accomplished in TMS-WrapLogic (mean ratings all above 4.3 on the 1 to 5 scale), and less difficult than initially anticipated. The mean SUS score was 88.33 (SD=6.26; Range = 80–97.5), far above the measure's benchmark for ease of usability. “Think aloud” and focus group results provided critical additional information regarding the system and the training that was provided. Users noted specific functions (e.g., rating success of progress toward family needs) and fields (e.g., youth height and weight in the crisis plan) that were missing. They also noted more global concerns (e.g., an unnecessary sequence of extra mouse clicks to enter POC elements; dialogue boxes with fonts too small to easily read), and provided useful suggestions (e.g., identify past iterations of the POC by date, not number).

Table 4.

Test users' levels of expected and experienced difficulty of TMS-WrapLogic operations.

	Expected Difficulty		Experienced Difficulty		Difference
	Mean	SD	Mean	SD
Scenario 1	3.95	0.78	4.43	0.87	+0.48
Scenario 2	3.83	0.56	4.75	0.53	+0.92
Scenario 3	3.48	0.51	4.30	0.47	+0.82
Average	3.75	0.68	4.49	0.71	+0.74

Note. Lower scores represent greater expected or experienced difficulty.

Commercialization Phase

In the first two years after the informational webinar was held in October 2012, 67 formal inquiries were received from prospective users. As of April 2015, 6 license agreements have been executed, four with small behavioral health organizations, one with a large regional initiative, and one with a statewide initiative that was part of the CMS PRTF waiver demonstration project. Many other potential licensees remain in negotiation. Among those who declined to license the software (from whom we were able to discern why they were not pursuing TMS-WrapLogic) 4 (7%) put the project out to bid and/or chose an alternative vendor (typically with systems that had more generic functionality); 3 (5%) reported that they planned to design their own system; 3 (5%) said lack of funds prohibited them from pursuing TMS-WrapLogic or will force them to continue with their old system; and 2 (3%) said security concerns prohibit them from using a cloud-based system. In sum, while some organizations attending the initial webinar and expressing interest have contracted for use of TMS WrapLogic, others continue their negotiations while some have opted for other HIT solutions to include remaining with their existing internal HIT, despite many perceiving them to be outdated and/or poorly aligned with their practice models.

Discussion

Despite a growing number of studies indicating positive outcomes, research continues to accumulate that care coordination is often delivered without attention to fidelity standards representative of empirically supported models, which in turn, can yield outcomes no better than traditional case management or uncoordinated care (Bruns et al., 2014; Bruns, 2015). Thus, a primary impetus for our research team was to develop an EBHIS that can promote model adherent wraparound as well as the range of proposed and demonstrated benefits of EHR described above, such as greater efficiency, enhanced communication, better teamwork, and outcomes monitoring and feedback.

As part of this effort, we developed a theory of positive impact for development of an EBHIS specific to wraparound (Figure 5), to guide development of the system and evaluation of its effects on provider and youth/family outcomes. As shown, we proposed that TMS-WrapLogic would incorporate functions capable of facilitating the achievement of several “common factors” (Barth et al., 2011) of research-based care, such as assessment and feedback and data-informed supervision; fidelity to the defined wraparound model; and research-based functions of EHR (e.g., information management and communication). Such features hold the potential to promote positive provider outcomes such as less duplication of effort and greater efficiency, better collaboration and teamwork, support for research- and data-based planning and decision-making, and greater accountability at a program level. Next in the logic chain, proximal family, youth, and team outcomes are proposed, either as a result of greater provider effectiveness or directly as a result of the EBHIS functions. All the above are proposed to promote overall success of wraparound teamwork, and more positive youth, family, and system outcomes.

To develop the system in keeping with best practices in digital technology design that emphasize user input during iterative system development (Courage & Baxter, 2005), we solicited input from experts, potential users, and national providers of technical assistance to the children's mental health field, engaged test users from multiple “real world” providers, and revised extensively based on their feedback on usability and feasibility. National presentations on TMS-WrapLogic nationwide over the past 2 years have generated substantial interest from behavioral health providers, managed care organizations, and statewide care coordination initiatives.

Barriers and Challenges to Uptake

As described above, successful diffusion of TMS-WrapLogic into the estimated 800 wraparound initiatives nationwide (Bruns, et al., 2011) might best be characterized as a slow but steady process. Despite positive ratings of potential interest, 67 formal inquiries from prospective users, and strong initial usability scores, only 6 license agreements have been executed to date. What has become apparent is that the actual contracting for use of an EBHIS does not happen quickly. In the experience of Social TecKnowledgy it can take an organization six months to two years to make a HIT contracting decision, with a typical timeframe being about nine months. Some of the considerations that go in to make a decision to contract for an HIT include comparison to existing HIT and or in-house solutions, a thorough exploration of all HIT offerings regardless of HIT “fit,” local or state regulations and requirements, and the perceived benefits against the costs of investing in HIT. In particular, due to the growing ubiquity of EHRs (Abraham, McCullough, Parente, & Gaynor, 2011), decisions about new technology adoption may increasingly reflect a need to either de-adopt a currently-installed system or implement an additional technology to operate in parallel to existing HIT. This complicates adoption decisions and introduces factors that fall outside of the scope of most existing HIT adoption models (e.g., Avgar, Litwin, & Pronovost, 2012; Michel-Verkerke & Spil, 2013). HIT implementation efforts may therefore benefit from a greater understanding of the existing technological infrastructure present in a destination context (Lyon et al., in press) and explicit model for innovation de-adoption (Prasad & Ioannidis, 2014).

Although only a small minority of potential users explicitly cited cost as a barrier to adoption, conversations with potential users indicate that a range of up-front costs of implementing a new EBHIS present a major barrier to adoption. Although we have been fortunate to be able to use federal resources to develop the system and attempted to ensure that workflow and nomenclature aligns with the defined wraparound model being used by hundreds of programs, license fees are still necessary to pay for training, user support, initial configuration, customizations to functionality and nomenclature, and server and administrative fees. Such costs have proven to be a barrier, especially for provider organizations who may have already invested in more generic IT systems that can meet basic needs for EHR across a range of community, outpatient, and inpatient services provided. A more prominent cost barrier, however, is the “human cost” of lost productivity of staff who must facilitate the installation of new HIT and/or transition from older HIT, and practitioners (such as wraparound care coordinators) who must be trained and supported to use a new system. Especially when other IT systems are required organization-wide, management is highly reluctant to invest in the efforts required to install a comprehensive EBHIS, even when it aligns with a specific practice model being used by the organization, and/or is viewed as a more effective solution to current methods. Considering that the broader EHR movement has yet to realize the oft-promised cost savings to service systems (Aaron & Carrol, 2015; Sidrov, 2006), such hesitation may be appropriate. Future research and development should continue to work to document efficiency gains and cost savings for EHR implementation.

Next steps for TMS-WrapLogic

To facilitate greater uptake of EBHIS such as TMS-WrapLogic and achieve the proposed positive outcomes of effective EHR, it will be important to continually refine systems to be as congruent with user needs as possible. For TMS-WrapLogic, several such steps are now underway. First, the team is in the process of certifying TMS-WrapLogic as a “meaningful use” (MU) behavioral health module HIT. This certification will assure that TMS offers the necessary technology capabilities, functionality, and security to meet federal standards around MU objectives and outcomes. In doing so, TMS would be able to exchange relevant health data with other MU-certified EHRs in use within a customer's health system (discussed further below). MU certification will also help users obtain certain federal resources and cost offsets under HITECH.

Second, a mobile application of the parent TMS software, compatible with all mobile operating systems (to include Apple-iOS, PC, Android, etc.), is now under development. Mobile health is a quickly growing sector of the clinical research and service delivery landscape, but also one fraught with its own unique implementation challenges (Steinhubl, Muse, & Topol, 2015). The initial offering of the mobile application of TMS will focus on user flexibility and increased efficiency, offering users touch screen as well as direct keystroke data entry. The mobile application will provide users with portability of the EBHIS in environments where “practice” is likely to occur, such as off-site team meeting locations, schools, family homes, and partner agency and provider offices. Such an application will potentially reduce one of the observed “human” costs of installing and using the system, which is the duplication of effort inherent in having to return to a desktop to enter data collected in the field with families and teams. Nevertheless, although the development of these kinds of mobile technologies has been identified as a potential way to facilitate the adoption of PHR and related technologies (Aborzahra & Tan, 2013), their impact has yet to be tested on a large scale.

Third, to respond to the inevitable challenges around interoperability and data exchange between systems, work has been completed on the ability of TMS WrapLogic to integrate data from multiple modalities to include data exchange between with similar EBHIS, practice management claims adjudication with public, private and other payee sources, and the ability to exchange data with Administrative Service Organizations for purposes of authorizations. In addition to service claims authorization and adjudication, TMS-WrapLogic now embeds software that allows for bidirectional data integration. If the data integration is with a MU-certified EHR, an HL-7 (Health Level 7 International) data exchange feature (based on standards for the exchange, integration, sharing, and retrieval of electronic health information that supports management, delivery, and evaluation of health services) can be used to facilitate data exchange/ consumption (HL7 International, 2007). If the integration is with a non MU-certified database and/or related to non-“meaningful” data variables, Social TecKnowledgy is now programming an add-on feature that allows for real-time and or scheduled data integration and exchange. Establishing this type of capability is expected to directly address the previously mentioned barrier posed by the presence of multiple required HIT systems within an organization.

Fourth, the research and development team is partnering with consultants to ensure that TMS-WrapLogic functionality facilitates effective intervention and decision-making. A first example is to integrate data from embedded standardized assessment instruments such as CANS with service encounter and administrative data to trigger “clinical alerts” to care coordinators and supervisors about individual youth whose profiles of strength and need, or lack of progress, indicate they may benefit from a specific type of research-based practice, or simply highlights their need for attention. This type of algorithm-based alert system has been used effectively within adult mental health to identify clients at risk for dropout for over a decade, with significant positive effects (Lambert et al., 2003). Similarly, algorithms that combine aggregate wraparound data to calculate difference in mean rates of positive change, service use, residential placements, or costs, can automatically illuminate important system processes and outcomes and direct the application of additional interventions or resources.

In addition, as part of our work to coordinate wraparound care management with evidence-based practice (Bruns et al., 2013), we will seek to connect TMS-WrapLogic functions to the Managing and Adapting Practice (MAP) knowledge management system (Southam-Gerow et al., 2013). Relevant MAP resources that could be connected to TMS-WrapLogic include an online searchable database of psychosocial youth treatments that returns treatment options based on user queries around problem area, age, gender, ethnicity, and/or treatment setting, and Practice Guides that summarize the evidence-based procedures returned by searches. Given the “Needs-Outcomes-Strategies” Section and billable strategy functionality of TMS-WrapLogic, it is possible that specific evidence-based practice elements could be identified based on information on youth needs and progress maintained in WrapLogic, and tracked as part of a broader quality assurance functions.

Finally, with continued refinement comes need for continued rigorous testing. As part of our Phase II STTR grant, we are currently undertaking a sequence of user testing activities with wraparound initiatives that represent a range of potential installation contexts (small provider, managed behavioral health care context, statewide initiative). Where possible, we are emulating the Contextualized Technology Adaptation Process (CTAP) described in this special issue (Lyon et al., in press), using repeated mixed methods assessments to guide revision and adaptation of TMS-WrapLogic to ensure high compatibility with the specific site or initiative. The iterative process of testing, adaptation, and revision will culminate in a controlled trial that tests the theory of change presented in Figure 1, evaluates feasibility of a range of measures of process and outcomes, and sheds light on the potential for an EBHIS such as TMS-WrapLogic to improve provider, youth, and family outcomes. Consistent with the CTAP and other models (e.g., Kaufman, Roberts, Merrill, Lai, & Bakken, 2006), we also anticipate that – like all HIT products – sustained use of TMS-WrapLogic will require continued system refinements over time to improve usability, streamline functionality, and meet user expectations as hardware infrastructure continues to evolve.

Figure 1.

Youth Plan of Care home page in TMS-WrapLogic.

Conclusion

It is clear from our experiences, as well as research on HIT more generally, that uptake of promising EHR systems will require more than work on the part of HIT developers to refine and improve their systems. For example, rather than exclude the behavioral health and children's services fields from mandates and motivation to invest in and implement electronic care management systems, government must actively lead to reduce barriers to adoption and increase incentives. Efforts will also be needed to ensure any mandates around EHR for youth with SEBD are implemented and effective. As described by Cimino (2013) and Morrow (2013), federal efforts to develop common standards and formats for EHR are needed to ensure that information exchange and interoperability across the myriad of electronic systems is possible. Given the barriers presented by the confusing and complex array of privacy laws for children and adolescents, review and clarification of the inconsistencies in these laws are needed to promote decision-making by providers and systems around EHR and information sharing.

Efforts to convene and educate policymakers and providers about the value of EHR systems, their optimal characteristics, and the myths and realities about investing in and using them are also needed. Expansion of the evaluation and research base on costs and outcomes of EHR will aid such education efforts substantially, as will examples of local and data sharing agreements that have facilitated information sharing and care coordination via electronic means. Finally, funding must be provided to the cause of promulgating effective and usable EHR systems (Morrow, 2013). As discussed above, a primary barrier to adoption of EHR is questions about the likelihood of return on investment in the face of up front and ongoing costs of these systems. Ironically, the complexity and cross-system nature of serving youths with SEBD increases both the costs of such EHR systems and the need for them. Without targeted funding, states and providers will be much less likely to take advantage of these critical technologies, potentially contributing to fragmented and ineffective care for the youth and families who most desperately need our systems' response to be focused, well-coordinated, and based on research for what works.

Table 3.

Summary of qualitative feedback from prototype TMS-WrapLogic test users.

Theme/Subtheme	N Statements	Percent of Statements
Ease of Use
Easy to use and navigate overall	7	12.2%
Aligned with local model/language	3	5.3%
Certain assessments easy to enter (CANS, Demos, Needs)	5	8.7%
Positive Input on Specific Functions/Features
Visual representations of progress (dashboards)	7	12.2%
Team/meeting tracking (who, when)	3	5.3%
Aids in supervision	2	3.5%
Reminders	1	1.7%
Tracking tasks separately from workflow	1	1.7%
Workflow	1	1.7%
Problems/Challenges
Some fields may duplicate current HER	4	7.0%
Assents confusing/subjective	4	7.0%
Moving past a screen if not all info is known or entered incorrectly	3	5.3%
Entering needs, strategies and tasks difficult	3	5.3%
May be hard to us in rural areas	2	3.5%
Slow speed	2	3.5%
Logging in and changing passwords	1	1.7%
Can't backdate	1	1.7%
Recommendations for Improvement
Cut back on number of screens/clicks to enter information	4	7.0%
Allow users to control what gets printed out	1	1.7%
An assessment related to phases of wraparound might be helpful	1	1.7%
Helpful if user could export assessments into Excel	1	1.7%
Total	57	100%